RU2805003C2 - Method and system for remote clothing selection - Google Patents

Abstract

FIELD: electronic trade; remote selection of clothing.

SUBSTANCE: invention relates to means for assisting the user in remote selection of goods to which the concept of sizes is applicable, by assessing the compliance of the product sizes with the shape and size of the user’s body. The method for remote selection of clothing includes determining the anthropometric parameters of the user, automatically assessing the suitability of the item of clothing to the shape and size of the user's body in real time, developing and providing the user with recommendations for choosing a particular item of clothing and, optionally, visualizing the item of clothing on a digital avatar of this item user in the virtual fitting room, including when changing his position.

EFFECT: invention improves the efficiency of remote selection of clothing by the user, as well as increases the accuracy of the user's body model.

54 cl, 44 dwg, 3 tbl

Description

Field of technology

The invention relates to the field of electronic commerce. In particular, it refers to means for assisting the user in remote selection of goods (clothing, footwear, accessories, etc.), to which the concept of sizes is applicable, by assessing the conformity of the product sizes to the shape and size of the user's body.

State of the art

In recent years, the use of the traditional simple size system, for example, [height]-[bust]-[waist]-[hip] or [height]-[shoulder size]-[hip size]) when selling clothes, [size] - [fullness] in the shoe trade is gradually being replaced by the use of more complex parametric models or geometric models of the user’s body, for compliance with which a particular item of clothing, shoes, accessories, etc. can be analyzed. (hereinafter referred to as clothes for brevity).

The trend towards the use of such models is caused by several factors, among which we should highlight the desire of retailers to reduce the number of returns of clothes that “did not fit” for one reason or another, and the transition of clothing manufacturers to the use of “computer” patterns instead of traditional “pattern” patterns. To obtain “computerized” clothing patterns, production computer models or physical mannequins are used, for which parametric models are defined. Such production models are usually created on the basis of statistical data obtained from sample representative measurements of the population in a certain area. In other words, production models usually fairly accurately describe at least the most common body types of people in a given country, corresponding to at least 80–90% of the population.

This makes it possible, by performing an optical scan of the user’s body and constructing a geometric (in particular, three-dimensional) and/or parametric model, to correlate this model with the corresponding production models used in sewing clothes, and to remotely select items of clothing that are most suitable for the user in size and body shape.

In addition, it makes it possible to visualize these garments on the user, i.e. present on the screen of an electronic device a realistic image of one item of clothing or several items of clothing in combination. At the same time, the user has the opportunity to examine in detail the depicted item of clothing (or himself in this clothing) from different angles and distances, against different situational backgrounds (which can be selected from the library), under different lighting (daytime, evening, artificial), etc. .

It is expected that all these measures should contribute to more meaningful clothing purchases and, accordingly, to a significant reduction in the number of returns.

Thus, the task of helping the user in remote selection of clothing can be divided into several stages: determining the anthropometric parameters of the user (in particular, his dimensional characteristics), determining the dimensional characteristics of clothing items, assessing the suitability of clothing items for the user’s body based on the analysis of these characteristics and, optionally ,visualization of clothing items on the user’s digital avatar.

Determining the user's dimensional attributes may include constructing a model of the user's body. Approaches to constructing a three-dimensional model of the user's body, determining its dimensional characteristics and comparing them with the dimensional characteristics of clothing items are described in many patent and non-patent documents.

The document US2015206341A1 (Method for providing a three dimensional body model) describes the use of segmentation by body parts on a 3D image, the use of an articulated model of the body skeleton, the construction of a canonical model, and also partially describes the determination of the shape parameters and canonical pose parameters of a parametric model by minimizing the function losses taking into account visible points, minimizing errors of an arbitrary model using the Gauss-Newton method and constructing an arbitrary three-dimensional surface model by changing the transformation parameters of the canonical model based on a 3D image (in an arbitrary shape and an arbitrary pose) according to the morphing field graph. This document does not describe the construction of a morphing field graph by defining the vertices of a parametric model and refining the shape parameters and canonical pose parameters of the parametric model taking into account the results of the canonical 3D surface model.

In documents EP2674913A1 (Three-dimensional object modeling fitting & tracking), EP3358527A1 (Apparatus and method to generate realistic rigged three dimensional (3D) model animation for view-point transform), EP3382648A1 (Three-dimensional model generating system, three-dimensional model method, and program), RU2014152713A1 (Body measurement), US2012019517A1 (Automatic generation of 3D character animation from 3D meshes), US2013187919A1 (3D body modeling, from a single or multiple 3D cameras, in the presence of motion), US2013286012 A1 ( 3D body modeling from one or more depth cameras in the presence of articulated motion), US2015356767A1 (Rapid avatar capture and simulation using commodity depth sensors), US2016110595A1 (Fast 3D model fitting and anthropometrics using synthetic data), US2017140578A1 (Depth camera-based human -body model acquisition method and network virtual fitting system), US2017337732A1 (Human body representation with non-rigid parts in an imaging system), WO2010019925A1 (Method and apparatus for estimating body shape), WO2013087084A1 (Method and device for estimating a pose), WO2014053198A1 (Co-registration - simultaneous alignment and modeling of articulated 3D shapes), WO2015021381A1 (Real-time reconstruction of the human body and automated avatar synthesis), WO2016207311A1 (Skinned multi-person linear model) describes the use of segmentation by body parts in 3D image, the use of an articulated model of the body skeleton, the construction of a canonical model, and also partially describes the determination of the shape parameters and canonical pose parameters of the parametric model by minimizing the loss function taking into account visible points and the construction of an arbitrary three-dimensional surface model by changing the transformation parameters of the canonical model based on the 3D image (in any form and any pose) according to the graph of the morphing field. These documents do not describe the construction of a morphing field graph by determining the vertices of a parametric model, minimizing the errors of an arbitrary model using the Gauss-Newton method, and refining the shape parameters and canonical pose parameters of the parametric model taking into account the results of the canonical 3D surface model.

The documents US2008180448A1 (Shape completion, animation and marker-less motion capture of people, animals or characters), US2013226528A1 (Perceptually guided capture and stylization of 3D human figures), US2013342527A1 (Avatar construction using depth camera) describe the use of an articulated model of the body skeleton, construction of a canonical model, and also partially describes the determination of the shape parameters and canonical pose parameters of a parametric model by minimizing the loss function taking into account visible points and the construction of an arbitrary three-dimensional surface model by changing the transformation parameters of the canonical model based on a 3D image (in an arbitrary shape and an arbitrary pose) according to the morphing field graph. These documents do not describe the use of segmentation by body parts on a 3D image, the construction of a morphing field graph by determining the vertices of a parametric model, minimizing errors of an arbitrary model using the Gauss-Newton method, and refining the shape parameters and canonical pose parameters of a parametric model taking into account the results of a canonical 3D surface model .

In documents RU2615911C1 (Method and system for constructing a realistic 3D user avatar for a virtual fitting room), US2010306082A1 (Garment fit portrayal system and method), US2014035901A1 (Animating objects using the human body), US2014375635A1 (Methods and systems for generating a three dimensional representation of a subject), US2015213646A1 (Method and system for constructing personalized avatars using a parameterized deformable mesh), WO2014037939A1 (System and method for deriving accurate body size measures from a sequence of 2D images), WO2016073841A1 (Scan data retrieval with depth sensor data) describes the construction canonical model, and also partially describes the determination of the shape parameters and canonical pose parameters of the parametric model by minimizing the loss function taking into account visible points and the construction of an arbitrary three-dimensional surface model by changing the transformation parameters of the canonical model based on a 3D image (in an arbitrary shape and an arbitrary pose) according to graph of the morphing field. These documents do not describe the use of an articulated model of the body skeleton, the use of segmentation by body parts on a 3D image, the construction of a morphing field graph by determining the vertices of a parametric model, the minimization of errors of an arbitrary model by the Gauss-Newton method, and the refinement of the shape parameters and canonical pose parameters of a parametric model with taking into account the results of the canonical three-dimensional surface model.

The documents US2019122424A1 (Generation of body models and measurements), WO2013174671A1 (A method and a system for generating a realistic 3D reconstruction model for an object or being) describe the use of segmentation by body parts in a 3D image, the use of an articulated model of the body skeleton, the construction canonical model, and also partially describes the determination of the shape parameters and canonical pose parameters of the parametric model by minimizing the loss function taking into account visible points. These documents do not describe the construction of a morphing field graph by determining the vertices of a parametric model, minimizing errors of an arbitrary model using the Gauss-Newton method, or constructing an arbitrary three-dimensional surface model by changing the transformation parameters of a canonical model based on a 3D image (in an arbitrary shape and an arbitrary pose) according to the graph morphing field and refinement of the shape parameters and canonical pose parameters of the parametric model, taking into account the results of the canonical three-dimensional surface model.

The document [A1] describes a method for generating a 3D model of the human body based on multiple images (depth maps) using segmentation and articulation by changing the shape and pose parameters.

Document [A2] describes a SCAPE model of the human body, formed based on one static image (depth maps) and multiple images using markers. The method includes segmentation, articulation, determination and modification of shape and pose parameters.

Paper [A3] provides an overview of various approaches to generating a 3D model of the human body.

Document [A4] describes the features of forming the skeletal structure of 3D models for animation.

The paper [A5] describes a method for generating a generalized 3D model of the human body (FAUST) using optical markers.

The paper [A6] describes a method for generating a 3D model of the human body using the DeepCut method for segmentation and articulation and the SMPL statistical model for changing shape and pose parameters. Combination of composition positions according to DeepCut with the SMPL model was performed by minimizing the Geman-McClure objective function.

Paper [A7] provides a comparative overview of 3D human body models: SCAPE, BlendSCAPE, Dyna, S-SCAPE, SMPL and RealtimeSCAPE. The vertex-based approach has been shown to be more accurate than the triangle-based approach.

The paper [A8] discusses the issues of rendering muscles when changing the parameters of the shape and pose of a 3D model of the human body.

The document [A9] discusses the formation of visible surfaces (skinning) of 3D models based on the skeletal approach.

The document [A10] discusses the formation of visible surfaces (skinning) of 3D models using Dual Quaternions.

The document [A11] describes the principle of operation of the DeepWrinkles neural network (Facebook AI Research) for 3D modeling of clothing items on a person.

The document [A12] discusses the formation of visible surfaces (skinning) of 3D models based on the skeletal approach. Laplace smoothing taking into account rigidity (rigidity Laplacian regularization) is discussed.

The paper [A13] compares the skeleton subspace deformation (SSD) and pose space deformation (PSD) methods for pose morphing.

The document [A14] describes a 3D model of the human body Skinned Multi-Person Linear model (SMPL).

The document [A15] discusses the formation of the surface of a 3D model of the human body from the point of view of the musculoskeletal model.

Paper [A16] describes a statistical 3D model of the human body S-SCAPE with pose normalization according to Wuhrer et al. (WSX) and Neophytou and Hilton (NH).

Paper [A17] describes a locally predictive approach to obtaining anthropometric measurements by transforming a point cloud using a random regression forest.

The paper [A18] describes the use of Generative Adversarial Networks (GANs) to generate surfaces of a 3D human body model.

The paper [A19] discusses approaches to implementing the shape and pose parameters of a 3D human body model.

The document [A20] discusses the issues of parameterizing the shape of 3D models of the human body.

The document [A21] describes an approach (SobolevFusion) for generating a 3D model of the human body from a single RGB-D data stream using a gradient stream in Sobolev space.

Paper [A22] describes how to obtain a 3D human body model for a virtual fitting room from data from a single consumer depth camera. The method includes segmentation and articulation followed by extraction from the database of the closest-sized variant. It is claimed that the method is efficient in real time.

The paper [A23] describes the use of a refined skeletal approach (using characteristic examples) to generate the surface of a 3D model.

The document [A24] describes obtaining a 3D model of the human body using the Semantic Parametric ReshapING method (SPRING) - using 23 semantic dimensional parameters.

Paper [A25] describes obtaining a 3D model of a human body from a single depth camera image.

The document [A26] proposes a method for obtaining a 3D model of the human body from data from a single depth camera (BodyFusion) based on skeleton-embedded surface fusion (SSF) and graph-node deformations. The method is said to be resistant to noise and missing data.

The document [A27] proposes a method for obtaining a 3D model of the human body from data from a single depth camera (DoubleFusion) using Double Node Graph and Joint motion tracking.

The paper [A28] describes a method for reconstructing the surface of a 3D human body model based on clothing images obtained from a depth camera. The method is based on minimizing the objective function determined by exposed parts of the body surface (without clothing).

The document [A29] describes a method for generating a 3D model of the human body using segmentation and articulation.

The paper [A30] describes a method for constructing a 3D model of the human body using one 3D camera and several inertial sensors mounted on the human body.

The non-patent documents mentioned above discuss the general approaches, algorithms and mathematical details of various ways to implement 3D models of the human body. A technical solution that solves the problem of the present invention using the same means has not been found in the prior art.

Methods for matching data about a wearer's body with data about an item of clothing are also described in numerous patent and non-patent documents.

The documents US2006287877A1 (Matching the fit of individual garments to individual consumers), US2009276291A1 (System and method for networking shops online and offline), US6546309B1 (Virtual fitting room) at least partially describe the presence of input data (1) about the user’s body and data (2) about garments, transformation and/or filtering of input data, comparison of data (1) and (2) by analyzing the objective function, imposing restrictions (boundary conditions) on the objective function, analysis results - determining the global optimum of the objective function, output data – recommendations to the user on the choice of clothing items based on the results of the analysis of the objective function. These documents do not describe the comparison of data (1) and (2) using a neural network and the presence in the neural network of at least two layers, the first layer implementing the internal representation of the network about the dimensional characteristics of a person, and the second layer implementing the internal representation of the network about the dimensional characteristics item of clothing (or brand).

In documents US2014040041A1 (Garment fitting system and method), US2017039622A1 (Garment size recommendation and fit analysis system and method), US2002188372A1 (Method and system for computer aided garment selection), WO0217160A2 (Method and system for generating a recommendation for a selection of a piece of clothing), US2003028436A1 (Method and system for selling clothes), US2004093105A1 (Method for custom fitting of apparel), US2014180864A1 (Personalized clothing recommendation system and method), US20090210320A1 (System and method for comparative sizing between a well-fitting source item and a target item), US2004083142A1 (System and method for fitting clothing), US2002138170A1 (System, method and article of manufacture for automated fit and size predictions), US2009193675A1 (Systems and methods for collecting body measurements, virtually simulating models of actual and target body shapes, ascertaining garment size fitting, and processing garment orders) at least partially describe the presence of input data (1) about the user’s body and data (2) about clothing items, comparing data (1) and (2) by analyzing the objective function, imposing restrictions (boundary conditions) on the objective function, analysis results - determining the global optimum of the objective function, output data - recommendations to the user on the choice of clothing items based on the results of the analysis of the objective function. These documents do not describe the transformation and/or filtering of input data, the comparison of data (1) and (2) using a neural network, and the presence of at least two layers in the neural network, the first layer implementing the internal representation of the network about the dimensional characteristics of a person, and the second The layer implements the network’s internal representation of the dimensional characteristics of a piece of clothing (or brand).

Documents US2005049816A1 (System and method for assisting shoe selection), US2011099122A1 (System and method for providing customers with personalized information about products) at least partially describe the presence of input data (1) about the user's body and data (2) about items of clothing , transformation and/or filtering of input data, comparison of data (1) and (2) by analyzing the objective function, the results of the analysis - determining the global optimum of the objective function, output data - recommendations for the user on the choice of clothing items based on the results of the analysis of the objective function. These documents do not describe the imposition of restrictions (boundary conditions) on the objective function, the comparison of data (1) and (2) using a neural network, and the presence of at least two layers in the neural network, with the first layer implementing the internal representation of the network about the dimensional characteristics of a person, and the second layer implements the network’s internal representation of the dimensional attributes of a piece of clothing (or brand).

In documents WO2010014599A1 (A distributed matching system for comparing garment information and buyer information), US2011184831A1 (An item recommendation system), US2012259581A1 (Apparatus, system and method for providing garment fitting identification), US20110295711A1 (Apparel fit advisory service), US20060059054A1 (Apparel size service), US2002178061A1(Body profile coding method and apparatus useful for assisting users to select wearing apparel), US2007198120A1 (Computer system for rule-based clothing matching and filtering considering fit rules and fashion rules), US2016155186A1 (Digital wardrobe using simulated forces on garment models), US2007022013A1 (Fitting systems), US2009234489A1 (Fitting systems), US6741728B1 (Footwear sizing database method), WO2009090391A1 (Garment filter generation system and method), US2011231278A1 (Garment sizing system), WO2008033138A1 (M atching the fit of garments to consumers ), US2008235114A1 (Matching the fit of individual garments to individual consumers), US2009287452A1 (Method and apparatus for accurate footwear and garment fitting), EP2752804A1 (Method and system for optimizing size consultation upon the ordering of trousers), US2010293076A1 (Method and system for providing fitting and sizing recommendations), US2015161707A1 (Method and system for recommending a size of a wearable item), US6701207B1 (Method for integrating information relating to apparel fit, apparel sizing and body form variance), US2011055054A1 (Method for online selection of items and an online shopping system using the same), US2014379515A1 (Method for providing a custom-like fit in ready-to-wear apparel), US2003076318A1 (Method of virtual garment fitting, selection, and processing), US2006020482A1 (Methods and systems for selling apparel ), US2019266654A1 (Methods and systems for virtual dressing rooms or hybrid stores), US2014279289A1 (Mobile application and method for virtual dressing room visualization), US20120084987A1 (Shaped fit sizing system), EP3171323A1 (System and method for determining and dispensing one or more tailored articles of clothing), US6879945B1 (System and method for sizing footwear over a computer network), US2005022708A1 (Systems and methods for improved apparel fit), US2019073335A1 (Using artificial intelligence to determine a size fit prediction), US2007005174A1 (Virtual apparel fitting) by the presence of input data (1) about the user’s body and data (2) about items of clothing, comparison of data (1) and (2) by analyzing the objective function, analysis results - determination of the global optimum of the objective function, output data - recommendations to the user are at least partially described on the choice of clothing items based on the results of the analysis of the objective function. These documents do not describe the transformation and/or filtering of input data, the imposition of restrictions (boundary conditions) on the objective function, the comparison of data (1) and (2) using a neural network, and the presence of at least two layers in the neural network, with the first layer implementing the internal representation of the network about the dimensional characteristics of a person, and the second layer implements the internal representation of the network about the dimensional characteristics of a piece of clothing (or brand).

Paper [C1] describes the use of a neural network to generate a realistic image of an item of clothing on a person's digital avatar.

Paper [C2] describes the use of a neural network to transfer an image of an item of clothing from one person's digital avatar to another person's digital avatar.

Paper [C3] describes the use of a nonlinear regression model based on a neural network to generate a realistic image of an item of clothing on a digital avatar of a person and the use of a recurrent neural network to calculate the folds of clothing over time.

Paper [C4] describes the use of a wavelet neural network for remote clothing sizing.

Paper [C5] provides an overview of approaches to implementing virtual fitting rooms. The use of neural networks is mentioned.

Paper [C6] provides an overview of approaches to implementing clothing recommender systems. The use of neural networks is mentioned.

Paper [C6] provides an overview of approaches to implementing clothing recommendation systems using self-reported images.

Paper [C7] describes the use of a convolutional neural network to generate a realistic image of an item of clothing on a digital avatar of a person in various poses.

Paper [C8] describes the use of spline transformation based on a convolutional neural network to generate a realistic image of a clothing item on a digital human avatar in various poses.

The paper [C9] describes the use of a multilayer neural network to implement a virtual fitting room.

Paper [C10] describes the use of a competitive neural network and a self-organizing neural network for shirt modeling to implement a virtual fitting room.

The paper [C11] describes the use of a convolutional neural network to generate a realistic image of an item of clothing on a digital human avatar.

The paper [C12] describes the use of a neural network to generate a realistic image of an item of clothing on a person's digital avatar.

Paper [C13] describes the use of a deep neural network to implement a recommender system for cosmetics.

The paper [C14] describes the use of a neural network to generate a realistic image of an item of clothing on a person's digital avatar.

The paper [C15] mentions the use of a neural network to generate a realistic image of a piece of clothing on a person's digital avatar.

Paper [C16] provides an overview of approaches to implementing virtual fitting rooms. The use of neural networks is mentioned.

Paper [C17] provides an overview of approaches to implementing virtual fitting rooms.

The above non-patent documents describe approaches to the use of neural networks, including multilayer ones, to solve problems associated with remote selection of clothing.

Thus, technical solutions of the prior art with greater or lesser success solve fragmented problems, one way or another related to the remote selection of clothing. However, no technical solution that comprehensively solves the problem of the present invention has been found in the prior art.

Disclosure of the Invention

The objective of the present invention is to determine the anthropometric parameters of the user, automatically assess the suitability of a piece of clothing to the shape and size of the user's body in real time, develop and provide the user with recommendations for choosing a particular piece of clothing and, optionally, visualize the piece of clothing on the digital avatar of this user in a virtual fitting room , including when changing his posture.

The technical result of the present invention is to increase the accuracy of the user's body model, increase the efficiency of the user's remote selection of clothes (speeding up and increasing the accuracy of the selection), improve the user experience from the remote purchase, increase the user's satisfaction from the purchase of clothing (customer satisfaction) and, ultimately account, an increase in the volume of online sales of clothing and a decrease in the share of clothing returned after purchase due to unsatisfactory fit to the shape and size of the user’s figure.

The problem of the invention is solved using a method for remote selection of clothing, including:

- determination of human dimensional characteristics;

- determination of dimensional characteristics of one or more items of clothing;

- comparison of the dimensional characteristics of a person with the dimensional characteristics of one or more items of clothing to determine the degree to which these items of clothing correspond to a person;

- optional visualization of the item of clothing;

- making a choice of clothing item.

Determining the dimensional characteristics of a person may include the formation of a parametric model of the person. A parametric model of a person can be formed on the basis of a geometric model of a person, on the basis of physical measurement data of the human body, on the basis of questionnaire data, on the basis of two-dimensional image data (for example, a photograph), on the basis of physical measurement data of a reference item of clothing, on the basis of optical measurement data standard item of clothing. A parametric human model can be formed using a neural network and a machine learning algorithm.

A geometric model of a person can be formed based on three-dimensional scanning data of the human body, including optical scanning. Optical scanning may involve the use of a depth camera, laser range finder, or other similar means to obtain data about the surface of the human body. A geometric model of a person can be formed using a neural network and a machine learning algorithm.

Dimensional features of a person can be determined using linear algebra and/or using a machine learning algorithm. In this case, an embodiment of the invention is possible in which the intermediate dimensional features of a person are first determined using linear algebra, and then the final dimensional features of a person are determined using a machine learning algorithm. A machine learning algorithm may use a regression model.

The sizing characteristics of a garment can be determined based on a production geometric model, which can be a physical mannequin, a three-dimensional digital model, a set of patterns, etc. Dimensional characteristics of a garment can also be determined based on a manufacturing parametric model, which may represent at least part of the technological documentation of the garment, for example, a technical specification (tech pack) and/or specification sheet (spec sheet) and/or specification of materials (bill of materials) and/or grading rules. The manufacturing parametric model may represent at least part of the brand's sizing chart.

Dimensional features of a garment can be determined based on physical measurement data of the garment or on the basis of optical measurement data on the clothing item, in particular, using a computer vision algorithm.

Comparison of dimensional characteristics of a person and a piece of clothing may include analysis of the objective function. In another case, matching the dimensional attributes of a person and an item of clothing may involve the use of a neural network and a machine learning algorithm. A neural network may contain at least two layers, one of which acts as an encoder, and the others act as a decoder. The layer that acts as an encoder can be trained separately from the layers that act as a decoder. In another case, the layer that acts as an encoder can be trained together with the layers that act as a decoder. For example, initial training of the encoder may be performed separately from the decoder, and further training may be performed in conjunction with the decoder.

Dimensional attributes of a person and/or article of clothing may include at least one of the following: height, distance, girth, and distance along a curved surface. To compare the dimensional characteristics of a person and an item of clothing, these dimensional characteristics can be reduced to a comparable type or data format.

The visualization of the clothing item can be performed on a virtual mannequin or on a user avatar. An avatar (digital twin) can be a 3D model of a person's body and can provide a realistic representation of the person's head. The avatar can take into account the individual characteristics of the user’s appearance - skin color, eye shape, hair type and color, age manifestations, etc. Visualization may include the ability to set and/or change the position of the avatar (posture, gestures, etc.), angle, scale (zooming in or out), situational background, lighting conditions, etc.

The selection of a clothing item may be made from a list of clothing items ranked by the degree of suitability of the clothing items to the user. Fit of a garment can include physical fit and stylistic fit. Selecting an item of clothing may include not selecting an item of clothing, i.e. selection of a clothing item is optional and may not be present in some embodiments of the invention (eg, if the user is unable to select the desired clothing item).

A method for determining the size characteristics of a person may include:

- construction of a geometric model of the human body;

- construction of a parametric model of the human body based on a geometric model;

- determination of dimensional characteristics of a person based on a parametric model of the human body.

Determining the size characteristics of a person can be supplemented by determining the type of human figure.

Constructing a geometric model of the human body may involve obtaining a depth map and converting the depth map to a point cloud. Converting the depth map to a point cloud may involve applying a bilateral filter. The construction of a geometric model of the human body can be performed for the canonical human pose.

Building a parametric model may include:

- calculation of the initial approximation of the pose parameters;

- clarification of pose parameters;

- clarification of posture and shape parameters;

- construction of a morphing field graph that ensures the transformation of a geometric model from a canonical pose and/or shape to an arbitrary pose and/or shape.

When calculating the initial approximation of pose parameters, information about the position of characteristic points of the human skeleton in the parametric model can be used. The characteristic points can correspond to the centers of the joints and/or those parts of the skeleton that are taken into account when segmenting the human body from the point of view of the possibilities of changing the pose.

Refining the pose parameters may involve minimizing an error function over the pose parameters. The error function for pose parameters may include the sum of squared angles between pairs of segments connecting characteristic points of the human skeleton in a geometric model and in a parametric model, and/or the sum of squared distances between pairs of points from a set of characteristic points in a geometric model and in a parametric model, see also section 2.2.

In this case, the error function for the pose parameters can first be used, equal to the sum of the squares of the angles between pairs of segments connecting the characteristic points of the human skeleton in the geometric model and in the parametric model, and then the error function for the pose parameters, equal to the sum of the squares of the distances between pairs of points from the set characteristic points in the geometric model and in the parametric model.

Refining the pose and shape parameters may involve minimizing an error function over the pose and shape parameters. The error function for the pose and shape parameters can be the following function:

,

where P is the set of visible points x of the geometric model, θ is the pose parameters, β is the shape parameters, U is the set of visible vertices parametric model with pose and shape parameters ( θ , β ), p – Huber function with parameter δ equal to 1, see also section 2.2.

The morphing field graph can have vertices containing coordinates in three-dimensional space, parameters of the Euclidean transform, and a weighting coefficient of the Euclidean transform. The graph may also have edges between the vertices that provide connectivity to the morphing field, i.e. smoothness of a geometric model of the required shape in an arbitrary pose, which can be built on the basis of a parametric model. The smoothness of such a transformation can be important when creating a user avatar for the purpose of high-quality visualization of a piece of clothing.

The formation of graph vertices may include:

- selection of the value δ of the neighborhood radius;

- division of space into voxels with a side of 2δ;

- determination of vertex coordinates as medoid coordinates for each voxel;

- assignment of a weighting coefficient equal to 2δ for each vertex, see also section 2.2.

Edges between vertices in a graph can be formed when those vertices are among the closest, for example, eight neighboring vertices along the shortest path in the graph.

The Euclidean transformation of an arbitrary point x may include:

- determination of the vertices closest to point x graph;

- calculating the weighted sum of double quaternions of these vertices, with the weight w of each double quaternion calculated using the following formula:

or is taken equal to 0 if Where vertex weight;

- normalization of the summation result;

- construction of an orthogonal transformation matrix R and a translation vector t , while the result of the transformation of point x is the point Rx + t .

A parametric model of the human body can be built using a neural network and a machine learning algorithm, which may include the use of the error function for determining characteristic points of the human skeleton in two-dimensional space, the error function for determining characteristic points of the human skeleton in three-dimensional space, the error function for determining the parameters of the SMPL model, and depth map error functions, see also section 2.2. A machine learning algorithm may use a regression model.

Determining dimensional attributes of a garment may include obtaining dimensional attributes of the garment from a production model. The manufacturing model may be a parametric model, and obtaining the dimensional attributes of the garment may involve converting the parameters of the manufacturing model into dimensional attributes of the garment. The manufacturing model may be a geometric model, and obtaining the dimensional attributes of a garment may involve converting the geometric manufacturing model to a parametric manufacturing model and then converting the parameters of the manufacturing model into dimensional attributes of the garment.

The production geometric model can be a physical mannequin, a 3D digital model, or a set of patterns. The manufacturing parametric model may represent at least part of the manufacturing documentation of the garment and/or at least part of the brand's sizing chart.

Determining the degree of human fit of a clothing item may include comparing dimensional attributes of the person and the clothing item using an objective function that takes into account the effect of each dimensional attribute on the degree of human fit of the clothing item.

The objective function may be a function

,

where M is the number of dimensional features, - a dimensional sign of a person, – dimensional indication of a piece of clothing, is a weighting factor and can be minimized.

Determining the degree to which an item of clothing fits a person may include determining the recommended clothing size S :

S = Σ(w _i s _i )/Σw _i +ds,

Wheres _i – size correspondingi-th dimensional characteristic,w _i - weight coefficienti-th dimensional characteristic,ds – at least one optional correction value. The correction value can be determined individually for each user or for a group of users, for example, with figures of the same type. The size calculated in this way can be additionally adjusted to the brand’s sizing chart.

The result may be a list of garments, in which garments may be ranked according to fit rating, and garments with the same fit rating may further be ranked by fit error value.

The dimensional characteristics of a person may be the dimensional characteristics of a reference item of clothing, i.e. such an item of clothing, the suitability of which to the user is beyond doubt. In particular, the dimensional characteristics of a reference item of clothing can be used directly in the analysis of another item of clothing, for example, the same brand belonging to the same sizing grid, or can be converted into dimensional characteristics in another dimensional grid if the ratios of dimensional characteristics in these dimensional grids are in advance known.

Determining how appropriate a garment is for a person may include taking into account stylistic guidelines, such as those related to the wearer's body type.

Determining the degree to which an item of clothing fits a person may involve matching the dimensional attributes of the person and the item of clothing using a neural network and a machine learning algorithm. The machine learning algorithm can be implemented as a classifier. In another case, the machine learning algorithm can be implemented as a decision tree gradient boosting algorithm.

Visualization of a garment may include:

- obtaining an image of the human body in an arbitrary pose;

- obtaining a parametric model of the human body;

- obtaining a parametric model of a garment;

- changing the parameters of a parametric model of the human body to correspond to an arbitrary pose;

- formation of an image of an item of clothing on a person in an arbitrary pose.

The user's image can be obtained from any device capable of generating such an image, for example, from a smartphone, tablet, game console, laptop, desktop computer, TV equipped with a camera, etc. The resulting image of the user in an arbitrary pose may contain a depth map and/or a two-dimensional image. The pose and/or shape parameters of the user's parametric body model can be changed so that the shape and pose of the user's avatar matches the resulting image. This processing can be done in real time.

It is assumed that the shape parameters of the user's parametric body model generally correspond to the shape of the user's body. However, if the processing process reveals a significant discrepancy between the user's avatar body shape and the user's body shape recorded in the resulting image, the parametric model can be adjusted. The need for such correction may be caused by both the limited accuracy of the formation of the parametric model, and actual changes in the shape of the user’s body after the formation of the parametric model, for example, an increase or decrease in body weight, targeted training of certain muscle groups, changes in posture, pregnancy, etc.

Brief description of drawings

A detailed description of various embodiments of the invention in practice is given below with reference to the accompanying drawings.

In fig. Figure 1 shows a block diagram of the algorithm for processing scanning results and constructing a model of the user’s body based on depth maps.

In fig. Figure 2 shows an example of changing the scene to match the input data, from top to bottom: the scene surface in the canonical state, the input data is the point cloud of the input frame, the scene surface transformed according to the input data.

In fig. Figure 3 shows an articulated model of the skeleton of the human body, in which red (large) points enter many key points of the skeleton, blue (small) points do not enter it, and green segments correspond to the bones of the arms, legs, spine, including the neck, as well as shoulder lines and pelvis.

In fig. Figure 4 illustrates the solution to the problem associated with the regularization of semantically unrelated vertices, in particular the vertices of the right and left legs, where from left to right are presented: the original model, the graph initialized by the proposed method, and the graph initialized by the usual procedure for searching for nearest neighbors. Particular attention should be paid to confusion of the legs during the normal procedure.

In fig. 5 on the left is a visualization of the morphing field, in the center is the surface obtained from the depth map of the next frame, on the right is the same surface after transformation to the canonical position.

In fig. Figure 6 illustrates the segmentation of the human body, where segments are indicated by different colors and adjacent segments are connected by lines.

In fig. 7, on the left is the result of rendering a surface into a depth map, and on the right is the original surface with visible vertices, indicated by red dots.

In fig. Figure 8 presents a set of models that can be obtained by transforming the original SMPL model by varying three shape parameters while keeping the pose parameters fixed. Red dots indicate the positions of skeletal joints.

In fig. Figure 9 shows scanning defects using the example of three-dimensional images of the CAESAR data set - the presence of areas of the body surface with missing information in the area of the feet, hands and the outer surface of the arms, shoulders, head, as well as “glued” areas.

In fig. Figure 10 shows the skeleton as a tree of vertices, where each vertex is characterized by a position in space and represents the point of articulation of the “bones” of the skeleton, and the zones of influence of each joint on the surface of the body model are shown in color.

In fig. Figure 11 illustrates the appearance of significant artifacts in the HumanShape model when changing the shape parameters of the human body due to their influence on the position of skeletal points.

In fig. Figure 12 illustrates the emergence of noise geometric structures located inside a model of the human body. One geometric structure (highlighted in green) falls into a separate connected component of the graph and can be relatively easily removed, and the other geometric structure, which is part of the surface of the left shoulder (highlighted in yellow), is part of the surface of the human body.

In fig. Figure 13 shows an example of a depth map, where the color indicates the distance to the point, and the color scale is graduated in meters. A depth map represents the distance from the camera's projection plane to the surface of a 3D scene.

In fig. Figure 14 illustrates the step-by-step application of shape parameters and pose parameters with a preliminary recalculation of the coordinates of skeletal joints, and then using the general transformation parameters - scaling and displacement. Next, the loss function is calculated based on the standard deviation from the nearest vertices.

In fig. Figure 15 shows a graph of the activation function used in calculating the component that penalizes for too large angles between the normals of adjacent triangles.

In fig. Figures 16–18 show a comparison of the results of the proposed modification of the NRD algorithm and the original algorithm [B15] on a three-dimensional image from the CAESAR set. From left to right are: the proposed version of the NRD, the original 3D image, and the original version of the NRD. The color indicates deviations from the three-dimensional image: white means zero deviation, red means a deviation of at least 15 mm.

In fig. Figure 19 shows the results of the NRD algorithm. From left to right: with classic loss function, 25 thousand vertices in a 3D image; with classic loss function, 250 thousand vertices in a 3D image; and with a modified loss function, 250 thousand vertices in a 3D image.

In fig. 20 shows registrations of the same three-dimensional image using different parametric models. From left to right: registration based on the HumanShape model (18659 vertices), original 3D image (250000 vertices), registration based on the SMPL model (18890 vertices).

In fig. 21A shows errors when registering a data set. Orange indicates results based on the HumanShape model, and blue indicates results based on the SMPL model. The solid line marks the results of initializing registration using the method described in Section 2.9.2, and the dotted line marks the results of improving registration using the proposed modification of the NRD algorithm.

In fig. 21B shows errors when registering the CAESAR data set. Blue color indicates the registration results obtained by the proposed method, and orange color indicates those obtained by the original method.

In fig. Figure 22 shows a graph of the dependence of the explained variance on the number of shape space parameters taken into account.

In fig. Figure 23 shows a comparison of the initial pose and shape of the SMPL parametric model (left) and the resulting parametric model (right) for a male and female body.

In fig. Figure 24 shows a visualization of the result of separate changes in the first four shape parameters while fixing the remaining parameters of the resulting female parametric model.

In fig. Figure 25 shows a graph of the amount of correction in the A-pose for each leg in the form of the difference between the actual length of the leg and its projection on the vertical axis.

In fig. Figure 26 shows a visualization of a dimensional feature in the form of an anthropometric point.

In fig. Figure 27 shows a visualization of a dimensional feature in the form of a distance between two anthropometric points.

In fig. Figure 28 shows a visualization of the dimensional attribute in the form of girth.

In fig. 29 illustrates a method for measuring a dimensional feature in the form of a distance along a surface.

In fig. Figure 30 shows a visualization of a dimensional feature in the form of a distance along the surface.

In fig. Figure 31 presents the result of comparing dimensional features obtained using the claimed algorithm with dimensional features obtained by manual body measurements.

In fig. Figure 32 shows the result of comparing the dimensional features obtained using the claimed algorithm with the dimensional features determined from a three-dimensional image.

In fig. Figure 33 shows a block diagram of an enlarged algorithm for remote selection of clothing.

In fig. 34 shows an illustrative example of a system for implementing a method for remote clothing selection in a B2C environment.

In fig. 35 shows an illustrative example of a system for implementing a method for remote clothing selection in a B2B environment.

In fig. 36 schematically shows the architecture of the Human Mesh Recover algorithm.

In fig. 37 schematically shows the architecture of discriminator networks for processing the parameters of the SMPL model.

In fig. Figure 38 schematically shows the architecture of the modified Human Mesh Recover algorithm and examples of vertex projection and distance maps.

In fig. Figure 39 shows an example of an error map of a predicted mesh parametric 3D model. Training the algorithm may include a penalty for errors.

In fig. 40 shows the result of the system. From left to right: depth map, segmentation, key points, reference model.

In fig. 41 shows an example of a simple installation for optical scanning of clothing.

In fig. 42 illustrates the operation of a computer vision algorithm that determines the type or category of clothing, identifies key points in the image, calculates the outline of the clothing and, based on this data, determines the dimensions.

In fig. 43 shows a block diagram of an enlarged algorithm for recommending the appropriate size for a specific clothing model.

In fig. 44 schematically shows the structure of a neural network for implementing a recommendation method based on a machine learning algorithm.

Carrying out the invention

The following describes non-limiting options for the practical implementation of the invention. The purpose of the following description is to illustrate approaches to the practical implementation of the invention, and not to limit the scope of the invention to the described embodiments.

To implement the approach described above, the following enlarged algorithm of actions is proposed (Fig. 33):

- (31) optical scanning of the user’s body;

- (32) processing scan results and constructing a model of the user’s body;

- (33) determination of the anthropometric parameters of the user’s body from the constructed model;

- (34) selection by anthropometric parameters of one or more of the closest production models from an array of production models from different clothing manufacturers;

- (35) search for garments made using selected production patterns or sewn in any way and pre-measured using a camera or by hand;

- (36) (optional) visualization of found items of clothing;

- (37) (optional) making a selection and (optional) purchasing selected items of clothing.

Instead of steps (31) and (32), it is possible to construct a model of the user's body based on user data obtained using one or more of the following methods:

- filling out a questionnaire by the user;

- photographing the user from one or several angles;

- physical measurements of the user's body.

In addition, in some cases, the body model building step in step (32) of the user can be omitted and the anthropometric parameters of the user's body can be obtained directly from the above-mentioned user data.

The parameters of a statistical model of the human body or previously known clothing sizes of selected brands suitable for the user can be used as anthropometric parameters of the user.

Instead of steps (34) and (35), a comparison can be made of the user’s anthropometric parameters with the brand’s dimensional grid, with the results of clothing measurements, etc. A detailed description of the proposed implementations is presented in sections 4 and 5.

Steps (36) and (37) can be implemented in a method used in online trading. One or both of these steps may be absent in the method used in other situations, for example, when creating a user’s virtual wardrobe, during deferred purchases, during individual tailoring with remote measurements, during statistical or marketing research, etc.

It should be noted that in this text the term “optical scanning” is used, but everything said is also true in a more general case: that is, throughout the text, instead of the term “optical scanning”, it is permissible to use the broader term “three-dimensional scanning”, correct all statements are completely preserved.

It should also be noted that, in addition to optical/three-dimensional scanning, everything said in this text is also true when capturing in any way any data necessary to reconstruct a human model, including photo and video shooting (for example, using the photogrammetry approach), as well as various measurements (height, weight, etc.).

It should also be noted that the search and selection can be carried out not only by anthropometric parameters, as indicated in step (34), but also by various other parameters (statistical model parameters, etc.). Thus, wherever the term “anthropometric parameters” (including, in particular, dimensional characteristics and body type) appears in this text, it can be replaced by the broader term “model parameters” with full preservation of the meaning.

It should also be noted that the term “production models” used in this text can mean physical mannequins, live models (mannequins and fashion models in production), computer models (mannequins, mannequins, fashion models).

In the case of clothing recommendations based on a comparison of the user’s anthropometric parameters and clothing measurements, clothing measurements can be measurements of sewn clothing, measurements of clothing patterns, or other parameters. They can be obtained from clothing production documentation (tech pack, grading rules) or from clothing measurements or clothing patterns.

If obtaining production documentation with clothing measurement tables from clothing manufacturers is impossible or takes a long time (business arrangements, signing agreements), it is possible to obtain clothing measurements using optical scanning.

Optical scanning of clothing is supposed to be carried out on a special installation. The installation consists of a table, for example, with a light tabletop and a static, pre-calibrated chamber (Fig. 41). For each photograph of clothing, a computer vision algorithm is applied (Fig. 42), which:

1) identifies the type of clothing;

2) calculates key points in the image;

3) calculates the outline of the clothing in the photo.

The algorithm used consists of a set of convolutional neural networks that perform each task separately: classifier, regressor and segmenter, respectively.

Based on the data obtained by the algorithm described, measurements characteristic of each type of clothing are calculated. To do this, pre-prepared rules for each type of clothing are used to convert key points and contours into target measurements. At the end of the algorithm, using the camera calibration parameters, the acquired measurements are transformed into metric space.

When recommending clothing sizes, it is assumed that there are:

1) dimensional characteristics of the user;

2) a database of clothing measurements, stratified by type of clothing;

3) rules for comparing the user’s dimensional characteristics and clothing measurements.

In this case, the algorithm for recommending a certain type of clothing based on size characteristics is an algorithm for comparing vectors in the Euclidean space hnsw or faiss .

1. Optical scanning of the user's body

Optical scanning of the user's body can be performed in a variety of ways, including laser scanning, structured illumination scanning, ToF scanning (time-of-flight), photography from fixed angles, video recording, etc. Scanning can be performed from one or multiple points. Each point from which scanning is performed can be fixed or can move along a specific path. In the example implementation of the invention described below, the model is built using the point cloud obtained during scanning, but this example is only illustrative and other types of input data, such as depth maps, images with projected structured illumination, etc., can be used to build the model.

The point cloud can be obtained using optical means (so-called “depth cameras”), capable of determining the depth of a three-dimensional scene with sufficient resolution. Such capabilities, in particular, are available in devices Stereolabs ZED, Carnegie Robotics MultiSense, Ensenso N35, Nerian SceneScan, Intel RealSense, Zivid One+, Arcure Omega, etc. It is also possible to use ASUS Xtion, ASUS Xtion2, Orbbec Astra, Orbbec Persee, Microsoft Kinect, Microsoft Kinect 2, Azure Kinect and similar devices. It is expected that in the foreseeable future, “depth cameras” of acceptable resolution (as well as acceptable noise levels) can be integrated into some models of smartphones and other mobile devices (iPhone X, iPad Pro (2018), Huawei Mate 20 Pro, etc.).

Next, two approaches to processing scanning results are described, resulting in a parametric model of the human body (section 2.8). The first approach is based on processing depth maps and is described in Section 2. An alternative approach allows one to obtain a parametric model of the human body using one or more color photographs of a person and is described in Section 3. These approaches can be used both independently and together. For example, a model of a person built from a photograph can be used to initialize a method that performs a three-dimensional reconstruction, if the recording device has the ability to simultaneously capture color images and depth maps.

2. Processing scan results and building a model of the user’s body based on depth maps

Processing of scanning results is based on the principle of three-dimensional reconstruction of dynamic scenes using information about the human body from a parametric model and is based on the ideas outlined in [B1], with changes aimed at expanding the capabilities of the algorithm, namely reducing errors arising from sudden movements and overlaps, as well as to increase the variety of possible starting user positions. Changes relate to the use of segmentation of input depth maps by body parts and to the position of key points of the skeleton, as well as to the initialization and optimization algorithm.

Graphically the proposed processing method is presented in Fig. 1. The algorithm begins work by filtering the depth maps supplied to the input and converting them into point clouds using the internal calibration matrix of the depth camera. The next step of the algorithm is to initialize the canonical shape and pose of the parametric model. Then changes in space transformation parameters are calculated, as well as changes in model pose parameters to best fit the next image frame. The calculated parameters are used to integrate the data into an internal representation that contains the person at the original (canonical) position. After integration, the algorithm performs fitting of the shape parameters and the canonical pose.

2.1. Pre-processing of input data

The algorithm's operation involves identifying correspondences between a point cloud in three-dimensional space corresponding to the current depth measurements and the scene surface obtained from previously received data, so the incoming depth maps must be converted into a point cloud. However, depth maps can be very noisy, for example, as shown in [B2], which negatively affects the stability of the algorithm. Based on the assumption that the surface of the human body is characterized by the absence of sharp changes, we can assume that changes in depth between adjacent pixels should be sharp only at the boundaries, therefore, to reduce noise, a bilateral filter [B3] can be applied, smoothing the depth map while preserving the boundaries unchanged. A predefined internal camera calibration matrix [B4] is used to convert the depth map into a point cloud.

2.2. Initializing parameters

To start the work of all algorithms based on the method of supplementing the internal representation of the scene, to which the proposed method also applies, it is necessary to determine the initial or, in other words, the canonical state of the scene, which will then be changed by a morphing field to ensure compliance with the current frame. In fig. Figure 2 shows an example of changing the scene to match the input data, from top to bottom: the scene surface in the canonical state, the point cloud of the input frame, the scene surface modified to suit the input data.

In algorithm [B1], for the algorithm to start working, the user must stand in a certain position, which in the context of applying the method in the consumer segment creates additional restrictions. This development proposes to circumvent this limitation by using an initialization algorithm based on the results of [B5].

The input of the algorithm is the first frame of the sequence, as well as information about segmentation and key points of the skeleton. The operation of the algorithm includes a number of steps: (1) calculating the initial approximation of the pose parameters, (2) iterative minimization of the error function over the shape and pose parameters, (3) initializing the morphing field graph over the model vertices.

At the stage of calculating the initial approximation, information about the position of key points of the skeleton in the parametric model and on the input frame is used. Since in general there is no clear correspondence between the key points determined on the input frame and the key points of the model, an auxiliary algorithm consisting of two steps was developed to determine the values of the model parameters from the key points received from the sensor.

At the first step, minimization is performed using the pose parameters of a function equal to the sum of the squares of the angles between pairs of segments: , Where angle between segments, – the i-th segment from an ordered set of segments drawn, respectively, between some key points of the model or some key points obtained from the sensor, k = | M _v | = | K _v |. Segments from the sets M _v and K _v correspond to the bones of the arms, legs, parts of the spine, as well as the lines of the shoulders and hips.

At the second step, using the pose parameters, the sum of squared distances between pairs of points is minimized from ordered sets M _J and K _J of key points of the sensor and model: , s = | M _j | = | K _j |.

An example of the sets M _v and M _J is shown in Fig. 3, where red (large) key points are included in the set M _J , blue (small) points are not included in it, and green segments (connecting large key points) are included in the set M _v .

At the second stage, to refine the pose parameters, as well as to determine the model’s shape parameters, minimization of the error function, defined as

(1)

where P is a cloud of points formed from the depth map supplied to the input, β are the model’s shape parameters, θ are the pose parameters, U is the part of the vertices of the parametric model with the pose and shape parameters visible from the camera’s point of view ( θ , β ). The problem of determining visible vertices is discussed in detail below.

The function p ( x ) is the Huber function Lδ ( x ) with parameter δ equal to 1 _:

As a result of the second stage of the algorithm, the model parameters in the canonical pose and shape become known, according to which the graph is initialized at the final stageG _f =β (V _f ,E _f ) morphing field. Similar to what was described in [B6], the verticesV _f the graph contains coordinates in three-dimensional space, the parameters of the Euclidean transform and its weight, and the edgesE _f between vertices are necessary to calculate regularization, ensuring a smooth transition between parts of space.

Coordinate initialization is performed by sampling the vertices of the parametric model in a canonical pose and shape. The sampling method is outlined below. First, the neighborhood radius δ is selected, which is also a parameter for the weight v _w of each vertex from the set of vertices V _f : v _w = 2δ. The vertices of the model V _m are divided into groups according to the coordinates of the voxel into which each vertex will fall if the space is divided into voxels with a side of 2δ. Then the coordinates of the vertices in each group are averaged and the model vertex closest to the result is selected. It becomes a member of the set V _f .

The edges between the vertices E _f are defined as follows: since not only the vertices, but also the edges are defined for the parametric model, it is also possible for it to define the edges connecting the vertices, which means it is possible to define an undirected graph G _m = ( V _m , E _m ) and assign weights to the edges of the graph equal to the distance between the corresponding vertices. Then vertices v and u ( v , u ∈ V _f ) are considered connected if u for v or v for u is one of the N nearest neighbors along the shortest path length in the graph G _m . The value of N is taken to be 8.

This procedure avoids the problems associated with regularizing semantically unrelated vertices, for example, the vertices of the right and left legs - see Fig. 4, where from left to right are presented: the original model, the graph initialized by the proposed method, and the graph initialized by the usual procedure for searching for nearest neighbors. Particular attention should be paid to confusion of the legs during the normal procedure.

2.3. Morphing Field

The vertices of the graph G _f initialized in the previous section define the structure of the morphing field, which is used to transform the scene surface from the canonical form to the form corresponding to the next frame. As mentioned earlier, each vertex v ∈ V _f contains coordinates v _c , weight v _w and a normalized double quaternion v _dq encoding the Euclidean transform.

To calculate the transformation of a specific point x in three-dimensional space, the double quaternion mixing algorithm [B7] is used, for which it is necessary to find the K vertices closest to x from the number V _f , and then calculate the weighted sum of the double quaternions of these vertices. The weight for each term is calculated using the formula or to optimize further calculations it is assumed to be equal to 0 if || x − v _c || ₂ > 3 v _w . The result of the summation is a double quaternion, but it is not normalized and, accordingly, is not a Euclidean transform, so the result is further normalized. After this, the orthogonal transformation matrix R and the translation vector t are constructed from the normalized double quaternion, and the transformed point is the point Rx + t .

The morphing field is used not only to transform the surface from the canonical state to the current one, but also to perform the inverse transformation at the stage of integrating new data. To do this, a change is introduced to the integration process detailed in [B2] - the center of each voxel undergoes this transformation before being projected onto the depth map. As a result of such integration, the inverse transformation is realized without the need for additional calculation of inverse matrices, see Fig. 5, where on the left is a visualization of the morphing field, in the center is a surface obtained from the depth map of the next frame, on the right is the same surface after transformation to the canonical position.

2.4. Parametric model

The SMPL model [B8] was used as a parametric model in the proposed algorithm. This model has the following advantages:

- the algorithm for calculating the grid model consists of matrix multiplication and skinning, which allows for efficient implementation on GPUs;

- parameters for pose and shape are separated;

- the source materials of the algorithm are open;

- the largest CAESAR database [B9], to which there is no public access, was used as a basis for representing the space of human body shapes.

It should be noted that instead of the SMPL model, you can use other parametric human models, for example, HumanShape, S-SCAPE, your own parametric human model created by Texel LLC, etc.

It should also be noted that it is possible to use both the CAESAR database and other databases of human models, including our own database of human models created by Texel LLC.

During the integration of the SMPL model, the following changes were made to the algorithm for constructing this model:

(1) a version of the algorithm was created to run on a GPU;

(2) in order to increase productivity, the automatic calculation of derivatives of vertices based on model parameters was replaced with a separate algorithm based on the results of manual calculation, and general optimization was carried out;

(3) added calculation of surface normals and their derivatives by parameters for each vertex;

(4) According to the results of [B1], the component that affects the shape when changing the pose parameters was removed from the model to improve the convergence at the error minimization stage.

2.5. Parameter optimization

After pre-processing the next frame and converting it into a point cloud, it is necessary to change the pose of the model and the parameters of the morphing field so that the surface of the scene in the current view corresponds to the new observations, and the pose corresponds to the current position of the user. To achieve these goals, minimization of the sum of functions has been implemented, which can be divided into several groups:

- morphing field error;

- model position error;

- separation of scene transformation and skinning;

- regularization of field parameters.

The proposed functions are based on the ideas of [B1] with modifications to take into account information about the position of individual body segments.

The morphing field and model position errors are defined as functions

where C is the set of correspondences, that is, the set of pairs ( v, u ), in which v is the vertex of the visible surface of the scene or model in the canonical state, u is a point from the point cloud of the current frame, n _v is the normal to the surface at point v , and ñ _v and vertices or normals transformed by the morphing field and the model skinning function, respectively.

The set of correspondences C is formed from pairs of points closest to each other that belong to the same or neighboring segments. The proximity of the segments is established in advance and each body segment is associated with a key point of the skeleton. In turn, key points have a hierarchical structure - each point, except the root one, has a parent. Therefore, segments are considered adjacent if their corresponding keypoints are related by a parent-child relationship, with the exception of two special segments in the shoulder region, which are manually determined. An example of segmentation and neighborhood relationship is shown in Fig. 6, where different segments have different colors and adjacent segments are connected by lines.

The dissociation and regularization functions are defined as functions

(4)

(5)

where W _x is the skinning weight for vertex x , which is determined by the parametric model for each of its vertices, p is the Huber weight function [B10] with parameter δ = 0.2, T _x is the 4x4 Euclidean transformation matrix corresponding to the double quaternion of the vertex x . This function has an intuitive meaning - ensuring a smooth change in the surface, while neighboring vertices must be transformed by this function in the same way as they transform themselves. The Huber weight function, which decreases as the argument increases, will reduce the influence of regularization if the vertices are on different parts of the body, since in this case their skinning weights will differ significantly. Regularization increases stability, but reduces the ability to adapt to sudden changes.

Thus, using the model pose parameters and field transformation parameters, the function is minimized

(6)

However, if you perform unconditional optimization on the morphing field transform parameters, which are normalized double quaternions, they become unnormalized and no longer represent a Euclidean transform. At the same time, specifying conditions for observing a unit norm often reduces performance and worsens convergence. To solve this problem, a logarithmic transformation [B11] is additionally performed on the field parameters, which allows them to be represented as vectors from where each such vector corresponds to a normalized double quaternion.

Note that function (6) can be represented as a sum of squares. Therefore, to minimize such a function, we apply the Gauss-Newton method [B12]. As in method [B2], due to the fact that each point transformed by a field is affected only by N = 5 nearest vertices, there is no need to store the entire Jacobian matrix; it is enough to store only 6 N = 30 values for each term of the function E _field . For functions that depend on pose parameters, it is necessary to additionally store 3 × 25 = 75 values, according to the number of model shape parameters.

The system J ^T Jx = Je, where J is the Jacobian, e is the vector of values of the terms of the error function, can be solved directly, for example, by the Gaussian method, or iteratively by the method of congruent gradients [B13], and in the latter case to increase productivity and additional regularization a limited number of iterations can be carried out, leaving the solution approximate. Both solutions are discussed below.

2.6. Determining Visible Vertices

When calculating the error function, only the visible part of the scene surface is taken into account. In order to approximately determine the subset of visible surface vertices, the following algorithm was developed and applied:

- the surface is rendered into a depth map, for which standard rendering is performed, then the depth buffer is read, while the camera parameters are set to the same as at the integration stage of the previous frame;

- surface vertices are projected onto the image plane, and the corresponding pixel of the previously produced depth map is calculated from the projection coordinates;

- the distance z _vert from the plane to the vertex is compared with the z _depth value in the corresponding pixel of the depth map: if z _vert < z _depth + ε, then the vertex is considered visible. In this case, the value ε = 10 ^–2 was used.

In fig. 7 on the left is the result of rendering the surface into a depth map, and on the right is the original surface with visible vertices indicated by red dots)

2.7. Correction of the model's shape and canonical pose

After the integration stage of the new frame, the internal representation of the scene is supplemented with new information, and therefore it becomes possible to refine the assumption of the canonical pose and shape of the model. For this, an optimization algorithm is used, which is part of [B1]: the algorithm minimizes function (7) according to parameters θ , β :

(7)

(8)

(9)

where V _canon is the set of model vertices with pose and shape parameters θ , β , parameters θ _prev , β _prev are the previous parameter values, the function D ( x ) gives the TSDF value of the internal representation at a given point using trilinear interpolation over the values of nearby voxels. A zero TSDF value corresponds to a surface or the absence of information about this point. Consequently, the minimum of function (7) will be achieved if all the vertices of the model are located on the plane or outside the observation area. In order to prevent the model from leaving the observation area, regularization is applied using previous parameter values. As in the case of function (6), function (7) is a sum of squares, and therefore can be minimized by the methods described earlier.

The result of the above steps is a 3D model of the user's body. It should be noted that scanning the user’s body and constructing its model is sufficient to perform once and save in the appropriate format on the user’s computer or mobile device or in a cloud service. In the future, the user's body model can be updated or refined by repeated scanning or manual correction, for example, by indicating physical measurements taken from the user.

It should also be noted that the user's body model may include a detailed model of the user's head (including the face) used for photorealistic rendering in visualizing the user's clothing appearance. This allows you to get a greater emotional response from the user and, ultimately, helps to increase the volume of purchases in monetary terms.

2.8. Parametric model of the human body

A parametric model of the human body allows you to obtain a 3D model of the body based on two low-dimensional vectors that determine the shape and pose of a person. An example of models that can be obtained as a result of varying shape parameters with fixed pose parameters can be seen in Fig. 8, where the models are obtained by varying the first three shape parameters of the parametric SMPL model [B14]. Red dots indicate the positions of skeletal joints.

The parametric model requires significantly less hardware resources for its storage and processing and significantly less time spent on associated calculations than the corresponding geometric model. In addition, parameters extracted from a parametric model require fewer transformations to allow them to be compared with the dimensional characteristics of clothing items, and in some cases such transformations may not be required at all.

Today, the task of parametric human modeling is extremely relevant. Parametric models find application in many algorithms and areas of computer vision. In particular, many approaches to the problem of reconstructing a human figure from partial data, such as video, color images, information obtained from depth sensors, require the presence of some low-dimensional representation of the shape of the human body, which is a parametric model.

Parametric 3D models of a person can be used in algorithms for automatically obtaining human anthropometric data, which are used in the production and sale of clothing, and are also used in systems for constructing a virtual representation of a person (the so-called virtual avatar). Problems related to design, ergonomics, determining body shape from partial data, such as photographs from one angle, based on semantic parameters (height, weight, etc.) are solved using such models. Parametric models can also be used to generate artificial training data sets to solve other problems related to computer vision.

As initial data for constructing three-dimensional parametric 3D models, sets of three-dimensional photographs of people are used, which are described by a set of vertices and triangles that unite the vertices into a single surface. In addition, some jobs require additional information about existing 3D images. Thus, the authors of the HumanShape model [B15] use information about the position of 69 key anthropometric points in space for each three-dimensional image in the process of constructing a parametric model. The use of anthropometric points is provided in many works on the construction of parametric models, in particular in [B16, B17]. Marking these points for each 3D image requires knowledge of the anatomy of the human body, making the process slow and expensive.

The authors of many works on creating parametric models [B14, B15, B17, B18] use data sets containing three-dimensional photographs of people in several poses to build the model. Such data sets can be difficult to obtain due to the design limitations of the machines that produce 3D images of people.

Carrying out scientific work on constructing parametric models is often complicated by the fact that there are practically no open collections of three-dimensional images of the human body containing a sufficient number of unique people. The size of the collection used is a very important factor, directly affecting the quality of the resulting human body shape space. So Brett Allen et al. [B16] used a total of 125 male and female models, the authors of the model [B17] used a total of 45 models. The construction of the S-SCAPE model [B19] was carried out on a collection containing about 100 models. One of the largest commercially available sets of 3D images of the human body is CAESAR [B20]. This set contains data from about 4300 unique people and is used by the authors of the HumanShape [15] and SMPL [B14] models to construct the shape space.

Many of the existing kits have scanning defects. For example, three-dimensional images of the CAESAR data set are characterized by the presence of areas of the body surface in the area of the feet, hands and the outer surface of the arms, shoulders, head with missing information, as well as “glued” areas (Fig. 9). If there are systematic defects (in the area of the same body parts), the shapes of the affected body parts may not be accurately depicted. In addition, existing parametric models contain a relatively small number of vertices (about 6000–7000), which does not allow the transfer of fine details of the body relief.

Therefore, it was necessary to develop an automated, fast and accurate approach to construct a highly detailed parametric 3D model of a person based on a set of 3D images of the human body.

2.8.1. Review of existing approaches

The general idea of constructing a parametric model of the human body using a statistical approach is to perform the so-called. “registrations” for each 3D body image in the dataset. The registration process consists of deforming a human body template so that it describes the registered three-dimensional image of the body as accurately as possible while maintaining the functional correspondence of the template points. If each point performs a similar function on each of the deformed templates, the shape space of the human body can be described using PCA (Principal component analysis).

Existing approaches to constructing parametric models can be classified into two groups according to the method of deformation of the template model. One of them, the SCAPE family of parametric models of the human body, describes the process of registering a scan using transformations of the triangles that make up the surface of the template. Another group of models describes the registration process using template vertex transformations.

The SCAPE family of models includes such models as SCAPE [B17], BlendSCAPE [B18], S-SCAPE [B19]. In 2005, a paper was published introducing a model called SCAPE [B17]. To create the parametric model, the authors used a set of three-dimensional photographs in which each person is represented in many poses. As mentioned above, this work uses the deformation of individual triangles to register individual 3D images. However, the process of obtaining a model from the triangle deformation space is a complex optimization problem that requires solving the Poisson system. The result was a model that realistically reflected a wide range of deformation not only in shape space, but also in pose. The pose space was also described using the deformation space of individual triangles and, therefore, the application of a specific animation of human movement was difficult, due to the lack of the possibility of using skeletal animation.

The authors of the parametric models BlendSCAPE [B18] and S-SCAPE [B19] proposed their own construction methods based on the approaches used in the SCAPE model. The S-SCAPE model, proposed in 2010, is an improved yet simplified version of the SCAPE model, offering a way to reconstruct a 3D model without the need to solve complex Poisson equations. BlendSCAPE, proposed in 2012, offers a way to more accurately register a person's body based on 3D images of the same person in different poses.

The second family of parametric models uses deformations of individual vertices to perform the registration process. One of the first parametric models using this approach is the model from Allen et al. 2002 [B16]. This approach was then developed in the work of Pishchulin et al. (HumanShape model) [B15]. The parametric model of the human body, SMPL [B14], which shows the best results to date, also uses this approach.

One of the first statistical parametric models was the model [B16]. Here the Non-Rigid Deformation (NRD) algorithm was applied for the first time. An optimization problem of transforming a template surface was formulated in relation to a three-dimensional photograph of the body of a specific person, and a method for solving it was proposed. This work used a highly detailed human model template, but the parametric model was built from only 250 3D images. Such an algorithm has been demonstrated to cope well with scanning defects such as missing vertices. The model could operate, among other things, with semantic parameters of the form, such as weight, height, etc. However, the work of [B16] practically does not dwell on the topic of creating a human pose space. The model was built on a set of three-dimensional photographs of people standing in the same pose, and as a result did not describe the pose space.

In 2013, the “HumanShape” model [B15] appeared, developing the idea of the model [B16]. The model was built iteratively using the existing parametric model and improving it using the NRD algorithm. It used the S-SCAPE (Simplified Scape) model [B19] as initialization, which was significantly less computationally expensive to construct. The HumanShape model was trained on a full set of CAESAR 3D images, allowing it to capture a wide range of shapes. The use of the Linear Blend Skinning (LBS) technique in this model made it possible to apply arbitrary animations to the parametric model. One of the goals of this work was to make the resulting open source code available to researchers. However, the 3D image registration process used information about the position of key anthropometric points, information about which is often not found in 3D image databases. Obtaining such information is a labor-intensive process that requires manual marking or filming of people with special markers on their bodies.

In 2015, the SMPL model [B14] appeared, showing the best results to date and used in most problems that require a parametric model. In this work, the authors show that a registration process based on vertex deformations leads to much more accurate results compared to triangle deformations by comparing the results of their work with the BlendSCAPE model [B18]. Similar to the method proposed by the authors of BlendSCAPE, a human pose space is first constructed using a dataset containing 3D photographs of the same people in different poses. Next, the shape space is generated using the CAESAR dataset. In addition, the researchers in this article consider the issue of deformations in the shape of the human body caused by changes in posture. In this way, their model links shape and pose spaces together and is able to capture changes such as muscle deformation. The authors implemented, among other things, a dynamic version of the parametric model - DMPL, which predicts shape deformation, including based on the previous time parameters of the pose during animation. This made it possible to implement deformation of soft tissues, which added realism to the model.

This development uses a registration process based on vertex deformations, noted by the authors of the SMPL model as the most efficient and accurate. Model registration is performed using the approach described in the HumanShape model - the Non-Rigid Deformation algorithm. However, the proposed modification of the algorithm makes it automated and does not require the presence of anthropometric points.

Next, we consider two options for a parametric model based on initialization using two parametric models - HumanShape and SMPL.

2.8.2. Model shape space

Most of the existing studies on the topic under consideration have similar construction algorithms. Since a set of points in 3D images is often unstructured, it is necessary to structure it. A certain template of the human body is selected, described as a set of vertices and triangles consisting of vertices forming a surface. Next, for each vertex of the template, a transformation is sought that allows the deformed template to describe as accurately as possible a three-dimensional image of a specific person. The process of deforming a template model to fit a 3D photograph of a specific person is called registration. In this case, the deformation must occur in such a way that the functional correspondence of each point is maintained in all registrations. For example, if the template point was responsible for the position of the elbow of any of the hands, then during registration it should perform the same function.

Then the set of template model points is encoded as a vector and PCA (Principal Component Analysis) decomposition is performed. During decomposition, a space is constructed from the eigenvectors of the sample covariance matrix. Next, a subspace is selected from the first few eigenvectors with the largest eigenvalues. Thus, the resulting set of vectors represents the shape space of the human body with the least deviations. Parameterization is performed using coordinates in the resulting basis:

(10)

where M are the coordinates of the vertices of the final model, B is the number of shape parameters left in the model, e _i is the i -th eigenvector obtained by PCA decomposition, β _i is the value of the i -th shape parameter, M̅ are the coordinates of the template vertices.

Each of the two works mentioned formulates and executes the registration process differently. The authors of the HumanShape model use the SSCAPE model as an initialization and then improve it using the NRD algorithm. In the SMPL model, the pose space is first constructed. Then, by selecting a person’s pose in the next 3D image, the authors directly optimize the position of the registration vertices to obtain maximum correspondence to the vertices of the 3D image.

2.8.3. Algorithm for constructing the shape space of the HumanShape model

The authors of the HumanShape model perform the process of registering 3D images of people in two stages. In the first stage, the 3D image registration is approximated using an existing parametric model. The authors use the S-SCAPE model as such a model. As part of this stage, it is proposed to select the pose and shape parameters of the existing parametric model so that it describes the registered three-dimensional image as accurately as possible. To do this, an iterative two-stage method is used, repeated until convergence is achieved:

(1) optimize pose parameters to reduce the standard deviation between the position of key anthropometric points in the 3D image and the parametric model;

(2) optimize the pose and shape parameters to reduce the standard deviation from points on the model to the nearest points in the 3D image.

Due to the need to use key points, this method is not applicable to solve the problem. The optimizer is the built-in MatLab fmincon optimizer, which uses finite-difference differentiation to find the gradient of a parametric model.

Next, the authors propose to refine the resulting registration using the Non-Rigid Deformation algorithm. A description of this algorithm used by the authors of HumanShape is given in the next section.

2.8.4. Non-Rigid Deformation Algorithm

As mentioned earlier, this algorithm was first described in [B16]. In the HumanShape article, it was used with minor modifications. The general idea is to obtain, for each vertex of the template T , an affine transformation for vertex i , described by the 4x4 matrix Ai for vertex i , such that the resulting template surface matches the surface of the current 3D image S as closely as possible.

The accuracy of approximation of a 3D image by a template is described using a set of the following three loss functions:

(i) Smoothness term function

(eleven)

where edges(T) is the set of all edges of the template, A _i is the 4x4 transformation matrix for vertex i , || · || _F is the Frobenius norm of the matrix, Es is the result of calculating the Smoothness term function. Since the summation of the difference in the norms of the transformation matrices occurs along the edges, this part of the loss function penalizes the model for large differences between transformations identified for neighboring vertices;

(ii) Landmark term function

(12)

where K is the number of key points used, k _i is the number of the template vertex, which is the i -th key point, p _ki is the coordinates of the k _{i -th} vertex of the original template, A _ki is the transformation corresponding to the k _i -th vertex of the template, l _i is the i coordinate -key point in the current 3D image. This implies the use of the Euclidean norm. This part of the loss function should penalize the model for discrepancies between the positions of key points in the template and the 3D image. Since the HumanShape model was created on the CAESAR data set, it used 64 key points that take into account the anatomical features of the human structure;

(iii) Data term function

(13)

where w _i is the weight of the i -th vertex in the template, A _i is the transformation matrix of the i -th vertex, p _i is the coordinate of the i -th vertex in the template, NN(i) is the scan vertex, the nearest neighbor of the i -th vertex of the template. This part of the loss function penalizes the model if the distance between the transformed template vertices and their closest vertices in the 3D image is too large. The weights of each vertex were determined as the confidence of the scanning installation in the position of the vertex, which was selected as the nearest neighbor.

The final loss function is a linear combination of the three loss functions mentioned above:

E = αE _d + βE _l + γE _s (14)

When constructing the model, the authors of HumanShape used the following parameters: α = 1, β = 10 ⁻³ , γ = 10 ⁶ . In this case, several iterations were selected, at each of which the weights of the components of the previous iterations were multiplied by certain coefficients. In particular, the value of α remained unchanged, while the values of β and γ decreased by a factor of 4. This optimization process continued until the condition γ ≥ 1000 α was satisfied or until the error ceased to decrease from one iteration to the next. In addition, an initialization stage was used _{, at} which the parameter α was set equal to zero and deformation occurred only under the influence of the functions El and Es .

The loss function was optimized using the quasi-Newtonian optimizer LBFGS-B. Between iterations, the nearest neighbors were recalculated, and the nearest neighbor was considered successfully found if the angle between the normals of the vertices did not exceed 60°, and the distance between the vertices did not exceed 5 centimeters.

2.8.5. Model pose space

Next, in models such as HumanShape and SMPL, the skeletal animation technique Linear Blend Skinning (LBS) is used to describe the pose space to the template with the applied shape parameters. In this case, a skeleton fits into the template - a tree of vertices, where each vertex is characterized by a position in space and represents the point of articulation of the “bones” of the skeleton (see Fig. 10, where the influence of each joint on the vertices of the model is shown in color). Using this character animation technique makes the model compatible with most 3D model animation and processing software.

More specifically, the LBS technique requires a matrix of weights Wheren – number of vertices in the model,k – number of joints. This matrix describes the degree of influence of the transformation of individual joints on a specific vertex. The position of skeletal joints is described by the matrix. The transformation of each joint is described using an axis and rotation angle relative to the ancestor of the joint in the skeleton tree. Let – axis around which rotation will occurj-th joint, and is the rotation amplitude, then the rotation matrix forjth joint can be described using Rodrigues' formula:

(15)

where I ₃ is the identity matrix, and – vector , written as a skew-symmetric matrix. Such a matrix describes only the “local” transformation of the joint position, without taking into account the influence of all joints along the path from the root of the skeleton tree to the current point. Global transformation matrices can be calculated using the following formula:

(16)

Where – global transformation matrix for the k -th joint, A(k) – ordered set of all ancestor joints for the k -th, in order from the root of the skeleton, joint, exp ( θ _k ) – local rotation matrix for the joint, calculated by the formula ( 15), J _k – initial position of the joint. Then the transformation for each vertex is described by a convex combination of transformation matrices of individual joints:

(17)

The greater the joint weight for a particular vertex, the greater the impact it has on changing its coordinates. It is usually assumed that only the nearest joints influence a vertex, and those farther from it have zero weight. This allows the weight matrix to be stored in sparse form. Thus, in the HumanShape and SMPL models, no more than 3–4 joints influence the apex.

Encoding pose transformations using Rodrigue's formula requires 3 parameters per skeletal joint (the normalized vector represents the axis of rotation, and its norm represents the amplitude). The SMPL model skeleton contains k = 24 joints and hence the human pose is encoded using 3x24 =72 parameters. The HumanShape model allows rotation of the skeleton bones only around the axes specified by the model, determined during the construction of S-SCAPE. It defines k = 15 unique skeletal joints, but for some of them (for example, for the hip and shoulder joints) rotation is possible around several axes. The local rotation matrix for them can be obtained by multiplying the matrices for each axis obtained using the Rodrigue formula (15).

Changing the shape parameters of the human body leads to a change in the position of skeletal points. Using the LBS technique with incorrect joint positions can lead to significant artifacts, such as with the HumanShape model (see Fig. 11). In this model, the authors propose to recalculate the positions of skeletal points using a matrix – the i -th column of the LBS weight matrix, p _i is the ancestor of the i -th joint in the skeleton, and min{·, ·} is taking the element-wise minimum. Recalculation of joint coordinates occurs as follows:

The authors of the SMPL model use a similar approach, however, the regressor matrix of joint positions L is selected during the construction of the pose space, together with the weight matrix . This approach shows better results, ensuring the absence of such artifacts.

2.8.6. Pose normalization

Performing PCA on many registrations without any changes will introduce a large amount of variance into shape space due to pose changes. Therefore, before decomposition, it is necessary to normalize the pose of each registration from the data set, leading to a standard one. Such standard poses can be, for example, A-pose or T-pose. The A-pose of a person is characterized by arms slightly spread to the side, legs shoulder-width apart and is more often used, for example, when measuring a person’s anthropometric parameters for the production of clothing. This pose is used as a standard pose in the HumanShape model. In the T-pose, a person stands with his arms extended to the side, his feet shoulder-width apart. This pose is more common in 3D graphics and animation software and is used by the authors of SMPL.

Since when constructing the shape space of the SMPL model, a pre-built pose space is used, at the time of model registration it is possible to quite accurately select the pose parameters, and therefore restore the initial registration pose. The HumanShape model uses the NRD algorithm to refine the registration process, which can result in significant pose changes. The authors use the method proposed by Wuhrer et al. to normalize the pose. [B21], based on the optimization of localized Laplace coordinates of the averaged body shape.

2.9. Own approach to building a parametric model

While almost all approaches to creating a parametric model require a data set containing three-dimensional photographs of the body of the same person in different poses, this development aims to describe the possible process of building a model under conditions of a limited data set. The approach proposed by the authors of this development allows us to refine the shape space of the existing parametric model based on a set of three-dimensional images, where each person occurs only once. In addition, the proposed process has a high operating speed and is easily scalable when new data appears, because does not require manual marking of additional information, such as the position of key points.

First, the 3D image is preprocessed, then the registration is initialized using a parametric model, then the registration is refined by the NRD algorithm, then the template pose is normalized.

2.9.1. Data Preprocessing

One of the problems with the existing data set is the appearance of noise geometric structures inside the human body if the person moved during scanning. Such structures can cause nearest neighbor search errors during the operation of the NRD algorithm and lead to registration defects. Therefore, the data preprocessing step includes filtering such geometric structures. It often turns out that similar structures in 3D images lie in separate connected components, if you imagine the 3D model as a graph. More formally, let there be an undirected graph G = (V, E) , where V is the set of vertices corresponding to the vertices of a three-dimensional image of the human body, E is the set of edges. We will say that an edge exists if there is a triangle belonging to the surface of a three-dimensional image, the side of which connects these two vertices. In this case, by running a depth-first search on such a graph, you can obtain a list of connected components. The component with the largest number of vertices will be the result of filtering.

In some cases this may not be enough. Such structures may actually be inside the body, but be part of the closed surface of the body. An example of such a case can be seen in Fig. 12, where the noise is marked in yellow and green. This image contains a geometric structure (A, highlighted in green) that falls into a separate connected component of the graph and therefore can be removed using the algorithm described above.

In addition to it, there is also a noise structure, which is part of the surface of the left shoulder (B, highlighted in yellow). If such a structure is part of the surface of the human body, then it cannot be filtered using the classical algorithm for checking whether a point is inside a non-convex figure, which counts the number of intersections of an arbitrary ray emitted from the point being checked.

To filter such noisy vertices and triangles, the following approach is proposed. During the algorithm, the visibility of each point for the observer is determined. To do this, the depth map of the model is drawn from different angles and the distances from the camera to the point are compared with the value of the depth map.

More formally, let – camera position in space, – camera normal (i.e. its direction), is the rotation angle of the filtered 3D model at the current step t , v _i is the position of the model vertices at the current step.

(1) The model is rendered into a depth map. It turns out a map where N×M is the rendering resolution. Let the position of the projection of point v _i on the projection plane. Then d _ij is the distance from to v _i . An example of a depth map is shown in Fig. 13, where the color indicates the distance to the point, and the scale is graduated in meters. Thus, the depth map represents the distance from the camera's projection plane to the surface of the model.

(2) The value d _i = ( – ) – the distance from the vertex to the corresponding point on the projection plane. If the resulting value d _i is greater than the corresponding value on the depth map, this means that the point is invisible from this angle.

(3) If a point is visible from at least one angle, then it is assumed to be visible.

The rotation angles of the model can be chosen arbitrarily. In this development, we use an iteration of each of the three rotation angles along a linear grid from 0 to 2π with a step of π/10.

This algorithm allows you to filter out all invisible vertices that are inside a three-dimensional image of a person, and therefore are noisy.

Next, the three-dimensional image is rotated around the y-axis so that the person is facing in the direction of the z- axis. This is necessary for good initialization of registration by the algorithm described in the next section of the description.

Lety _min ,y _max – minimum and maximum axis coordinatesy among all points of the rotated 3D model andh =y _max −y _min – height of the model. Then, if you filter out all points whose height satisfies the conditiony _i <y _min + 0.03h, most of such a set will consist of vertices related to human feet. Then clustering into two components is performed using the KMeans algorithm. Resulting centroidsc _low1 Andc _low2 are the centers of human feet. If we take the area of points above, for example,y _min + 0.09h <y _i <y _min + 0.12h, then such an area would contain the lower part of a person's legs. Similarly, using the KMeans algorithm you can find the centroidsc _high1 Andc _high2 . Now you need to select the angle of rotation so that the straight lines passing through the pointsc _low1 ,c _low2 , Andc _high1 ,c _high2 , were perpendicular to the axisz, while straightc _high1 ,c _high2 would have smaller coordinatesz.

2.9.2. Initializing registration using a parametric model

At this stage, the registration of a three-dimensional image is approximated using a certain parametric model. As part of this development, the HumanShape and SMPL models were tested. For a parametric model, shape parameters are selected , poses , as well as general transformation parameters such as offset and scaling factor s .

The calculation graph when initializing registration by the parametric model is presented in Fig. 14, which shows the step-by-step application of shape parameters and pose parameters, first recalculating the coordinates of skeletal joints, and then applying general transformation parameters such as scaling and translation. Next, the loss function is calculated based on the standard deviation from the nearest vertices.

In this approach, the standard deviation between the vertices of the target point cloud and the closest vertices of the parametric model is used as a loss function. The nearest vertices of the template are recalculated at each iteration of gradient descent, but this is not a differentiable function. The nearest vertex is assumed to be constant within one descent iteration. In fig. 14 you can see a visualization of the calculation graph. If a characteristic pose is known for the models in the data set, it is possible to initialize the pose parameters of the parametric model accordingly, as well as introduce regularization on the pose parameters:

(18)

where v _i is the position of the vertex of the i -th three-dimensional image, T is the set of template vertices, NNT (v _i ) is the coordinate of the _vertex closest to vi from T , λ is the regularization coefficient, I is the set of numbers of joints subject to regularization, – pose parameters, – model shape parameters, – displacement of the model relative to the origin, s – scale of the model.

To perform optimization, it is proposed to use the Adam gradient optimizer. To speed up the convergence rate and avoid local minima, it is proposed to use the following optimization technique.

(1) Perform n iterations of gradient descent using the Adam optimizer, optimizing the loss function only according to the transformation parameters.

(2) Perform n iterations of gradient descent using the Adam optimizer, optimizing the loss function only for the pose parameters.

(3) Using four optimizers, alternately perform 10 gradient descent steps with each optimizer (in the following order: pose, shape, transformation, all parameters) until the required level of convergence is achieved or until 4n iterations are reached at this step.

In this approach, the value n = 100 is used. Since the values of the parameters β _i , can be quite large, a scaling factor has been applied to them. An important factor is the selection of learning rate values for each optimizer. These values are shown in Table 1 below.

2.9.3. Modification of the Non-Rigid Deformation algorithm

The implementation of the NRD algorithm performed in this work has the following features:

- the original algorithm uses the quasi-Newtonian LBFGS-B as an optimizer, while the proposed approach uses the Adam gradient optimizer. The following criterion is used as a criterion for stopping gradient descent at each iteration: stopping occurs if during the patience iterations the loss function has not improved by more than tolerance or the maximum number of steps max_iter has been reached;

- to calculate the normals to the vertices of the template, the original algorithm uses a truncated calculation along the normal of one randomly selected triangle containing the vertex. In the proposed approach, a complete calculation of normals is performed: the normal is determined by averaging all the normals of the triangles adjacent to a given vertex;

- due to the fact that most 3D laser images contained missing points related to the feet, their optimization was prohibited in the original algorithm of the authors of the HumanShape model. The collection available to the authors of this development does not contain such artifacts, therefore, their optimization was allowed.

The authors of the HumanShape model presented the results of registrations on the CAESAR data set, which made it possible to calculate for each triangle of the template its average area over all registrations. The presence of large triangles does not make it possible to make the surface of the human body smooth during registration or to convey small details. Therefore, the n statistically largest triangles in the template were divided into four parts by adding new vertices to the middle of each side of the triangle.

The loss function in the original algorithm optimizes the positions of the vertices, without taking into account the fact that the cloud of resulting points forms a surface. As a result, the coordinates of the template vertices turned out to be precisely selected, but when they were connected into a surface by triangles, relief defects arose - excessively sharp drops, depressions and bumps.

To take into account the features of the surface formed by the template, a new component has been added to the loss function, which penalizes for too large angles between the normals of neighboring triangles:

(19)

where | edges | – number of edges in the template, edges ( T ) – set of edges of the template, n _i – normal vector for triangle i , – the angle between the normals, F (·) – application of the activation function to the angle value.

The following function was used as an activation function (see also Fig. 15):

The use of a better method of initializing registration made it possible to avoid the use of anthropometric points in the operation of the NRD algorithm, due to which full automation of the method was achieved. The new loss function minimized during the algorithm has the form:

(20)

2.9.4. Pose normalization

At the parametric model registration initialization stage, pose parameters were selected for the SMPL model. Then, as a result of applying the NRD algorithm, the person's pose could change slightly if the pose parameters were not selected accurately. In this case, it is necessary to refine the pose by optimization:

(21)

where T _nrd,i is the position of the vertex of the template i obtained as a result of registration using the NRD algorithm, T _i is the position of the vertex i of the parametric model.

Using the obtained pose parameters, you can calculate the Linear Blend Skinning matrices, and then, having calculated the inverse ones, multiply them by the coordinates of the vertices T _nrd . In this way, registration in the initial pose will be obtained. For the SMPL model, the starting pose is the T-pose, where the person's arms are spread to the side and the legs are spread shoulder-width apart.

The HumanShape model contains pose leakage into the shape parameters, that is, changing the shape parameters causes the pose of the parametric model to change. This causes the initial pose for the parametric model to change as the shape parameters change. Therefore, the above approach is not applicable for the HumanShape based model. In this work, it is proposed to correct this effect by constructing a skeleton in a neutral pose, and then optimizing the positions of the skeletal points:

(22)

where Ĵ _i is the position of joint i of the constructed skeleton in A-pose, J _nrd,i is the position of joint i of the registration skeleton obtained using the NRD algorithm. Here, optimization is performed based on the parameters of the registration pose obtained as a result of the NRD algorithm.

2.10. Experimental study

The following describes the conditions for experimental testing of the described approach.

2.10.1. Dataset used

The collection obtained using the TEXEL Portal 3D human body scanner was used as the initial set of 3D images. This scanner is highly accurate, with a scanning error of about 1.4 mm. For the experiments, a set of 500 3D photographs of people was selected, containing approximately the same number of male and female 3D models. The presented images were highly detailed and contained 250 thousand vertices and 500 thousand triangles each.

In addition, to test and compare the quality of the resulting parametric model, the CAESAR data set was used - a set of 3D images used by the authors of the SMPL and HumanShape parametric models to construct pose space. It contains three-dimensional photographs of 4,394 different people, each of whom is presented in three poses. In this case, only photographs were used where a person stands in an A-pose. No supporting meta-information, including information about the positions of key anthropometric points, was used during the registration process.

2.10.2. Implementation of the registration initialization algorithm

Initialization of registration using a parametric model is an important part of the proposed approach to registration of 3D images. The quality of the resulting vectors describing the shape space of the parametric model depends on the quality of this stage. If the pose parameters are selected incorrectly during initialization, then when applying the NRD algorithm, incorrect vertices of the 3D image will be selected as nearest neighbors, as a result of which the functional correspondence of the template vertices will be violated. Optimization problem (18) is a complex problem with a large number of local minima. To avoid the Adam optimizer from falling into local minima, it is important to select learning rate values. The parameter values used in this work are given in Table 1.

Since the templates of the parametric models used contain a small number of vertices (the SMPL template contains 6890 vertices, the HumanShape template contains 6449 vertices), there is no need to use all the vertices of a 3D image at this stage. In the proposed approach, before performing optimization, a set of 7000 vertices of the original 3D image was randomly selected. Then the optimization problem (18) was solved for the resulting set of vertices.

Since the approximate pose of all people in the available set of 3D images is known, the parametric model was initialized with A-pose, while the rotations of the head and foot joints were regularized using the value λ = 10.

2.10.3. Testing the implementation of the Non-Rigid Deformation algorithm

Since the new data collection does not contain information about the confidence of the scanning installation in the position of the vertices of three-dimensional images, the corresponding weighting coefficients of formula (13) were set equal to one for all vertices. As a stopping criterion, we used the previously described criterion with parameter values patience = 70, tolerance = 10 ⁻² and the maximum number of gradient descent steps per iteration max_iter = 300.

When searching for nearest vertices, it was considered that the nearest neighbor was found successfully if it was located at a distance of no more than 3 cm, and the angle between the normal of the 3D image vertex and the normal of the parametric model vertex was less than 45°. Optimizing the position of the template's hand vertices when initializing with the parametric SMPL model was prohibited because in many of the 3D images the fingers are clenched into a fist, and the parametric SMPL model has a straight palm.

Optimizing the position of the template's head vertices was also prohibited because many 3D images contain voluminous hairstyles that distort the shape of the head. The weighting coefficients used for the loss function (20) are presented in Table 2. The coefficients by which the weights of the loss function were multiplied are marked in the table as r _α , r _β , r _γ .

In fig. Figures 16–18 show a comparison of the results of the proposed modification of the NRD algorithm and the original algorithm [B15] on a three-dimensional image from the CAESAR set. From left to right are: the proposed improved version of the NRD, the original 3D image and the original version of the NRD. The three-dimensional image contains gaps - areas with missing data. The quality of the algorithm's work was determined by the distance from the vertices of the obtained registration to the vertices of the three-dimensional image. In fig. 16–18, the color of each registration vertex indicates the distances to the nearest vertices of the three-dimensional image. White means zero deviation, while red means deviations of 15 mm or more. In this experiment, the number of vertices in the template did not increase during the operation of the proposed algorithm.

It can be seen that better results were obtained for most of the body surface, as indicated by lighter areas of the surface. The advantage of the proposed algorithm is especially noticeable on the topography of the upper back and abdomen. The tops of the feet appear in red because the original 3D image has scan defects in that area of the body. It almost completely lacks vertices corresponding to the feet. The tops of the hands also have a high registration error here, since in the original 3D image the fingers are straight, but in the template they are clenched into a fist.

In addition, performance measurements were made. The original implementation of the NRD algorithm takes 235–239 seconds to build a registration, while the proposed implementation takes 7–8 seconds. Initialization of registration by a parametric model lasts 26–29 minutes in the original implementation of the authors of HumanShape, and the proposed implementation spends 13 seconds on this stage.

The operation of the new component (19) is illustrated in FIG. 19, where from left to right the results of the NRD algorithm are presented: with a classical loss function, 25 thousand vertices in a three-dimensional image; with classic loss function, 250 thousand vertices in a 3D image; and a modified loss function, 250 thousand vertices in a 3D image. You can see that in the right image the surface of the body is smoother, and in the left image the surface of the body is angular, with relief differences that are not characteristic of the human body. In addition, without using component (19), intersections of body surface triangles arise, which are shown in black in the left image.

The registration process was performed on full-resolution 3D images (250 thousand vertices), while the original implementation of the NRD algorithm [B15] randomly selected 19347 vertices from a 3D image and worked only on them. To build a parametric model, we used an increase in the number of vertices in the template to 18–19 thousand.

An example of a three-dimensional image used in constructing the parametric model, as well as the results of its registration based on initialization by the HumanShape and SMPL parametric models can be seen in Fig. 20, which presents a comparison of registrations of the same three-dimensional image using different parametric models. From left to right: HumanShape-based registration (18659 vertices), original 3D image (250000 vertices), SMPL-based registration (18890 vertices). The SMPL parametric model template has better detail of the head, fingers, and toes, so registration based on this parametric model appears more realistic. However, the registrations obtained based on both parametric models - HumanShape and SMPL - have fairly small deviations from the original images.

The authors of the HumanShape model use in their work the distance from the template points to the closest points of the three-dimensional image as a metric for registration success. This metric reflects the quality of the approximation of a 3D image by a template point cloud. In this case, the surface relief formed by the template is not taken into account. To take into account the surface topography of the resulting registration, the following metric is proposed:

(23)

where T _S is the set of all template triangles, t is the surface triangle under consideration, v _i is vertex i of the three-dimensional image, d ( v _i , t ) is the distance from vertex v _i to triangle t , err _i is the deviation for vertex i of the three-dimensional image.

Thus, formula (23) calculates the distances from each vertex of the three-dimensional image to the surface of the 3D model obtained as a result of registration. It is proposed to calculate these values for each vertex of a three-dimensional image, and then, for example, group them by quantiles.

The results of registering your own data set can be seen in Fig. 21. In this graph, all deviations for one scan were divided into quantiles, and then the values of individual quantiles were averaged across all three-dimensional images. Registration results obtained based on the HumanShape parametric model are indicated in orange, and registration results obtained from the SMPL parametric model are indicated in blue. The solid line marks the results of initializing the registration using the method described in Section 2.9.2, and the dotted line marks the results of improving the registrations using the proposed modification of the NRD algorithm.

In fig. 21A shows that the SMPL parametric model approximates the body shape of people in existing 3D images better than the HumanShape parametric model. The use of the proposed modification of the NRD algorithm provides a significant improvement in the quality of registration on both models. At the same time, registrations based on the SMPL parametric model also have better quality compared to registrations based on the HumanShape parametric model.

Together with the software implementation of the algorithms, the authors of HumanShape made the results of registration of the CAESAR data set publicly available. These registration results can be used to directly compare the performance of algorithms. However, publicly available CAESAR registrations have a neutral posture. To carry out the experiment, for each of the registrations, pose parameters were selected in accordance with three-dimensional photographs by minimizing the functional (18) only by , . The comparison results with the method proposed in this work are presented in Fig. 21B, where blue indicates the registration results obtained by the proposed method, and orange indicates those obtained by the original method.

From the experimental results, we can conclude that the method for registering three-dimensional images proposed in this approach has significantly better quality and speed than the original method of the authors of the HumanShape parametric model. The proposed method of increasing the number of vertices in the template makes it possible to more accurately describe the relief of the human body, taking into account small details.

2.10.4. Results of constructing a parametric model

As described earlier, the construction of a new shape space of the parametric model was carried out by performing PCA decomposition on the set of all available registrations. As a result, based on the SMPL parametric model, two parametric models (male and female bodies) were built. While the original SMPL model template contains 6890 vertices, the resulting parametric model templates contain 18890 vertices. Since adding new vertices was done by adding a vertex to the midpoints of the edges of the template surface triangles, the Linear Blend Skinning weight vector (formula 17) was calculated as , where w _i , w _j are vectors of LBS weights for the vertices forming the edge. The regressor of the position of skeletal joints remained unchanged, i.e. the influence of new vertices on the positions of skeletal joints was assumed to be zero.

During PCA decomposition, the eigenvalues λ _i of the sample covariance matrix are calculated. The squares of the obtained eigenvalues λ _i ² can be interpreted as the amount of dispersion of the human body shape explained by the parameter i . In this case, you can calculate the proportion of explained variance in the human body shape using the first k parameters (Explained Variance Ratio):

(24)

A graph of the dependence of the EV R metric on the number of components used is presented in Fig. 22. Here, the percentage of variance explained is calculated separately for the resulting male and female parametric models.

In fig. Figure 22 shows that about 97.5% of the variance in body shape is explained by the first 10 parameters of the shape space. Moreover, the use of 20 parameters allows us to explain almost 99% of the variance. This means that using 10–20 shape space parameters should be sufficient to describe any three-dimensional image of the human body. Thus, the resulting representation is indeed low-dimensional.

In fig. Figure 23 shows a comparison of the initial pose and shape of the SMPL parametric model and the resulting parametric model for the male and female body. It can be seen that the resulting parametric model is more detailed. In particular, the chest is reflected more smoothly and accurately, and small details of the body structure, such as collarbones, are present.

However, within the framework of this work, the head shape space of the parametric model was not specified. During the operation of the proposed modification of the NRD algorithm, the positions of the vertices did not change. This decision was made because many of the original 3D images of people contained voluminous hairstyles that distorted the shape of the head as a result of the NRD algorithm. As a development of this work, it is proposed to add segmentation of hair and clothing details in three-dimensional photographs of people using color information. The 3D image vertices related to hairstyle elements should have lower confidence values in the loss function of the NRD algorithm (formula 13).

Visualization of the result of changing the first four shape parameters with the remaining parameters of the resulting female parametric model fixed is presented in Fig. 24.

Thus, the proposed modifications of the NRD algorithm and initialization of registration using a parametric model made it possible to fully automate the method. The proposed approach does not require the use of information about the position of key anthropometric points, the manual marking of which is an extremely labor-intensive process.

It has been experimentally proven that there is a significant increase in the quality of registration of three-dimensional images of the human body compared to the original algorithm [B15]. At the same time, the speed of the algorithm has increased significantly. Increasing the detail of the template made it possible to reflect fine details of the body relief on the registrations of three-dimensional images.

Based on the SMPL parametric model, our own parametric model was obtained, which is a refinement of the human body shape space on a new data set.

2.10.5. Evaluating scan results

Determining the degree of correspondence between the resulting three-dimensional model and the user of the system is a complex task, which is best resolved by comparing the result with a three-dimensional model of the user, taken as a reference. To obtain three-dimensional images in reality, three-dimensional scanners or systems consisting of many synchronized RGB cameras are used. However, there is a problem in obtaining images that are suitable as models for testing in the problem under consideration. The use of three-dimensional scanners designed for static objects is impossible in this case for two reasons: firstly, due to the fact that a person cannot stand absolutely still, artifacts will appear in the image, distorting the assessment, and secondly, even without taking into account scanning artifacts , which may not exist, if the scanning occurs quickly enough, the person will not be able to exactly repeat the pose in which he stood in the first frame of the filmed sequence. Therefore, for this task there is a need to use multi-chamber systems. The system must obtain a 3D image of a person at the moment when the depth sensor receives the first frame of the sequence, which results in synchronization difficulties. In addition, it is worth noting that such systems are not available everywhere due to their high cost and the need for fine tuning.

Taking into account the above problems, as well as due to the lack of public

With the availability of sets containing the data necessary for the assessment, a tool was developed to obtain a synthetic data set close to real ones. The tool receives as input a three-dimensional image of a person created, for example, using a three-dimensional scanner. Before using the imaging tool, the skeleton is determined using the rigging technique. The tool then applies skeletal animation, which is transferred to the image using the LBS skinning algorithm. After this, the necessary data is generated for each animation frame: the depth map is rendered from the point of view of the virtual camera, and the position of key points of the skeleton is recorded. In addition, for each frame, semantic segmentation is generated according to the following algorithm: the number of the key point with the greatest influence on the skinning for those vertices that fall into a given pixel when projected onto the image plane is recorded in each pixel. After generation is complete, Gaussian noise can be added to the depth maps to simulate the errors encountered when shooting with a real sensor.

The system ensures the generation of all the necessary data for the operation of the algorithm, and also generates a three-dimensional model of a person in a position that exactly matches the position in the first frame of the sequence, which makes it possible to make an accurate assessment. At the same time, the universal interface allows you to use arbitrary skeletal animation to test the algorithm under different user behavior. Examples of generated data are shown in FIG. 40: from left to right – depth map, segmentation, key points, reference model.

The database created for experimental testing in this work contains about 2000 frames with a resolution of 420x880 pixels and added noise and consists of 4 video sequences reflecting various situations of using the algorithm:

-a person stands and makes small involuntary movements (breathing, hand movements);

- a similar scene with a rotation around 1° per frame;

- a man waves his hand;

- a person walking.

The first two sequences are aimed at testing a scanning scenario where a person tries to maintain the same position, the subsequent ones are designed to test under conditions of fast natural movements. To conduct an experimental evaluation of the algorithm, the following two metrics were used to evaluate different components of the system:

where V , S , M are the sets of vertices of the resulting surface, the reference surface and the surface of the parametric model with canonical parameters, respectively, – closest to the top vertex of S , – closest to the top vertex from M , and – normal to the surface at a point .

The first metric is the average value of the distance from the vertices of the result to the surface plane of the reference model, the second is the averaging of similar distance metrics from the model to the reference and vice versa.

2.11 Creating a 3D user avatar

To visualize clothing on a person, as well as for a more visual representation of the results of the algorithm, it is proposed to create a 3D avatar of a person using data obtained at the optical scanning stage. A 3D avatar is a three-dimensional textured model of a person’s body, consistent with his anthropometric data. To extract texture, data from a color camera is used, processed in conjunction with depth maps at the stage of creating a user model.

To speed up texture extraction, you can use an approach based on deep neural networks, an alternative to the classical texture extraction method. The neural network approach allows you to quickly and with acceptable accuracy extract the user's texture from one or more color images. To extract texture, a neural network is used, trained on a set of correspondences between the user's photo and his texture. During the training process, images of different people from different angles with a known texture map are fed to the input of the neural network, and the error in restoring such a texture map is penalized.

To generate realistic body texture, it is important to determine skin tone from a photograph or a 3D head shot. In the case of a three-dimensional image, the face is rendered from the front, and the problem is reduced to the case of a two-dimensional image. The algorithm for determining skin tone can be described by the following basic steps.

1. Preparing the photo (scaling and cropping to 512x512 pixels), or rendering a 3D model of the head into an image of 512x512 pixels.

2. Neural network segmentation of the face using ready-made or newly trained models to highlight segments such as facial skin, hair, neck, eyes, eyebrows, etc.

3. Formation of a mask for facial skin based on the segmentation results.

4. Morphological mask erosion to reduce artifacts at the edges of the area

5. (Optional) Apply a bilateral filter or Gaussian blur to the original image.

6. Isolating pixels from the original (preprocessed) image using a mask and converting colors into Lab space.

7. Clustering of pixel colors in the Lab space using the K Means method with the number of clusters equal to 3.

8. The centroid of the largest cluster is taken as the skin tone.

9. Coloring the texture in the Lab color space while maintaining the relative illumination of the areas:

- the average value of L is determined over the entire texture;

- the average value L is subtracted from the L component of each texture pixel;

- the L value of a certain skin tone is added to each texture pixel;

- The ab channels for each texture pixel are taken from the skin tone.

3. Processing scan results and building a model of the user’s body based on color images

This section provides a brief description of the basic Human Mesh Recovery (HMR) algorithm proposed in “End-to-end Recovery of Human Shape and Pose” [B37], and also reveals proposed modifications. More details on the proposed approach can be found in [B33].

3.1 Human Mesh Recovery

The Human Mesh Recovery (HMR) algorithm [B37] relies on the SMPL parametric 3D human model [B39], but can work with an arbitrary parametric model. Section 3.1.1 contains a description of the HMR architecture and the method for restoring the parameters of a 3D model. To train this method, an adversarial training procedure is used, the main element of which is a discriminator network, the architecture of which is described in Section 3.1.2. Section 3.1.3 provides a description of the loss functions used to train the HMR and the discriminator neural network.

3.1.1 Algorithm for restoring parameters of a human model

Let us consider the architecture of the Human Mesh Recovery (HMR) algorithm [B37] in more detail; its diagram is shown in Fig. 36. The ResNet-50 v2 architecture [B36] with removed fully connected layers, pre-trained on the ImageNet classification task [B38], is used as a network for extracting parameters from the source image. Algorithm 1 below describes the procedure for recovering the parameters of a 3D SMPL model based on the extracted parameters. The parameters extracted from the image are stretched line by line into a vector of 2048 elements ( ), which is concatenated with the standard parameters of the SMPL model ( – average shape parameters calculated from the collection of 3D images used to train the parametric model, and – posture parameters corresponding to the human T-pose: feet shoulder-width apart, arms extended to the sides at right angles relative to the body) and fed to the input of the iterative parameter recovery module ( ), which consists of two fully connected layers separated by a dropout and a fully connected layer that provides a bias for the shape and pose parameters of the SMPL model ( ).

3.1.2 Discriminators

The parameters of the SMPL model obtained by the neural network algorithm are fed to the input of several discriminator networks, the architecture of which is shown in Fig. 37. To determine the realism of a shape, one fully connected neural network with two layers is used, which receives a vector of shape parameters as input and outputs a number reflecting the realism of the shape parameters. For pose parameters, 24 fully connected discriminator neural networks are used (one for pose parameters for each skeletal joint and one common) with a division of some parameters. To begin with, the Rodrigue rotation formula is applied to the three parameters of the pose (they set the direction of the rotation axis and the magnitude of the angle) for each joint, which converts the parameters into a matrix representation (rotation matrix ). After this, the resulting representation is fed as input to two fully connected layers of size , common to all pose parameters, transforming the input vector into a common parametric dimension space The resulting parameter vectors are fed to the input of 23 neurons, which produce predictions for each joint separately, and are also concatenated among themselves and transferred to two fully connected layers of size 1024, the output of which is a prediction for all pose parameters together. The values obtained from all discriminators are concatenated, giving the output a vector of length 25, which describes the realism of the predicted parameters of the 3D model.

3.1.3 Training

To train the algorithm, the authors proposed using a loss function consisting of four penalties. The output of the algorithm for each input image is the pose and shape parameters of the SMPL model ( ), discriminator prediction vector ( ), camera parameters, as well as coordinates of the 3D human pose ( ) extracted from the SMPL model based on the predicted parameters. To model the camera, the authors use a weak perspective projection model, for which the following projection formula is valid:

(25)

Where – coordinates of a point in three-dimensional space, – coordinate of the projected point, – global rotation predicted by the algorithm, And are the scale and shift vector in two-dimensional space, also predicted by the algorithm, and P is the orthographic projection. For penalties for deviation of the predicted 2D pose from the reference one, the reprojection error with the L1 norm is used:

(26)

Where – number of examples, – number of joints in the skeleton model under consideration, – visibility of the j- th key point of the i- th example (taken from the reference marking), and – coordinates of the reference and predicted j- th key point of the i- th example. Also, if there are 3D pose markings and SMPL model parameters, the corresponding penalties are applied (standard deviation for both groups of parameters):

(27)

Another term in the error function is a penalty on the discriminator's predictions, aimed at making them as close to 1 as possible (that is, so that the discriminator considers all predicted parameters to be realistic):

(28)

Where – a vector of 1 , matching in size to the concatenated output of the discriminators. The overall loss function for training the HMR algorithm [37] is given by the following formula:

(29)

Where – coefficient selected using the validation set, – indicator function equal to 1 only for examples that have markings of 3D pose and SMPL model parameters.

To train discriminator networks, along with the predicted parameters of the SMPL 3D model, reference ones obtained from labeled data collections are used. The loss function for training the ith discriminator is as follows:

(thirty)

Where – number of examples from the reference collection, is the number of examples predicted by the HMR model.

3.2 Proposed modifications of the Human Mesh Recovery algorithm

The proposed modifications of the Human Mesh Recovery method [37] are aimed at eliminating the shortcomings identified during the analysis of the basic algorithm. To improve the accuracy of the method, including improving the quality of human figure reconstruction, a modification of the architecture (section 3.2.1) and a modification of the training procedure (section 3.2.2) were proposed.

3.2.1 Architecture modifications

Human segmentation is a fairly simple, yet informative representation that allows you to obtain information about a person’s figure. To increase the accuracy of restoring the parameters of a 3D model of a human figure and pose, it is proposed to restore a segmentation map for the predicted 3D model and use the resulting segmentation map to train the algorithm.

A number of current methods use approximate differentiable rendering [B40] or learnable rendering [B41] to obtain a segmentation map based on a predicted 3D model, but this increases the training time and also complicates the training procedure itself. Instead, it is proposed to use a different approach, similar to that used in some early two-step methods for restoring parameters of a 3D human model, for example, in [B35].

A point-by-point projection of the vertices of a parametric 3D mesh model onto the original image is the simplest way to obtain an approximation of the silhouette (the resulting single-channel image contains a value of 255 in the pixels to which the vertices were projected, and 0 in all others). But without preprocessing, such an approximation is difficult to use, since there will be empty space between the projected points, which, when using a pixel-by-pixel penalty with a reference segmentation map, will be interpreted as an algorithm error. In addition to the projection of vertices, it is proposed to calculate a distance transform for the resulting single-channel image using a chamfer distance, as a result of which a single-channel image will be obtained, each pixel of which will contain the distance to the nearest projected point. This distance map will allow the penalty to be weighted proportionally to the distance. A diagram of the modified method and examples of vertex projections and distance maps are shown in Fig. 38.

3.2.2 Modifications to the training procedure

Based on the calculated distance map (Fig. 39, ) and the reference segmentation map of Fig. 39, ) you can obtain a map of the areas of the reference map that are most distant from the predicted points by element-wise multiplication of two maps (Fig. 39, ). This produces a single-channel image, each pixel of which contains 0 in those areas where the reference class is background, and for areas with the reference class person, the greater the value, the further from this pixel the nearest projected vertex of the predicted mesh parametric 3D model is located. By penalizing high values in the distance map calculated in this way, it is possible to reduce the area of the reference human silhouette area not covered by the predicted silhouette.

To achieve the opposite effect: reducing the area of the predicted silhouette area not covered by the reference silhouette, it is proposed to use a distance map (Fig. 39, ), which is calculated based on the predicted segmentation map (Fig. 39, ) and distance maps for the reference segmentation map (Fig. 39, ), obtained similarly to the distance map . Thus, the total result based on the obtained distance maps can be expressed by the following formula:

(31)

Where – predicted segmentation map, – reference segmentation map, – distance map calculated based on the predicted segmentation map, is the distance map calculated for the reference segmentation map, and the symbol denotes element-wise multiplication. Visualization of the stages of calculating formula (31) is shown in Fig. 39.

The final expression for the proposed segmentation-based penalty is as follows:

(32)

Where – height and width of segmentation maps. The final loss function used to train the modified algorithm (HMR-seg) takes the form:

(33)

Where – coefficient selected using the validation set, – indicator function equal to 1 only for examples that have segmentation markings.

4. Determination of anthropometric parameters of the user’s body

The algorithm for determining the anthropometric parameters of the user's body includes (a) obtaining dimensional characteristics of a person based on a fitted model and a three-dimensional image and (b) determining the type of human figure.

4.1. Obtaining user dimensional attributes

Dimensional features can be calculated by various methods: linear algebra methods, machine learning methods, combined methods. Determining measurements using linear algebra methods is similar to measuring measurements from the human body manually, only it is carried out from a three-dimensional photograph of the human body. Machine learning algorithms are built from models trained on pre-labeled data. Combined methods are implemented partly on the basis of linear algebra algorithms, partly on machine learning algorithms. For example, intermediate anthropometric parameters are calculated based on a 3D image, which are then used to train a regression model.

It is also possible to use different methods to calculate different measurements, i.e. each measurement can be calculated using the method that gives the minimum error on the test sample.

4.1.1. Obtaining dimensional features based on linear algebra methods

Dimensional characteristics were determined on the basis of numerical values called dimensions. When carrying out measurements, the following standards were taken into account: ISO 8559, ISO 7250, GOST 17522-72, GOST 31396-2009, OST 17-326-81 [23–30]. Often measurements from different standards are duplicated, and sometimes measurement techniques contradict each other [B31]. In such cases, preference was given to the ISO 8559 standard. A total of 89 measurements were carried out, of which 25 measurements were symmetrical, i.e. consisting of three dimensions: right, left and middle. The full list of dimensional characteristics (DS) is given in Table 3.

Table 3

RP number Name of RP Relevant RP
in ISO 8559 standard 1 Back shoulder width 5.4.3. Across back shoulder width 2 (symmetrical) Arm length to wrist 5.7.8 Outer arm length 3 Rear neck to waist distance 5.4.5 Back neck point to waist 5 Chest girth third 5.3.4 Bust gig 6 Subgluteal fold height N/A 9 (symmetrical) Knee circumference 5.3.22 Knee girth 10 Hip circumference at hip height 5.3.13 Hip girl eleven Neck circumference at the base of the neck 5.3.3 Neck base girth 12 Height 5.1.1 Stature 13 (symmetrical) Hip circumference 5.3.20 Thigh girth 14 Overall Crotch Length 5.4.18 Total crotch length 15 (symmetrical) Shoulder circumference 5.3.16 Upper arm girth 16 Waist circumference 5.3.10 Waist girth 17 (symmetrical) Wrist circumference 5.3.19 Wrist birthday 18 Chest girth fourth 5.3.8 Under bust girth 21 (symmetrical) Calf circumference 5.3.24 Calf girl 22 (symmetrical) Elbow circumference 5.3.17 Elbow girl 26 Distance from neck to waist in front 5.4.8 Front neck point to waist 30 (symmetrical) Distance from the base of the neck at the back to the nipple point 5.4.12 Back neck point to bust point 31 (symmetrical) Distance from the back of the neck to the waistline 5.4.13 Back neck point to waist level 32 (symmetrical) Distance from the waist line to the floor at the side 5.4.22 Outside leg length 33 Distance from the base of the neck at the back to the level of the armpits 5.4.6 Scye depth length 34 Back shoulder width in a straight line 5.4.2 Back shoulder width 35 (symmetrical) Shoulder point height N/A 36 (symmetrical) Shoulder width 5.4.1 Shoulder Length 37 (symmetrical) Side waist to hip distance 5.4.21 Side waist to hip 38 (symmetrical) Knee point height 5.1.16 Knee height 40 Front neck height 5.1.4 Front neck height 41 Back neck height 5.1.5 Back neck height 42 Nipple point height 5.1.7 Bust height 43 (symmetrical) Waistline height 5.1.10 Waist height 44 Chest circumference contour 5.3.5 Bust Girth Contoured 45 Chest girth second 5.3.6 Chest Girth 46 Bust girth first 5.3.7 Upper chest girth 47 (symmetrical) Clavicular point height N/A 48 (symmetrical) Height of the base of the neck N/A 49 (symmetrical) Thigh height 5.1.13 Hip height 50 (symmetrical) Distance from the base of the neck on the side to the nipple point 5.4.10 Side neck point to bust point 51 (symmetrical) Ankle circumference 5.3.25 Minimum leg girth 53 Distance between nipple points 5.2.3 Bust point width 54 (symmetrical) Distance from the base of the neck on the side to the waist line 5.4.11 Side neck point to waist level 55 (symmetrical) Distance from back of neck to wrist 5.4.17 Back neck point to wrist 56 (symmetrical) Arm width at armhole level 5.2.4 Armscye front to back width 80 (symmetrical) Leg height along the inner surface 5.1.15 Inside leg height 86 (symmetrical) Leg length from heel to inner surface 5.7.6 Inside leg length 87 Mid neck circumference 5.3.2 Neck (middle) 88 Body height 5.7.3 Torso height 89 Distance from neck to pelvis in back N/A

Each measurement is implemented as a separate function and is based on one of the measurement functions of the given type. Each measurement can be made either from a three-dimensional photograph or from a fitted model of the human body. 3D photo measurements are the closest to real world measurements and are expected to provide more accurate results. Sometimes 3D images contain scanning artifacts, which manifest themselves in arms stuck to the body or legs stuck together. In such cases, it turns out to be more convenient to determine part of the measurements from the model, for example, hip girth, upper arm girth, etc. Sometimes a person is scanned with long flowing hair and it is not possible to obtain some measurements, for example, neck girth, only on the basis of a three-dimensional image. In this case, the model is also measured.

To get measurements you need to call the get_all_measurements method of the Measurements class. The proposed method measures all dimensions listed in the measurement_functions dictionary. Each dictionary field specifies one dimension, and also specifies the model_cut and dependencies parameters. The model_cut parameter specifies the point cloud that needs to be measured - a 3D image or a model, and the dependencies parameter specifies the list of measurements on which this measurement depends. Before making a measurement, it is checked whether the measurement has dependencies, and if so, then first the measurements on which the given measurement depends are recursively performed. This method allows you to save time on taking measurements that partially coincide with other measurements.

As a result, the output of the get_all_measurements method generates a dictionary with measurement results and visualizations of measurements in three-dimensional space.

By using a parametric model that has the property of functional vertex reduction (the same vertices are in the same places on the human body for all three-dimensional images), anthropometric points were marked directly on the model. In other words, each anthropometric point used in the measurement algorithm was assigned a point on the model with a fixed index.

The measurements are divided into four types:

(1) height (relative to floor level);

(2) distances (between two given points);

(3) girths;

(4) distances along the body surface.

4.1.1.1. Heights

Dimensional features of this type correspond to distances from fixed levels on the human body to the floor level. To measure each dimensional characteristic from this group, one anthropometric point corresponding to the RP is used.

Additionally, a correction is applied to all height measurements, taking into account the person's posture. The standards describing the measurement of RP of this group assume that a person stands with his legs together, while the implemented algorithm uses three-dimensional photographs of a person in an A-pose, when the legs are shoulder-width apart and the arms are straight and located at an angle of about 20 degrees relative to the torso.

The following algorithm is used for correction. The angles of the legs in relation to the vertical are calculated based on the joints of the model “right hip”, “right ankle” and “left thigh”, “left ankle”. Then the correction value is calculated for each leg as the difference between the actual length of the leg and its projection on the vertical axis. For all points above hip level, the measurement value is adjusted by a fixed amount equal to the average value of the correction values for both legs. For points below hip level, the measurement value is adjusted to a smaller amount in accordance with that shown in Fig. 25 schedule. Visualization of this type of measurement is the coordinate of the point (Fig. 26).

4.1.1.2. Distances

The distance between two anthropometric points is calculated using the well-known formula as the root of the sum of the squares of the coordinate differences. A visualization of this type of measurement is two points connected by an edge (Fig. 27).

4.1.1.3. Girths

This dimensional feature corresponds to the length of a flat closed contour located at a certain level of the body or passing through given anthropometric points. It is measured on the human body with a measuring tape.

To determine this type of RP, the following algorithm is used. The plane in which the measuring tape should lie to obtain the desired girth is determined. The point cloud is cut through the given plane. The result is one or more closed loops. For example, if a cloud of points is cut by a plane at waist level, then three contours will be obtained: the waist and two arms. To select the desired contour, the distance from the contour to a given point is used and the contour with the minimum distance is selected. After that, a convex contour is built on the selected contour, simulating the behavior of a centimeter tape. The length of the convex contour will be the desired girth value.

In the “girth” type measurement function, the ability to specify the required number of contours for a section is implemented. If there are fewer or more contours, the plane is shifted by a small amount until the number of contours is equal to the specified value. This allows you to increase accuracy when choosing a plane and avoid additional errors. For example, this functionality is used to find chest girth, where the plane may be just above the armpits. Then the selected contour will capture the shoulder and thus give an inflated result. In this RP, the parameter for the required number of circuits is set to three. Visualization of this type of measurement is a set of points of a convex contour connected by edges (Fig. 28).

4.1.1.4. Distances over the body surface

To measure this measurement on the human body, you need to connect two or more anthropometric points (sequentially) on the surface of the body with a measuring tape and determine the resulting length.

The algorithm for implementing this type of RP in the proposed method can be divided into several steps (Fig. 29). The first two steps for this type of measurement and the girth type measurement are similar, but the subsequent steps are different.

1. The cloud of points is cut by a plane, resulting in one or more contours. If there are fewer or more contours than the specified number, the plane is shifted by a small amount until the number of contours is equal to the specified number.

2. When receiving multiple contours, the contour C ( contour ) closest to the middle_point is selected.

3. The contour is divided into two parts at the end points point 1 and point 2 .

4. The distance from each part of the contour to the middle_point is compared, and the part of the contour L ( line ) closest to the point is selected.

5. Using the selected part of the contour L , a convex contour CH ( convex hull ) is constructed.

6. The convex contour CH is divided by the end points point 1 and point 2 into two parts, one of them will be located on the outside of the body, and the other on the inside of the body.

7. Select the part of the convex contour external to the body. To determine this, a special method was developed, described below.

Method for determining the outer part of a convex contour: each point of the convex contour corresponds to one of the points on the line. A point on the convex contour CH is determined, corresponding to one of the end points of line L. Next, it is checked whether the neighboring points of the convex contour are equal to the neighboring points on the line. On the side where they are not equal (for at least one side they will not be equal), the convex contour “jumps” over the points of the line, i.e. the corresponding segment of the convex contour is on one side of the line. Therefore, it is further checked whether the midpoint of a given segment ( HI ) is outside or inside the contour C. If it is located outside, then the part of the convex contour CH to which the segment belongs is located outside the body and is the desired one. If it is inside, then the remaining second part of the convex contour is the one we are looking for. Visualization of this type of measurement is a set of points of a selected segment of a convex contour, connected by edges (Fig. 30).

4.1.2. Correction of measurement results

It may turn out that some anthropometric points are not specified exactly. This may increase the error in measurement results obtained using linear algebra methods.

To minimize this effect, it is proposed to use a measurement adjustment method. This requires a database of 3D photographs of people and manually measured measurements. For each dimensional feature, the average value of the algorithm error from the true values is determined. Then, for each measurement, a correction is introduced to compensate for the average error.

This method can reduce the average measurement error by approximately 40–60%.

4.1.3. Obtaining dimensional features based on machine learning methods

It is proposed to use a regression model. The model is trained as follows. First, a model of the human body is fitted to a three-dimensional photograph of a person, which has parameters of physique and pose. Then a regression model is trained, providing a transformation from model parameters to measurement values.

To obtain the measurements of a person, you need to fit a model of the human body to his three-dimensional photograph and, using a trained regression model, obtain the required measurements.

The model was trained using data from the CAESAR digital image database, which includes more than 4,000 3D images of people from various regions. For each person in this database there was a set of measurements taken manually from the human body.

The advantages of this method are the speed of determining measurements, but the disadvantages include the dependence of the accuracy of measurements on the size of the training sample, and the limitation of the set of measurements to those presented in the training sample.

4.1.4. Obtaining dimensional features based on combined methods

These algorithms can be implemented in various ways. The general idea is to first obtain some intermediate anthropometric parameters using linear algebra methods, which are then used to determine target measurements.

Intermediate anthropometric parameters can be, for example, girths of the human torso or other part of a three-dimensional image, taken at different heights and angles to the floor plane. Another option for intermediate anthropometric parameters can be the distances between the points of intersection of the straight line and the contour of the projection of the human body onto one of the vertical axes.

Next, a set of intermediate parameters can be processed using both functional analysis methods (search for maximum, minimum, inflection points, etc.) and machine learning methods.

4.1.5. Determination of anthropometric characteristics of the user based on a questionnaire

As an alternative to the methods described above, it is possible to determine the user's anthropometric parameters based on filling out a questionnaire. This method can be implemented as an application on a smartphone, as a service on an Internet site, or in another way. The questions may include information about height, weight, age, body shape or any anthropometric parameters of the body. Answers can be entered as a real number or by selecting one of several options provided. Also, optionally, the user can add one or more body photographs to the entered data.

The data entered into the questionnaire is transferred to a special module, which calculates the anthropometric parameters of a person based on these data.

The task of such a module is to determine unknown anthropometric body parameters based on the entered parameters. This problem can be solved using statistical models. For example, knowing the gender, height, weight, and body type of the user, it is possible to construct the most probable model of the user’s body for such parameters, which will correspond to certain dimensional characteristics.

The model can be built using regression analysis methods, using classification and clustering methods, as well as using deep learning or using mixed methods.

As questions with multiple answer options, qualitative characteristics of a person can be requested, for example, the curvature of the back (degree of stoop), the ratio of muscle and fat body mass (athletic, normal, obese), etc.

If the user considers the calculated dimensional features to be incorrect, he can leave feedback by entering the correct values of the dimensional features into the system. Feedback data can be used to fine-tune the module, taking into account real statistical data about users.

The incoming data can contain both continuous and discrete quantities, as well as photographic images. Several approaches are proposed to process this data.

1. Translation of categorical features into continuous ones. In this case, only the user’s gender can remain a categorical attribute. The statistical model is built either separately for men and women, or one for both sexes. Using the regression method, unknown parameters are determined from known parameters.

2. Translation of continuous features into categorical ones. In such a case, the space of continuous parameters is mapped into clusters using voxelization or another clustering method. For each cluster, the most probable values of dimensional characteristics are determined. Based on the data entered by the user, the cluster to which the user belongs is determined, and after that his anthropometric characteristics are clarified using additional information. The result is the user's dimensional attributes.

3. Processing of full-length photographs. Using deep learning techniques, it is possible to extract either continuous or categorical anthropometric features from photographs. For example, the ratio of waist circumference to hip circumference, leg length to height, or other ratios can be continuous. Categorical ones can be, for example, body type. This data can be used to clarify the user’s dimensional characteristics described above in paragraphs 1 and 2.

4. Mixed options. The original problem can also be solved using combined options from paragraph 1, paragraph 2 and paragraph 3.

4.2. Determining the user's body type

To determine the user's body type, various methods can be used, including both purely algorithmic methods and machine learning methods.

The human figure typing algorithm has as input the person’s gender and a table of measurements taken from him. Body typing for women and men is implemented with various functions.

4.2.1. Typing men

To type men, the algorithm uses information from three measurements: chest circumference, waist circumference and hip circumference, and classifies the figure into one of three types: thin, normal and fat.

First, it is checked whether the figure is thin. It is believed that thin men are those who do not have a pronounced pelvis, abdomen or trapezius muscles. Therefore, it is first checked whether all three measurements are within the specified limits. If yes, then the figure is of the “thin” type.

Next, it is checked whether the person belongs to the full type. A man is considered obese if he has excess weight in the abdominal or pelvic area, so it is checked whether the waist or hip measurement exceeds the threshold for a fat man. If at least one of these measurements exceeds the threshold value, then the figure is classified as “fat”.

The remaining figures are considered to be of the “normal” type. That is, these are men of relatively average weight, while a person may have large muscle mass (for example, pronounced trapezius muscles) or not have it.

On a test base of three-dimensional images of 126 men, the following results were obtained: “thin” – 23, “normal” – 80, “fat” – 23. During visual inspection, the men’s body types corresponded to the results of automatic testing.

4.2.2. Typing women

For women, typing also occurs based on measurements of the chest, waist and hips. In this case, five body types are used: “thin”, “normal with a defined waist”, “normal without a defined waist”, “full with a defined waist”, “full without a defined waist”.

Four parameters are used for typing: thin_thresh , flat_coef , wide_contoured_thresh , wide_flat_thresh . The verification of belonging to a specific type is carried out step by step and at each subsequent step only the figures remaining after the previous steps are typed.

First, just like for men, it is checked whether the woman is thin, which will be carried out provided that all three measurements are less than thin_thresh . Next, a check is made for compliance with the figure of a plump woman: the hips should be larger than wide_contoured_thresh and the waist-to-hip ratio should be less than flat_coef . After this, compliance with the figure type “full without a pronounced waist” is checked: the waist is greater than wide_flat_thresh . Next, it remains to type the “normal” figures. If the remaining models have a waist-to-hip ratio less than flat_coef , then the figure will be “normal with a defined waist.” If none of the above conditions are met, then we consider that the figure type is “normal without a pronounced waist.”

The test results based on 146 female three-dimensional images are as follows: “thin” – 8.9% (13), “normal with a pronounced waist” – 43.8% (64), “normal without a pronounced waist” – 27.4% ( 40), “overweight with a defined waist” – 5.5% (8), “overweight without a defined waist” – 14.4% (21). These data correspond to statistical data. When visually checked, the women's body types also corresponded to the results of automatic testing.

4.3. Data analysis and results

The quality of the algorithm's work was determined on a test sample consisting of three-dimensional images (3D scans; 3D scanograms). Comparisons were made with two types of reference data:

(a) dimensional characteristics (RF) of a person, measured manually along the body;

(b) dimensional attributes (DF) measured from a 3D image using clothing design software (CAD). Examples of such CAD systems include CLO, Optitex, Browzwear, etc.

A comparison of RPs obtained using the algorithm described above with RPs measured manually over the body is presented in Fig. 31. The average error was 23.3 mm. A comparison of the RPs obtained using the algorithm described above with the RPs measured from a three-dimensional image is presented in Fig. 32. The average error was 21.2 mm. In fig. 31 and fig. 32 along the abscissa axis are the numbers of measurements in the internal system, along the ordinate axis is the difference between the measurement results obtained using the proposed algorithm and the results measured manually from a person in reality or from a three-dimensional image. Each point corresponds to the error in the result of a specific measurement of a specific person.

Thus, the average error in all measurements turned out to be less than 2.5 cm. It can be seen that the distribution centers are shifted relative to zero, which indicates an inaccurate choice of the positions of anthropometric points; in addition, the diagram contains outliers that can be associated with errors algorithm and with reference data errors. In the future, it is planned to test the algorithm on a larger labeled sample, which will allow us to select the optimal parameters of the algorithm.

5. Selection of the closest production models

Next, a selection is made based on anthropometric parameters of one or more of the closest production models from the array of production models from different clothing manufacturers.

5.1. Parameterization of production models

To match the manufacturing models to the user's body model, the 3D manufacturing models are "registered" in a manner similar to the 3D models obtained from scanning the user's body as described above.

If digital models are used, then such registration can be performed directly from the digital model or after the necessary data conversions. The task of such data conversion is trivial for a specialist, so the details of the conversion are omitted here for brevity. If physical mannequins are used, they are scanned in the same way as the user's body.

In the case of using parametric production models, RP can be obtained from such models - either directly (if the methods for determining RP coincide with those used in the proposed approach) or by transformation. Moreover, the method of such transformation depends on the specific implementation of the parameters of a particular production model.

5.2. Matching production models with user body model

Matching production models to the user's body model can be done using an objective function

(34)

where F is the target divergence function to be minimized, M is the number of dimensional features (DF), – dimensional sign of the user, – dimensional attribute of the production model corresponding to the dimensional attribute of the user, – a weighting factor that takes into account the importance of each attribute in terms of its influence on the “fit” of a particular item of clothing on the user, i.e. on the user’s subjective assessment of whether this clothing “fits” him well or poorly. The number M can represent both the number of all dimensional attributes of the user and/or production model, and a subset of dimensional attributes, including only those dimensional attributes that are significant for a particular item of clothing (for example, only the RP of the legs when choosing jeans or only the RP of the hands when choosing gloves).

The criterion for comparing models can be based, for example, on the estimate F < ε ₀ , where ε ₀ is a threshold value, the value of which can be unified (i.e. valid for all types of clothing, all clothing manufacturers and all users) or differentiated (t .i.e. set individually for different types of clothing, different clothing manufacturers and different users). It will be clear to one skilled in the art that the above criterion is provided for illustrative purposes only and that the invention includes a variety of possible criteria.

It is worth noting that the comparison can be performed not only using the objective function (34), but also by more general known methods, for example, by registering the two compared models with respect to each other and calculating the discrepancy of the three-dimensional surfaces of the models. This allows you to compare models not only in selected places (waist circumference, hip circumference, etc.), but also to compare models as a whole.

Production models selected by comparison criteria can be recorded in any suitable form on any suitable storage medium, as suitable for a specific user, and can subsequently be used in the process of remote selection of clothing. Such recording can be performed in an appropriate format on the user’s computer or mobile device, or on a cloud service server.

The above describes an algorithm that assumes the existence of a manufacturing model (geometric or parametric) suitable for comparison with the user's body model. However, there may be situations where a production model is missing or where the existing production model contains significant errors that lead to an unsatisfactory model comparison result. In this case, the construction of a production model and/or its verification may be impossible or economically infeasible.

In this case, remote selection of clothing is possible without using a production model based on a comparison of the user’s dimensional characteristics with the dimensional grid of a particular brand or based on a comparison of the user’s dimensional characteristics with the measurement results of the item of clothing.

5.3. Comparison user's dimensional characteristics with the brand's sizing chart

In this approach, the user selects a clothing model, and the system determines the size suitable for the user for this model. Here the term “clothing model” does not mean a mathematical or computer model, but a sample of clothing embodied in the material, characterized by certain sizes, silhouette, cut, compositional and design solutions, materials and decorative elements.

An enlarged algorithm for recommending the appropriate size for a specific clothing model is presented in Fig. 43. The algorithm involves performing the following actions:

- (41) the user selects the model or category of clothing he is interested in;

- (42) if the user selects a clothing model, then a category (subcategory) is determined for it;

- (43) dimensional characteristics of a person are determined or introduced, on the basis of which a suitable size can be determined;

- (44) the user’s dimensional characteristics are compared with the dimensional grid of the brand that owns the selected clothing model;

- (45) the comparison results are processed to determine the recommended size of the selected clothing model;

- (46) The recommended size is communicated to the user.

The preliminary selection of a model or category of clothing is carried out in any suitable way, for example, using an electronic catalog. It is also possible to use for this purpose paper catalogs, flyers, publications on social networks, video clips and other information and advertising materials that allow identifying the item of clothing that interests the user.

An electronic catalog can be mono-brand or multi-brand, functional or thematic. All clothing models in the catalog are pre-distributed into categories and, if necessary, into subcategories. For each category (subcategory) of clothing, those dimensional characteristics of a person are preset, on the basis of which the appropriate size can be determined.

These features are determined by or input into the system (if they have been predefined) or extracted from a geometric or parametric model of the user's body (for example, if contained in the user's profile). Determination of the user's dimensional characteristics can be performed, for example, by the methods described above in section 4.1.

The dimensional grid can be obtained either directly or by calculating the dimensional characteristics of production mannequins.

For each size attribute, the corresponding size is calculated numerically (for example, as a real number) using the brand’s sizing grid. Next, based on the obtained sizes, the recommended clothing size is calculated using the formula.

In the simplest case, to make recommendations, you can use a size grid containing only two dimensional characteristics, for example, chest girth and hip girth. In this case, the size related to the upper body will be determined by the chest girth, and the size related to the lower body will be determined by the hip girth.

A more accurate recommendation can be provided by using a size chart with a large number of dimensional characteristics, for example, for fitted shirts, jackets, dresses, etc. It may be advisable to take into account the circumference of the neck, chest and waist, shoulder width, as well as the person's height and sleeve length.

For a more accurate recommendation, a dimensional grid containing a larger number of dimensional characteristics is needed.

The list and characteristics of the dimensional characteristics of the human body are given above in Table 3.

Determining size based on several dimensional characteristics is possible using various approaches. For example, for each dimensional attribute, you can enter a coefficient that determines the degree of influence of this attribute on the recommended clothing size. In this case, the size can be calculated using the weighted arithmetic average formula:

S = Σ(w _i s _i )/Σw _i , (35)

Wheres _i – size (for example, in the form of a real number) corresponding toi-th dimensional characteristic,w _i - weighti-th dimensional characteristic when determining the size,S – the recommended size (for example, in the form of a real number) of clothing in numerical form.

For some clothing models, it may turn out that the user's preferences do not fully correspond to the clothing designer's intentions. It may also be that the material a garment is made from influences individual fit preferences. To account for such cases, one or more correction values can be entered for each clothing model. Then formula (35) takes the form:

S = Σ(w _i s _i )/Σw _i +ds ₁+ds ₂ +... , (36)

where ds ₁ is the first correction value, ds ₂ is the second correction value, etc. The number, types and magnitudes of correction values may be individual for each user or for some users. In particular, individual correction values can offset the lack (small number) of dimensional features taken into account during comparison, taking into account the characteristics of the user’s figure.

Next, you need to convert these numerical size values into size values according to the brand's sizing chart. This can be done, for example, by rounding the resulting numerical size value to the nearest size in the sizing chart. The rules for such rounding may vary, for example, up, down, or to the nearest value.

In the simplest case, the size of clothing can be calculated in this way based on length and width. The result of the recommendation algorithm in this case is a pair of numerical sizes: length size and width size.

If necessary, the clothing size recommendation algorithm can be expanded by additionally implementing a stylistic recommendations algorithm (discussed in more detail in section 5.7). To prepare stylistic recommendations, the expert stylist must first determine the rules for matching body characteristics with the characteristics of suitable clothing. Body characteristics can be based on the user's age, body type, skin color, hair type, color and length, eye shape and color, etc. The algorithm for stylistic recommendations calculates the necessary characteristics of a person’s body, determines the clothing parameters appropriate for a given person, and recommends the most suitable clothing in style and style.

5.4. Comparison dimensional characteristics of the user with the results of measuring the item of clothing

In the absence of information about model ranges and sizing charts of brands, recommendations can be developed based on a comparison of the user’s dimensional characteristics with the measurement results of specific items of clothing.

If the garment is available for physical measurement, then the measurement can be done explicitly, for example using a traditional tailor's yardstick. In most cases, physical measurement may not be possible or economically feasible due to the high labor intensity and high cost of manual labor. Therefore, in queuing systems, the measurement of a garment can be performed indirectly, for example by optical scanning using conventional two-dimensional photographic images of the garment being measured.

Optical scanning of clothing can be performed on a special automated installation. The installation may consist of a working surface and a static chamber. The work surface may have markings that automatically calibrate the system to determine dimensions based on images received from the camera. In the simplest case, such markings can be a grid with a square cell of a predetermined size. In other cases, the marking may consist of reference points located at predetermined locations at a predetermined distance from each other.

To determine the size of a piece of clothing, a computer vision algorithm can be used, which determines the type or category of clothing, identifies key points in the image, calculates the outline of the clothing and, based on this data, determines the dimensions (Fig. 42). For this purpose, rules for converting key points and contours into measurement results can be applied in advance for each type (category) of clothing.

The measurement results can then be converted into dimensional attributes that can be used for comparison directly or from which a parametric model of the garment can be formed if necessary. Essentially, these dimensional characteristics or parametric model describe the production mannequin on which this clothing model is sewn.

The dimensional features of the user's body can be compared with the dimensional features of the item of clothing in the same way as described in paragraph 5.3 with reference to formulas (35) and (36).

The following describes an illustrative example of an algorithm for recommending clothing of a given category. In this example, the system treats different sizes of the same clothing model as different clothing models and returns a list of clothing items sorted in descending order of fit rating.

A system that implements a recommendation algorithm may contain:

- a clothing database containing a dictionary of clothing categories, a dictionary of clothing subcategories for each category, a list of clothing models for each subcategory, measurement results (hereinafter referred to as measurements) and the corresponding sizes for each clothing model;

- rules for converting human dimensional characteristics and clothing measurements, allowing to bring human dimensional characteristics and clothing measurements to a single format;

- rules for calculating missing measurements (in particular, for those sizes of a particular clothing model for which measurement data is not available);

- rules defining the importance of measurement results for each clothing model, while for each model it is indicated whether this measurement result should be equal (equal) or exceed the corresponding size characteristic of a person (greater); any combination of rules can be specified for a specific clothing model;

- rules for grading clothing by size.

The user's dimensional attributes are fed to the input of the algorithm. The database of clothing available for selection is searched taking into account the number M of recommended clothing models and the target category (subcategory) in which the search should be carried out. The number M and one or more categories (subcategories) for search can be specified by the user or determined by the system itself based on various criteria. Such criteria may be user preferences identified based on an analysis of this user’s past actions in the system, current system load, etc.

The output of the algorithm is a list of no more than M clothing models, indicating the size of each model, ranked in descending order of fit rating (i.e., a coefficient showing how suitable a particular item of clothing is for the user).

The recommendation algorithm involves initialization, data preprocessing, and calculation of the clothing fit rating.

During the initialization phase, data on clothing items is imported from an external data source, converting the format of the imported data to the system data format. Sources of apparel data can be numerous and varied in terms of data formats. Format conversion allows you to create a database of clothing items for a recommendation system. Garment data may represent garment measurements. If the imported data does not contain measurements for the entire size range of a particular clothing model, then the missing measurements are calculated by interpolation or extrapolation based on the available measurements according to predefined rules.

Pre-processing of the data, if necessary, includes converting the format of the user's dimensional characteristics and/or the format of clothing measurements into a format that allows them to be compared. In particular, garment measurements can be converted into a format that describes the production mannequin on which the garment is sewn. The sizing characteristics of the production mannequin correspond to the ideal fit of the garment. Since such a format can be the same as the user's body model, matching them can be done directly. However, another option is possible when the user’s dimensional characteristics are converted into a clothing measurement format, or another option when both formats are converted into some third format that provides comparison of dimensional characteristics.

The result of the comparison can be expressed as a fit rating - a numerical assessment of the suitability of an item of clothing to the wearer's figure. In particular, the fit rating can be calculated for two vectors - a vector of dimensional attributes of the user and a vector of dimensional attributes of the garment.

Due to the specifics of such a task, using L-norms as a landing rating is not possible for the following reasons.

First, each garment measurement has an acceptable range within which the measurement can vary without compromising the technical fit of the garment. Accordingly, deviation of a dimensional characteristic from the corresponding measurement within this range should not degrade the fit rating.

Secondly, if one of the clothing measurements is smaller than the corresponding size characteristic of the user, taking into account the acceptable range, the clothing cannot be recommended to the user.

Thirdly, in general, not all clothing measurements have the same meaning for a comfortable fit. For example, for a dress shirt, freedom at the waist may not be as important as shoulder width and neck circumference.

Therefore, to calculate the fit rating, an algorithm was developed that calculates a scalar value whose value ranges from 0 to 1, where “0” means that the garment does not fit the wearer at all and “1” means that the garment fits the wearer perfectly .

Before calculating the fit rating, the difference values between the corresponding vector coordinates are divided into 3 groups: “fit”, “small” and “big”, where:

- “fit” means the fulfillment of at least one of the following conditions: the value of the dimensional attribute of the clothing item is in the acceptable range and the value of the dimensional attribute of the clothing item is greater than the value of the corresponding dimensional attribute of the user, but is in the “greater” group of importance rules;

- “small” means that the size of the item of clothing is less than the acceptable range;

- “big” means that the size attribute of the item of clothing is greater than the acceptable range and is in the “equal” group of importance rules.

If at least one measurement of a clothing item of a certain size is in the “small” group, then it is excluded from the search results. In the simplest case, the fit rating can be equal to the number of measurements included in the “fit” group divided by the total number of measurements of the garment.

To rank garments with an acceptable fit rating, a magnitude of fit error may additionally be calculated. To do this, the L2-norm is calculated for each of the groups “fit”, “big” and “small” separately. The calculation results are summed using weighting factors, for example, 1*fit + 2*big and the sum is the fit error value. The fit error value allows you to rank fully fitted clothing (for which the fit rating is 1) and, if there are items of clothing with measurements in the “big” category in the recommended list, raise the position in the list of clothes with the smallest deviation from the acceptable range.

Thus, the following data is supplied to the input of the ranking algorithm:

- K vectors of clothing measurements (one vector for each item of clothing);

- one vector of user dimensional attributes;

- Information on the ideal fit for each item of clothing.

Vectors of clothing measurements and a vector of dimensional attributes consist of N coordinates. The coordinates of all vectors are in the same order (i.e., expressed in a single coordinate system).

To iterate through all instances of clothing, the “bruteforce” method is used for all clothing models from a given category (subcategory). At each i -th enumeration step, the fit rating is calculated for the i -th vector of clothing measurements and the vector of user dimensional attributes.

After calculating the fit error and fit rating for each item of clothing in the sample, two sequential sorts are performed: sorting by fit rating in descending order and sorting by fit error in ascending order (optional).

At the last step of the algorithm, the first M elements are selected from the sorted sample. Thus, the output of the algorithm is a selection of no more than M elements, ordered in descending order of accuracy of the technical fit of clothing.

It should be clear to a specialist that the described algorithms can replace and/or complement each other and can be used both to compare the user’s dimensional characteristics with the results of measuring an item of clothing, and for comparison with the brand’s sizing grid. To do this, it is enough to swap the measurements of the item of clothing and the dimensional characteristics of the brand’s size grid at the input to the algorithm.

5.5. Comparison of the sizes or measurements of a reference item of clothing with the sizes or measurements of a selected item of clothing

The system may be able to implement a recommendation algorithm even in the absence of anthropometric measurements of the user's body. In this case, instead of the user’s dimensional characteristics, the input of the algorithm can be supplied with measurement data of reference items of clothing, i.e. garments that are guaranteed to have an acceptable fit on the wearer. One skilled in the art will appreciate that this version of the recommendation algorithm has certain limitations, but may provide satisfactory results for items of clothing of the same category (subcategory).

5.6. Recommendations based on machine learning algorithm

The recommendation principles described above are implemented on the basis of heuristic algorithms, so the accuracy of such recommendations can greatly depend on the human factor - the developers' ideas about the ideal fit of clothes and their ability to express these ideas in the form of rules for the operation of the algorithm.

To get rid of this drawback and increase the accuracy of recommendations, it is possible to use machine learning algorithms based on the use of neural networks (MLA, Machine Learning Algorithm), based on previous experience, selecting rules and dependencies that can best recommend clothes to the user.

The main limitation of this approach is that such systems require a large set of training data. Therefore, it is possible to initially use the system described above, and when sufficient data is accumulated to train the MLA algorithm, the heuristic algorithm can be replaced by the MLA algorithm described below. At the same time, clothing recommendations using the MLA algorithm can be developed by comparing the user’s dimensional characteristics with both the brand’s dimensional grid and measurement results.

The MLA algorithm can be implemented as a classifier, the input of which is the size characteristics of the user, and the output is scalar values indicating the likelihood that a particular size of a particular brand (or a particular item of clothing) will suit the user.

Below, as an illustrative example, we describe the principles of operation of the MLA algorithm, which implements a brand size recommendation.

The MLA algorithm is based on a neural network with two or more layers. The need to implement a multilayer neural network is caused by the fact that different brands do not have the same dimensional grids. The multilayer structure of the neural network makes it possible to separate the network's representation of the user's dimensional characteristics and transform the network's representation of the user's dimensional characteristics into a size recommendation for a specific brand.

In this case, one layer, responsible for the network's representation of the user's size characteristics, acts as an encoder, and another layer, responsible for converting the network's representation of the user's size characteristics into a size recommendation for a specific brand, acts as a decoder (see Fig. 44). The role of an encoder can be performed by several layers of a neural network. In this case, the encoder and decoder can be trained separately. This allows you to train the encoder on large brand-independent training data sets, and train the decoder on a subset of brand-specific training data. Then, to introduce a new brand into the system, it is enough to additionally train the encoder on a new set of training data for this brand and train the decoder on a new part of the training data. This approach makes it possible to increase the accuracy of recommendations and at the same time speed up the addition of new brands to the product range, accelerating their time to market.

For initial training of a neural network (i.e., for training an encoder), a database can be used containing a data set containing a set of dimensional characteristics of one person, a set of sizes of one item of clothing, and data on whether this item of clothing fit or did not fit this person. Many such data sets can cover all body types and body types of users and the size ranges of many brands. Body types and body types in training data sets can be determined taking into account national and/or regional population size statistics.

Subsequent training of the neural network can be carried out for one or more decoders, each of which is turned on only when the encoder is fed a set of training data corresponding to this decoder (i.e. data related to a specific brand). Thus, the decoder is trained on the corresponding brand's training data, and the encoder is eventually trained on the full training data set.

Training on brand-specific data can be done on labeled training data sets. Each data set may contain a set of dimensional characteristics of one person, a set of sizes of one piece of clothing from a particular brand, and data on whether this piece of clothing fits or does not fit this person. In another embodiment, each data set may contain one size attribute for one person, one size for one item of clothing from a particular brand, and data about whether that item of clothing fits or does not fit that person.

As an alternative to using encoder and decoder layers in a neural network, the MLA algorithm can be implemented as a Gradient Boosted Decision Tree (GBRT) algorithm based on decision trees, using which the predictive model is formed as a forest of decision trees. The forest is being built in stages. Each subsequent decision tree of such a forest focuses learning on those parts of the previous decision tree that led to an unfulfilled prediction or a significant error. In general, boosting is intended to improve the quality of forecasting by the MLA algorithm. In this case, the system uses the forecast not of one trained algorithm (one decision tree), but of many trained algorithms (forest of decision trees) and makes the final decision based on the set of forecast results of these algorithms.

5.7. Recommending clothes based on user-supplied images

To simplify the user experience and to be able to deploy a recommendation system without the use of stationary scanners, a clothing selection system based on several photographs can be used.

Such a system works on the basis of several photographs of the user, which he can take independently, using devices available to him, such as a smartphone or laptop with a web camera. Additionally, the user is required to enter several parameters, for example, height, weight, etc.

The system receives as input several images of the user from different angles, for example, in profile and frontal. The image data is input to a convolutional neural network, which converts the user's image data into a segmented map, where each pixel has a value of either 0 or 1. In this case, pixels that are not related to the person have a value of 0, and pixels that are related to him have a value of 1. After this, the image data is fed to the input of a convolutional neural network, which converts it into a feature vector. After that, a correspondence is sought for this vector in the clustered space of parametric models. Next, for this cluster, the corresponding size from the size grid or clothing size from those previously mapped to this cluster is searched.

The image segmentation map is obtained using a UNet-like convolutional neural network containing an encoder and a decoder. One color image of the user is supplied to the network input, and a segmentation map is expected at the output. A detailed description of the UNet can be found at https://arxiv.org/abs/1505.04597.

To train such a network, a set of images is used, in which each image corresponds to an accurate segmentation map.

The loss function uses a weighted sum

,

Where: – Dice similarity coefficient,

– binary cross-entropy, p – predicted value, y – real value, – scalar coefficients selected by enumerating parameters.

Let us assume that the neural network receives N images of size X×Y×3, where 3 is the number of colors in a pixel. In this case, the neural network acts as a transformation operator , Where is the image space, and feature vector space.

The very acquisition of the feature vector is carried out by a neural network of conventional architecture. To do this, take a regular encoder consisting of a number of convolutional layers, for example, ResNet, Inception, or MobileNet. The output data is smoothed into a vector by the encoder and fed to the fully connected layer. The output of this fully connected layer is a feature vector.

A neural network with any non-zero weights will perform feature vector extraction. However, better cluster matching is achieved if the neural network is trained on an appropriate dataset containing images similar to those that will be input.

Therefore, the neural network is trained on existing scans, supplemented with random noise in the model’s pose parameters. In this case, the preparation of one data set is carried out according to the following algorithm.

1. Select one scan of a person and the parametric model inscribed in it.

2. Random noise with pre-calculated parameters is added to the figure parameters of this parametric model. These parameters are calculated so that the distribution is similar to the distribution of the real pose parameters from the dataset.

3. A random pose of the model is selected from the entire data set and random noise is also added to it.

4. The model is placed at the center of coordinates.

5. The camera is initialized with random parameters c _x , c _y , f _x , and f _y .

6. The camera is positioned so that the model is looking strictly at the camera.

7. A random small offset is applied to the model.

8. Small random transformations are applied to the camera position, to the observation point and to the vector looking to the right in the camera coordinate system.

9. This model is rendered.

10. The camera is rotated at a random angle from –100° to –80° or from 80° to 100° around the model and steps 7, 8, and 9 are repeated.

This renders the model into a binary mask, where each pixel has a value of either 0 or 1, where 0 means that the pixel belongs to the background, and 1 means that the pixel belongs to the model. These images are subsequently fed to the input of the neural network.

After training the neural network, an algorithm is trained that matches the cluster to the feature vector.

Since a parametric model is matched for each cluster, the matching algorithm is trained on a data set obtained as follows.

1. A model is supplied to the input of the neural network.

2. Using the parameters of this model, the cluster number is searched.

3. The parameters of the found cluster are paired with the feature vector.

Using the obtained data set, it becomes possible to validate algorithms for matching feature vectors and clusters.

5.8. Finding suitable clothing items

The search for garments made using selected production patterns is carried out in any suitable way. The implementation of any such method is quite obvious to a specialist, so its description is omitted for brevity. Suffice it to say that this requires that for each garment, information about the production model used in its design and/or cutting is available.

The order in which these garments are issued may depend on the magnitude of the divergence function F (Equation 34) or may be determined by the user based on personal preference. The interface of the system for remote selection of clothing may provide for prompt switching of the principle of forming the order of issuing clothing items. This switching can be done manually by the user (for example, if the default order does not provide sufficiently relevant results) or automatically (for example, based on analysis of user activity statistics).

Also, the search for suitable items of clothing can be carried out using body typing, described in section 4.2 above, which allows you to select items of clothing not just in accordance with body parameters (so that, for example, clothes do not sting or press, or, conversely, do not hang baggy) , but also in accordance with the body type (so that the clothes stylistically decorate the figure), recommending, for example, for women with a figure classified as “pear”, empire line dresses and items of clothing that visually enlarge the shoulders (with the help of shoulder pads, puffed sleeves, bolero etc.) etc. (see, for example, typical stylistic clothing recommendations for 7 body types on the pages https://www.joyofclothes.com/style-advice/shape-guides/body-shapes-overview.php or in similar stylist references).

5.9. Visualization of found clothing items

Visualization of garments for which the used production model is known is also a trivial task for the specialist, so the description of its practical implementation is omitted for the sake of brevity. Suffice it to say that the item of clothing may be rendered as such (ie, without the image of a mannequin or the image of a person) or it may be rendered on a mannequin or on a person.

In particular, one embodiment of the invention may involve rendering an item of clothing on a 3D model of the wearer, where the rendering of the image may include rendering a realistic image of the wearer's head to enhance the emotional effect. However, the 3D head model, including the user’s face and hairstyle, may not be part of the user’s body model in the sense described in the algorithm description, and may be intended solely for decorative purposes.

The interface of the remote clothing selection system may provide for prompt switching of the clothing visualization principle. As with the order in which search results for clothing items are displayed, this switching can be done manually by the user (for example, if the user wants to view the inside (lining) of a garment, he can turn off the visualization of a mannequin or a person) or automatically (for example, based on analysis of user activity statistics) .

The interface of the remote clothing selection system may allow the user to control the visualization using controls by changing the angle, zooming in or out of the image, etc. to allow the user to properly view the clothing.

As mentioned earlier, visualization of clothing is possible on different situational backgrounds (which can be selected from the library or can be loaded by the user), when illuminated from different points (in particular, with the choice of the number and/or position of virtual lamps), with different intensities, with different diffuseness (for example, the light can be directional or diffuse) and/or with a different spectrum (for example, the light can be artificial, daytime, evening, imitate the rising or setting sun), etc. The interface of the remote clothing selection system may allow the user to control all of these features. The implementation of these capabilities will also not cause significant difficulties for specialists, therefore the details of the corresponding technical solutions are omitted for brevity.

6. Making the selection and purchase of selected clothing items

After evaluating the proposed clothing options, the user can make a choice and make a purchase in the traditional way. The principles of operation of online stores are well known in the art, so details of the corresponding technical solutions are also omitted for the sake of brevity.

It should be noted that the technology described above can be used not only for the remote sale of clothing items (for example, in an online store), but also for the remote rental of clothing (for example, formal or elegant clothes for celebrations, etc.). Moreover, the technology described above, with minor modifications, can be used for remote selection of other products other than clothing (for example, medical products - corsets, bandages, etc., which require matching the patient’s figure).

It is also worth noting that the user can make a purchase or rental not only remotely (online, via the Internet), but also in an offline store or salon. In this case, it is also possible to help an assistant bring things to the user after the user selects these things in the application interface. In this case, the assistant can either transfer things into the hands of the user or use special cells (boxes, baskets, etc.), due to which the assistant does not see the user (which can be convenient, for example, if the user is in the fitting room and in currently undressed). Also, the assistant can be either a person or a robotic system.

7. System for remote selection of clothing

The following describes a system that implements the above-described algorithm for remote selection of clothing.

The system for performing remote clothing selection contains computing devices that perform the actions of the above-described algorithm for remote clothing selection. In addition, this system may contain the equipment mentioned above for optical scanning of the user's body, used at the initial stage of the algorithm.

In particular, in a business-to-consumer (B2C, Business-to-Customer) environment, the system contains the equipment of the potential user, hereinafter referred to as the user's device, and the equipment of the service provider associated with the remote selection of clothing.

The user device can be any computer hardware capable of executing programs suitable for solving a given task. Some (non-limiting) examples of user devices include personal computers (desktops, laptops, netbooks, etc.), smartphones and tablets, and network equipment (routers, switches, gateways, etc.).

Service provider equipment may also be any computer hardware capable of executing programs suitable for the task at hand. Some (non-limiting) examples include individual computers and computer systems. Computer systems can be concentrated or distributed.

In a business-to-business environment (B2B, Business-to-Business), the system may contain an additional link, conventionally called retailer equipment, functionally connected to the user’s device and the service provider’s equipment. In this case, the remote clothing selection algorithm may largely be performed by the retailer's equipment and the service provider's equipment, and the user's device may be used primarily in terminal mode, for example, for visualization and/or for receiving control commands through the user interface.

In some embodiments of the invention, the system may have a client-server architecture, wherein the user device may be a client device and the service provider equipment may include a server device. As used herein, the term "server" means a computer program executed by associated hardware and capable of receiving requests (eg, from client devices) over a network and executing or initiating execution of those requests. The term "client" means a computer program executed by appropriate hardware and capable of generating requests to the server, receiving a response from the server and processing it.

It should be noted that a device that functions as a client device can also function as a server device for other client devices. The use of the expression “client device” does not exclude the use of multiple client devices to receive, send, perform, or initiate execution of any task or request, or the results of any tasks or requests, or steps of any method described herein.

In other embodiments of the invention, the system may have a peer-to-peer architecture, for example, based on a peer-to-peer network. The system can also have any hybrid architecture.

In fig. 34 shows an illustrative example of a system (20) for performing remote clothing selection in a B2C environment, comprising a user device (21) executing a program (22) and a service provider equipment (23) executing the program (24). The user device (21) is connected to the service provider equipment (23) via a network connection (25). In some embodiments of the invention, the network connection (25) may be implemented using the Internet. In other embodiments of the invention, the network connection (25) may be implemented using any wide area communications network, local communications network, private communications network, or the like. The implementation of the network connection (25) depends, among other things, on the implementation of the user device (21) and the service provider equipment (23). The system (20) is functionally connected to the clothing supplier's system (30). In some embodiments of the invention, systems (20) and (30) can be combined into one system. In other embodiments, the system (20) may be operably linked to multiple systems (30) from different clothing suppliers.

In fig. 35 shows another illustrative example of a system (20) for performing remote clothing selection in a B2B environment, in which the system (20) includes retailer equipment (26) executing a program (28) linked to service provider equipment (23) via a network connection. (25), and with the user’s device (21) through a local network connection (27).

As an example, in those embodiments of the invention where the user device (21) is implemented as a wireless communication device (eg, a smartphone), the network connection (25) may be implemented using a wireless communication link (eg, a 3G network link, a communication networks 4G, WiFi, Bluetooth, etc.). In other embodiments of the invention, where the user device (21) is implemented as a remote computer, the network connection (25) may be implemented using a wireless communication link (such as WiFi, Bluetooth, etc.), a wired communication link, or an optical communication link (such as an Ethernet connection). Network connection (27) can be implemented in a similar way.

In fig. 34 and fig. 35, for the sake of simplicity, only one user device (21), one service provider piece of equipment (23), and one clothing provider system (30) are shown. In practice, the system (20) may contain multiple user devices (21) (e.g., hundreds, thousands, or millions of such devices), multiple service provider equipment (23) (e.g., tens or hundreds of regional service provider sites), and may be associated with multiple systems (30) of clothing suppliers (for example, with tens or hundreds of clothing suppliers). In this case, the software implementation of the remote clothing selection algorithm is distributed between programs (22), (24) and (28). In fig. 34 and fig. 35, for simplicity, equipment for optical scanning of the user's body is not shown.

It should be noted that the above description reflects only those actions that are most essential to achieve the purpose of the invention. It should be clear to a specialist that for the operation of the system (20), other necessary actions must be performed, for example, connecting the equipment, its initialization, launching the corresponding software, sending and receiving commands and confirmations, exchanging service data, synchronization, etc., the description of which is omitted for brevity.

The devices and their parts mentioned in the description, drawings and claims are software and hardware, and the hardware parts of some devices may be different, partially the same or completely the same as the hardware parts of other devices, unless otherwise expressly indicated. Hardware parts of devices may be located in different parts of other devices, unless otherwise explicitly stated. Program parts (modules) can be implemented in the form of program code contained in a storage device of any type.

The sequence of actions in the description of the method is illustrative in nature and in various embodiments of the invention this sequence may differ from that described, provided that the function performed and the result achieved are preserved.

The features of this invention can be combined in various embodiments of the invention if they do not contradict each other. The embodiments described above are for illustrative purposes only and are not intended to limit the scope of the present invention as defined by the claims. All reasonable modifications, upgrades and equivalent substitutions in the composition, construction and operation of the present invention, made within the scope of the present invention, are within the scope of the present invention.

The following are information sources, all contents of which are incorporated by reference into this application.

Sources

A1. Brett Allen, Curless Brian, Popovic Zoran, Hertzmann Aaron. Learning a correlated model of identity and pose-dependent body shape variation for real-time synthesis. Eurographics/SIGGRAPH Symposium on Computer Animation 2006, DOI: 10.2312/SCA/SCA06/147-156.

A2. Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, James Davis. SCAPE: shape completion and animation of people. ACM SIGGRAPH 2005 Papers, pp. 408–416, DOI: 10.1145/1186822.1073207.

A3. Alexandru O. Balan. Detailed Human Shape and Pose from Images. Doctoral Theses, Department of Computer Science at Brown University, Providence, Rhode Island, May 2010

A4. Ilya Baran, Jovan Popovic. Automatic rigging and animation of 3D characters. ACM SIGGRAPH 2007 papers, article 72, DOI: 10.1145/1275808.1276467.

A5. Federica Bogo, Javier Romero, Matthew Loper, Michael J. Black. FAUST: Dataset and evaluation for 3D mesh registration. IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23–28 June 2014, DOI: 10.1109/CVPR.2014.491.

A6. Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, Michael J. Blac. Keep it SMPL: Automatic Estimation of 3D human pose and shape from a single image. Computer Vision – ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part V (pp.561–578), DOI: 10.1007/978-3-319-46454-1_34.

A7. Zhi-Quan Cheng, Yin Chen, Ralph R. Martin, Tong Wu, Zhan Song. Parametric modeling of 3D human body shape – a survey. Computers & Graphics. Volume 71, April 2018, pp. 88–100, DOI: 10.1016/j.cag.2017.11.008.

A8. Nils Hasler, Carsten Stoll, Martin Sunkel, Bodo Rosenhahn, Hans-Peter Seidel. A statistical model of human pose and body shape. Computer Graphics Forum (Proc. Eurographics 2009), Munich, Germany, March 2009, DOI: 10.1111/j.1467-8659.2009.01373.x.

A9. Doug L. James, Christopher D. Twigg. Skinning mesh animations. ACM Transactions on Graphics (SIGGRAPH 2005), Volume 24, Issue 3, July 2005, pp. 399–407, DOI: 10.1145/1186822.1073206.

A10. Ladislav Kavan, Steven Collins, Jirí Zára, Carol O'Sullivan. Geometric skinning with approximate dual quaternion blending. ACM Transactions on Graphics (TOG) Volume 27 Issue 4, October 2008, Article No. 105, DOI: 10.1145/1409625.1409627.

A11. Zorah Lahner, Daniel Cremers, Tony Tung. DeepWrinkles: Accurate and Realistic Clothing Modeling. Computer Vision – ECCV 2018, pp. 698–715, DOI:10.1007/978-3-030-01225-0_41.

A12. Binh Huy Le, Zhigang Deng. Robust and accurate skeletal rigging from mesh sequences. ACM Transactions on Graphics (TOG) Volume 33, Issue 4, July 2014, Article No. 84, DOI: 10.1145/2601097.2601161.

A13. JP Lewis, Matt Cordner, Nickson Fong. Pose space deformation: a unified approach to shape interpolation and skeleton-driven deformation. SIGGRAPH '00. Proceedings of the ^27th annual conference on Computer graphics and interactive techniques, pp. 165–172, DOI: 10.1145/344779.344862.

A14. Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, Michael J. Black. SMPL: A skinned multi-person linear model. ACM Transactions on Graphics (TOG) Volume 34 Issue 6, November 2015 Article No. 248, DOI: 10.1145/2816795.2818013.

A15. Akihiko Murai, Yui Endo, Mitsunori Tada. Anatomical volumetric skin-musculoskeletal model and its kinematic deformation with surface-based SSD. IEEE Robotics and Automation Letters, Volume: 1, Issue: 2, July 2016, pp. 1103–1109, DOI: 10.1109/LRA.2016.2524069.

A16. Leonid Pishchulin, Stefanie Wuhrer, Thomas Helten, Christian Theobalt, Bernt Schiele Building statistical shape spaces for 3D human modeling. Pattern Recognition, Volume 67, July 2017, Pp. 276–286, DOI: 10.1016/j.patcog.2017.02.018.

A17. Thomas Probst, Andrea Fossati, Mathieu Salzmann, Luc Van Gool. Efficient model-free anthropometry from depth data. 2017 International Conference on 3D Vision (3DV), Qingdao, China, 10–12 October 2017, DOI: 10.1109/3DV.2017.00062.

A18. Yassir Saquil, Qun-Ce Xu, Yong-Liang Yang, Peter Hall. Rank3DGAN: Semantic mesh generation using relative attributes. https://arxiv.org/abs/1905.10257.

A19. Hyewon Seo, Frederic Cordier, Nadia Magnenat-Thalmann. Synthesizing animatable body models with parameterized shape modifications. Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, pp. 120–125, DOI: 10.2312/SCA03/120-125.

A20. Chang Shu, Stefanie Wuhrer, Pengcheng Xi. 3D anthropometric data processing. Int. J. Human Factors Modeling and Simulation, Vol. 3, No. 2, 2012, DOI: 10.1504/IJHFMS.2012.051093.

A21. Miroslava Slavcheva, Maximilian Baust, Slobodan Ilic. SobolevFusion: 3D reconstruction of scenes undergoing free non-rigid motion. IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018, DOI:10.1109/CVPR.2018.00280.

A22. Qiaosong Wang, Vignesh Jagadeesh, Bryan Ressler, Robinson Piramuthu. Im2Fit: Fast 3D model fitting and anthropometrics using single consumer depth camera and synthetic data. Electronic Imaging, 3D Image Processing, Measurement (3DIPM), and Applications 2016, pp. 3DIPM-045.1–3DIPM-045.7, DOI: 10.2352/ISSN.2470-1173.2016.21.3DIPM-045.

A23. Ofir Weber, Olga Sorkine, Yaron Lipman, Craig Gotsman. Context-aware skeletal shape deformation. Computer Graphics Forum, Volume 26, Issue 3, September 2007, pp. 265–274, DOI: 10.1111/j.1467-8659.2007.01048.x.

A24. Yipin Yang, Yao Yu, Yu Zhou, Sidan Du, James Davis, Ruigang Yang. Semantic parametric reshaping of human body models. 2nd International Conference on 3D Vision, Tokyo, Japan, 8–11 December 2014, DOI: 10.1109/3DV.2014.47.

A25. Mao Ye, Xianwang Wang, Ruigang Yang, Liu Ren, Marc Pollefeys. Accurate 3D pose estimation from a single depth image. 2011 International Conference on Computer Vision, Barcelona, Spain, 6–13 November 2011, DOI: 10.1109/ICCV.2011.6126310.

A26. Tao Yu, Kaiwen Guo, Feng Xu, Yuan Dong, Zhaoqi Su, Jianhui Zhao, Jianguo Li, Qionghai Dai, Yebin Liu. BodyFusion: Real-time capture of human motion and surface geometry using a single depth camera. IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22–29 October 2017, DOI: 10.1109/ICCV.2017.104.

A27. Tao Yu, Zerong Zheng, Kaiwen Guo, Jianhui Zhao, Qionghai Dai, Hao Li, Gerard Pons-Moll, Yebin Liu. DoubleFusion: Real-Time capture of human performances with inner body shapes from a single depth sensor. IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018, DOI:10.1109/CVPR.2018.00761.

A28. Chao Zhang, Sergi Pujades, Michael J. Black, Gerard Pons-Moll. Detailed, accurate, human shape estimation from clothed 3D scan sequences. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 4191–4200, DOI: 10.1109/CVPR.2017.582.

A29. Yingliang Zhang, Xi Luo, Wei Yang, Jingyi Yu. Fragmentation guided human shape reconstruction. IEEE Access, Volume 7, pp. 45651–45661, DOI: 10.1109/ACCESS.2019.2905879.

A30. Zerong Zheng, Tao Yu, Hao Li, Kaiwen Guo, Qionghai Dai, Lu Fang, Yebin Liu. HybridFusion: Real-time performance capture using a single depth sensor and sparse IMUs. The European Conference on Computer Vision (ECCV), 2018, part 9, pp. 389–406, DOI: 10.1007/978-3-030-01240-3_24.

B1. Tao Yu, Zerong Zheng, Kaiwen Guo, Jianhui Zhao, Qionghai Dai, Hao Li, Gerard Pons-Moll, Yebin Liu. DoubleFusion: Real-Time capture of human performances with inner body shapes from a single depth sensor. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 18–23 June 2018, Salt Lake City, UT, USA, DOI:10.1109/CVPR.2018.00761.

B2. Richard A. Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J. Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, Andrew Fitzgibbon. Kinectfusion: Real-time dense surface mapping and tracking. ^10th IEEE International Symposium on Mixed and Augmented Reality, 26–29 October 2011, Basel, Switzerland, DOI: 10.1109/ISMAR.2011.6092378.

B3. Sylvain Paris, Pierre Kornprobst, Jack Tumblin, Fredo Durand. Bilateral filtering: Theory and applications Foundations and Trends in Computer Graphics and Vision, Vol. 4, No. 1 (2009), pp. 1–73, DOI: 10.1561/0600000020.

B4. Konushin A. Geometric properties of several images // Computer graphics and multimedia, 2006, no. 4, p. 3.

B5. Guzov V.V. Calculation of the parameters of the pose and shape of a three-dimensional human model using depth sensor data // Collection of abstracts of the XXVI International Scientific Conference of Students, Postgraduate Students and Young Scientists "Lomonosov-2019": section "Computational Mathematics and Cybernetics". – M.: Publishing department of the Faculty of Computational Mathematics and Mathematics of Moscow State University, 2019, pp. 45–47.

B6. Richard A. Newcombe, Dieter Fox, Steven M. Seitz. DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time. Conference on Computer Vision and Pattern Recognition (CVPR), 7–12 June 2015, Boston, MA, USA, DOI: 10.1109/CVPR.2015.7298631.

B7. Ladislav Kavan, Steven Collins, Jiri Zara, Carol O'Sullivan. Skinning with dual quaternions. Symposium on Interactive 3D Graphics, SI3D 2007, April 30 – May 2, 2007, Seattle, Washington, USA, DOI: 10.1145/1230100.1230107.

B8. Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, Michael J. Black. SMPL: A skinned multi-person linear model. ACM Transactions on Graphics (TOG), Volume 34, Issue 6, November 2015, Article No. 248, DOI: 10.1145/2816795.2818013.

B9. Robinette K. M., Daanen H., Paquet E. The CAESAR project: a 3-D surface anthropometry survey. Second International Conference on 3-D Digital Imaging and Modeling, 8 October 1999, Ottawa, Ontario, Canada, DOI: 10.1109/IM.1999.805368.

B10. Kirk MacTavish, Timothy D. Barfoot. At all costs: A comparison of robust cost functions for camera correspondence outliers. ^12th Conference on Computer and Robot Vision, 3–5 June 2015, Halifax, NS, Canada, DOI: 10.1109/CRV.2015.52.

B11. Jose Luis Blanco Claraco. A tutorial on SE(3) transformation parameterizations and on-manifold optimization. University of Malaga, Technical report #012010, 2010, Last update: 03/18/2019.

B12. Ake Bjorck. Numerical methods for least squares problems. Linkoping University, Linkoping, Sweden, 1996, DOI: 10.1137/1.9781611971484.

B13. Hestenes M.R., Stiefel E. Methods of Conjugate Gradients for Solving Linear Systems. Journal of Research of the National Bureau of Standards, 49, pp. 409–435, DOI: 10.6028/jres.049.044.

B14. Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, Michael J. Black. SMPL: A skinned multi-person linear model. ACM Transactions on Graphics (TOG) Volume 34 Issue 6, November 2015 Article No. 248, DOI: 10.1145/2816795.2818013

B15. Leonid Pishchulin, Stefanie Wuhrer, Thomas Helten, Christian Theobalt, Bernt Schiele. Building statistical shape spaces for 3D human modeling. Pattern Recognition, Volume 67, July 2017, Pages 276–286, DOI: 10.1016/j.patcog.2017.02.018

B16. Allen Brett, Curless Brian, Popovic Zoran. Articulated body deformation from range scan data. ACM Transactions on Graphics (TOG), Volume 21 Issue 3, July 2002, Pages 612–619, DOI: 10.1145/566570.566626.

B17. Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, James Davis. SCAPE: shape completion and animation of people. ACM SIGGRAPH 2005 Papers, Pages 408–416, DOI: 10.1145/1186822.1073207.

B18. David A. Hirshberg, Matthew Loper, Eric Rachlin, Michael J. Black. Coregistration: Simultaneous Alignment and Modeling of Articulated 3D Shape. In: Fitzgibbon A., Lazebnik S., Perona P., Sato Y., Schmid C. (eds) Computer Vision – ECCV 2012. ECCV 2012. Lecture Notes in Computer Science, vol. 7577, Springer, Berlin, Heidelberg, DOI: 10.1007/978-3-642-33783-3_18.

B19. Arjun Jain, Thorsten Thormahlen, Hans-Peter Seidel, Christian Theobalt. MovieReshape: Tracking and Reshaping of Humans in Videos. ACM Transactions on Graphics (TOG), Volume 29, Issue 6, December 2010, Article No. 148, DOI:10.1145/1866158.1866174.

B20. Kathleen M. Robinette, Sherri Blackwell, Hein Daanen, Mark Boehmer, Scott Fleming. Civilian American and European Surface Anthropometry Resource (CAESAR), Final Report. Volume 1. Summary. 2002, DOI: 10.21236/ada406704.

B21. Wuhrer Stefanie, Shu Chang, Xi Pengcheng. Posture-invariant statistical shape analysis using Laplace operator. Computers & Graphics, 36(5), August 2012, Pages 410–416, DOI: 10.1016/j.cag.2012.03.026.

B22. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, Adam Lerer. Automatic differentiation in PyTorch. 31 ^st Conference on Neural Information Processing Systems (NIPS 2017), 4–9 December, 2017, Long Beach, CA, USA.

B23. ISO 8559-1:2017 Size design of clothes. Part 1: Anthropometric definitions for body measurement.

B24. ISO 8559-2:2017 Size design of clothes. Part 2: Primary and secondary dimension indicators.

B25. ISO 7250-1:2017 Basic human body measurements for technological design. Part 1: Body measurement definitions and landmarks.

B26. ISO/TR 7250-2:2017 Basic human body measurements for technological design. Part 2: Statistical summaries of body measurements from national populations.

B27. ISO 7250-3:2017 Basic human body measurements for technological design. Part 3: Worldwide and regional design ranges for use in product standards.

B28. GOST 17522-72 Typical figures of women. Dimensional characteristics for clothing design.

B29. GOST 31396-2009 Classification of typical female figures by height, size and weight groups for clothing design.

B30. OST 17-326-81 Sewing, knitted, fur products. Typical figures of women. Dimensional characteristics for clothing design.

B31. Daisy Veitch. Where is the human waist? Definitions, manual compared to scanner measurements. Work 41 (2012), pp. 4018–4024, DOI: 10.3233/WOR-2012-0065-4018.

B32. Guzov Vladimir Vladimirovich. Constructing a three-dimensional model of a person from video from a depth sensor. Master's dissertation. Faculty of Computational Mathematics and Cybernetics of the Federal State Budgetary Educational Institution of Higher Education "Moscow State University named after M.V. Lomonosov", Department of Intelligent Information Technologies, defense 06/01/2019.

B33. Petrov Ilya Alekseevich. Neural network algorithm for restoring the parameters of a 3D model of a human figure and pose from images. Master's dissertation. Faculty of Computational Mathematics and Cybernetics of the Federal State Budgetary Educational Institution of Higher Education "Moscow State University named after M.V. Lomonosov", Department of Intelligent Information Technologies, defense 06/01/2019.

B34. Patakin Nikolay. Development of a parametric 3D model of the human body. Federal State Autonomous Educational Institution of Higher Education “National Research University “Higher School of Economics”, Faculty of Computer Science, defense 06/06/2019.

B35. Balan A. O. et al. Detailed human shape and pose from images. // 2007 IEEE Conference on Computer Vision and Pattern Recognition. – IEEE, 2007, pp. 1–8.

B36. He K. et al. Identity mappings in deep residual networks. //European conference on computer vision. – Springer, Cham, 2016, pp. 630–645.

B37. Kanazawa A. et al. End-to-end recovery of human shape and pose // The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.

B38. Krizhevsky A., Sutskever I., Hinton G. E. Imagenet classification with deep convolutional neural networks. // Advances in neural information processing systems, 2012, pp. 1097–1105.

B39. Loper M. et al. SMPL: A skinned multi-person linear model. // ACM Transactions on Graphics (TOG), 2015, Vol. 34, No. 6, p. 248.

B40. Pavlakos G. et al. Learning to estimate 3D human pose and shape from a single color image //Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 459–468.

B41. Tan V., Budvytis I., Cipolla R. Indirect deep structured learning for 3d human body shape and pose prediction, 2018.

C1. Xuaner Cecilia Zhang, Lam Yuk Wong. Virtual fitting: real-time garment simulation for online shopping. ACM SIGGRAPH 2014 Posters, July 2014, Article No. 41, DOI: 10.1145/2614217.2633388.

C2. Amit Raj, Patsorn Sangkloy, Huiwen Chang, James Hays, Duygu Ceylan, Jingwan Lu. SWAPNET: image based garment transfer. In: Ferrari V., Hebert M., Sminchisescu C., Weiss Y. (eds) Computer Vision – ECCV 2018. ECCV 2018. Lecture Notes in Computer Science, vol. 11216, DOI: 10.1007/978-3-030-01258-8_41.

C3. Igor Santesteban, Miguel A. Otaduy, Dan Casas. Learning-based animation of clothing for virtual try-on. Eurographics 2019 (submitted on March 17, 2019), DOI: 10.1111/cgf.13643.

C4. Luo Ronglei; He Wenjie; Li Chengyi. An algorithm based on wavelet neural network for garment size selection. Telkomnika. Dec 2014, Vol. 12, Issue 4, pp. 1073–1078, DOI: 10.12928/TELKOMNIKA.v12i4.788.

C5. Suthar Dhruvi J. A review on virtual try and suggest system. Journal of the Gujarat Research Society Vol. 21, Issue 13, Dec 2019.

C6. Guan, Congying, Qin, Sheng-feng, Ling, Wessie and Ding, Guofu. Apparel recommendation system evolution: An empirical review. International Journal of Clothing Science and Technology, 28(6). pp. 854–879, DOI: 10.1108/IJCST-09-2015-0100.

C7. Haoye Dong, Xiaodan Liang, Xiaohui Shen, Bochao Wang, Hanjiang Lai, Jia Zhu, Zhiting Hu, Jian Yin. Towards multi-pose guided virtual try-on network. IEEE International Conference on Computer Vision (ICCV), 2019, pp. 9026–9035.

C8. Bochao Wang, Huabin Zheng, Xiaodan Liang, Yimin Chen, Liang. Lin, Meng Yang. Toward characteristic-preserving image-based virtual try-on network. European Conference on Computer Vision, September 8 – 14, 2018, DOI: 10.1007/978-3-030-01261-8_36.

C9. Thanasiri Muttulingam. A neural network approach to the virtual dressing room. Master's thesis, Department of Systems and Computer Engineering, the Faculty of Graduate Studies and Research, Ottawa-Carleton Institute for Electrical and Computer Engineering, 2007.

C10. J. Q. Yan, V. E. Kuzmichev. Virtual technology of made-to-measure men's shirt. IOP Conference Series: Materials Science and Engineering, Volume 460, conference 1, DOI: 10.1088/1757-899X/460/1/012014.

C11. Ruiyun Yu, Xiaoqi Wang, Xiaohui Xie. VTNFP: An image-based virtual try-on network with body and clothing feature preservation. The IEEE International Conference on Computer Vision (ICCV), 2019, pp. 10511–10520.

C12. Chia-Wei Hsieh, Chieh-Yun Chen, Chien-Lung Chou, Hong-Han Shuai, Jiaying Liu, Wen-Huang Cheng. FashionOn: Semantic-guided image-based virtual try-on with detailed human and clothing information. Proceedings of the 27th ACM International Conference on Multimedia October 2019, pp. 275–283, DOI: 10.1145/3343031.3351075.

C13. Taleb Alashkar, Songyao Jiang, Shuyang Wang, Yun Fu. Examples-rules guided deep neural network for makeup recommendation. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence 2017, Pages 941–947.

C14. Hyug Jae Lee, Rokkyu Lee, Minseok Kang, Myounghoon Cho, Gunhan Park. LA-VITON: A network for looking-attractive virtual try-on. The IEEE International Conference on Computer Vision (ICCV), 2019.

C15. Surgan Jandial, Ayush Chopra, Kumar Ayush, Mayur Hemani, Abhijeet Kumar, Balaji Krishnamurthy SieveNet: A unified framework for robust image-based virtual try-on. Accepted at IEEE WACV 2020 (Submitted on 17 Jan 2020).

C16. Pascal Volino, Frederic Cordier, Nadia Magnenat-Thalmann. From early virtual garment simulation to interactive fashion design. Computer-Aided Design 37 (2005) pp. 593–608, DOI: 10.1016/j.cad.2004.09.003.

C17. Shantseva O.A., Petrosova I.A., Andreeva E.G., Ivanova A.A. A study of existing virtual clothing fitting systems. Modern problems of science and education, No. 2–3, 2015, p. 10.

Claims (95)

1. A method for determining the dimensional characteristics of a person, including: - construction of a geometric model of the human body based on three-dimensional optical scanning data of the surface of the human body in a canonical pose; - construction of a parametric model of the human body based on the constructed geometric model; - determination of dimensional characteristics of a person based on the constructed parametric model of the human body, while: - construction of a parametric model includes: - calculation of the initial approximation of the pose parameters; - refinement of pose parameters and/or refinement of pose and shape parameters, where refinement of pose parameters includes minimizing the error function for pose parameters equal to the sum of squared angles between pairs of segments connecting characteristic points of the human skeleton in the geometric model and in the parametric model, and/ or equal to the sum of squared distances between pairs of points from a set of characteristic points in the geometric model and in the parametric model; - construction of a morphing field graph that ensures the transformation of a geometric model from an initial pose and/or shape into an arbitrary pose and/or shape when changing the parameters of the pose and/or shape; - the graph of the morphing field has vertices containing coordinates in three-dimensional space, parameters of the Euclidean transformation and a weighting coefficient, and also has edges between the vertices that ensure the connectivity of the morphing field; - The Euclidean transformation of an arbitrary point x includes the construction of an orthogonal transformation matrix R and a translation vector t, and the result of the transformation of point x is the point Rx + t. 2. The method according to claim 1, in which the construction of the geometric model includes obtaining a depth map. 3. The method according to claim 2, in which the construction of the geometric model includes converting the depth map into a point cloud. 4. The method of claim 3, wherein converting the depth map to a point cloud includes applying a bilateral filter. 5. The method according to claim 4, in which the construction of a geometric model is performed for the canonical human pose. 6. The method according to claim 1, in which when calculating the initial approximation of the pose parameters, information about the position of the characteristic points of the skeleton in the parametric model is used. 7. The method according to claim 1, in which first the error function for the pose parameters is used, equal to the sum of the squares of the angles between pairs of segments connecting the characteristic points of the human skeleton in the geometric model and in the parametric model, and then the error function for the pose parameters is used, equal to the sum squared distances between pairs of points from a set of characteristic points in the geometric model and in the parametric model. 8. The method according to claim 1, in which the refinement of the pose and shape parameters includes minimizing the error function over the pose and shape parameters. 9. The method according to claim 8, in which the following error function for the parameters of pose and shape is used: , where P is the set of visible points x of the geometric model, θ is the pose parameters, β is the shape parameters, U is the set of visible vertices v of the parametric model with pose and shape parameters (θ, β), p is the Huber function with parameter δ equal to 1. 10. The method according to claim 1, in which the formation of graph vertices includes: - selection of the value δ of the neighborhood radius; - division of space into voxels with a side of 2δ; - determination of vertex coordinates as medoid coordinates for each voxel; - assignment of a weight coefficient equal to 2δ for each vertex. 11. The method according to claim 1, in which edges between the vertices of the graph are formed if these vertices are one of the closest vertices along the shortest path in the graph. 12. The method according to claim 11, in which the number of nearest vertices is chosen to be eight. 13. The method according to claim 1, in which the Euclidean transformation of an arbitrary point x includes: - determination of the vertices v closest to point x _from the graph; - calculating the weighted sum of double quaternions of these vertices, with the weight w of each double quaternion calculated using the following formula: or is taken equal to 0 if ||R − v _c || ₂ > 3v _w , where v _w is the weight coefficient of the vertex; - normalization of the summation result. 14. The method according to claim 1, in which a parametric model of the human body is built using a machine learning algorithm. 15. The method according to claim 14, in which the machine learning algorithm includes using an error function for determining characteristic points of the human skeleton in two-dimensional space, an error function for determining characteristic points of the human skeleton in three-dimensional space, an error function for determining the parameters of the SMPL model, and a map determination error function depths. 16. The method according to claim 1, in which the dimensional characteristics of a person are determined using linear algebra. 17. The method according to claim 1, in which the dimensional characteristics of a person are determined using a machine learning algorithm. 18. The method according to claim 1, in which the intermediate dimensional features of a person are first determined using linear algebra, and then the final dimensional features of a person are determined using a machine learning algorithm. 19. The method of claim 17 or 18, wherein the machine learning algorithm uses a regression model. 20. The method according to claim 1, further including determining the type of human figure. 21. A method for determining the dimensional characteristics of an item of clothing, including: - obtaining a production geometric model used in the production of this item of clothing; - transformation of the production geometric model into a parametric production model based on the Non-Rigid Deformation algorithm using the following loss function: , where E _data is the Data term loss function, E _smooth is the Smoothness term loss function, E _normal is the normalization loss function, which penalizes for large angles between the normals of adjacent triangles, α, β, γ are weighting coefficients; - determination of dimensional characteristics of a garment from a parametric production model. 22. The method of claim 21, wherein the manufacturing geometric model is derived from the physical mannequin. 23. The method of claim 21, wherein the manufacturing geometric model is derived from a human model. 24. The method according to claim 21, in which the production geometric model is obtained based on the computer model. 25. The method of claim 21, wherein the production geometric model is obtained based on at least one of the following: technical specifications, specifications, and size grading rules. 26. The method according to claim 21, in which the loss function E _data is the following function: , where w _i is the weight of the i-th vertex in the production parametric model, A _i is the transformation matrix of the i-th vertex, p _i is the coordinate of the i-th vertex in the production parametric model, NN(i) is the vertex of the production geometric model, the nearest neighbor of i th vertex of the production parametric model. 27. The method according to claim 21, in which the loss function E _smooth is the following function: , where edges(T) is the set of all edges of the production parametric model, A _i is the 4×4 transformation matrix for vertex i, || · || _F is the Frobenius norm of the corresponding matrix. 28. The method according to claim 21, in which the loss function E _normal is the following function: , where |edges| - number of edges in the production parametric model, edges(T) - set of edges in the production parametric model, n _i - normal vector for triangle i, - angle between normals, F(·) - activation function to the angle value . 29. The method according to claim 21, wherein the weighting coefficient α is equal to the first predetermined value at the beginning of the algorithm, and then takes the second predetermined value. 30. The method of claim 21, wherein the first predetermined value is 0 and the second predetermined value is 1. 31. The method of claim 21, wherein the weighting coefficient β is first equal to the first predetermined value and then equal to the second predetermined value. 32. The method of claim 21, wherein the first preset value is 10 ^-3 and the second preset value is decreased by a factor of 4 at each step. 33. The method according to claim 21, in which the second predetermined value is changed until the condition β ≥ 10 ^-6 α is satisfied or until the error stops decreasing. 34. The method of claim 21, wherein the weighting coefficient γ is first equal to the first predetermined value and then takes the second predetermined value. 35. The method of claim 21, wherein the first preset value is 10 ⁶ and the second preset value is decreased by a factor of 4 at each step. 36. The method according to claim 21, in which the second predetermined value is changed until the condition γ ≥ 10 ³ α is satisfied or until the error stops decreasing. 37. A method for determining the degree of suitability of an item of clothing for a person, performed using a computer device and including: - selection of dimensional characteristics for comparison based on the category of the item of clothing; - obtaining dimensional characteristics of a garment using at least one of the following: optical scanning of a finished garment, measurement of a finished garment, measurement of a pattern for sewing a garment, measurement of a mannequin for sewing a garment, analysis of production documentation for sewing a garment, analysis of a digital model item of clothing; - obtaining dimensional characteristics of a person using at least one of the following: optical scanning of the surface of the human body, measuring a reference item of clothing; - comparison of dimensional characteristics of a person and a piece of clothing using the objective function , where M is the number of dimensional features, - dimensional sign of a person, - dimensional indication of the item of clothing, - weighting factor; - determination of the recommended clothing size S: S = Σ(w _i s _i ) / Σw _i + ds, where s _i is the size corresponding to the i-th dimensional feature, w _i is the weighting coefficient of the i-th dimensional feature, ds is at least one correction value to compensate for known deviations. 38. The method according to claim 37, in which the objective function is minimized. 39. The method according to claim 37, in which the correction value ds is determined individually for each person. 40. The method according to claim 37, in which the correction value ds is determined individually for each type of human figure. 41. The method according to claim 37, in which the recommended size S is additionally adjusted to the size chart. 42. The method according to claim 41, in which the correction value ds is used to bring the recommended size S into line with the sizing chart. 43. The method of claim 37, including generating a list of garments in which the garments are ranked according to a fit rating, and the garments with the same fit rating are further ranked by a fit error value. 44. The method of claim 37, wherein the fit rating is determined based on a vector of dimensional attributes of the garment and a vector of dimensional attributes of the person. 45. The method according to claim 37, in which the value of the fit error is determined based on the L2-norm for groups of the difference between the coordinates of the vector of dimensional characteristics of a garment and the vector of dimensional characteristics of a person, and weighting coefficients are applied to the groups. 46. The method according to claim 37, additionally including taking into account stylistic recommendations. 47. The method according to claim 37, in which the comparison of the dimensional characteristics of a person and a piece of clothing is performed using a neural network containing at least two layers, one of which acts as an encoder representing the dimensional characteristics of a person, and the other layers act as a decoder providing comparison. 48. The method of claim 47, wherein the encoder layer is first trained on a brand-neutral apparel dataset separate from the decoder layers. 49. The method of claim 47, wherein the encoder layer is then trained on the brand-specific garment data set in conjunction with the decoder layers. 50. The method of claim 47, wherein the brand-neutral clothing item data set comprises sizing data for a person, sizing data for an item of clothing, and data regarding whether a particular item of clothing fits or does not fit a particular individual. 51. The method of claim 47, wherein the data is based on national and/or regional statistics on population size characteristics. 52. The method of claim 47, wherein the brand-specific clothing item data set contains sizing data for a population, sizing data for a brand's clothing items, and data regarding whether a particular clothing item of that brand fits or does not fit a particular person. 53. The method according to claim 47, in which the neural network is implemented as a classifier. 54. The method according to claim 37, in which the comparison of dimensional attributes of a person and a piece of clothing is performed using a decision tree gradient boosting algorithm.