CN112401369B - Body parameter measurement method, system, device, chip and medium based on human body reconstruction - Google Patents
Body parameter measurement method, system, device, chip and medium based on human body reconstruction Download PDFInfo
- Publication number
- CN112401369B CN112401369B CN202011320729.9A CN202011320729A CN112401369B CN 112401369 B CN112401369 B CN 112401369B CN 202011320729 A CN202011320729 A CN 202011320729A CN 112401369 B CN112401369 B CN 112401369B
- Authority
- CN
- China
- Prior art keywords
- human body
- model
- camera
- points
- template
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000691 measurement method Methods 0.000 title claims abstract description 11
- 238000000034 method Methods 0.000 claims abstract description 37
- 230000000007 visual effect Effects 0.000 claims abstract description 16
- 239000011159 matrix material Substances 0.000 claims description 48
- 230000009466 transformation Effects 0.000 claims description 43
- 210000000988 bone and bone Anatomy 0.000 claims description 30
- 238000004422 calculation algorithm Methods 0.000 claims description 23
- 230000033001 locomotion Effects 0.000 claims description 12
- 239000000284 extract Substances 0.000 claims description 11
- 238000012549 training Methods 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 8
- 238000013528 artificial neural network Methods 0.000 claims description 7
- 238000004364 calculation method Methods 0.000 claims description 6
- 238000001514 detection method Methods 0.000 claims description 6
- 238000013527 convolutional neural network Methods 0.000 claims description 5
- 238000013507 mapping Methods 0.000 claims description 5
- 238000004891 communication Methods 0.000 claims description 4
- 230000001360 synchronised effect Effects 0.000 claims description 4
- 238000005457 optimization Methods 0.000 claims description 3
- 230000011218 segmentation Effects 0.000 claims description 3
- 238000013519 translation Methods 0.000 claims description 3
- 238000005259 measurement Methods 0.000 abstract description 25
- 241000220225 Malus Species 0.000 description 9
- 238000012545 processing Methods 0.000 description 6
- 238000005070 sampling Methods 0.000 description 5
- 238000003860 storage Methods 0.000 description 5
- 230000001276 controlling effect Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000036541 health Effects 0.000 description 3
- 238000005452 bending Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 235000021016 apples Nutrition 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000009792 diffusion process Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000877 morphologic effect Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000005096 rolling process Methods 0.000 description 1
- 230000007306 turnover Effects 0.000 description 1
Classifications
-
- A—HUMAN NECESSITIES
- A41—WEARING APPAREL
- A41H—APPLIANCES OR METHODS FOR MAKING CLOTHES, e.g. FOR DRESS-MAKING OR FOR TAILORING, NOT OTHERWISE PROVIDED FOR
- A41H1/00—Measuring aids or methods
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Measuring devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/107—Measuring physical dimensions, e.g. size of the entire body or parts thereof
- A61B5/1072—Measuring physical dimensions, e.g. size of the entire body or parts thereof measuring distances on the body, e.g. measuring length, height or thickness
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Measuring devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/107—Measuring physical dimensions, e.g. size of the entire body or parts thereof
- A61B5/1075—Measuring physical dimensions, e.g. size of the entire body or parts thereof for measuring dimensions by non-invasive methods, e.g. for determining thickness of tissue layer
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Measuring devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/107—Measuring physical dimensions, e.g. size of the entire body or parts thereof
- A61B5/1077—Measuring of profiles
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Measuring devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/107—Measuring physical dimensions, e.g. size of the entire body or parts thereof
- A61B5/1079—Measuring physical dimensions, e.g. size of the entire body or parts thereof using optical or photographic means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Biophysics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Biomedical Technology (AREA)
- Public Health (AREA)
- Animal Behavior & Ethology (AREA)
- Dentistry (AREA)
- Veterinary Medicine (AREA)
- Surgery (AREA)
- Medical Informatics (AREA)
- Heart & Thoracic Surgery (AREA)
- Pathology (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Theoretical Computer Science (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Evolutionary Computation (AREA)
- Data Mining & Analysis (AREA)
- Artificial Intelligence (AREA)
- Textile Engineering (AREA)
- Image Analysis (AREA)
Abstract
Body parameter measurement method, system, device, chip and medium based on human body reconstruction, comprising: collecting color images of different visual angles around a human body; extracting image features to obtain a human body model; non-rigid deformation is carried out on the human body model, and the human body model is consistent with the template model; and extracting vertex coordinates of the deformed template model, and calculating to obtain body parameters. The method for measuring various body circumference parameters such as the body waistline and the like by fitting the body model through the body template model can measure the length and circumference information of a plurality of parts of the whole body, and has high measurement speed and high precision.
Description
Technical Field
The invention relates to the fields of computer vision and computer graphics, in particular to a body parameter measurement method, system, equipment, chip and medium based on human body reconstruction.
Background
Body parameter measurement is used as a primary link of the clothing customization industry, and measurement accuracy and convenience of measurement operation are important. The human body is used as a non-rigid body with extremely large deformation, so that the difficulty of measurement is increased. The traditional measurement method mainly uses a flexible tape for manual measurement, the measurement precision is extremely easy to be influenced by subjective factors of a measurer (such as difficulty in accurately finding out a measurement part, difficulty in controlling the tightness degree of the tape and the like), the measurement mode is very troublesome, and a measurer with rich experience is required to find out the measurement part of the body of the measurer, and then uses the flexible tape for measurement from position to position; in addition, the measurement by using the tape manually inevitably results in the situation that the measurer is in close contact with the limb of the measurer, which causes some discomfort. The method realizes quick and accurate non-contact body parameter measurement, can be also used in the fields of medical health management and the like, and has great commercial value. CN201910631825.6 patent application discloses a method and device for measuring human parameters by using multi-depth camera, which comprises the following steps: collecting a current object depth image; fusing the depth images into regular point clouds; fitting a parameterized human body template, and solving morphological parameters and attitude parameters of the parameterized human body template, so that the SMPL human body template deformed based on the parameters is fitted with the obtained point cloud as much as possible; and acquiring the human body parameters of the current object by a human body parameter acquisition mode defined by the standard parameterized human body template. In the three-dimensional human model reconstruction stage, a mode of carrying out data fusion reconstruction by using 16 depth cameras to acquire point clouds is adopted, so that the hardware cost is high, the setting is complex, and the application is not extensive; body parameter measurement phase: the method of directly extracting grid vertices on the SMPL template model at the girth of the girth and the like, then calculating Euclidean distances between adjacent points one by one, and summing the calculated Euclidean distances as the girth can bring great errors at the girth and the like with complex topological structures. The reason is that the directly extracted grid vertices are not on the same plane, so that the curves forming the chest circumference are not on the same plane, and the normal measuring body circumference is measured on the same plane, which can lead to the measuring tape bending up and down for a circle, and the measuring result is far larger than the actual result.
Disclosure of Invention
The invention aims to solve the existing problems and aims to provide a body parameter measurement method, system, equipment, chip and medium based on human body reconstruction.
In order to achieve the above object, the technical solution of the present invention provides a body parameter measurement method based on human body reconstruction, including:
Step one, a camera collects color images of a plurality of different visual angles around a human body;
Extracting image features through the position and the posture of the camera and the internal and external parameter matrixes, predicting depth information of the image features and fusing the depth information to obtain a human body model;
Thirdly, carrying out non-rigid deformation on the human body model to enable the human body model to be consistent with the template model;
and step four, extracting vertex coordinates of the deformed template model, and calculating to obtain body parameters.
Further, in the first step, the camera surrounds the human body and captures an image every 120 °.
In the first step, the background of the portrait in the color image is removed in real time, the internal parameters and the external parameter matrix of the camera are output at the same time, and the height information is input.
Further, semantic segmentation is performed by adopting a convolutional neural network, namely, a target detection algorithm is executed to extract the position of a person, and then saliency detection is performed to obtain a human body image with the background removed.
In the second step, the camera extracts image features from the color image in real time by means of synchronous positioning and mapping based on monocular vision and an inertial sensor, and the image features are used for estimating the position and the posture of the camera.
Further, in the second step, the neural network model is trained through the color images of a plurality of visual angles, the camera internal reference matrix, the camera pose matrix and the high-precision human body three-dimensional model corresponding to the training set, so that the algorithm model can return to the three-dimensional human body model from any multi-visual angle human body color image.
In the third step, the mass centers of the template model and the human body model are overlapped, and after the corresponding points are extracted, the iterative optimization of the nearest neighbor points is carried out, so that coarse alignment is realized.
Further, after coarse alignment, the shape difference of the template model and the human body model is optimized, and the template model which can be used for expressing the human body model after denoising and deformation is obtained.
Further, after the bone registration, performing skin operation, namely establishing an association relation between bone points and grid vertices, and deforming the grid surface to adapt to the current bone posture.
Further, in the fourth step, the body parameters can be obtained by measuring the length of the curve formed by the vertices. The present invention also provides a system for measuring a body parameter, comprising:
the acquisition module acquires color images of a plurality of different visual angles around a human body;
The modeling module extracts image features through the position and the posture of the camera and the internal and external parameter matrix, predicts the depth information of the image features and fuses the depth information to obtain a human body model;
the deformation module is used for carrying out non-rigid deformation on the human body model to enable the human body model to be consistent with the template model;
and the calculation module is used for extracting vertex coordinates of the deformed template model and calculating body parameters.
Further, the acquisition module adopts APPLEARKIT platforms.
Further, the acquisition module shoots an RGB image around a human body at intervals of 120 degrees, simultaneously carries out real-time portrait matting at the mobile phone end through a neural network image segmentation algorithm to remove the background, shoots and outputs the internal parameters and the external parameters of the current camera, and reminds a user to input height information.
Further, APPLE ARKIT of the acquisition module extracts image features in real time from images acquired by the video stream based on the synchronous positioning and mapping modes of the monocular vision and the inertial sensor, and the image features are used for estimating the position and the posture of the user.
Further, the modeling module directly regresses and predicts the high-precision three-dimensional human body model by inputting parameters of the background-removed color images, the camera internal reference matrix and the camera pose matrix of three different visual angles into a neural network algorithm model trained in advance.
Further, the neural network algorithm model of the reconstruction module is obtained by training 500 high-precision human body three-dimensional models acquired by hundreds of color camera arrays, the training core is that three-view color images, camera internal reference matrixes and camera pose matrixes are rendered by using OpenGL to obtain human body three-dimensional models of a training set, 5000 3D points are sampled from the surface of the human body three-dimensional models and the inner and outer vicinity of the surface of the human body three-dimensional models, the corresponding relation between image pixel points and the sampled 3D points is obtained through the camera internal reference matrixes and the camera pose matrixes, the training model enables the human body three-dimensional models to predict the probability of the 3D points in the human body three-dimensional models from the images, and then the three-dimensional human body models are reconstructed.
Further, the deformation module calculates the mass centers of the template model and the human body model, then performs a rigid transformation, coincides the mass centers to the same point, respectively extracts the limb end points and the vertex points of the template model and the human body model as corresponding points, and performs a nearest neighbor point iterative algorithm to perform the rigid transformation so as to realize coarse alignment.
Further, after coarse alignment, a local rotation transformation matrix is executed at the joint where the template model needs to be deformed, the local rotation transformation matrix on the motion chain is multiplied, so that a global transformation matrix of the joint point can be obtained, and the bone deformation registration can be realized by minimizing the Euclidean distance between the corresponding joint points of the two.
Further, after bone registration, the deformation module performs skin operation, and the bone points and mesh vertex transformation formula:
w is the weight W of each bone point and the grid vertexes around the bone point; t is a transformation matrix T of each skeleton point, v p is a mesh vertex before transformation, and v' p is a vertex of the mesh after transformation.
Further, after grid gesture alignment, non-rigid deformation registration is performed: each vertex on the template model is subjected to a rotational-translational transformation such that each vertex thereon is positioned closest to a corresponding point on the manikin.
Further, the calculation module obtains vertex coordinates through the topological structure of the template model, and calculates body parameters. The invention provides a device for measuring a body parameter, comprising:
A camera that captures color images of several different perspectives around a human body;
The camera is in communication connection with the server; the server comprises a processor, and a memory for storing executable instructions of the processor, which when executed performs any of the methods described above. Further, the camera is an apple iphone mobile phone with APPLE ARKIT platforms.
The present invention provides a chip comprising a processor for calling and running a computer program from a memory, such that a device on which the chip is mounted performs any of the methods described above.
The present invention provides a computer readable medium having stored thereon computer program instructions which, when processed and executed, implement a method as described in any of the above.
Compared with the prior art, the method for measuring various body circumference parameters such as the waistline of the body by fitting the human body model containing noise information such as clothes folds through the human body template model can measure the length and circumference information of a plurality of parts of the whole body, has high measurement speed and high precision, is full-automatic non-contact measurement, can be widely used for tailoring clothes, managing and monitoring medical health, and has wide market application prospect.
The three-view human body image based on ARKit acquisition sparsity is adopted in the invention to reconstruct a high-precision human body three-dimensional model, the input is simple, the data acquisition software is very friendly to users, the manual intervention is less, other data processing and three-dimensional model reconstruction are all completed by the full automation of algorithm codes, and the invention has a very large application market.
Drawings
FIG. 1 is an explanatory diagram of capturing color images;
FIG. 2 is an explanatory diagram of capturing color images;
FIG. 3 is a schematic illustration of a phantom reconstructed from a color image, including noise;
FIG. 4 is a schematic illustration of a stencil model (TEMPLATE MESH);
FIG. 5 is a schematic illustration of a template model after pose fitting;
FIG. 6 is a schematic diagram of a template model after shape fitting;
Fig. 7 is an explanatory diagram of measurement results.
Detailed Description
The invention will now be further described with reference to the accompanying drawings.
Referring to fig. 1 to 5, fig. 1 to 5 illustrate an embodiment of the present invention, which aims at measuring a body parameter, mainly comprising:
step one, collecting color images of a plurality of different visual angles around a human body by using a camera;
Extracting image features through the position and the posture of the camera and the internal and external parameter matrixes, predicting depth information of the image features and fusing the depth information to obtain a human body model;
Thirdly, carrying out non-rigid deformation on the human body model to enable the human body model to be consistent with the template model;
and step four, extracting vertex coordinates of the deformed template model, and calculating to obtain body parameters.
Wherein, the equipment that adopts includes: a camera for capturing color images of several different perspectives around a human body. The camera is preferably an apple iphone mobile phone with APPLE ARKIT platforms; ARKit is an AR development platform as proposed by apples, where developers can create augmented reality applications using iPhone or iPad with ARKit tools. In this embodiment, ARKit implements functions such as data acquisition.
The camera is in communication connection with the server; the server comprises a processor, and a memory for storing executable instructions of the processor, which processor, when run, performs the method described above.
First, a color image of a human body is acquired. And can develop a mobile phone app of the iphone end based on the requirement of the method, the app has the following main functions:
Referring to fig. 1, an RGB image can be photographed every 120 ° around a human body, and simultaneously real-time portrait matting is performed through APPLE ARKIT to remove the background, and the internal parameters and external parameters of the current camera are output while photographing the image, and then the user is reminded to input height information. And transmitting the data to a server of a remote cloud through an http communication protocol.
The method for shooting the image every 120 degrees comprises the following steps: the mobile phone can rotate by 120 degrees each time by reading the rolling, yaw and pitching angles of the mobile phone acquired by the sensor through the sensor of the iPhone, such as a triaxial gyroscope, namely reminding a user of acquiring images.
Referring to fig. 2, in order to remove the influence of the background on the three-dimensional reconstruction of the human body, matting is required. The method for removing the portrait matting background comprises the following steps: performing semantic segmentation by adopting a convolutional neural network; firstly, a target detection algorithm is executed to extract the position of a person, and then saliency detection is carried out to obtain a human body image with the background removed.
The method for acquiring and processing the camera internal and external parameter matrix comprises the following steps: APPLE ARKIT can extract image features in real time from images acquired from video streams based on the slam mode of monocular vision and inertial sensors for estimating self position and pose.
Since the definition of the camera coordinate system in APPLE ARKIT is: the y-axis direction is opposite to the gravity direction, the x-axis direction is horizontal to the right, the z-axis direction meets the right-hand rule, and the definition of a camera coordinate system in an opencv library used in the subsequent reconstruction is as follows: the horizontal right and vertical downward are respectively the x-axis and the y-axis, and the z-axis meets the rule of the right hand, so that the coordinate system transformation is firstly carried out, and the coordinate system is unified.
Next, referring to fig. 3, a three-dimensional human body is reconstructed. Training data is rendered first. When three images are shot, the positions of the camera pose determine the positions of the camera pose in a world coordinate system, so 500 grids in a training set are required to be converted to the public visible positions of the three camera poses through rigid transformation, and then a camera projection matrix is obtained according to the camera internal reference matrix and the camera pose acquired in the stage one; and using an OpenGL render module, according to the projection matrix, three-dimensional points can be projected onto the 2D image from the grid, and images with three viewing angles can be generated. In order to increase the diversity of data and improve the generalization capability of an algorithm in the subsequent three-dimensional human body reconstruction, the embodiment also performs data enhancement work, namely Gaussian disturbance noise is respectively added to the distance from a camera to a grid, the pitch angle, the turnover angle and the yaw angle of the camera; in addition, the size and the orientation of the grid are randomly disturbed to a certain extent, and the camera pose and the internal reference matrix corresponding to the human body image generated after the noise disturbance is added are recalculated. Finally, a training set consisting of 500 sets of training data was obtained. Each set of data comprises: the method comprises the steps of removing background human body color pictures under three visual angles, corresponding camera internal reference matrixes, relative pose between the cameras and 3D point coordinates of the inside and the outside of a grid obtained by up-sampling the grid.
And obtaining a rough three-dimensional human body model through visual hull algorithm by using the background-removing images of three different visual angles, the corresponding camera internal reference matrix and the camera pose matrix. To obtain a high-precision three-dimensional mannequin, the neural network is trained: firstly, extracting image features through a convolutional neural network Hourglass, then sampling 3D points from grids, and then establishing a corresponding relation between the 3D points and 2D points of an image according to a camera projection matrix obtained by a camera pose and an internal reference matrix. On the basis, the multi-layer perceptron MLP predicts the probability that the 3D point corresponding to the image pixel is inside or outside the grid, three probabilities can be obtained each time due to the three-view image, the three probability values need to be averaged, and then the L2 distance between the predicted value and the true value is iteratively optimized in a supervised mode. And judging whether the result learned by the algorithm is good or not according to the intersection ratio between the predicted result and the real result, and indicating that the learning result is better when the IOU is closer to 1. Finally, the algorithm can well predict which points belong to points on the surface of the human body, and the final three-dimensional human body model can be reconstructed through the marking cube algorithm by using the discrete points. Then, the human body model is deformed in a non-rigid manner so as to conform to the template model.
Referring to fig. 4 and 5, bone deformation registration is performed first. In order to deform the human body model to the framework of the template model to be consistent, the human body model and the template model are required to be roughly aligned, the method of roughly aligning is to calculate the respective mass centers firstly, then execute a rigid body transformation, coincide the mass centers of two grids to the same point, respectively extract five points which are the four limb end points and the vertex point on the human body model and the template model as corresponding points, execute a nearest neighbor point iterative algorithm, and execute the rigid body transformation to realize the rough alignment.
After the rough alignment is completed, skeletal joint point deformation is needed, and skeletal actions of the human body model are deformed to be the same as those of the template model. The human body joint can move randomly within a certain range, and cascade relations exist among all the joint points, namely the movement of a father node can influence the movement of the human body joint and the movement of a son node of the father node. In order to realize bone deformation registration, the 3D coordinates of each joint point of a template model are known, then a local rotation transformation matrix is executed at the joint needing deformation, the local rotation transformation matrix on a motion chain is multiplied, the global transformation matrix of the joint point can be obtained, and the bone deformation registration can be realized by minimizing Euclidean distance between corresponding joint points of a human body model and the template model.
After bone registration, a skin operation is required, namely, an association relation between bone points and grid vertices is established, and the grid indication is deformed to adapt to the current bone posture. Each skeletal point may be associated with a number of vertices, and each vertex may be weighted differently. When a bone point undergoes rotational translational transformation, its associated mesh vertex also undergoes a different degree of positional transformation. Deformation of the whole grid can be controlled by controlling the rotation vector of the skeleton points, and the transformation formula of the grid vertex is as follows:
firstly, obtaining 3D coordinates of skeleton points of a template model, and calculating the weight w of each skeleton point and the surrounding mesh vertexes through a thermal diffusion equation; the basic principle is that the smaller the Euclidean distance from a bone point, the larger the weight, and the weight range is between 0 and 1. In the skeleton deformation registration stage, a transformation matrix T of each skeleton point can be obtained, v p is a grid vertex before transformation, v' p is a vertex of the transformed grid, and the coordinate of each vertex on the grid after gesture transformation can be obtained through the formula, so that a new grid is obtained.
Referring to fig. 6, after the grid is aligned, the shape alignment can be achieved by non-rigid deformation registration. The method for registering the non-rigid deformation comprises the following steps: and carrying out rotary translation transformation on each vertex on the template model so that the position of each vertex on the template model is closest to the corresponding point on the human body model, and simultaneously, the whole template model is kept smooth locally. In order to reduce the calculation amount and improve the fitting speed, the embodiment adopts a method of deforming the picture (deformation graph), namely, firstly taking the geodesic distance (geodesic distance) as a measurement standard on the human body model, uniformly and equally sampling all vertexes at fixed geodesic distance coefficients, and only optimizing the key points after sampling when subsequently optimizing the errors of corresponding points of the human body model and the template model, wherein the related points controlled by the key points only need to be ensured to be smooth during global optimization.
Finally, referring to fig. 7, parameters of the body circumference are measured. Since the topology of the phantom reconstructed each time is always changed, but the topology of the template model is not changed, the IDs of the vertices of the parts to be measured such as the arm circumference and the waistline can be easily extracted.
Because the number of the vertexes forming the waistline and other parts is very dense, and the distribution of the vertexes is basically on the same circumference plane, the distance between two adjacent points can be directly calculated and summed, and the waistline and other parts can be approximately expressed by polygons formed by dense straight line segments, so that a high-precision measurement result can be obtained.
However, the same point-taking method brings great errors at chest circumference with complex topological structure. The reason is that the vertices directly extracted are not on the same circumferential plane, so that the curves forming the chest circumference are not on the same plane, which can lead to a circle of the measuring tape bending up and down, resulting in a measurement result far greater than the actual result. The method and the device have the advantages that interpolation processing is carried out on the parts with the complex topological structures, so that the points forming the chest circumference are distributed in one plane as much as possible, and the measurement accuracy of the parts is greatly improved.
Referring to fig. 2, the present invention further provides a model optimized deployment system, comprising: the acquisition module acquires color images of a plurality of different visual angles around a human body;
The modeling module extracts image features through the position and the posture of the camera and the internal and external parameter matrix, predicts the depth information of the image features and fuses the depth information to obtain a human body model;
the deformation module is used for carrying out non-rigid deformation on the human body model to enable the human body model to be consistent with the template model;
and the calculation module is used for extracting vertex coordinates of the deformed template model and calculating body parameters.
Preferably, the acquisition module employs a APPLEARKIT platform.
Preferably, the acquisition module shoots an RGB image around a human body at intervals of 120 degrees, simultaneously carries out real-time portrait matting to remove the background through APPLE ARKIT, shoots and outputs the internal parameters and the external parameters of the current camera at the same time, and reminds a user to input height information.
Preferably, APPLE ARKIT of the acquisition module extracts image features in real time from images acquired by the video stream based on the synchronous positioning and mapping modes of the monocular vision and the inertial sensor, and the image features are used for estimating the position and the posture of the user.
Preferably, the modeling module obtains a rough three-dimensional human body model through visual hall algorithm through three background-removed color images with different visual angles, a camera internal reference matrix and a camera pose matrix; and extracting image features through a convolutional neural network Hourglass, sampling 3D points from the grid, establishing a corresponding relation between the 3D points and the 2D points of the image, and obtaining 3D points belonging to the surface of the human body through predicting depth information so as to reconstruct a human body model.
Preferably, the deformation module calculates the mass centers of the template model and the human body model, then performs a rigid transformation, coincides the mass centers to the same point, respectively extracts the limb end points and the vertex points of the template model and the human body model as corresponding points, and performs a nearest neighbor point iterative algorithm to perform the rigid transformation so as to realize coarse alignment.
Preferably, after coarse alignment, a local rotation transformation matrix is executed at the joint where the template model needs to be deformed, the local rotation transformation matrix on the motion chain is multiplied, so that a global transformation matrix of the joint point can be obtained, and the bone deformation registration can be realized by minimizing the Euclidean distance between the corresponding joint points of the two.
Preferably, after the bone registration, the deformation module performs a skinning operation, and the bone points and mesh vertex transformation formula:
w is the weight W of each bone point and the grid vertexes around the bone point; t is a transformation matrix T of each skeleton point, v p is a mesh vertex before transformation, and v' p is a vertex of the mesh after transformation.
Preferably, after the grid gestures are aligned, the deformation module performs non-rigid deformation registration: each vertex on the template model is subjected to a rotational-translational transformation such that each vertex thereon is positioned closest to a corresponding point on the manikin.
Preferably, the calculation module obtains vertex coordinates through the topological structure of the template model, and calculates the body parameters. In some implementations, the memory referred to in this example stores the following elements, an upgrade package, an executable unit, or a data structure, or a subset thereof, or an extended set thereof: an operating system and application programs.
The operating system includes various system programs, such as a framework layer, a core library layer, a driving layer, and the like, and is used for realizing various basic services and processing hardware-based tasks. And the application programs comprise various application programs and are used for realizing various application services. The program for implementing the method of the embodiment of the invention can be contained in an application program.
In the embodiment of the present invention, the processor is configured to execute the above method steps by calling a program or an instruction stored in the memory, specifically, a program or an instruction stored in an application program.
The embodiment of the invention also provides a chip for executing the method. Specifically, the chip includes: and a processor for calling and running the computer program from the memory, so that the device on which the chip is mounted is used for executing the above method.
The present invention also provides a computer readable storage medium having a computer program stored thereon, which when executed by a processor implements the steps of the above-described method of the present invention.
For example, machine-readable storage media may include, but are not limited to, various known and unknown types of non-volatile memory.
Embodiments of the present invention also provide a computer program product comprising computer program instructions for causing a computer to perform the above method.
Those of skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In embodiments of the present application, the disclosed systems, electronic devices, and methods may be implemented in other ways. For example, the division of the units is only one logic function division, and other division manners are also possible in actual implementation. For example, multiple units or components may be combined or may be integrated into another system. In addition, the coupling between the individual units may be direct coupling or indirect coupling. In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or may exist alone physically, or the like.
The functions, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored on a machine-readable storage medium. Accordingly, aspects of the present application may be embodied in a software product, which may be stored on a machine-readable storage medium, which may include instructions for causing an electronic device to perform all or part of the processes of the aspects described in embodiments of the present application. The storage medium may include a ROM, a RAM, a removable disk, a hard disk, a magnetic disk, or an optical disk, etc. various media in which program codes can be stored.
It should be especially noted that, unlike the method of measuring parameters of multiple depth cameras and the device described in the foregoing description of the multiple depth cameras, the present embodiment directly optimizes parameters of controlling actions and shapes of the motion and shape parameters in the SMPL template model, so as to implement a method of fitting the motion and shapes of the motion and shape parameters in the SMPL template model to the depth camera array reconstruction. The method has the following two problems that the SMPL template model is poor in fitting the human body model:
1) The SMPL template model is not well matched with the Chinese posture;
2) The parameters of the SMPL template model for controlling the shape change of the human body are only 10, and the accurate change of the shape of the human body cannot be realized.
When the shape of the template model cannot be accurately fitted to the original human body model, larger errors are brought to the measurement of the body parameters, and the application is invalid.
The template model used in this embodiment is preferably manually trimmed by an animator using MAYA software on the basis of SMPL so that it better fits the average of the usual posture of chinese.
More importantly, the embodiment fits the motion in a bone-driven mode, but in the shape fitting part, non-rigid deformation registration is carried out on the template model, so that the local geometric detail surface of the template model can be optimized to be consistent with the original human model, the template model can express the shape of the human model more accurately, and the measurement accuracy is ensured.
In summary, the embodiment can fit a human body model containing noise information such as clothes folds through the human body template model, and then measure various body circumference parameters such as body waistline, and the like, so that the length and circumference information of a plurality of parts of the whole body can be measured, the measurement speed is high, the precision is high, the full-automatic non-contact measurement is realized, the method can be widely used for tailoring clothes, medical health management and monitoring, and the market application prospect is wide.
The data acquisition APP based on ARKit development adopted in the embodiment is used for acquiring sparse three-view human body images, and a high-precision human body three-dimensional model is reconstructed. The human body three-dimensional reconstruction method is simple in input, the data acquisition software is very friendly to users, the manual intervention is less, other data processing, three-dimensional model reconstruction and geometric texture mapping are all completed by the algorithm codes in a full-automatic mode, and the method has a large application market.
The embodiments of the present invention have been described above with reference to the accompanying drawings and examples, which are not to be construed as limiting the invention, but rather as modifications, variations or adaptations thereof may be made by those skilled in the art within the scope of the appended claims.
Claims (8)
1. A body parameter measurement method based on human body reconstruction, characterized by comprising:
Firstly, a camera collects color images of a plurality of different visual angles around a human body, removes the background of the human image in the color images in real time, shoots and outputs an internal parameter matrix and an external parameter matrix of the camera at the same time, and inputs height information;
Extracting image features through the position and the posture of the camera, an internal parameter matrix and an external parameter matrix, predicting depth information of the image features, and fusing the depth information to obtain a human body model;
training a neural network algorithm model through a color image with a plurality of visual angles, a camera internal parameter matrix, a camera pose matrix and a high-precision human body three-dimensional model, so that the neural network algorithm model can reconstruct a human body model from the multi-visual angle color image;
Thirdly, carrying out non-rigid deformation on the human body model to enable the human body model to be consistent with the template model;
Overlapping the mass centers of the template model and the human body model, extracting corresponding points of the limbs and the top of the head, and then performing iterative optimization on nearest neighbor points to realize coarse alignment; executing a local rotation transformation matrix at a joint of the template model to be deformed, multiplying the local rotation transformation matrix on the motion chain to obtain a global transformation matrix of the joint points, and realizing bone deformation registration by minimizing Euclidean distance between corresponding joint points of the human body model and the template model; performing skin operation after bone deformation registration, namely establishing an association relation between bone points and grid vertexes, and deforming the grid to adapt to the current bone posture; carrying out non-rigid deformation registration, namely carrying out rotation translation transformation on each vertex on the template model so that the position of each vertex on the template model is closest to a corresponding point on the human body model;
and step four, extracting vertex coordinates of the deformed template model, and calculating to obtain body parameters.
2. The body parameter measurement method based on human body reconstruction according to claim 1, wherein: in step one, the camera takes an image around the human body at every 120 °.
3. The body parameter measurement method based on human body reconstruction according to claim 1, wherein: the background of the portrait in the color image is removed by adopting a convolutional neural network to perform semantic segmentation, namely, a target detection algorithm is executed to extract the position of the person, and then saliency detection is performed to obtain the human body image with the background removed.
4. A body parameter measurement method based on human reconstruction according to claim 3, characterized in that: in the second step, the camera extracts image features from the color image in real time through a synchronous positioning mapping mode based on monocular vision and an inertial sensor, and the image features are used for estimating the position and the posture of the camera.
5. A system for measuring a body parameter, comprising:
the acquisition module acquires color images of a plurality of different visual angles around a human body; removing the background of the portrait in real time, shooting and outputting an internal parameter matrix and an external parameter matrix of the current camera at the same time, and reminding a user to input height information; extracting image features in real time for estimating the position and the posture of the user;
The modeling module extracts image features through the position and the gesture of the camera, an internal parameter matrix and an external parameter matrix, predicts the depth information of the image features and fuses the depth information to obtain a human body model; the method comprises the steps of inputting background-removed color images of three different visual angles, an internal parameter matrix of a camera and parameters of a pose matrix of the camera into a neural network algorithm model trained in advance, and directly regressively predicting a high-precision three-dimensional human body model;
the deformation module is used for carrying out non-rigid deformation on the human body model to enable the human body model to be consistent with the template model;
Calculating the mass centers of the template model and the human body model, overlapping the mass centers to the same point, respectively extracting the limb end points and the vertex points of the template model and the human body model as corresponding points, and executing a nearest neighbor point iterative algorithm to perform rigid body transformation so as to realize coarse alignment; executing a local rotation transformation matrix at the joint of the template model needing to be deformed, and multiplying the local rotation transformation matrix on the motion chain to obtain a global transformation matrix of the joint points, wherein the bone deformation registration is realized by minimizing Euclidean distance between the corresponding joint points of the two; performing skin operation after bone deformation registration, namely establishing an association relation between bone points and grid vertexes, and deforming the grid to adapt to the current bone posture; carrying out non-rigid deformation registration, namely carrying out rotation translation transformation on each vertex on the template model so that the position of each vertex on the template model is closest to a corresponding point on the human body model;
and the calculation module is used for extracting vertex coordinates of the deformed template model and calculating body parameters.
6. An apparatus for measuring a body parameter, comprising:
A camera that captures color images of several different perspectives around a human body;
The camera is in communication connection with the server; the server comprising a processor, and a memory for storing executable instructions of the processor, which processor, when run, performs the method of any of claims 1-4.
7. A chip comprising a processor for calling and running a computer program from a memory, such that a device on which the chip is mounted performs the method of any of claims 1-4.
8. A computer-readable medium, characterized by: on which computer program instructions are stored which, when processed and executed, implement the method of any of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011320729.9A CN112401369B (en) | 2020-11-23 | 2020-11-23 | Body parameter measurement method, system, device, chip and medium based on human body reconstruction |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011320729.9A CN112401369B (en) | 2020-11-23 | 2020-11-23 | Body parameter measurement method, system, device, chip and medium based on human body reconstruction |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112401369A CN112401369A (en) | 2021-02-26 |
| CN112401369B true CN112401369B (en) | 2024-10-15 |
Family
ID=74778080
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011320729.9A Active CN112401369B (en) | 2020-11-23 | 2020-11-23 | Body parameter measurement method, system, device, chip and medium based on human body reconstruction |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112401369B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113096144B (en) * | 2021-03-23 | 2022-07-29 | 清华大学 | Method and device for generating dynamic human body free viewpoint video based on neural network |
| CN113362446B (en) * | 2021-05-25 | 2023-04-07 | 上海奥视达智能科技有限公司 | Method and device for reconstructing object based on point cloud data |
| CN113658303A (en) * | 2021-06-29 | 2021-11-16 | 清华大学 | A method and device for generating virtual human based on monocular vision |
| CN113545773B (en) * | 2021-07-21 | 2022-05-31 | 合肥的卢深视科技有限公司 | Human body size measuring method and device, electronic equipment and readable storage medium |
| CN113743283A (en) * | 2021-08-30 | 2021-12-03 | 成都数字天空科技有限公司 | Mesh topology structure acquisition method and device, electronic equipment and storage medium |
| CN115346017A (en) * | 2022-01-24 | 2022-11-15 | 美的集团(上海)有限公司 | Measurement method of human body size, electronic equipment and computer storage medium |
| CN118506252B (en) * | 2024-07-17 | 2024-09-20 | 成都航空职业技术学院 | Dynamic target behavior prediction system based on 3D convolution and cyclic neural network |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108629294A (en) * | 2018-04-17 | 2018-10-09 | 华南理工大学 | Human body based on deformation pattern and face net template approximating method |
| CN109655011A (en) * | 2018-12-13 | 2019-04-19 | 北京健康有益科技有限公司 | A kind of method and system of Human Modeling dimension measurement |
| CN110123331A (en) * | 2019-04-17 | 2019-08-16 | 平安科技（深圳）有限公司 | Human body body and constitution collecting method, device and storage medium |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10321728B1 (en) * | 2018-04-20 | 2019-06-18 | Bodygram, Inc. | Systems and methods for full body measurements extraction |
| CN109242954B (en) * | 2018-08-16 | 2022-12-16 | 叠境数字科技（上海）有限公司 | Multi-view three-dimensional human body reconstruction method based on template deformation |
| CN109800653A (en) * | 2018-12-24 | 2019-05-24 | 重庆第二师范学院 | A kind of characteristics of human body's parameter extracting method and system based on image analysis |
| CN109409348A (en) * | 2018-12-29 | 2019-03-01 | 北京卡路里信息技术有限公司 | A kind of determination method, apparatus, equipment and the storage medium of user's sign |
| CN109938737A (en) * | 2019-03-01 | 2019-06-28 | 苏州博慧智能科技有限公司 | A kind of human body body type measurement method and device based on deep learning critical point detection |
| CN110175897A (en) * | 2019-06-03 | 2019-08-27 | 广东元一科技实业有限公司 | 3D synthesis fitting method and system |
-
2020
- 2020-11-23 CN CN202011320729.9A patent/CN112401369B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108629294A (en) * | 2018-04-17 | 2018-10-09 | 华南理工大学 | Human body based on deformation pattern and face net template approximating method |
| CN109655011A (en) * | 2018-12-13 | 2019-04-19 | 北京健康有益科技有限公司 | A kind of method and system of Human Modeling dimension measurement |
| CN110123331A (en) * | 2019-04-17 | 2019-08-16 | 平安科技（深圳）有限公司 | Human body body and constitution collecting method, device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112401369A (en) | 2021-02-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112401369B (en) | Body parameter measurement method, system, device, chip and medium based on human body reconstruction | |
| US10679046B1 (en) | Machine learning systems and methods of estimating body shape from images | |
| Zheng et al. | Hybridfusion: Real-time performance capture using a single depth sensor and sparse imus | |
| CN109461180B (en) | Three-dimensional scene reconstruction method based on deep learning | |
| US11210804B2 (en) | Methods, devices and computer program products for global bundle adjustment of 3D images | |
| US9235928B2 (en) | 3D body modeling, from a single or multiple 3D cameras, in the presence of motion | |
| CN108154550B (en) | RGBD camera-based real-time three-dimensional face reconstruction method | |
| CN109074660B (en) | Method and system for real-time 3D capture and instant feedback from a monocular camera | |
| CN103733227B (en) | Three-dimensional object modelling fitting & tracking | |
| Stoll et al. | Fast articulated motion tracking using a sums of gaussians body model | |
| US11928778B2 (en) | Method for human body model reconstruction and reconstruction system | |
| CN104992441B (en) | A kind of real human body three-dimensional modeling method towards individualized virtual fitting | |
| CN111862299B (en) | Human body three-dimensional model construction method, device, robot and storage medium | |
| US20170278302A1 (en) | Method and device for registering an image to a model | |
| CN116309880A (en) | Object pose determining method, device, equipment and medium based on three-dimensional reconstruction | |
| US20170287162A1 (en) | Method and system for scanning an object using an rgb-d sensor | |
| WO2022240745A1 (en) | Methods and systems for representing a user | |
| CN108010122B (en) | Method and system for reconstructing and measuring three-dimensional model of human body | |
| EP2800055A1 (en) | Method and system for generating a 3D model | |
| CN114663983A (en) | Mesh topology acquisition method, device, electronic device and storage medium | |
| US20230215096A1 (en) | System and method for mapping the skin | |
| Cordea et al. | 3D head pose recovery for interactive virtual reality avatars | |
| CN118096890B (en) | 6D pose real-time estimation method and device for out-of-cabin astronauts | |
| CN116958450B (en) | A three-dimensional human body reconstruction method based on two-dimensional data | |
| US20240412452A1 (en) | Systems and methods for 3d human model estimation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |