US20170053422A1 - Mobile device human body scanning and 3d model creation and analysis - Google Patents

Mobile device human body scanning and 3d model creation and analysis Download PDF

Info

Publication number
US20170053422A1
US20170053422A1 US15/237,850 US201615237850A US2017053422A1 US 20170053422 A1 US20170053422 A1 US 20170053422A1 US 201615237850 A US201615237850 A US 201615237850A US 2017053422 A1 US2017053422 A1 US 2017053422A1
Authority
US
United States
Prior art keywords
user
image
camera
images
model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US15/237,850
Inventor
Fabien CHOJNOWSKI
Yuri Peter KIZIMOVICH
Frederic Tachet
Remy BLANCHARD
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Individual
Original Assignee
Individual
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Individual filed Critical Individual
Priority to US15/237,850 priority Critical patent/US20170053422A1/en
Publication of US20170053422A1 publication Critical patent/US20170053422A1/en
Priority to US16/442,823 priority patent/US10460517B2/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/261Image signal generators with monoscopic-to-stereoscopic image conversion
    • G06T7/602
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/10Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/20Finite element generation, e.g. wire-frame surface description, tesselation
    • G06T7/0065
    • G06T7/0079
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/50Depth or shape recovery
    • G06T7/55Depth or shape recovery from multiple images
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/60Analysis of geometric attributes
    • G06T7/62Analysis of geometric attributes of area, perimeter, diameter or volume
    • H04N13/0055
    • H04N13/0221
    • H04N13/026
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/189Recording image signals; Reproducing recorded image signals
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00Indexing scheme for image data processing or generation, in general
    • G06T2200/08Indexing scheme for image data processing or generation, in general involving all processing steps from image acquisition to 3D model generation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20092Interactive image processing based on input by user
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/204Image signal generators using stereoscopic image cameras
    • H04N13/207Image signal generators using stereoscopic image cameras using a single 2D image sensor
    • H04N13/221Image signal generators using stereoscopic image cameras using a single 2D image sensor using the relative movement between cameras and objects

Definitions

  • the present invention is in the field of creating and analyzing 3D models of the human body with mobile devices from 2D images captured by device's single camera.
  • 3D cameras as disclosed in the current state of the art, are not required by the present invention.
  • a 2D camera is different from a 3D camera as a 3D camera consists of either: (1) two 2D cameras and special hardware to synchronize the images obtained via the 2D cameras; (2) one 2D camera plus infrared sensors; or (3) other technology comprising an array of sensors.
  • 3D cameras use both image capture and creation of a depth map of the image.
  • the present invention does not require a depth map of the image.
  • 3D cameras are not included in the mass market available for mobile devices, except in some niche markets, so most mobile devices only comprise an included 2D camera.
  • the mass market available to most mobile device does not currently provide a solution to scan/image a human body using a 2D camera to create and analyze a human 3D body digital model, as well display it in 3D and share the 3D object in a web-enabled 3D format, providing an ability to access it with full 3D visual and metric information from any device, anytime, and anywhere.
  • the present invention addresses this problem.
  • the invention uses any 2D camera for the purpose of performing 3D full body self-scanning and body metrics analysis.
  • a mobile device camera may take 2 or more, or 3 or more, 2D images, while the mobile device's accelerometer is used to determine the angular camera position.
  • This information is shared with a CPU processor (cloud, server, etc.) to form an accurate 3D body model of a user.
  • These steps may all be performed in a non-controlled environment (for an example home conditions) and in self-mode (i.e., performed by the user alone).
  • the method further comprises performing an automatic segmentation of the 2D images to create a 3D model reconstruction, which can be further adjusted by the user.
  • the 3D model is shared via cloud and social media, and it is also used to assist in shopping while ensuring accurate measurements for the particular user.
  • the digital model of one or more products designed for the target consumer body may be automatically adjusted and shown as a 3D image on the target consumer's 3D body image.
  • the 3D model may be directly shared with businesses for traditional or additive manufacturing (e.g., 3D printing) of the made-to-measure clothing or any special medical/sport product using 3D morphology and precise measurements of the consumer's body or particular body parts (legs, arms, etc.).
  • the method for creating one or more 3D images of a user comprises placing a 2D camera on a planar surface, capturing a first 2D image of the user, capturing a second 2D image image of the user, the second 2D image being a different perspective than the first image, the user remaining on a single axis in both images, segmenting the first and second images to form a plurality of segmented 2D images, constructing a 30 image from a combination of all three of the plurality of segmented images, data extracted from that plurality of images, and information on camera position and angle, however slight.
  • the method further comprises the step of manually adjusting the data extracted from the plurality of segmented 2D images, wherein the manual adjusting is performed by the user.
  • the method further comprises the step of automatically extracting measurements of the user's body from the 3D image constructed.
  • the method further comprises the step of adding one or more desired measurements for user height based on the 3D image.
  • the method further comprises the step of adding one or more desired measurements for user girth based on the 3D image.
  • the method further comprises the step of adding one or more desired measurements for a geodesic distance between two or more points on the user's 3D image.
  • 2 or more 3D images are constructed, and the method further comprises the step of saving each 3D image such that the user may track one or more bodily features over time.
  • the 3D image is compared to an ideal 3D model.
  • the method further comprises the step of sending measurements to an authorized recipient.
  • the method further comprises the step of tailoring an outfit for the user based on measurements sent to an authorized recipient.
  • the method further comprises the step of comparing the extracted measurements to known fitment data of a desired product.
  • the method further comprises the step of suggesting a product based on information retrieved from a comparing step.
  • the geodesic distance is the distance between the ground and a point on the circumference of the user's body.
  • the method further comprises the step of measuring, using a virtual measurement tape, a distance between two points on the 3D image.
  • the method further comprises the step of measuring a circumference of the user's body.
  • the method further comprises the step of controlling a camera position using an accelerometer coupled to the camera.
  • the method further comprises the step of generating feedback using an accelerometer coupled to the camera, the feedback correcting the user's positioning.
  • the computed camera position is obtained via an accelerometer coupled to the camera, the computed camera position comprising the camera's sensor plane position relative to the user.
  • the method further comprises measuring one or more dimensions comprising depth.
  • the method further comprises the step of creating a real time transformation of the user's body, said real time transformation being designed exactly once per categorical body type.
  • the real time transformation may then be used without further input to test varying garments or other wearable products on the user's particular body in a virtual manner.
  • FIG. 1 shows an example of a 3D model created by the method of the present invention.
  • FIG. 2 shows an example of a view due to manipulation and rotation of a 3D model.
  • FIG. 3 shows an example of a user's measurement extracted from a 3D model. This particular example shows a specific circumference of a user's body being measured, as well as a distance being measured from the ground to the particular circumference of a user's body.
  • FIG. 4 shows an example of a user interface with exemplary extracted measurement information.
  • FIG. 5 shows an example of a history of 3D models for a user to compare.
  • FIG. 6 is a flowchart demonstrating one method of 3D model creation according to the present invention.
  • FIG. 7 is a flowchart demonstrating one method for measurement extraction from a 3D model according to the present invention.
  • FIG. 8 is a flowchart demonstrating an exemplary connection of a mobile device with social media or a cloud server according to the present invention.
  • FIG. 9 is a flowchart demonstrating exemplary storage features for 3D models according to the present invention.
  • FIG. 10 is a flowchart demonstrating another exemplary 3D model creation and measurement extraction according to the present invention.
  • FIG. 11 is a flowchart demonstrating an exemplary 3D model creation and construction according to the present invention.
  • FIG. 12 shows an example of a frontal image of a 3D image created by the present invention.
  • FIG. 13 shows an example of an image of a user comprising a piece of clothing on the user's body and a virtual representation of how that piece of clothing would look on the user.
  • FIG. 14 shows an example of how extremity points are obtained in order to be used for 3D mapping with known camera parameters and device accelerometer data, using a subject's 2D front and side images, according to the present invention.
  • FIG. 15 shows an example of collecting training points for computation of the mapping function, F m , using a subject's 2D front and side images, according to the present invention.
  • the invention comprises a method which uses a single 2D camera, for example, a mobile device's camera, to take multiple (i.e., two or more) 2D pictures and make a 3D body model.
  • the method requires 2 images of a body, from which the method creates a 3D model, and without requiring a depth measurement.
  • the camera is ideally built-in to the mobile device. In one embodiment the camera is external to the mobile device and connected via a wired or wireless connection.
  • the mobile device is ideally a smartphone, PDA, tablet, or other small mobile device, but it could also be a laptop, desktop computer, or other device equipped with a camera.
  • invention may further comprise using a built-in accelerometer providing angles for camera positioning.
  • the method comprises taking three or more 2D pictures to form a 3D body model.
  • the subject is positioned in two poses (manners) in front of the camera in an uncontrolled (e.g., home) environment: the first pose may be, e.g., a front image, and the second pose may be, e.g,. a side image (profile).
  • the subject may be positioned in a third (i.e., the other side, left or right) position.
  • the subject may further turn from a first pose to a second pose via a conventional body rotation around a single axis, such as a vertical axis.
  • a sequence of the two images is captured for construction of a 3D model. Only two positions and two pictures are generally needed for construction of a 3D model.
  • the camera may be programmed with an application and/or timer which provides the subject with enough time to change positions.
  • any conventional 2D camera may be used from any mass market device, and usually the camera is incorporated into a mass market mobile device.
  • the 2D images are captured in any uncontrolled environment (basic lighting conditions, no special background or other special conditions for image taking environment).
  • the device may be, e.g. placed on a plane surface and the subject stands in front of the camera. No special conditions or extra equipment, calibration landmarks, or other information, equipment, or technology is needed.
  • the image capture can be performed in a home environment with any mobile device and personally by the subject/user. During image capture, the user is ideally wearing tight, form-fitting clothing, underwear, or no clothing, so as to facilitate the best possible measurements and creation of the most accurate 3D model.
  • a 3D camera is not required for formation of a 3D image.
  • a 2D camera is different and more simple than a 3D camera, as a 3D camera consists of either: (1) two 2D cameras and special hardware to synchronize the images; (2) one 2D camera plus infrared sensors; or (3) some other technology as an array of sensors, 3D cameras use both image capture and creation of a depth map of the image.
  • the present invention does not require a depth map of the image.
  • 3D cameras are not included in the mass market available for mobile devices, except in some niche markets, so most mobile devices only have an included 2D camera.
  • the present invention allows for creation of 3D images using 20 cameras, wherein the camera is required to take only two 2D images.
  • the method may further comprise obtaining a depth map of the pictures taken, which would allow the method to be applicable to devices comprising 3D cameras. The depth map would allow for further initial image processing.
  • the present invention may further use an accelerometer incorporated in the mobile device to control camera position and generate feedback if necessary to correct user positioning.
  • the present invention may also use an accelerometer incorporated in the mobile device (or otherwise coupled to the camera) to compute a camera's sensor plane position relative to the subject/user in a 3D coordinate system to allow for a full 3D reconstruction including a calculation of the depth.
  • the mobile device in the present invention may also compute camera calibration, including the exact spatial position of the camera, using a user's height as a parameter in combination with the obtained accelerometer data.
  • the present invention then also obtains 3D coordinates of the set of body feature points using the camera calibration data.
  • the method of the present invention can obtain and tune a mathematical model of the 3D free form deformation of a standard human body 3D explicit model.
  • a 3D free form deformation is obtained from pre-calculated and stored data which is adapted to the current body using a special set of universal 3D shape functions.
  • the method of the present invention doesn't require any databases of human shape data and/or any related statistical data.
  • a single explicit 3D description of one example for a male and one example for a female is all that is required.
  • the present invention uses a 3D body explicit shape data super compression technique to work with a data size of 50-100 kB for the full 3D body image and for communication between a client device and a cloud/server for storage.
  • the mobile device can be in communication with a cloud storage server via a wireless or wired network connection.
  • a cloud-based computational facility can be established to accelerate computations necessary to solve minimization problems related to 3D shape recovery, which allows for use of the method with a wide spectrum of mobile devices without requiring a high device processing power.
  • the present invention may further include providing an interface for interactive feedback from the user. The user can use this interface in order to more tightly fit the model to their 2D images, as well as interact with an application which has social media, shopping, history, and other features. The user can tailor feedback to fit the images to their body for greater precision of the 3D model reconstruction and has the ability to use images taken without special provisions for conditions such as background contrast, lighting, etc.
  • the present invention allows for 3D image capture anytime and anywhere, via use of a 2D camera; the ability to capture, store, and automatically calculate body metrics, which may be displayed and labelled in a 3D image created from 2D images; the ability to perform additional body 3D analysis; and the ability to access and share 3D digital models of the user's human body as a customer when shopping.
  • the present invention further allows makers of clothing or medical/sport wearable products to access customer body metrical and morphological data, customize products, and improve products, and gather feedback on the product, all potentially before the product is even manufactured.
  • the present invention further stores and tracks a history of 3D models of a user's body. This means a user is able to follow changes in shape, size, and other body metrics, using the history of 3D models.
  • This feature has applications for fitness and health tracking, tracking growth, and so on.
  • the present invention allows a user to share their 3D models with other users and via social media.
  • the present invention also allows a user to compare their model to an ideal model. This feature can be used by a user who has an ideal body shape they are attempting to achieve and allows the user to track progress while they move toward that ideal shape.
  • the present invention also allows for the user to zoom in and out of their 3D model and to rotate their 3D model as they desire.
  • the present invention allows automatic computations of the body measurements from the 3D model Users may view and interrogate, in 3D, any measurement taken from the computed 3D digital body using virtual measurement tape.
  • any 3D body measurement presented as a body circumference, length between two circumferences, length between a circumference and the floor, or a geodesic distance between 2 or more points selected by the user on the 3D digital body may be computed and stored in real time. These measurements can then be used to determine a clothing fit or size. This feature allows a user to use a digital 3D model to facilitate purchase or alteration of clothing.
  • the present invention provides, if desired, sizing information for multiple clothing sizing system, international and otherwise, in combination with the display of the 3D image of the customer body on personal devices or any device connected to the Internet.
  • the present invention can be used with any manufacturer's sizing or measurement information in order to determine if a desired item will fit and whether the fit will be tight, loose, etc.
  • the present invention also provides a fit index computation and may display the index computation on the 3D image of the customer body.
  • the present invention further allows the 3D model to be directly shared with businesses for traditional or additive manufacturing (3D printing) of custom made clothing or any special medical/spurt product using 3D morphology and requiring precise measurements of the customer body or body parts (such as legs, arms, etc.).
  • the present invention further allows for the creation of an implicit model of the subject's body along with an explicit body surface presentation, based on a set of special shape functions.
  • a model may be used for real time transformation of 3D models of products (e.g., clothes, special medical and sports wearables) in order to fit a particular subject's body.
  • the product may be tested and/or custom-designed to fit to the particular user's body by combining information on a categorical body shape and information from the 3D image obtained via the method described above, in order to instantaneously and virtually examine the product on the user for style, fit, etc.
  • a 3D image and correspondent mapping function is obtained via the method described above, which further allows for the creation of a real time transformation of 3D models of varying products (e.g., clothes, special medical, sports wearables etc.) in order to fit a particular subject's body.
  • the virtual product (which is designed once for a categorical (i.e., particular) body) may be tested and custom-designed to fit to the particular user's body by combining information on the product designed for a categorical body shape and information from the 3D image obtained via the method described above.
  • the product may be instantaneously and virtually examined as a 3D image (3D product on the 3D user) for style, fit, etc.
  • FIG. 1 shows an example of a 3D image which is constructed according to the method of present invention.
  • FIG. 2 shows an example of how a user may manipulate and rotate a 3D model created by the method of the present invention.
  • FIG. 2 shows an angular view from a point above the subject's image.
  • the present invention allows for a user to rotate and view the 3D image in any manner in order to see particularly desired aspects of their body.
  • FIG. 3 shows an example of a user's measurement extracted from a 3D model. This particular example shows a specific circumference of a user's body being measured, as well as a distance being measured from the ground to the particular circumference of a user's body. This figure further shows yet another view of the same body as FIG. 2 , rotated as desired by the user.
  • FIG. 4 shows an example of a user interface according to the present invention, with exemplary extracted measurement information.
  • the information depicted comprises a size range, a size category (small, or “s”), choice of country for correct reference, and type of clothing.
  • the options displayed correspond to a female, but male options are similarly available.
  • FIG. 5 shows an example of a history of 3D models for a user to compare.
  • a woman is depicted in two images. In the first image, the woman is imaged as prior to/after pregnancy, while in the second image, the same woman is imaged as pregnant. Such images were obtained at different times but both are saved by the invention such that comparisons may be made whenever desired by the user in order to track progress.
  • FIG. 6 is a flowchart demonstrating one method of 3D model creation according to the present invention.
  • a user first initializes the 3D model creation (e.g., by opening/starting the program) 61 . Then, the user places the mobile device/camera on a planar surface 62 . A frontal 2D image is captured 63 , then the user rotates on a single axis 64 , and when the user reaches a side view (relative to the camera), a side 2D image is captured by the camera 65 . Once both images are captured, a 3D model is generated from the two images 66 . The user may also further customize the created 3D image to adjust and more accurately represent the dimensions of the user 67 .
  • FIG. 7 is a flowchart demonstrating examples of measurement extraction from a 3D model according to the present invention, and particular applications for the extracted information.
  • a user may select particular measurements to be extracted from the 3D body image/model, or alternatively, body measurements are automatically extracted by the program 71 . From this point, the user may send measurements to an authorized recipient 72 , and the measurements may be used for tailoring or fitting a particularly desired product without requiring the user to be physically present for fitting 73 .
  • the user's measurements may be compared to known fitment data of particularly desired products 74 , after which, based on the comparisons performed, size suggestions and fitment matches are notated or brought to the user's attention 75 .
  • FIG. 8 is a flowchart demonstrating an exemplary connection of a mobile device with social media or a cloud server according to the present invention.
  • the mobile device 81 is connected to a network 82 , which in turn is connected to a social media server or cloud server 83 .
  • FIG. 9 is a flowchart demonstrating exemplary storage features for 3D models according to the present invention.
  • Examples of applications comprising the ability to store various 3D models of a user's body through time 91 include but are not limited to allowing the user to view a history of 3D models to track progress 92 , and allowing the user to compare 3D models of the user's body relative to an ideal 3D model 93 .
  • FIG. 10 is a flowchart demonstrating another example of 3D model creation and measurement extraction according to the present invention.
  • a user initializes the 3D model creation 101 .
  • the camera is placed on a planar surface 102 .
  • a frontal 2D image is captured first 103 , the user rotates on a single axis 104 until the camera can capture a side 2D image 105 .
  • the front and side images are automatically segmented 106 .
  • the user may adjust segmentation information on the 2D images for better accuracy 107 , especially if the initial two images are insufficient to accurately represent the user in 3D.
  • a 3D model is reconstructed using the 2D image segmentation information and the computed camera position 108 .
  • Body measurements are then automatically extracted 109 , and the user may add desired measurements for height, girth, and/or geodesic distance between two or more body points in the interactive 3D model 110 .
  • FIG. 11 is a flowchart demonstrating another example of 3D model creation and construction according to the present invention.
  • the user initializes the 3D model creation 111 .
  • the camera is placed on a planar surface 112 .
  • a frontal 2D image is captured first 113 , the user rotates on a single axis 114 until the camera can capture a side 2D image 115 .
  • the front and side images are automatically segmented 116 .
  • the user may adjust segmentation information on the 2D images for better accuracy 117 , especially if the initial two images are insufficient to accurately represent the user in 3D.
  • a 3D model is reconstructed using the 2D image segmentation information and the computed camera position 118 .
  • FIG. 12 shows an example of a frontal image of a 3D image created by the present invention.
  • This virtual product with or without a garment, is made possible by the present invention, and each such image is required to be designed exactly and only once for a categorical body.
  • the 3D image created may be tested or further customized to fit the particular subject's body by combining information on a product (e.g., a garment) designed for a categorical body shape with information from the 3D image of the subject (obtained as described herein). Thereafter, the product can be instantaneously and virtually examined on the subject for style. Alternatively, the user may similarly examine the subject body without a garment (e.g., for fitness purposes).
  • FIG. 12 shows a 3D image of a subject without a garment
  • FIG. 13 shows an example of a 3D image of a digital model of a garment on the same user's body, i.e. a virtual and instantaneous representation of how that piece of clothing would look on the user.
  • the garment is designed for a categorical body shape and reflects specific designs intended, such as style, body lay out, etc.
  • the user instantly sees a 3D image of the garment on her body while all original design intentions and proportions are preserved in the image, making it optimal and realistic. This is due to a revoked mapping between the user's body information obtained and stored and the categorical body information used and stored for design of the garment.
  • FIG. 14 shows an example of how extremity points are obtained in order to be used for 3D mapping with known camera parameters and device accelerometer data, using a subject's 2D front and side images, according to the present invention. Such extremity points may be obtained in order to further customize the user's body, as compared to categorical body information.
  • FIG. 15 shows an example of collecting training points for computation of the mapping function, F m , using a subject's 2D front and side images, according to the present invention.
  • Such training points may be obtained in order to further customize the user's body, as compared to categorical body information.
  • the mapping function is computed exactly and only once per categorical body type. The unique body type of each user is entered/obtained once, after which the user may try on any clothing/garments to determine how the clothing/garments look on the user in a virtual setting. The only requirement is that the dimensions of the clothing are known or suggested. Such dimensions may be entered by the designers of the clothing/garments and are generally easily obtainable.
  • a user may try on any garment on which information is available, and designers of garments and other wearable products may receive information from the users of the method for additional feedback (e.g., transforming garments to fit different categorical bodies, determining whether clothing designed for a particular body type may also look well on other body types previously not considered, etc.).
  • additional feedback e.g., transforming garments to fit different categorical bodies, determining whether clothing designed for a particular body type may also look well on other body types previously not considered, etc.
  • Example 1 A 3D image of a subject/user is obtained via the following steps:
  • Step 1 The subject uses a mobile device (e.g., smartphone) to make a self-scan in uncontrolled conditions using an application on the mobile device.
  • a mobile device e.g., smartphone
  • Step 2 The mobile device is placed on any plane surface, preferably a flat surface.
  • Step 3 The subject stands between 2 and 2.5 meters away from the mobile device and facing the front of the camera located on the mobile device. If the surface on which the mobile device is placed is not exactly flat, the interface of application located on the mobile device may assist the user to stand approximately parallel to the mobile device's surface. The interface may also assist the subject to be centered for an optimal view and image capture.
  • the camera captures a first 2D image of the subject facing the camera.
  • Step 4 The subject turns clockwise or counter-clockwise until standing in profile view relative to the camera.
  • the camera then captures a second 2D image of either the left or right profile image of the subject.
  • Step 5 The subject exits the scene captured by the camera, and the camera captures a background image without the subject in the image.
  • the mobile device application runs one or more image segmentation algorithms, which extract(s) extremity feature points 60 from the front and side images of the subject (top, bottom). See FIG. 14 .
  • the image segmentation algorithms further obtain the camera's angular positions using an accelerometer of the mobile device.
  • the application runs another set of algorithms/functions which allow for the computation of a sensed camera position and a map between image coordinates corresponding to the extremity points (x, y, z) 60 in a world coordinate system.
  • Step 7 Further image segmentation allows for segmentation of the front and side silhouettes of the subject, and a map between image coordinates can be constructed as described in Step 6 above, wherein the image coordinates now correspond to boundary points 70 of both silhouettes. See FIG. 15 . More points 70 can be included in the set of images (both along the boundary of as well as within the silhouettes), using machine learning on the data base of information from a user's models, wherein relative spatial positions for such points 70 are continuously being analyzed and determined.
  • Step 8 A model body (test model) with an explicit surface boundary is used.
  • the model body is represented by a polygon mesh for both genders.
  • X1, y1, and z1 coordinate values are obtained for all topological corresponding points using the information from Step 6 and Step 7.
  • Step 9 A mapping function, F m , is introduced.
  • the mapping function, F m maps the surface of the test model (the model body) to the subject's body model as a linear combination of the 3D shape functions obtained in Steps 6-8.
  • Step 10 The subject's 3D model is represented by a final function, F f , which transforms the test model surface polygon mesh into a subject's body model surface polygon mesh.
  • F f a final function
  • the latter can be created on demand and based on multiple purposes. Among these are: 3D rendering, body metric analysis, product fit computations, etc.
  • example or “exemplary” are used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, use of the words “example” or “exemplary” is intended to present concepts in a concrete fashion.
  • the term “or” is intended to mean an inclusive “or” rather than an exclusive “or”. That is, unless specified otherwise, or clear from context, “X employs A or B” is intended to mean any of the natural inclusive permutations.

Abstract

A 2D camera is used to create a 3D full body image. A camera takes 3 or more 2D images, an accelerometer is used to calculate camera position, and a CPU is employed to construct a 3D body model. This may be performed in a non-controlled environment and by the user alone. An automatic segmentation of the 2D images creates special information for 3D model reconstruction. Once the 3D model measurements are extracted, the user has the option to further specify measurements. In one embodiment, the 3D model is shared via cloud and social media, and also used to assist in shopping while ensuring accurate measurements for the user. In another embodiment, the digital model of products designed for the target consumer body is automatically adjusted and shown as a 3D image on the user's body. The 3D model may be shared with businesses for manufacturing using 3D morphology.

Description

    CROSS REFERENCE TO RELATED APPLICATIONS
  • The present invention claims priority to and incorporates fully by reference Provisional Patent Application Ser. No. 62/205,832, filed Aug. 17, 2015.
  • FIELD OF THE INVENTION
  • The present invention is in the field of creating and analyzing 3D models of the human body with mobile devices from 2D images captured by device's single camera.
  • BACKGROUND OF THE INVENTION
  • 3D cameras, as disclosed in the current state of the art, are not required by the present invention. A 2D camera is different from a 3D camera as a 3D camera consists of either: (1) two 2D cameras and special hardware to synchronize the images obtained via the 2D cameras; (2) one 2D camera plus infrared sensors; or (3) other technology comprising an array of sensors. 3D cameras use both image capture and creation of a depth map of the image. The present invention does not require a depth map of the image. 3D cameras are not included in the mass market available for mobile devices, except in some niche markets, so most mobile devices only comprise an included 2D camera.
  • The mass market available to most mobile device does not currently provide a solution to scan/image a human body using a 2D camera to create and analyze a human 3D body digital model, as well display it in 3D and share the 3D object in a web-enabled 3D format, providing an ability to access it with full 3D visual and metric information from any device, anytime, and anywhere. The present invention addresses this problem.
  • SUMMARY OF THE INVENTION
  • The invention uses any 2D camera for the purpose of performing 3D full body self-scanning and body metrics analysis. A mobile device camera, for example, may take 2 or more, or 3 or more, 2D images, while the mobile device's accelerometer is used to determine the angular camera position. This information is shared with a CPU processor (cloud, server, etc.) to form an accurate 3D body model of a user. These steps may all be performed in a non-controlled environment (for an example home conditions) and in self-mode (i.e., performed by the user alone). The method further comprises performing an automatic segmentation of the 2D images to create a 3D model reconstruction, which can be further adjusted by the user. Once the 3D model is created, measurements are automatically extracted and the user has an option to further select or specify measurements to be extracted. In one embodiment, the 3D model is shared via cloud and social media, and it is also used to assist in shopping while ensuring accurate measurements for the particular user. In another embodiment, the digital model of one or more products designed for the target consumer body may be automatically adjusted and shown as a 3D image on the target consumer's 3D body image. Optionally the 3D model may be directly shared with businesses for traditional or additive manufacturing (e.g., 3D printing) of the made-to-measure clothing or any special medical/sport product using 3D morphology and precise measurements of the consumer's body or particular body parts (legs, arms, etc.).
  • In some aspects, the method for creating one or more 3D images of a user comprises placing a 2D camera on a planar surface, capturing a first 2D image of the user, capturing a second 2D image image of the user, the second 2D image being a different perspective than the first image, the user remaining on a single axis in both images, segmenting the first and second images to form a plurality of segmented 2D images, constructing a 30 image from a combination of all three of the plurality of segmented images, data extracted from that plurality of images, and information on camera position and angle, however slight.
  • In some aspects, the method further comprises the step of manually adjusting the data extracted from the plurality of segmented 2D images, wherein the manual adjusting is performed by the user.
  • In some aspects, the method further comprises the step of automatically extracting measurements of the user's body from the 3D image constructed.
  • In some aspects, the method further comprises the step of adding one or more desired measurements for user height based on the 3D image.
  • In some aspects, the method further comprises the step of adding one or more desired measurements for user girth based on the 3D image.
  • In some aspects, the method further comprises the step of adding one or more desired measurements for a geodesic distance between two or more points on the user's 3D image.
  • In some aspects, 2 or more 3D images are constructed, and the method further comprises the step of saving each 3D image such that the user may track one or more bodily features over time.
  • In some aspects, the 3D image is compared to an ideal 3D model.
  • In some aspects, the method further comprises the step of sending measurements to an authorized recipient.
  • In some aspects, the method further comprises the step of tailoring an outfit for the user based on measurements sent to an authorized recipient.
  • In some aspects, the method further comprises the step of comparing the extracted measurements to known fitment data of a desired product.
  • In some aspects, the method further comprises the step of suggesting a product based on information retrieved from a comparing step.
  • In some aspects, the geodesic distance is the distance between the ground and a point on the circumference of the user's body.
  • In some aspects, the method further comprises the step of measuring, using a virtual measurement tape, a distance between two points on the 3D image.
  • In some aspects, the method further comprises the step of measuring a circumference of the user's body.
  • In some aspects, the method further comprises the step of controlling a camera position using an accelerometer coupled to the camera.
  • In some aspects, the method further comprises the step of generating feedback using an accelerometer coupled to the camera, the feedback correcting the user's positioning.
  • In some aspects, the computed camera position is obtained via an accelerometer coupled to the camera, the computed camera position comprising the camera's sensor plane position relative to the user.
  • In some aspects, the method further comprises measuring one or more dimensions comprising depth.
  • In some aspects, the method further comprises the step of creating a real time transformation of the user's body, said real time transformation being designed exactly once per categorical body type. The real time transformation may then be used without further input to test varying garments or other wearable products on the user's particular body in a virtual manner.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • The present invention will now be discussed in further detail below with reference to the accompanying figures in which:
  • FIG. 1 shows an example of a 3D model created by the method of the present invention.
  • FIG. 2 shows an example of a view due to manipulation and rotation of a 3D model.
  • FIG. 3 shows an example of a user's measurement extracted from a 3D model. This particular example shows a specific circumference of a user's body being measured, as well as a distance being measured from the ground to the particular circumference of a user's body.
  • FIG. 4 shows an example of a user interface with exemplary extracted measurement information.
  • FIG. 5 shows an example of a history of 3D models for a user to compare.
  • FIG. 6 is a flowchart demonstrating one method of 3D model creation according to the present invention.
  • FIG. 7 is a flowchart demonstrating one method for measurement extraction from a 3D model according to the present invention.
  • FIG. 8 is a flowchart demonstrating an exemplary connection of a mobile device with social media or a cloud server according to the present invention.
  • FIG. 9 is a flowchart demonstrating exemplary storage features for 3D models according to the present invention.
  • FIG. 10 is a flowchart demonstrating another exemplary 3D model creation and measurement extraction according to the present invention.
  • FIG. 11 is a flowchart demonstrating an exemplary 3D model creation and construction according to the present invention.
  • FIG. 12 shows an example of a frontal image of a 3D image created by the present invention.
  • FIG. 13 shows an example of an image of a user comprising a piece of clothing on the user's body and a virtual representation of how that piece of clothing would look on the user.
  • FIG. 14 shows an example of how extremity points are obtained in order to be used for 3D mapping with known camera parameters and device accelerometer data, using a subject's 2D front and side images, according to the present invention.
  • FIG. 15 shows an example of collecting training points for computation of the mapping function, Fm, using a subject's 2D front and side images, according to the present invention.
  • DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT
  • The invention comprises a method which uses a single 2D camera, for example, a mobile device's camera, to take multiple (i.e., two or more) 2D pictures and make a 3D body model. The method requires 2 images of a body, from which the method creates a 3D model, and without requiring a depth measurement. The camera is ideally built-in to the mobile device. In one embodiment the camera is external to the mobile device and connected via a wired or wireless connection. The mobile device is ideally a smartphone, PDA, tablet, or other small mobile device, but it could also be a laptop, desktop computer, or other device equipped with a camera. For higher accuracy of the 3D reconstruction, invention may further comprise using a built-in accelerometer providing angles for camera positioning. In some embodiments, the method comprises taking three or more 2D pictures to form a 3D body model.
  • During initial 2D image capture, the subject is positioned in two poses (manners) in front of the camera in an uncontrolled (e.g., home) environment: the first pose may be, e.g., a front image, and the second pose may be, e.g,. a side image (profile). In the case of substantial limb symmetry, the subject may be positioned in a third (i.e., the other side, left or right) position. The subject may further turn from a first pose to a second pose via a conventional body rotation around a single axis, such as a vertical axis. A sequence of the two images is captured for construction of a 3D model. Only two positions and two pictures are generally needed for construction of a 3D model. The camera may be programmed with an application and/or timer which provides the subject with enough time to change positions. The key is that any conventional 2D camera may be used from any mass market device, and usually the camera is incorporated into a mass market mobile device. The 2D images are captured in any uncontrolled environment (basic lighting conditions, no special background or other special conditions for image taking environment). The device may be, e.g. placed on a plane surface and the subject stands in front of the camera. No special conditions or extra equipment, calibration landmarks, or other information, equipment, or technology is needed. The image capture can be performed in a home environment with any mobile device and personally by the subject/user. During image capture, the user is ideally wearing tight, form-fitting clothing, underwear, or no clothing, so as to facilitate the best possible measurements and creation of the most accurate 3D model.
  • A 3D camera is not required for formation of a 3D image. A 2D camera is different and more simple than a 3D camera, as a 3D camera consists of either: (1) two 2D cameras and special hardware to synchronize the images; (2) one 2D camera plus infrared sensors; or (3) some other technology as an array of sensors, 3D cameras use both image capture and creation of a depth map of the image. The present invention does not require a depth map of the image. 3D cameras are not included in the mass market available for mobile devices, except in some niche markets, so most mobile devices only have an included 2D camera. The present invention allows for creation of 3D images using 20 cameras, wherein the camera is required to take only two 2D images. It should be noted that, in some embodiments, although not required, the method may further comprise obtaining a depth map of the pictures taken, which would allow the method to be applicable to devices comprising 3D cameras. The depth map would allow for further initial image processing.
  • The present invention may further use an accelerometer incorporated in the mobile device to control camera position and generate feedback if necessary to correct user positioning. The present invention may also use an accelerometer incorporated in the mobile device (or otherwise coupled to the camera) to compute a camera's sensor plane position relative to the subject/user in a 3D coordinate system to allow for a full 3D reconstruction including a calculation of the depth. The mobile device in the present invention may also compute camera calibration, including the exact spatial position of the camera, using a user's height as a parameter in combination with the obtained accelerometer data. The present invention then also obtains 3D coordinates of the set of body feature points using the camera calibration data.
  • The method of the present invention can obtain and tune a mathematical model of the 3D free form deformation of a standard human body 3D explicit model. Such a 3D free form deformation is obtained from pre-calculated and stored data which is adapted to the current body using a special set of universal 3D shape functions. The method of the present invention doesn't require any databases of human shape data and/or any related statistical data. A single explicit 3D description of one example for a male and one example for a female is all that is required. The present invention uses a 3D body explicit shape data super compression technique to work with a data size of 50-100 kB for the full 3D body image and for communication between a client device and a cloud/server for storage. The mobile device can be in communication with a cloud storage server via a wireless or wired network connection. A cloud-based computational facility can be established to accelerate computations necessary to solve minimization problems related to 3D shape recovery, which allows for use of the method with a wide spectrum of mobile devices without requiring a high device processing power. The present invention may further include providing an interface for interactive feedback from the user. The user can use this interface in order to more tightly fit the model to their 2D images, as well as interact with an application which has social media, shopping, history, and other features. The user can tailor feedback to fit the images to their body for greater precision of the 3D model reconstruction and has the ability to use images taken without special provisions for conditions such as background contrast, lighting, etc.
  • The present invention allows for 3D image capture anytime and anywhere, via use of a 2D camera; the ability to capture, store, and automatically calculate body metrics, which may be displayed and labelled in a 3D image created from 2D images; the ability to perform additional body 3D analysis; and the ability to access and share 3D digital models of the user's human body as a customer when shopping. The present invention further allows makers of clothing or medical/sport wearable products to access customer body metrical and morphological data, customize products, and improve products, and gather feedback on the product, all potentially before the product is even manufactured.
  • The present invention further stores and tracks a history of 3D models of a user's body. This means a user is able to follow changes in shape, size, and other body metrics, using the history of 3D models. This feature has applications for fitness and health tracking, tracking growth, and so on. The present invention allows a user to share their 3D models with other users and via social media. The present invention also allows a user to compare their model to an ideal model. This feature can be used by a user who has an ideal body shape they are attempting to achieve and allows the user to track progress while they move toward that ideal shape.
  • The present invention also allows for the user to zoom in and out of their 3D model and to rotate their 3D model as they desire.
  • The present invention allows automatic computations of the body measurements from the 3D model Users may view and interrogate, in 3D, any measurement taken from the computed 3D digital body using virtual measurement tape. In addition, any 3D body measurement presented as a body circumference, length between two circumferences, length between a circumference and the floor, or a geodesic distance between 2 or more points selected by the user on the 3D digital body, may be computed and stored in real time. These measurements can then be used to determine a clothing fit or size. This feature allows a user to use a digital 3D model to facilitate purchase or alteration of clothing. The present invention provides, if desired, sizing information for multiple clothing sizing system, international and otherwise, in combination with the display of the 3D image of the customer body on personal devices or any device connected to the Internet. The present invention can be used with any manufacturer's sizing or measurement information in order to determine if a desired item will fit and whether the fit will be tight, loose, etc. The present invention also provides a fit index computation and may display the index computation on the 3D image of the customer body.
  • The present invention further allows the 3D model to be directly shared with businesses for traditional or additive manufacturing (3D printing) of custom made clothing or any special medical/spurt product using 3D morphology and requiring precise measurements of the customer body or body parts (such as legs, arms, etc.).
  • The present invention further allows for the creation of an implicit model of the subject's body along with an explicit body surface presentation, based on a set of special shape functions. Such a model may be used for real time transformation of 3D models of products (e.g., clothes, special medical and sports wearables) in order to fit a particular subject's body. The product may be tested and/or custom-designed to fit to the particular user's body by combining information on a categorical body shape and information from the 3D image obtained via the method described above, in order to instantaneously and virtually examine the product on the user for style, fit, etc. A 3D image and correspondent mapping function is obtained via the method described above, which further allows for the creation of a real time transformation of 3D models of varying products (e.g., clothes, special medical, sports wearables etc.) in order to fit a particular subject's body. The virtual product (which is designed once for a categorical (i.e., particular) body) may be tested and custom-designed to fit to the particular user's body by combining information on the product designed for a categorical body shape and information from the 3D image obtained via the method described above. Thus, the product may be instantaneously and virtually examined as a 3D image (3D product on the 3D user) for style, fit, etc.
  • FIG. 1 shows an example of a 3D image which is constructed according to the method of present invention.
  • FIG. 2 shows an example of how a user may manipulate and rotate a 3D model created by the method of the present invention. FIG. 2 shows an angular view from a point above the subject's image. Essentially, the present invention allows for a user to rotate and view the 3D image in any manner in order to see particularly desired aspects of their body.
  • FIG. 3 shows an example of a user's measurement extracted from a 3D model. This particular example shows a specific circumference of a user's body being measured, as well as a distance being measured from the ground to the particular circumference of a user's body. This figure further shows yet another view of the same body as FIG. 2, rotated as desired by the user.
  • FIG. 4 shows an example of a user interface according to the present invention, with exemplary extracted measurement information. The information depicted comprises a size range, a size category (small, or “s”), choice of country for correct reference, and type of clothing. In this case the options displayed correspond to a female, but male options are similarly available.
  • FIG. 5 shows an example of a history of 3D models for a user to compare. In this particular example, a woman is depicted in two images. In the first image, the woman is imaged as prior to/after pregnancy, while in the second image, the same woman is imaged as pregnant. Such images were obtained at different times but both are saved by the invention such that comparisons may be made whenever desired by the user in order to track progress.
  • FIG. 6 is a flowchart demonstrating one method of 3D model creation according to the present invention. In the method according to FIG. 6, a user first initializes the 3D model creation (e.g., by opening/starting the program) 61. Then, the user places the mobile device/camera on a planar surface 62. A frontal 2D image is captured 63, then the user rotates on a single axis 64, and when the user reaches a side view (relative to the camera), a side 2D image is captured by the camera 65. Once both images are captured, a 3D model is generated from the two images 66. The user may also further customize the created 3D image to adjust and more accurately represent the dimensions of the user 67.
  • FIG. 7 is a flowchart demonstrating examples of measurement extraction from a 3D model according to the present invention, and particular applications for the extracted information. For example, via the interface, a user may select particular measurements to be extracted from the 3D body image/model, or alternatively, body measurements are automatically extracted by the program 71. From this point, the user may send measurements to an authorized recipient 72, and the measurements may be used for tailoring or fitting a particularly desired product without requiring the user to be physically present for fitting 73. Alternatively, the user's measurements may be compared to known fitment data of particularly desired products 74, after which, based on the comparisons performed, size suggestions and fitment matches are notated or brought to the user's attention 75.
  • FIG. 8 is a flowchart demonstrating an exemplary connection of a mobile device with social media or a cloud server according to the present invention. The mobile device 81 is connected to a network 82, which in turn is connected to a social media server or cloud server 83.
  • FIG. 9 is a flowchart demonstrating exemplary storage features for 3D models according to the present invention. Examples of applications comprising the ability to store various 3D models of a user's body through time 91 include but are not limited to allowing the user to view a history of 3D models to track progress 92, and allowing the user to compare 3D models of the user's body relative to an ideal 3D model 93.
  • FIG. 10 is a flowchart demonstrating another example of 3D model creation and measurement extraction according to the present invention. First, a user initializes the 3D model creation 101. Then, the camera is placed on a planar surface 102. A frontal 2D image is captured first 103, the user rotates on a single axis 104 until the camera can capture a side 2D image 105. Then, the front and side images are automatically segmented 106. The user may adjust segmentation information on the 2D images for better accuracy 107, especially if the initial two images are insufficient to accurately represent the user in 3D. Then, a 3D model is reconstructed using the 2D image segmentation information and the computed camera position 108. Body measurements are then automatically extracted 109, and the user may add desired measurements for height, girth, and/or geodesic distance between two or more body points in the interactive 3D model 110.
  • FIG. 11 is a flowchart demonstrating another example of 3D model creation and construction according to the present invention. In this example, the user initializes the 3D model creation 111. Then, the camera is placed on a planar surface 112. A frontal 2D image is captured first 113, the user rotates on a single axis 114 until the camera can capture a side 2D image 115. Then, the front and side images are automatically segmented 116. The user may adjust segmentation information on the 2D images for better accuracy 117, especially if the initial two images are insufficient to accurately represent the user in 3D. Then, a 3D model is reconstructed using the 2D image segmentation information and the computed camera position 118.
  • FIG. 12 shows an example of a frontal image of a 3D image created by the present invention. This virtual product, with or without a garment, is made possible by the present invention, and each such image is required to be designed exactly and only once for a categorical body. The 3D image created may be tested or further customized to fit the particular subject's body by combining information on a product (e.g., a garment) designed for a categorical body shape with information from the 3D image of the subject (obtained as described herein). Thereafter, the product can be instantaneously and virtually examined on the subject for style. Alternatively, the user may similarly examine the subject body without a garment (e.g., for fitness purposes).
  • While FIG. 12 shows a 3D image of a subject without a garment, FIG. 13 shows an example of a 3D image of a digital model of a garment on the same user's body, i.e. a virtual and instantaneous representation of how that piece of clothing would look on the user. The garment is designed for a categorical body shape and reflects specific designs intended, such as style, body lay out, etc. The user instantly sees a 3D image of the garment on her body while all original design intentions and proportions are preserved in the image, making it optimal and realistic. This is due to a revoked mapping between the user's body information obtained and stored and the categorical body information used and stored for design of the garment.
  • FIG. 14 shows an example of how extremity points are obtained in order to be used for 3D mapping with known camera parameters and device accelerometer data, using a subject's 2D front and side images, according to the present invention. Such extremity points may be obtained in order to further customize the user's body, as compared to categorical body information.
  • FIG. 15 shows an example of collecting training points for computation of the mapping function, Fm, using a subject's 2D front and side images, according to the present invention. Such training points may be obtained in order to further customize the user's body, as compared to categorical body information. It should be noted that the mapping function is computed exactly and only once per categorical body type. The unique body type of each user is entered/obtained once, after which the user may try on any clothing/garments to determine how the clothing/garments look on the user in a virtual setting. The only requirement is that the dimensions of the clothing are known or suggested. Such dimensions may be entered by the designers of the clothing/garments and are generally easily obtainable. Thus, a user may try on any garment on which information is available, and designers of garments and other wearable products may receive information from the users of the method for additional feedback (e.g., transforming garments to fit different categorical bodies, determining whether clothing designed for a particular body type may also look well on other body types previously not considered, etc.). The key feature which makes such functionality, analysis, and adaptability possible is the relative mapping functions obtained and employed by the method.
  • The following example provides one embodiment of the present invention:
  • Example 1. A 3D image of a subject/user is obtained via the following steps:
  • Step 1. The subject uses a mobile device (e.g., smartphone) to make a self-scan in uncontrolled conditions using an application on the mobile device.
  • Step 2. The mobile device is placed on any plane surface, preferably a flat surface.
  • Step 3. The subject stands between 2 and 2.5 meters away from the mobile device and facing the front of the camera located on the mobile device. If the surface on which the mobile device is placed is not exactly flat, the interface of application located on the mobile device may assist the user to stand approximately parallel to the mobile device's surface. The interface may also assist the subject to be centered for an optimal view and image capture. The camera captures a first 2D image of the subject facing the camera.
  • Step 4. The subject turns clockwise or counter-clockwise until standing in profile view relative to the camera. The camera then captures a second 2D image of either the left or right profile image of the subject.
  • Step 5. The subject exits the scene captured by the camera, and the camera captures a background image without the subject in the image.
  • Step 6. The mobile device application runs one or more image segmentation algorithms, which extract(s) extremity feature points 60 from the front and side images of the subject (top, bottom). See FIG. 14. The image segmentation algorithms further obtain the camera's angular positions using an accelerometer of the mobile device. Using the algorithmic data, the camera's optical data, the photo sensor parameters, and a reported height of the subject, the application runs another set of algorithms/functions which allow for the computation of a sensed camera position and a map between image coordinates corresponding to the extremity points (x, y, z) 60 in a world coordinate system.
  • Step 7. Further image segmentation allows for segmentation of the front and side silhouettes of the subject, and a map between image coordinates can be constructed as described in Step 6 above, wherein the image coordinates now correspond to boundary points 70 of both silhouettes. See FIG. 15. More points 70 can be included in the set of images (both along the boundary of as well as within the silhouettes), using machine learning on the data base of information from a user's models, wherein relative spatial positions for such points 70 are continuously being analyzed and determined.
  • Step 8. A model body (test model) with an explicit surface boundary is used. The model body is represented by a polygon mesh for both genders. X1, y1, and z1 coordinate values are obtained for all topological corresponding points using the information from Step 6 and Step 7.
  • Step 9. A mapping function, Fm, is introduced. The mapping function, Fm, maps the surface of the test model (the model body) to the subject's body model as a linear combination of the 3D shape functions obtained in Steps 6-8.
  • Step 10. The subject's 3D model is represented by a final function, Ff, which transforms the test model surface polygon mesh into a subject's body model surface polygon mesh. The latter can be created on demand and based on multiple purposes. Among these are: 3D rendering, body metric analysis, product fit computations, etc.
  • It is noted that all 3D body models created via the process described in Example 1 above have a unique mapping function F corresponding to a particular general test model. This relationship will further allow for the creation of a 3D image containing all possible mutual mappings inside the data base pertaining to that particular test model.
  • The description of a preferred embodiment of the invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Obviously, many modifications and variations will be apparent to practitioners skilled in this art. It is intended that the scope of the invention be defined by the following claims and their equivalents.
  • Moreover, the words “example” or “exemplary” are used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, use of the words “example” or “exemplary” is intended to present concepts in a concrete fashion. As used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or”. That is, unless specified otherwise, or clear from context, “X employs A or B” is intended to mean any of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, then “X employs A or B” is satisfied under any of the foregoing instances. In addition, the articles “a” and “an” as used in this application and the appended claims should generally be construed to mean “one or more” unless specified otherwise or clear from context to be directed to a singular form.

Claims (20)

What is claimed is:
1. A method for creating one or more three-dimensional (3D) images of a user, comprising:
placing a two-dimensional (2D) camera on a planar surface,
capturing a first 2D image of the user,
capturing a second 2D image of the user, the second 2D image being an image of a different perspective than the first 2D image, wherein the user remains on a single axis in both the first and second 2D images,
segmenting the first and second 2D images, forming a plurality of segmented 2D images, and
constructing a 3D image from a combination of all 3 of:
said plurality of segmented 2D images,
data extracted from said plurality of segmented 2D images, and
information comprising a computed camera position.
2. The method of claim 1, further comprising the step of manually adjusting said data extracted from said plurality of segmented 2D images, wherein said manual adjusting is performed by the user.
3. The method of claim 1, further comprising the step of automatically extracting measurements of the user's body from the 3D image constructed.
4. The method of claim 1, further comprising the step of adding one or more desired measurements for user height based on the 3D image.
5. The method of claim 1, further comprising the step of adding one or more desired measurements for user girth based on the 3D image.
6. The method of claim 1, further comprising the step of adding one or more desired measurements flu a geodesic distance between two or more points on the user's 3D image.
7. The method of claim 1, wherein 2 or more 3D images are constructed, further comprising the step of saving each 3D image such that the user may track one or more bodily features over time.
8. The method of claim 1, wherein the 3D image is compared to an ideal 3D model.
9. The method of claim 3, further comprising the step of sending said measurements to an authorized recipient.
10. The method of claim 9, further comprising the step of tailoring an outfit for the user based on the measurements sent to the authorized recipient.
11. The method of claim 3, further comprising the step of comparing the extracted measurements to known fitment data of a desired product.
12. The method of claim 11, further comprising the step of suggesting a product based on information retrieved from said comparing step.
13. The method of claim 6, wherein the geodesic distance is between the ground and a point on the circumference of the user's body.
14. The method of claim 1, further comprising the step of measuring, using a virtual measurement tape, a distance between two points on said 3D image.
15. The method of claim 1, further comprising the step of measuring a circumference of the user's body.
16. The method of claim 1, further comprising the step of controlling a camera position using an accelerometer coupled to said camera.
17. The method of claim 1, further comprising the step of generating feedback using an accelerometer coupled to said camera, said feedback correcting the user's positioning.
18. The method of claim 1, wherein said computed camera position is obtained via an accelerometer coupled to the camera, the computed camera position comprising the camera's sensor plane position relative to the user.
19. The method of claim 18, further comprising measuring one or more dimensions comprising depth.
20. The method of claim 1, further comprising creating a real time transformation of the user's body, said real time transformation being designed exactly once per categorical body type.
US15/237,850 2015-08-17 2016-08-16 Mobile device human body scanning and 3d model creation and analysis Abandoned US20170053422A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US15/237,850 US20170053422A1 (en) 2015-08-17 2016-08-16 Mobile device human body scanning and 3d model creation and analysis
US16/442,823 US10460517B2 (en) 2015-08-17 2019-06-17 Mobile device human body scanning and 3D model creation and analysis

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201562205832P 2015-08-17 2015-08-17
US15/237,850 US20170053422A1 (en) 2015-08-17 2016-08-16 Mobile device human body scanning and 3d model creation and analysis

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/442,823 Continuation-In-Part US10460517B2 (en) 2015-08-17 2019-06-17 Mobile device human body scanning and 3D model creation and analysis

Publications (1)

Publication Number Publication Date
US20170053422A1 true US20170053422A1 (en) 2017-02-23

Family

ID=58158449

Family Applications (2)

Application Number Title Priority Date Filing Date
US15/237,850 Abandoned US20170053422A1 (en) 2015-08-17 2016-08-16 Mobile device human body scanning and 3d model creation and analysis
US16/442,823 Active US10460517B2 (en) 2015-08-17 2019-06-17 Mobile device human body scanning and 3D model creation and analysis

Family Applications After (1)

Application Number Title Priority Date Filing Date
US16/442,823 Active US10460517B2 (en) 2015-08-17 2019-06-17 Mobile device human body scanning and 3D model creation and analysis

Country Status (1)

Country Link
US (2) US20170053422A1 (en)

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108269302A (en) * 2018-01-19 2018-07-10 浙江大学 A kind of 3 D human body fast reconstructing method based on ordinary surveying clothes
US20190058857A1 (en) * 2017-08-15 2019-02-21 International Business Machines Corporation Generating three-dimensional imagery
WO2020034738A1 (en) * 2018-08-16 2020-02-20 Oppo广东移动通信有限公司 Three-dimensional model processing method and apparatus, electronic device and readable storage medium
US10832472B2 (en) * 2018-10-22 2020-11-10 The Hong Kong Polytechnic University Method and/or system for reconstructing from images a personalized 3D human body model and thereof
US10953602B2 (en) * 2014-04-30 2021-03-23 Materialise N.V. Systems and methods for customization of objects in additive manufacturing
US11036989B1 (en) * 2019-12-11 2021-06-15 Snap Inc. Skeletal tracking using previous frames
US20220078339A1 (en) * 2019-01-03 2022-03-10 Idiction Co., Ltd. Method for obtaining picture for measuring body size and body size measurement method, server, and program using same
US11465304B2 (en) * 2016-03-01 2022-10-11 Koninklijke Philips N.V. System and method for automated hairstyle processing and hair cutting device
US20220386874A1 (en) * 2017-06-21 2022-12-08 H3Alth Technologies Inc. System, method and kit for 3d body imaging
WO2023279163A1 (en) * 2021-07-07 2023-01-12 Hearables 3D Pty Ltd Method, program, and system for 3d scanning

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TWM606669U (en) * 2019-05-02 2021-01-21 八豐盟有限公司 Three-dimensional human body scanning device
WO2022051460A1 (en) * 2020-09-02 2022-03-10 Roblox Corporation 3d asset generation from 2d images
US11941770B2 (en) * 2020-12-30 2024-03-26 Beijing Wodong Tianjun Information Technology Co., Ltd. System and method for 3D try-on based on human pose and body shape estimation
US11752902B1 (en) 2023-02-15 2023-09-12 Ayro, Inc. Vehicle seating system and methods thereof
US11850984B1 (en) 2023-02-15 2023-12-26 Ayro, Inc. Vehicle seating system and methods thereof

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100111370A1 (en) * 2008-08-15 2010-05-06 Black Michael J Method and apparatus for estimating body shape
US20130215116A1 (en) * 2008-03-21 2013-08-22 Dressbot, Inc. System and Method for Collaborative Shopping, Business and Entertainment

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9177410B2 (en) * 2013-08-09 2015-11-03 Ayla Mandel System and method for creating avatars or animated sequences using human body features extracted from a still image

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130215116A1 (en) * 2008-03-21 2013-08-22 Dressbot, Inc. System and Method for Collaborative Shopping, Business and Entertainment
US20100111370A1 (en) * 2008-08-15 2010-05-06 Black Michael J Method and apparatus for estimating body shape

Cited By (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10953602B2 (en) * 2014-04-30 2021-03-23 Materialise N.V. Systems and methods for customization of objects in additive manufacturing
US11465304B2 (en) * 2016-03-01 2022-10-11 Koninklijke Philips N.V. System and method for automated hairstyle processing and hair cutting device
US20220386874A1 (en) * 2017-06-21 2022-12-08 H3Alth Technologies Inc. System, method and kit for 3d body imaging
US11813040B2 (en) * 2017-06-21 2023-11-14 H3Alth Technologies, Inc. System, method and kit for 3D body imaging
US10735707B2 (en) * 2017-08-15 2020-08-04 International Business Machines Corporation Generating three-dimensional imagery
US10785464B2 (en) 2017-08-15 2020-09-22 International Business Machines Corporation Generating three-dimensional imagery
US20190058857A1 (en) * 2017-08-15 2019-02-21 International Business Machines Corporation Generating three-dimensional imagery
CN108269302A (en) * 2018-01-19 2018-07-10 浙江大学 A kind of 3 D human body fast reconstructing method based on ordinary surveying clothes
WO2020034738A1 (en) * 2018-08-16 2020-02-20 Oppo广东移动通信有限公司 Three-dimensional model processing method and apparatus, electronic device and readable storage medium
US10832472B2 (en) * 2018-10-22 2020-11-10 The Hong Kong Polytechnic University Method and/or system for reconstructing from images a personalized 3D human body model and thereof
US20220078339A1 (en) * 2019-01-03 2022-03-10 Idiction Co., Ltd. Method for obtaining picture for measuring body size and body size measurement method, server, and program using same
US11036989B1 (en) * 2019-12-11 2021-06-15 Snap Inc. Skeletal tracking using previous frames
US11594025B2 (en) 2019-12-11 2023-02-28 Snap Inc. Skeletal tracking using previous frames
WO2023279163A1 (en) * 2021-07-07 2023-01-12 Hearables 3D Pty Ltd Method, program, and system for 3d scanning

Also Published As

Publication number Publication date
US10460517B2 (en) 2019-10-29
US20190304173A1 (en) 2019-10-03

Similar Documents

Publication Publication Date Title
US10460517B2 (en) Mobile device human body scanning and 3D model creation and analysis
US10813715B1 (en) Single image mobile device human body scanning and 3D model creation and analysis
CN107111833B (en) Fast 3D model adaptation and anthropometry
US8571698B2 (en) Simple techniques for three-dimensional modeling
EP3332547B1 (en) Virtual apparel fitting systems and methods
Pujades et al. The virtual caliper: rapid creation of metrically accurate avatars from 3D measurements
RU2551731C1 (en) Method of virtual selection of clothes
CN108475439B (en) Three-dimensional model generation system, three-dimensional model generation method, and recording medium
US9715759B2 (en) Reference object for three-dimensional modeling
TWI488071B (en) A non-contact three-dimensional data acquisition system and method body, which on the surface in terms of the system
CN105608737B (en) A kind of human foot three-dimensional rebuilding method based on machine learning
EP3745352B1 (en) Methods and systems for determining body measurements and providing clothing size recommendations
WO2020114035A1 (en) Three-dimensional feature extraction method and apparatus based on machine vision
CN106952335B (en) Method and system for establishing human body model library
CN106462738B (en) Method for constructing a model of a person's face, method and apparatus for analyzing a pose using such a model
CN103535960A (en) Human body three-dimensional measurement method based on digital images
Kaashki et al. Deep learning-based automated extraction of anthropometric measurements from a single 3-D scan
Lunscher et al. Point cloud completion of foot shape from a single depth map for fit matching using deep learning view synthesis
JP2017194301A (en) Face shape measuring device and method
EP3452956A1 (en) Method and system for body scanning and display of biometric data
EP3801201A1 (en) Measuring surface distances on human bodies
CN111445570B (en) Customized garment design production equipment and method
CN111435069B (en) Method and device for measuring volume
TWI764393B (en) Manufacturing method of pressure garment
US11721035B2 (en) System and method of use of augmented reality in measuring body circumference for the use of apparel production

Legal Events

Date Code Title Description
STPP Information on status: patent application and granting procedure in general

Free format text: FINAL REJECTION MAILED

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION