WO2023059663A1

WO2023059663A1 - Systems and methods for assessment of body fat composition and type via image processing

Info

Publication number: WO2023059663A1
Application number: PCT/US2022/045706
Authority: WO
Inventors: Amit Khera; Saaket AGRAWAL; Marcus KLARQVIST; Puneet BATRA
Original assignee: The Broad Institute, Inc.; The General Hospital Corporation
Priority date: 2021-10-04
Filing date: 2022-10-04
Publication date: 2023-04-13

Abstract

The subject matter disclosed herein relates to utilizing the silhouette of an individual to measure body fat volume and distribution. Particular examples relates to providing a system, a computer-implemented method, and a computer program product to utilize a binary outline, or silhouette, to predict the individual's fat depot volumes with machine learning models.

Description

SYSTEMS AND METHODS FOR ASSESSMENT OF BODY FAT COMPOSITION AND TYPE VIA IMAGE PROCESSING

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63/252, 112, filed October 4, 2021. The entire contents of the above-identified application are hereby fully incorporated herein by reference.

TECHNICAL FIELD

[0002] The subject matter disclosed herein is related to utilizing the silhouette of an individual to measure body fat volume and distribution. Particular examples relates to providing a system, a computer-implemented method, and a computer program product to utilize a binary outline, or silhouette, to predict the individual’ s fat depot volumes with machine learning models.

BACKGROUND

[0003] The prevalence of obesity has reached epidemic proportions with 40% of adults afflicted in the United States. It is expected that the number of adults affected with obesity will rise to half of the US population in the next decade. Obesity is solely defined by body mass index (BMI) but at any given BMI, cardiometabolic risk can drastically vary according to fat distribution. Despite its clinical importance, fat distribution is not a routinely quantified clinical metrics, largely because the most accurate methods in body-composition analysis require whole-body magnetic resonance imaging (MR) or computed tomography scans (can cite recent society statement on fat distribution that says not indicated). However, these methods are expensive, time consuming, require skilled operators, and can neither scale easily into the hundreds of thousands to many millions of subjects nor be introduced into routine clinical care for practical and financial reasons. Facing these practical limitations with conventional approaches, a need exists for a more cost-effective method and device that is more accessible and readily available for individuals and in clinical settings for determining fat distribution. Specifically, a method, system, and computer program product are needed that can utilize a binary outline, or silhouette, representing a crude representation of an individual’s body morphology, to predict the individual’s fat depot volumes with training deep learning models. [0004] Citation or identification of any document in this application is not an admission that such a document is available as prior art to the present invention.

SUMMARY

[0005] A computer-implemented method to determine body fat composition from imaging data, comprising: receiving, by a user device, one or more images from an imaging device, the user device communicatively coupled with an acquisition engine; transferring, by the acquisition engine, the one or more images to a deployed machine learning network communicatively coupled to the acquisition engine; processing the one or more images with the deployed machine learning network, the deployed machine learning network generated and deployed from a training machine learning network; and transferring, by the deployed machine learning network, the processed one or more images as output to a diagnosis engine communicatively coupled to the deployed machine learning network; generating, by the diagnosis engine, a body fat composition analysis; and transmitting, by the diagnosis engine, the body fat composition analysis to a user device associated with a user, the diagnosis engine being communicatively coupled to the user device.

[0006] The computer-implemented method may further comprising transferring, by the acquisition engine, the one or more images to a reconstruction engine communicatively coupled to the acquisition engine, before transferring the one or more images to the machine learning network, and reconstructing the one or more images with the reconstruction engine. [0007] In one example embodiment, the one or more images comprise water or fat phases of MRI image, grayscale DEXA image, CT image, or ultrasound images. The images may be silhouettes or converted to silhouettes. In one example embodiment, the images may be coronal silhouettes and/or sagittal silhouettes.

[0008] The machine learning may comprise unsupervised learning, supervised learning, semi-supervised learning, reinforcement learning, transfer learning, incremental learning, curriculum learning, and learning to learn. The machine learning method may further comprise linear classifiers, logistic classifiers, Bayesian networks, random forest, neural networks, matrix factorization, hidden Markov model, support vector machine, K-means clustering, or K-nearest neighbor. In one example embodiment, the neural network method is a deep learning method. [0009] The fat composition assess may comprise adipose tissue volume, adipose tissue distribution, adipose tissue type, and/or BMI. The adipose tissue may comprise one or more of visceral adipose tissue (VAT), dermal adipose tissue (DAT), and/or subcutaneous adipose tissue (SAT) depots, or any combination thereof, such as ratios of these parameters or values adjusted for BMI or similar characteristics (e.g. VATadjBMI). The VAR may comprise one or more of epicardial VAT (EV AT), omental VAT (OVAT), perirenal VAT (PVAT), retroperitoneal VAT (RVAT), mesenteric VAT (MV AT), gonadal (GV AT). The SAT may comprise cranial SAT (CSAT), upper body SAT (USAT), abdominal SAT (ASAT), gluteal SAT (GSAT), femoral SAT (FSAT).

[0010] In one example embodiment, a computer implement method of training a machine learning process comprises training any of the above recited machine learning methods with any of the above recited image types.

[0011] In another aspect, a system to measure body fat composition may comprise: a storage device; and a processor communicatively coupled to the storage device, wherein the processor executes application code instructions that are stored in the storage device to cause the system to: receive one or more images of a subject from a user device by an acquisition engine communicatively coupled to the user device; transfer the one or more images with the acquisition engine communicatively coupled to a deployed machine learning network; process the one or more images with a deployed machine learning network, the deployed machine learning network generated and deployed from a training machine learning network; transfer the processed one or more images as output to a diagnosis engine communicatively coupled to the deployed machine learning network; generate a body fat composition analysis with the diagnosis engine communicatively coupled to the deployed machine learning network; and transmit the body fat composition analysis to a device associated with a user.

[0012] The one or more images may first be transferred by the acquisition engine to the reconstruction engine communicatively coupled to the acquisition engine and reconstructing the one or more images with the reconstruction engine.

[0013] The one or more images may comprise water or fat phases of MRI images, grayscale DEXA image, CT image, or ultrasound image.

[0014] In one example embodiment, the images may be silhouettes or converted to silhouettes. The silhouettes may be coronal silhouettes and/or sagittal silhouettes. [0015] The machine learning may comprise unsupervised learning, supervised learning, semi-supervised learning, reinforcement learning, transfer learning, incremental learning, curriculum learning, and learning to learn. The machine learning method further comprises linear classifiers, , logistic classifiers, Bayesian networks, random forest, neural networks, matrix factorization, hidden Markov model, support vector machine, K-means clustering, or K-nearest neighbor. In one example embodiment, the neural network may comprise a deep learning method.

[0016] The fat composition analyzed by the system may comprises adipose tissue volume, adipose tissue distribution, adipose tissue type, and/or BMI. The adipose tissue comprises one or more of visceral adipose tissue (VAT), dermal adipose tissue (DAT), and/or subcutaneous adipose tissue (SAT) depots, or any combination thereof, such as ratios of these parameters or values adjusted for BMI or similar characteristics (e.g. VATadjBMI). The VAT may comprise one or more of epicardial VAT (EV AT), omental VAT (OVAT), perirenal VAT (PVAT), retroperitoneal VAT (RVAT), mesenteric VAT (MV AT), gonadal (GV AT). The may SAT comprise cranial SAT (CSAT), upper body SAT (USAT), abdominal SAT (ASAT), gluteal SAT (GSAT), femoral SAT (FSAT).

[0017] In another aspect , a computer program product for measuring body fat composition comprises: a non-transitory computer-readable storage device having computer-executable program instructions embodied thereon that when executed by a computer cause the computer to measure body fat composition of a subject, the computer-executable program instructions comprising: computer-executable program instructions to receive one or more images of a user from an imaging device; computer-executable program instructions to transfer the one or more images with an acquisition engine communicatively coupled to the imaging device to a deployed machine learning network; computer-executable program instructions to process the one or more images with the deployed machine learning network, the deployed machine learning network generated and deployed from a training machine learning network and communicatively coupled to the acquisition engine; computer-executable program instructions to transfer the processed one or more images as output to a diagnosis engine communicatively coupled to the deployed machine learning network; computer-executable program instructions to generate a fat composition analysis with a diagnosis engine communicatively coupled to the deep learning network; and computer-executable program instructions to transmit the body fat composition analysis to the user. [0018] In one example embodiment, the one or more images are first transferred by the acquisition engine to the reconstruction engine communicatively coupled to the acquisition engine and reconstructing the one or more images with the reconstruction engine.

[0019] The one or more images comprise water or fat phases of MRI images, grayscale DEXA image, CT image, or ultrasound image. The images may be silhouettes or are converted to silhouettes. The images may be coronal silhouettes and/or sagittal silhouettes.

[0020] The machine learning comprises unsupervised learning, supervised learning, semisupervised learning, reinforcement learning, transfer learning, incremental learning, curriculum learning, and learning to learn.

[0021] The machine learning method may comprise linear classifiers, logistic classifiers, Bayesian networks, random forest, neural networks, matrix factorization, hidden Markov model, support vector machine, K-means clustering, or K-nearest neighbor. In one example embodiment, the neural network method is a deep learning method.

[0022] The fat composition comprises adipose tissue volume, adipose tissue distribution, adipose tissue type, and/or BMI. The adipose tissue may comprise one or more of visceral adipose tissue (VAT), dermal adipose tissue (DAT), and/or subcutaneous adipose tissue (SAT) depots, or any combination thereof, such as ratios of these parameters or values adjusted for BMI or similar characteristics (e.g. VATadjBMI). The VAT may comprise one or more of epicardial VAT (EV AT), omental VAT (OVAT), perirenal VAT (PVAT), retroperitoneal VAT (RVAT), mesenteric VAT (MV AT), gonadal (GV AT). The SAT comprises cranial SAT (CSAT), upper body SAT (USAT), abdominal SAT (ASAT), gluteal SAT (GSAT), femoral SAT (FSAT).

[0023] These and other aspects, objects, features, and advantages of the example embodiments will become apparent to those having ordinary skill in the art upon consideration of the following detailed description of example embodiments.

[0024] These and other aspects, objects, features, and advantages of the example embodiments will become apparent to those having ordinary skill in the art upon consideration of the following detailed description of example embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

[0025] An understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention may be utilized, and the accompanying drawings of which:

[0026] FIG. 1 - A block diagram depicting a portion of a communications and processing architecture of a typical system to acquire user associated information from a database and perform machine learning on the user associated images, in accordance with certain examples of the technology disclosed herein.

[0027] FIG. 2 - A block flow diagram depicting methods to measure body fat volume and distribution with machine learning, in accordance with certain examples of the technology disclosed herein.

[0028] FIG. 3 - A block diagram depicting a computing machine and modules, in accordance with certain examples of the technology disclosed herein.

[0029] FIG. 4 - FIG. 4a) Silhouettes were created from whole-body magnetic resonance images by segmenting the outline of axial acquisitions followed by projecting the resulting volume into a 2-dimensional surface map followed by pixel binarization. More details are available in Online Methods; FIG. 4b) VAT, ASAT, and GF AT, as predicted using an ensemble of nested 5-fold cross validation models, all with the sample size N=40,032, plotted against previously described “gold standard” measurements calculated from whole-body MRI (ref). The fat depot volumes VAT (coefficient of determination R2 = 0.885), ASAT (R2 = 0.934), and GF AT (R2 = 0.932) could be estimated with high accuracy. The solid black lines denote the linear fits (ordinary least squares regression) and the solid grey lines correspond to the identity line. Additional sex-stratified performance metrics and Bland-Altman plots are available in Supplemental Information; FIG. 4c) Sex-stratified comparison of Applicants silhouette-based deep learning model and a series of linear models based on readily available anthropometric traits such as height, weight, hip and waist circumference, when combined with age and sex, and to a more sophisticated non-linear model constructed for VAT, including several bioelectric impedance measurements, recently proposed by Karlsson et al (ref). These sex-specific formulas were used to fit new models to each individual fat depot using their formulas as described. Models have very similar but not exactly the same sample sizes caused by missingness in certain features (Supplemental Table X-X). Correlation confidence intervals for all models, with the exception of the deep learning-based silhouette models, were estimated using nested 5-fold cross-validation repeated 100 times. [0030] FIG. 5 - Silhouettes estimate VAT/ASAT ratio with good performance. FIG. 5a) 2D MRI projections and silhouettes of an age, sex, BMI, and waist circumference matched pair of participants with drastic differences in abdominal fat distribution are displayed. While both participants have an elevated waist circumference for their sex- and BMI-group, participant 1 primarily has ASAT-driven central obesity, while participant 2 primarily has VAT-driven central obesity. FIG. 5b) A deep learning model trained on silhouettes can predict VAT/ASAT with good performance (R2 = 0.81 in sex-combined analyses). In sex-stratified analyses, silhouettes outperformed waist circumference by 0.48 in males and 0.32 in females, c) Waist circumference is strongly correlated with silhouette-predicted VAT (VATsiL) and silhouette- predicted AS AT (ASATsiL) (R^ 0.72-0.76), but nearly independent of silhouette-predicted VAT/ASAT (VAT/ASATsiL) (R^ 0.07-0.20). Complete correlograms including silhouette predictions and anthropometric measures are shown in supplemental figures X-X.

[0031] FIG. 6 - Association of silhouette-predicted VAT/ASAT with type 2 diabetes and coronary artery disease. FIG. 6a) Odds ratio for Type 2 and Coronary Artery Disease using VAT/ASAT and waist circumference. FIG. 6b) Males and females with Type 2 diabetes as a function of waist circumference.

[0032] The figures herein are for illustrative purposes only and are not necessarily drawn to scale.

DETAILED DESCRIPTION OF THE EXAMPLE EMBODIMENTS

General Definitions

[0033] Unless defined otherwise, technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. Definitions of common terms and techniques in molecular biology may be found in Molecular Cloning: A Laboratory Manual, 2^nd edition (1989) (Sambrook, Fritsch, and Maniatis); Molecular Cloning: A Laboratory Manual, 4^th edition (2012) (Green and Sambrook); Current Protocols in Molecular Biology (1987) (F.M. Ausubel et al. eds.); the series Methods in Enzymology (Academic Press, Inc.): PCR 2: A Practical Approach (1995) (M.J. MacPherson, B.D. Hames, and G.R. Taylor eds.): Antibodies, A Laboratory Manual (1988) (Harlow and Lane, eds.): Antibodies A Laboratory Manual, 2^nd edition 2013 (E.A. Greenfield ed.); Animal Cell Culture (1987) (R.I. Freshney, ed.); Benjamin Lewin, Genes IX, published by Jones and Bartlet, 2008 (ISBN 0763752223); Kendrew et al. (eds.), The Encyclopedia of Molecular Biology, published by Blackwell Science Ltd., 1994 (ISBN 0632021829); Robert A. Meyers (ed.), Molecular Biology and Biotechnology: a Comprehensive Desk Reference, published by VCH Publishers, Inc., 1995 (ISBN 9780471185710); Singleton etal., Dictionary of Microbiology and Molecular Biology 2nd ed., J. Wiley & Sons (New York, N.Y. 1994), March, Advanced Organic Chemistry Reactions, Mechanisms and Structure 4th ed., John Wiley & Sons (New York, N.Y. 1992); and Marten H. Hofker and Jan van Deursen, Transgenic Mouse Methods and Protocols, 2^nd edition (2011). [0034] As used herein, the singular forms “a”, “an”, and “the” include both singular and plural referents unless the context clearly dictates otherwise.

[0035] The term “optional” or “optionally” means that the subsequent described event, circumstance or substituent may or may not occur, and that the description includes instances where the event or circumstance occurs and instances where it does not.

[0036] The recitation of numerical ranges by endpoints includes all numbers and fractions subsumed within the respective ranges, as well as the recited endpoints.

[0037] The terms “about” or “approximately” as used herein when referring to a measurable value such as a parameter, an amount, a temporal duration, and the like, are meant to encompass variations of and from the specified value, such as variations of +/-10% or less, +/-5% or less, +/-1% or less, and +/-0.1% or less of and from the specified value, insofar such variations are appropriate to perform in the disclosed invention. It is to be understood that the value to which the modifier “about” or “approximately” refers is itself also specifically, and preferably, disclosed.

[0038] As used herein, a “biological sample” may contain whole cells and/or live cells and/or cell debris. The biological sample may contain (or be derived from) a “bodily fluid”. The present invention encompasses embodiments wherein the bodily fluid is selected from amniotic fluid, aqueous humour, vitreous humour, bile, blood serum, breast milk, cerebrospinal fluid, cerumen (earwax), chyle, chyme, endolymph, perilymph, exudates, feces, female ejaculate, gastric acid, gastric juice, lymph, mucus (including nasal drainage and phlegm), pericardial fluid, peritoneal fluid, pleural fluid, pus, rheum, saliva, sebum (skin oil), semen, sputum, synovial fluid, sweat, tears, urine, vaginal secretion, vomit and mixtures of one or more thereof. Biological samples include cell cultures, bodily fluids, cell cultures from bodily fluids. Bodily fluids may be obtained from a mammal organism, for example by puncture, or other collecting or sampling procedures. [0039] The terms “subject,” “individual,” and “patient” are used interchangeably herein to refer to a vertebrate, preferably a mammal, more preferably a human. Mammals include, but are not limited to, murines, simians, humans, farm animals, sport animals, and pets. Tissues, cells and their progeny of a biological entity obtained in vivo or cultured in vitro are also encompassed.

[0040] Various embodiments are described hereinafter. It should be noted that the specific embodiments are not intended as an exhaustive description or as a limitation to the broader aspects discussed herein. One aspect described in conjunction with a particular embodiment is not necessarily limited to that embodiment and can be practiced with any other embodiment(s). Reference throughout this specification to “one embodiment”, “an embodiment,” “an example embodiment,” means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, appearances of the phrases “in one embodiment,” “in an embodiment,” or “an example embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment, but may. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner, as would be apparent to a person skilled in the art from this disclosure, in one or more embodiments. Furthermore, while some embodiments described herein include some but not other features included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the invention. For example, in the appended claims, any of the claimed embodiments can be used in any combination.

[0041] The term “fat composition” as used herein may refer to fat volume, fat type, and fat distribution.

[0042] All publications, published patent documents, and patent applications cited herein are hereby incorporated by reference to the same extent as though each individual publication, published patent document, or patent application was specifically and individually indicated as being incorporated by reference.

OVERVIEW

[0043] In one aspect, technologies herein provide methods to determine an individual’s fat composition from imaging data. These methods provide for more reliable diagnosis of obesity and associated diseases where current methods fail. The most well-known method of determining fat composition is the body mass index (BMI). However, BMI does not differentiate between body lean mass and body fat mass. As a result, there exists relatively poor correlation between BMI and associated diseases such as diabetes. Furthermore, the current methods to determine fat composition accurately and simply are non-existent. For example, methods to estimate percent body fat, such as underwater weighing, air displacement, and other density measuring methods require assumptive values in the measurement method to quantify the percent body fat. However, these assumptions introduce significant error in populations outside the assumption. Bioelectric impedance suffers similar problems and introduces errors as well. Furthermore, none of the previously mentioned methods can identify fat type and distribution. More complex methods such as DEXA and CT, which can determine fat distribution, are not commonly available as they are cost and prohibitive and not widely available. See e.g. Nuttall, F. Q. Body Mass Index. Nutrition Today 2015, 50 (3), 117-128. Embodiments herein combine machine learning with a user’s physical representation in the form of a silhouette to accurately determine fat and to diagnose obesity and associated diseases [0044] The embodiments described herein include computer-implemented methods, computer program products, and systems to use imaging data to provide an assessment of body fat composition of a subject. In some examples of the technology, an imaging device receives one or more images from a user. The imaging device is coupled with an acquisition device, which transfers the images to a machine learning network that is generated and deployed from a training machine learning network. The processed images are transmitted to a diagnostic engine, communicatively coupled to the machine learning network, which generates an analysis of fat composition of the subject to a diagnostic engine. The diagnostic engine transmits the analysis of fat composition to a user device communicatively coupled to the diagnostic engine. The transmitted fat composition analysis may be used to diagnose obesity and associated diseases.

[0045] In some examples of the technology, the images are silhouettes of a subject’s body morphology, for example coronal silhouettes and/or sagittal silhouettes, are converted to silhouettes created from whole body magnetic resonance (MRI) images. The images may comprise water or fat phases of the MRI image, grayscale DEXA image, CT image, or ultrasound images. In some examples, the machine learning comprises unsupervised learning, supervised learning, semi-supervised learning, reinforcement learning, transfer learning, incremental learning, curriculum learning, and learning to learn. In some examples, the machine learning method may comprise linear classifiers, logistic classifiers, Bayesian networks, random forest, neural networks, matrix factorization, hidden Markov model, support vector machine, K-means clustering, or K-nearest neighbor. In some examples, the neural network method may be a deep learning method. The diagnostic engine comprises any software or hardware individually or in combination described herein that is capable of determining adipose tissue characteristics. In some examples, the diagnosis engine determines adipose tissue volume, adipose tissue distribution, adipose tissue type, BMI, or any combination thereof. In some examples, the diagnostic engine may determine one or more of visceral adipose tissue (VAT), dermal adipose tissue (DAT), and/or subcutaneous adipose tissue (SAT) depots, or any combination thereof, such as ratios of these parameters or values adjusted for BMI or similar characteristics. In example embodiments, VAT comprises one or more of epicardial VAT (EV AT), omental VAT (OVAT), perirenal VAT (PVAT), retroperitoneal VAT (RVAT), mesenteric VAT (MV AT), gonadal (GV AT). In example embodiments, SAT comprises cranial SAT (CSAT), upper body SAT (USAT), abdominal SAT (ASAT), gluteal SAT (GSAT), femoral SAT (FSAT).

[0046] In another aspect, embodiments disclosed herein includes diagnostic applications for users to operate on user computing devices. The diagnostic application may be a downloadable application or application programming interface for use on a smartphone or other user computing device that receives one or more images of user and other data from a user. The image data may include any form of imaging data. Imaging data may come from a camera, the camera may be located on a user associated device. Imaging data may come from other devices that measure other forms of electromagnetic radiation, such as MRI, DEXA, CT, or ultrasound. The diagnostic application, after receiving user associated image data, can process and administer a diagnosis of fat composition, obesity, and associated diseases.

Example Systems Architectures

[0047] Turning now to the drawings, in which like numerals represent like (but not necessarily identical) elements throughout the figures, example embodiments are described in detail.

[0048] FIG. 1 is a block diagram depicting a system 100 to determine fat composition and perform machine learning on a representation of a user’s body. In one example embodiment, a user 101 associated with a user computing device 110 must install an application, and or make a feature selection to obtain the benefits of the techniques described herein. [0049] As depicted in FIG. 1, the system 100 includes network computing devices/ systems 110, 120, and 130 that are configured to communicate with one another via one or more networks 105 or via any suitable communication technology.

[0050] Each network 105 includes a wired or wireless telecommunication means by which network devices/sy stems (including devices 110, 120, and 130) can exchange data. For example, each network 105 can include a local area network (“LAN”), a wide area network (“WAN”), an intranet, an Internet, a mobile telephone network, storage area network (“SAN”), personal area network (“PAN”), a metropolitan area network (“MAN”), a wireless network (“WiFi;”), wireless access networks, a wireless local area network (“WLAN”), a virtual private network (“VPN”), a cellular or other mobile communication network, Bluetooth, near field communication (“NFC”), ultra-wideband, wired networks, telephone networks, optical networks, or any combination thereof or any other appropriate architecture or system that facilitates the communication of signals and data. Throughout the discussion of example embodiments, it should be understood that the terms “data” and “information” are used interchangeably herein to refer to text, images, audio, video, or any other form of information that can exist in a computer-based environment. The communication technology utilized by the devices/sy stems 110, 120, and 130 may be similar networks to network 105 or an alternative communication technology.

[0051] Each network computing device/system 110, 120, and 130 includes a computing device having a communication module capable of transmitting and receiving data over the network 105 or a similar network. For example, each network device/system 110, 120, and 130 can include a server, desktop computer, laptop computer, tablet computer, a television with one or more processors embedded therein and/or coupled thereto, smartphone, handheld or wearable computer, personal digital assistant (“PDA”), wearable devices such as smartwatches or glasses, or any other wired or wireless, processor-driven device. In the example embodiment depicted in Figure 1, the network devices/sy stems 110, 120, and 130 are operated by user 101, data acquisition system operators, and mapping system operators, respectively.

[0052] The user computing device 110 includes a user interface 114. The user interface 114 may be used to display a graphical user interface and other information to the user 101 to allow the user 101 to interact with the data acquisition system 120, the machine learning system 130, and others. The user interface 114 receives user input for data acquisition and/or machine learning and displays results to user 101. In another example embodiment, the user interface 114 may be provided with a graphical user interface by the data acquisition system 120 and or the machine learning system 130. The user interface 114 may be accessed by the processor of the user computing device 110. The user interface 114 may display a webpage associate with the data acquisition system 120 and/or the machine learning system 130. The user interface 114 may be used to provide input, configuration data, and other display direction by the webpage of the data acquisition system 120 and/or the machine learning system 130. In another example embodiment, the user interface 114 may be managed by the data acquisition system 120, the learning system 130, or others. In another example embodiment, the user interface 114 may be managed by the user computing device 110 and be prepared and displayed to the user 101 based on the operations of the user computing device 110.

[0053] The user 101 can use the communication application 112 on the user computing device 110, which may be, for example, a web browser application or a stand-alone application, to view, download, upload, or otherwise access documents or web pages through the user interface 114 via the network 105. The user computing device 110 can interact with the web servers or other computing devices connected to the network, including the data acquisition server 125 of the data acquisition system 120 and the image representation and reconstruction server 135 of the machine learning system 130. In another example embodiment, the user computing device 110 communicates with devices in the data acquisition system 120 and/or the machine learning system 130 via any suitable technology, including the example computing system described below.

[0054] The user computing device 110 also includes a data storage unit 113 accessible by the user interface 114, the communication application 112, or other applications. The example data storage unit 113 can include one or more tangible computer-readable storage devices. The data storage unit 113 can be stored on the user computing device 110 or can be logically coupled to the user computing device 110. For example, the data storage unit 113 can include on-board flash memory and/or one or more removable memory accounts or removable flash memory. In another example embodiments, the data storage unit 113 may reside in a cloudbased computing system.

[0055] An example data acquisition system 120 comprises a data storage unit 123 and an acquisition server 125. The data storage unit 123 can include any local or remote data storage structure accessible to the data acquisition system 120 suitable for storing information. The data storage unit 123 can include one or more tangible computer-readable storage devices, or the data storage unit 123 may be a separate system, such as a different physical or virtual machine or a cloud-based storage service.

[0056] In one aspect, the data acquisition server 125 communicates with the user computing device 110 and/or the machine learning system 130 to transmit requested data. The data may include an image of a user’s body.

[0057] An example mapping system 130 comprises a machine learning system 133, an image representation and reconstruction server 135, and a data storage unit 137. The image representation and reconstruction server 135 communicates with the user computing device 110 and/or the data acquisition system 120 to request and receive data. The data may comprise the data types previously described in reference to the data acquisition server 125.

[0058] The machine learning system 133 receives an input of data from the image representation and reconstruction server 135. The machine learning system 133 can comprise one or more functions to implement any of the previously mentioned training methods to learn fat composition. In a preferred embodiment, the machine learning program may comprise deep learning. Any suitable architecture may be applied to learn fat composition of a user’s body.

[0059] The data storage unit 137 can include any local or remote data storage structure accessible to the machine learning system 130 suitable for storing information. The data storage unit 137 can include one or more tangible computer-readable storage devices, or the data storage unit 137 may be a separate system, such as a different physical or virtual machine or a cloud-based storage service.

[0060] In an alternate embodiment, the functions of either or both of the data acquisition system 120 and the machine learning system 130 may be performed by the user computing device 110.

[0061] It will be appreciated that the network connections shown are examples, and other means of establishing a communications link between the computers and devices can be used. Moreover, those having ordinary skill in the art having the benefit of the present disclosure will appreciate that the user computing device 110, data acquisition system 120, and the machine learning system 130 illustrated in FIG. 1 can have any of several other suitable computer system configurations. For example, a user computing device 110 embodied as a mobile phone or handheld computer may not include all the components described above.

[0062] In example embodiments, the network computing devices and any other computing machines associated with the technology presented herein may be any type of computing machine such as, but not limited to, those discussed in more detail with respect to FIG. 3. Furthermore, any modules associated with any of these computing machines, such as modules described herein or any other modules (scripts, web content, software, firmware, or hardware) associated with the technology presented herein may by any of the modules discussed in more detail with respect to FIG. 3. The computing machines discussed herein may communicate with one another as well as other computer machines or communication systems over one or more networks, such as network 105. The network 105 may include any type of data or communications network, including any of the network technology discussed with respect to FIG. 3

Example Processes

[0063] The example methods illustrated in FIG. 2 is described hereinafter with respect to the components of the example architecture 100. The example methods also can be performed with other systems and in other architectures including similar elements.

Diagnosing Fat Composition

[0064] Referring to FIG. 2, and continuing to refer to FIG. 1 for context, a block flow diagram illustrates methods 200 to determine body fat composition using machine learning, in accordance with certain examples of the technology disclosed herein. The resulting fat composition can be further processed into a diagnosis of obesity and risk of related diseases. [0065] In block 210, the machine learning system 130 receives an input of one or more user associated image data. The machine learning system 130 may receive the image data from the user computing device 110, the data acquisition system 120, or any other suitable source of imaging data via the network 105, discussed in more detail in other sections herein. The acquisition engine comprises any software or hardware individually or in combination described herein that is capable of communicating with a user device, such as fetching, receiving, or sending information, thereby allowing access to the imaging data or the body fat composition analysis by the machine learning system 130 or the data acquisition system 120.

Image Types

[0066] The user associated image (UAI) may be taken with a device capable of measuring electromagnetic radiation, such as the radio, visible, near-visible, and/or x-ray or a device capable of measuring magnetic fields. The UAI may be multi-dimensional (e.g. 2- or 3- dimensional) and the UAI may be a dark image, black-and-white image, or color image. The acquired UAI may be silhouette images such as coronal and/or sagittal or the UAI images are converted into silhouette images after acquisition. Silhouettes, in the context of this application, are dark outlines and shapes of a user. Coronal silhouettes, in the context of this application, are dark images of a user’s body in the plane running through the body from side to side, while sagittal silhouettes comprise the plane running through the body from front to back.

[0067] In an example embodiment, the UAI is taken with a digital camera. The image format may comprise any known image file format such as JPG, GIF, TIFF, PNG, or BMP files. In an example embodiment, the digital camera is associated with a computer device. In example embodiments, the camera device is associated with a user’s device. In an example embodiment, the computer device is a portable computer device.

[0068] In an example embodiment, the UAI is a magnetic resonance imaging (MRI) image. The MRI image may come from any MRI device with a field strength of 0.5 to 3.0 tesla (T). The MRI device may comprise wide bore, open, or high-field open devices and may be for laying, standing, or sitting. In an example embodiment, the UAI is dual energy x-ray absorption (DEXA) image. In one example embodiment, the UAI is a computerized tomography (CT) image. In one example embodiment, the UAI is an ultrasound image.

[0069] In block 220, the UAI is transferred over a network 105 via the transfer engine from the user associated device 100 or the data acquisition system 120 to the machine learning system 130. The transfer engine comprises any software and/or hardware that participates in exchanging information from one location to another either within a device or between devices. The acquisition system 120 comprises software and/or hardware capable of storing information either permanently or temporarily and either locally or across a network. The machine learning system 130 comprises the machine learning engine. The machine learning engine comprises a machine learning module, described in more detail below, and hardware capable of carrying out the process of determining the fat composition of a user from image data of the user. Example machine learning processes are described in further detail herein but are generally developed to transform initially indiscriminate information into valuable information for a user, such as a diagnosis of fat composition, obesity, and risk of associated diseases. In an example embodiment, the machine learning engine is configured to transform 2-dimensional representations of a user into the fat composition report. The 2-dimensional representations may further comprise silhouettes of a user.

[0070] The machine learning system 130 further comprises a reconstruction server 135. In example embodiments, the UAI does not comprise a coronal and/or sagittal silhouettes of a user and is first transferred to the reconstruction server 135 within the machine learning system 130 via the transfer engine. The reconstruction server 135 comprises a reconstruction engine. The reconstruction engine can reconstruct the UAI data through any known method of image manipulation software/code into coronal and/or sagittal silhouettes of a user. In a straightforward example, if the UAI comprises coronal and/or sagittal color images of a user, then the reconstruction engine can convert the color image data of the user, comprising color identification units such as HEX notation, into dark or black image data, such as changing any variation of OxOOOOff (blue), OxfTOOOO (red), and OxOOffOO (green) into 0x000000 (black). In addition, the background of the image of the user may be converted into Oxffffff (white).

Machine Learning Module

[0071] The machine learning module comprising algorithms that may learn from existing data by analyzing, categorizing, or identifying the data. Such machine-learning algorithms operate by first constructing a model from training data to make predictions or decisions expressed as outputs. In example embodiments, the training data includes data for one or more identified features and one or more outcomes, for example fat distribution, fat volume, and/or fat type such as visceral subcutaneous adipose tissue (VAT), abdominal subcutaneous adipose tissue (ASAT), and gluteofemoral adipose tissue (GFAT) depots. Although example embodiments are presented with respect to a few machine-learning algorithms, the principles presented herein may be applied to other machine-learning algorithms.

[0072] Data supplied to a machine learning algorithm can be considered a feature, which can be described as an individual measurable property of a phenomenon being observed. The concept of feature is related to that of an independent variable used in statistical techniques such as those used in linear regression. The performance of a machine learning algorithm in pattern recognition, classification and regression is highly dependent on choosing informative, discriminating, and independent features. Features may comprise numerical data, categorical data, time-series data, strings, graphs, or images. Features of the invention may comprise representations of a user’s body. These representations my include images such as of coronal silhouettes and/or sagittal silhouettes. In one example embodiment, the coronal silhouettes and/or sagittal silhouettes are from water or fat phases of MRI images, DEXA images, CT images, or ultrasound images.

[0073] In general, there are two categories of machine learning problems: classification problems and regression problems. Classification problems, also referred to as categorization problems, aim at classifying items into discrete category values. Training data teaches the classifying algorithm how to classify. In example embodiments, features to be categorized include representations of a user’s body, which can be provided to the classifying machine learning algorithm and then placed into categories of, for example, fat composition. Regression algorithms aim at quantifying and correlating one or more features. Training data teaches the regression algorithm how to correlate the one or more features into a quantifiable value. In example embodiments, features such as representations of a user’s body can be provided to the regression machine learning algorithm resulting in one or more continuous values, for example fat composition.

Embedding

[0074] In one example embodiment, the machine learning module may use embedding to provide a lower dimensional representation, such as a vector, of features to organize them based off respective similarities. In some situations, these vectors can become massive. In the case of massive vectors, particular values may become very sparse among a large number of values (e.g., a single instance of a value among 50,000 values). Because such vectors are difficult to work with, reducing the size of the vectors, in some instances, is necessary. A machine learning module can learn the embeddings along with the model parameters. In example embodiments, features such as representations of a user’s body can be mapped to vectors implemented in embedding methods. In example embodiments, embedded semantic meanings are utilized. Embedded semantic meanings are values of respective similarity. For example, the distance between two vectors, in vector space, may imply two values located elsewhere with the same distance are categorically similar. Embedded semantic meanings can be used with similarity analysis to rapidly return similar values. In example embodiments, a representation of a user is embedded. The representation of a user may comprise, for example, image data. The image data may be processed into vector space organized by properties such as pixel location, orientation, and/or color. In example embodiments, the methods herein are developed to identify meaningful portions of the vector and extract semantic meanings between that space.

Training Methods

[0075] In example embodiments, the machine learning module can be trained using techniques such as unsupervised, supervised, semi-supervised, reinforcement learning, transfer learning, incremental learning, curriculum learning techniques, and/or learning to learn. Training typically occurs after selection and development of a machine learning module and before the machine learning module is operably in use. In one aspect, the training data used to teach the machine learning module can comprise input data such as representations of a user’s body and the respective target output data such as fat composition.

[0076] In one example embodiment, unsupervised learning is implemented. Unsupervised learning can involve providing all or a portion of unlabeled training data to a machine learning module. The machine learning module can then determine one or more outputs implicitly based on the provided unlabeled training data. In one example embodiment, supervised learning is implemented. Supervised learning can involve providing all or a portion of labeled training data to a machine learning module, with the machine learning module determining one or more outputs based on the provided labeled training data, and the outputs are either accepted or corrected depending on the agreement to the actual outcome of the training data. In some examples, supervised learning of machine learning system(s) can be governed by a set of rules and/or a set of labels for the training input, and the set of rules and/or set of labels may be used to correct inferences of a machine learning module.

[0077] In one example embodiment, semi-supervised learning is implemented. Semisupervised learning can involve providing all or a portion of training data that is partially labeled to a machine learning module. During semi-supervised learning, supervised learning is used for a portion of labeled training data, and unsupervised learning is used for a portion of unlabeled training data. In one example embodiment, reinforcement learning is implemented. Reinforcement learning can involve first providing all or a portion of the training data to a machine learning module and as the machine learning module produces an output, the machine learning module receives a “reward” signal in response to a correct output. Typically, the reward signal is a numerical value, and the machine learning module is developed to maximize the numerical value of the reward signal. In addition, reinforcement learning can adopt a value function that provides a numerical value representing an expected total of the numerical values provided by the reward signal over time.

[0078] In one example embodiment, transfer learning is implemented. Transfer learning techniques can involve providing all or a portion of a first training data to a machine learning module, then, after training on the first training data, providing all or a portion of a second training data. In example embodiments, a first machine learning module can be pre-trained on data from one or more computing devices. The first trained machine learning module is then provided to a computing device, where the computing device is intended to execute the first trained machine learning model to produce an output. Then, during the second training phase, the first trained machine learning model can be additionally trained using additional training data, where the training data can be derived from kernel and non-kernel data of one or more computing devices. This second training of the machine learning module and/or the first trained machine learning model using the training data can be performed using either supervised, unsupervised, or semi-supervised learning. In addition, it is understood transfer learning techniques can involve one, two, three, or more training attempts. Once the machine learning module has been trained on at least the training data, the training phase can be completed. The resulting trained machine learning model can be utilized as at least one of trained machine learning module.

[0079] In one example embodiment, incremental learning is implemented. Incremental learning techniques can involve providing a trained machine learning module with input data that is used to continuously extend the knowledge of the trained machine learning module. Another machine learning training technique is curriculum learning, which can involve training the machine learning module with training data arranged in a particular order, such as providing relatively easy training examples first, then proceeding with progressively more difficult training examples. As the name suggests, difficulty of training data is analogous to a curriculum or course of study at a school.

[0080] In one example embodiment, learning to learn is implemented. Learning to learn, or meta-learning, comprises two levels of learning: quick learning of a single task and slower learning across many tasks. For example, a machine learning module is first trained and comprises of a its first set of parameters or weights. During or after operation of the first trained machine learning module, the parameters or weights are adjusted by the machine learning module. This process occurs iteratively on the success of the machine learning module. In another example, an optimizer, or another machine learning module, is used wherein the output of a first trained machine learning module is fed to an optimizer that constantly learns and returns the final results. Other techniques for training the machine learning module and/or trained machine learning module are possible as well.

[0081] In some examples, after the training phase has been completed but before producing predictions expressed as outputs, a trained machine learning module can be provided to a computing device where a trained machine learning module is not already resident. In other words, after training phase has been completed, the trained machine learning module can be downloaded to a computing device. For example, a first computing device storing a trained machine learning module can provide the trained machine learning module to a second computing device. Providing a trained machine learning module to the second computing device may comprise one or more of communicating a copy of trained machine learning module to the second computing device, making a copy of trained machine learning module for the second computing device, providing access to trained machine learning module to the second computing device, and/or otherwise providing the trained machine learning system to the second computing device. In example embodiments, a trained machine learning module can be used by the second computing device immediately after being provided by the first computing device. In some examples, after a trained machine learning module is provided to the second computing device, the trained machine learning module can be installed and/or otherwise prepared for use before the trained machine learning module can be used by the second computing device.

[0082] After a machine learning model has been trained it can be used to output, estimate, infer, predict, or determine. For simplicity these terms will collectively be referred to as results. A trained machine learning module can receive input data and operably generate results. As such, the input data can be used as an input to the trained machine learning module for providing corresponding results to kernel components and non-kernel components. For example, a trained machine learning module can generate results in response to requests. In example embodiments, a trained machine learning module can be executed by a portion of other software. For example, a trained machine learning module can be executed by a result daemon to be readily available to provide results upon request.

[0083] In example embodiments, a machine learning module and/or trained machine learning module can be executed and/or accelerated using one or more computer processors and/or on-device co-processors. Such on-device co-processors can speed up training of a machine learning module and/or generation of results. In some examples, trained machine learning module can be trained, reside, and execute to provide results on a particular computing device, and/or otherwise can make results for the particular computing device.

[0084] Input data can include data from a computing device executing a trained machine learning module and/or input data from one or more computing devices. In example embodiments, a trained machine learning module can use results as input feedback. A trained machine learning module can also rely on past results as inputs for generating new results. In example embodiments, input data can comprise representations of a user’s body and, when provided to a trained machine learning module, results in output data such as fat composition. The fat composition output data can then be provided to determine confidence in the output data or additional diagnosis information. The confidence of user’s fat composition can be determined, thereby instructing the user on the likelihood of further pursing health advice. The additional diagnosis information can use the user’s fat composition to determine the risk of diseases associated with the user’s fat composition, thereby further diagnosing the health risks of the user. As such, the image data-related technical problem of determining a user’s fat composition can be solved using the herein-described techniques that utilize machine learning to produce the user’s fat composition used to determine confidence in the output data or additional diagnosis information. The resulting predictions of fat deposits can then be provided to a postprocessing step to assess robustness of predictions, for example, but not limited to, comparing a secondary predicted variable (like BMI) against precise fat distributions to identify outliers. Further, postprocessing can be used to implement models that make additional downstream predictions (like clinical risk).

[0085] In one example embodiment, training data may comprise image data of user’s representation. The training image data may comprise MRI, DEXA, CT, or ultrasound image data, or image data from a camera, such as a digital camera wherein any of the image data may comprise silhouettes of the user and may further be coronal and/or sagitta silhouettes as disclosed in more detail in the Examples section below.

ML Processes

[0086] Different machine-learning processes have been contemplated to carry out the embodiments discussed herein. For example, linear regression (LiR), logistic regression (LoR), Bayesian networks (for example, naive-bayes), random forest (RF), neural networks (NN) (also known as artificial neural networks), matrix factorization, a hidden Markov model (HMM), support vector machines (SVM), K-means clustering (KMC), K-nearest neighbor (KNN), a suitable statistical machine learning processes, and/or a heuristic machine learning system for classifying or evaluating representations of a user’s body are contemplated.

Linear Regression (LiR)

[0087] In one example embodiment, linear regression machine learning is implemented. LiR is typically used in machine learning to predict a result through the mathematical relationship between an independent and dependent variable, such as representations of a user’ s body and fat composition, respectively. A simple linear regression model would have one independent variable (x) and one dependent variable (y). A representation of an example mathematical relationship of a simple linear regression model would be y = mx + b. In this example, the machine learning algorithm tries variations of the tuning variables m and b to optimize a line that includes all the given training data.

[0088] The tuning variables can be optimized, for example, with a cost function. A cost function takes advantage of the minimization problem to identify the optimal tuning variables. The minimization problem preposes the optimal tuning variable will minimize the error between the predicted outcome and the actual outcome. An example cost function may comprise summing all the square differences between the predicted and actual output values and dividing them by the total number of input values and results in the average square error.

[0089] The machine learning module may use, for example, gradient descent methods to select new tuning variables to reduce the cost function. An example gradient descent method comprises evaluating the partial derivative of the cost function with respect to the tuning variables. The sign and magnitude of the partial derivatives indicate whether the choice of a new tuning variable value will reduce the cost function, thereby optimizing the linear regression algorithm. A new tuning variable value is selected depending on a set threshold. Depending on the machine learning module, a steep or gradual negative slope is selected. Both the cost function and gradient descent can be used with other algorithms and modules mentioned throughout. For the sake of brevity, both the cost function and gradient descent are well known in the art.

[0090] LiR models may have many levels of complexity comprising one or more independent variables. Furthermore, in an LiR function with more than one independent variable, each independent variable may have the same one or more tuning variables or each, separately, may have their own one or more tuning variables. The number of independent variables and tuning variables will be understood to one skilled in the art for the problem being solved. In example embodiments, representations of a user’s body are used as the independent variables to train a LiR machine learning module, which, after training, is used to determine, for example, a user’s fat composition.

Logistic Regression (LoR)

[0091] In one example embodiment, logistic regression machine learning is implemented. Logistic Regression, often considered a LiR type model, is typically used in machine learning to classify information, such as representations of a user’s body into categories such as a user’s fat composition. LoR takes advantage of probability to predict an outcome from input data. However, what makes LoR different from a LiR is that LoR uses a more complex logistic function, for example a sigmoid function. In addition, the cost function can be a sigmoid function limited to a result between 0 and 1. For example, the sigmoid function can be of the form /(x)= l/( l+e'^x), where x represents some linear representation of input features and tuning variables. Similar to LiR, the tuning variable(s) of the cost function are optimized (typically by taking the log of some variation of the cost function) such that the result of the cost function, given variable representations of the input features, is a number between 0 and 1, preferably falling on either side of 0.5. As described in LiR, gradient descent may also be used in LoR cost function optimization and is an example of the process. In example embodiments, representations of a user’s body are used as the independent variables to train a LoR machine learning module, which, after training, is used to determine a user’s fat composition.

Bayesian Network

[0092] In one example embodiment, a Bayesian Network is implemented. BNs are used in machine learning to make predictions through Bayesian inference from probabilistic graphical models. In BNs, input features are mapped onto a directed acyclic graph forming the nodes of the graph. The edges connecting the nodes contain the conditional dependencies between nodes to form a predicative model. For each connected node the probability of the input features resulting in the connected node is learned and forms the predictive mechanism. The nodes may comprise the same, similar, or different probability functions to determine movement from one node to another. The nodes of a Bayesian network are conditionally independent of its nondescendants given its parents thus satisfying a local Markov property. This property affords reduced computations in larger networks by simplifying the joint distribution.

[0093] There are multiple methods to evaluate the inference, or predictability, in a Bayesian network but only two are mentioned for demonstrative purposes. The first method involves computing the joint probability of a particular assignment of values for each variable. The joint probability can be considered the product of each conditional probability and, in some instances, comprises the logarithm of that product. The second method is Markov chain Monte Carlo (MCMC), which can be implemented when the sample size is large. MCMC is a well- known class of sample distribution algorithms and will not be discussed in detail herein. [0094] The assumption of conditional independence of variables forms the basis for Naive Bayes classifiers. This assumption implies there is no correlation between different input features. As a result, the number of computed probabilities is significantly reduced as well as the computation of the probability normalization. While independence between features is rarely true, this assumption exchanges reduced computations for less accurate predictions, however the predictions are reasonably accurate. In example embodiments, representations of a user’s body are mapped to the BN graph to train the BN machine learning module, which, after training, is used to determine a user’s fat composition.

Random Forest

[0095] In one example embodiment, random forest is implemented. RF consists of an ensemble of decision trees producing individual class predictions. The prevailing prediction from the ensemble of decision trees becomes the RF prediction. Decision trees are branching flowchart-like graphs comprising of the root, nodes, edges/branches, and leaves. The root is the first decision node from which feature information is assessed and from it extends the first set of edges/branches. The edges/branches contain the information of the outcome of a node and pass the information to the next node. The leaf nodes are the terminal nodes that output the prediction. Decision trees can be used for both classification as well as regression and is typically trained using supervised learning methods. Training of a decision tree is sensitive to the training data set. An individual decision tree may become over or under-fit to the training data and result in a poor predictive model. Random forest compensates by using multiple decision trees trained on different data sets. In example embodiments, representations of a user’s body are used to train the nodes of the decision trees of a RF machine learning module, which, after training, is used to determine a user’s fat composition.

Neural Networks

[0096] In one example embodiment, Neural Networks are implemented. NNs are a family of statistical learning models influenced by biological neural networks of the brain. NNs can be trained on a relatively-large dataset (e.g., 50,000 or more) and used to determine, approximate, or predict an output that depends on a large number of inputs/features. NNs can be envisioned as so-called “neuromorphic” systems of interconnected processor elements, or “neurons”, and exchange electronic signals, or “messages”. Similar to the so-called “plasticity” of synaptic neurotransmitter connections that carry messages between biological neurons, the connections in NNs that carry electronic “messages” between “neurons” are provided with numeric weights that correspond to the strength or weakness of a given connection. The weights can be tuned based on experience, making NNs adaptive to inputs and capable of learning. For example, an NN for determining local fat depots is defined by a set of input neurons that can be given input data such as representations of a user’s body. The input neuron weighs and transforms the input data and passes the result to other neurons, often referred to as “hidden” neurons. This is repeated until an output neuron is activated. The activated output neuron makes a prediction. In example embodiments, representations of a user’s body are used to train the neurons in a NN machine learning module, which, after training, is used to determine a user’s fat composition.

Deep Learning

[0097] In example embodiments, deep learning is implemented. Deep learning expands the neural network by including more layers of neurons. A deep learning module is characterized as having three “macro” layers: (1) an input layer which takes in the input features, and fetches embeddings for the input, (2) one or more intermediate (or hidden) layers which introduces nonlinear neural net transformations to the inputs, and (3) a response layer which transforms the final results of the intermediate layers to the prediction. In example embodiments, representations of a user’s body are used to train the neurons of a deep learning module, which, after training, is used to determine a user’s fat composition.

Matrix Factorization

[0098] In example embodiments, Matrix Factorization is implemented. Matrix factorization machine learning exploits inherent relationships between two entities drawn out when multiplied together. Generally, the input features are mapped to a matrix F which is multiplied with a matrix R containing the relationship between the features and a predicted outcome. The resulting dot product provides the prediction. The matrix R is constructed by assigning random values throughout the matrix. In this example, two training matrices are assembled. The first matrix X contains training input features, and the second matrix Z contains the known output of the training input features. First the dot product of R and X are computed and the square mean error, as one example method, of the result is determined. The values in R are modulated and the process is repeated in a gradient descent style approach until the error is appropriately minimized. The trained matrix R is then used in the machine learning model. In example embodiments, representations of a user’s body are used to train the relationship matrix R in a matrix factorization machine learning module. After training, the relationship matrix R and input matrix F, which comprises vector representations of a user’s body, results in the prediction matrix P comprising a user’s predicted fat composition.

Hidden Markov Model

[0099] In example embodiments, a hidden Markov model is implemented. A HMM takes advantage of the statistical Markov model to predict an outcome. A Markov model assumes a Markov process, wherein the probability of an outcome is solely dependent on the previous event. In the case of HMM, it is assumed an unknown or “hidden” state is dependent on some observable event. A HMM comprises a network of connected nodes. Traversing the network is dependent on three model parameters: start probability; state transition probabilities; and observation probability. The start probability is a variable that governs, from the input node, the most plausible consecutive state. From there each node i has a state transition probability to node j. Typically the state transition probabilities are stored in a matrix Mij wherein the sum of the rows, representing the probability of state i transitioning to state j, equals 1. The observation probability is a variable containing the probability of output o occurring. These too are typically stored in a matrix N_oj wherein the probability of output o is dependent on state j. To build the model parameters and train the HMM, the state and output probabilities are computed. This can be accomplished with, for example, an inductive algorithm. Next, the state sequences are ranked on probability, which can be accomplished, for example, with the Viterbi algorithm. Finally, the model parameters are modulated to maximize the probability of a certain sequence of observations. This is typically accomplished with an iterative process wherein the neighborhood of states is explored, the probabilities of the state sequences are measured, and model parameters updated to increase the probabilities of the state sequences. In example embodiments, representations of a user’s body are used to train the nodes/states of the HMM machine learning module, which, after training, is used to determine a user’s fat composition.

Support Vector Machine

[0100] In example embodiments, support vector machines are implemented. SVMs separate data into classes defined by n-dimensional hyperplanes (n-hyperplane) and are used in both regression and classification problems. Hyperplanes are decision boundaries developed during the training process of a SVM. The dimensionality of a hyperplane depends on the number of input features. For example, a SVM with two input features will have a linear (1- dimensional) hyperplane while a SVM with three input features will have a planer (2- dimensional) hyperplane. A hyperplane is optimized to have the largest margin or spatial distance from the nearest data point for each data type. In the case of simple linear regression and classification a linear equation is used to develop the hyperplane. However, when the features are more complex a kernel is used to describe the hyperplane. A kernel is a function that transforms the input features into higher dimensional space. Kernel functions can be linear, polynomial, a radial distribution function (or gaussian radial distribution function), or sigmoidal. In example embodiments, representations of a user’s body are used to train the linear equation or kernel function of the SVM machine learning module, which, after training, is used to determine a user’s fat composition.

K-means clustering (KMC)

[0101] In one example embodiment, K-means clustering is implemented. KMC assumes data points have implicit shared characteristics and “clusters” data within a centroid or “mean” of the clustered data points. During training, KMC adds a number of k centroids and optimizes its position around clusters. This process is iterative, where each centroid, initially positioned at random, is re-positioned towards the average point of a cluster. This process concludes when the centroids have reached an optimal position within a cluster. Training of a KMC module is typically unsupervised. In example embodiments, representations of a user’s body are used to train the centroids of a KMC machine learning module, which, after training, is used to determine a user’s fat composition.

K-nearest neighbor (KNN)

[0102] In one example embodiment, K-nearest neighbor is implemented. On a general level, KNN shares similar characteristics to KMC. For example, KNN assumes data points near each other share similar characteristics and computes the distance between data points to identify those similar characteristics but instead of k centroids, KNN uses k number of neighbors. The k in KNN represents how many neighbors will assign a data point to a class, for classification, or object property value, for regression. Selection of an appropriate number of k is integral to the accuracy of KNN. For example, a large k may reduce random error associated with variance in the data but increase error by ignoring small but significant differences in the data. Therefore, a careful choice of k is selected to balance overfitting and underfitting. Concluding whether some data point belongs to some class or property value k. the distance between neighbors is computed. Common methods to compute this distance are Euclidean, Manhattan or Hamming to name a few. In some embodiments, neighbors are given weights depending on the neighbor distance to scale the similarity between neighbors to reduce the error of edge neighbors of one class “out-voting” near neighbors of another class. In one example embodiment, k is 1 and a Markov model approach is utilized. In example embodiments, representations of a user’s body are used to train a KNN machine learning module, which, after training, is used to determine a user’s fat composition.

[0103] To perform one or more of its functionalities, the machine learning module may communicate with one or more other systems. For example, an integration system may integrate the machine learning module with one or more email servers, web servers, one or more databases, or other servers, systems, or repositories. In addition, one or more functionalities may require communication between a user and the machine learning module. [0104] Any one or more of the module described herein may be implemented using hardware (e.g., one or more processors of a computer/machine) or a combination of hardware and software. For example, any module described herein may configure a hardware processor (e.g., among one or more hardware processors of a machine) to perform the operations described herein for that module. In some example embodiments, any one or more of the modules described herein may comprise one or more hardware processors and may be configured to perform the operations described herein. In certain example embodiments, one or more hardware processors are configured to include any one or more of the modules described herein.

[0105] Moreover, any two or more of these modules may be combined into a single module, and the functions described herein for a single module may be subdivided among multiple modules. Furthermore, according to various example embodiments, modules described herein as being implemented within a single machine, database, or device may be distributed across multiple machines, databases, or devices. The multiple machines, databases, or devices are communicatively coupled to enable communications between the multiple machines, databases, or devices. The modules themselves are communicatively coupled (e.g., via appropriate interfaces) to each other and to various data sources, to allow information to be passed between the applications to allow the applications to share and access common data.

[0106] In block 240, the output of the machine learning server 133 is transferred to the diagnosis server 133 via the transfer engine, wherein the output data from the machine learning server is processed into user comprehensible information comprising the user’s fat composition, obesity, and/or risk of associated diseases. In example embodiments, the machine learning and diagnosis server 133 are communicatively coupled and may reside within a local server system or across a network of servers. In example embodiments the machine learning server and diagnosis server may comprise the same server. In example embodiments, the machine learning server and diagnosis server comprise the same server and the machine learning engine and diagnosis engine comprise one engine. In example embodiments, the machine learning engine and diagnosis engine comprise one engine and the output from the machine learning engine comprises user comprehensible information.

[0107] In block 250, the diagnosis engine in the diagnosis server 133 generates user comprehensible information comprising the user’s fat composition, obesity, and/or risk of associated diseases. In example embodiments, the output from the machine learning engine comprises information either incomprehensible by the user or not wanted by the user. In example embodiments, the diagnosis engine processes the output from the machine learning engine into diagnosis information regarding fat composition, obesity, and/or risk of associated diseases. In an example embodiment, the output from the machine learning engine comprises the user’s fat composition such as fat distribution throughout the body, the diagnosis engine may process the information into information pertaining to areas of the body with the largest, smallest, and/or average amount of body fat. In example embodiments, the output information from the machine learning engine comprises user’s fat volume, the diagnosis engine may process the information into the proportion of population the user is categorized. In example embodiments, the machine learning engine outputs fat type, such as visceral, subcutaneous, and/or dermal or any subgroup of adipose tissue. The diagnosis engine may process the information into ratios comprising different fat types and/or combine the amount of body fat in subgroups to create a total amount of group type body fat.

[0108] These determinations can then be used to diagnose obesity or disease risk of the user. Although biochemical and metabolic features differentiate fat and adipose tissue, both will be used interchangeably herein. The fat volume determined by the machine learning system may be described as a numerical value that falls within a range, for example, the range may comprise a first end associated with a normal fat volume and a second end with a severe obesity volume. These ranges may have, for example, levels or groups in between the end ranges. In the previous example, there may be levels/groups such as less overweight after healthy (first end) or obesity before severe obesity (second end). The fat volume may also be described as a percentage, such as percent body fat. In example embodiments, the user’s body fat volume is given as a numerical value associated with a type of body fat. Common types of body fat consist of adipose tissue, which includes visceral adipose tissue (VAT), dermal adipose tissue (DAT), and/or subcutaneous adipose tissue (SAT) depots. VAT may comprises of epicardial VAT (EVAT), omental VAT (OVAT), perirenal VAT (PVAT), retroperitoneal VAT (RVAT), mesenteric VAT (MV AT), and gonadal (GV AT). SAT may comprise cranial SAT (CSAT), upper body SAT (USAT), abdominal SAT (ASAT), gluteal SAT (GSAT), femoral SAT (FSAT), and gluteofemoral adipose tissue (GF AT). In example embodiments, the fat distribution of a user is determined. The fat distribution may be described as a percentage of a particular body fat type or may be a ratio of body fat types such as VAT/ASAT. In one example embodiment, the user’s fat composition is used to compute where the user falls, in the form of a percentage, relative to a group, e.g., a local or global population. For more detail on measuring body fat see e.g., Shuster, A., et al. The Clinical Importance of Visceral Adiposity: A Critical Review of Methods for Visceral Adipose Tissue Analysis. BJR 2012, 85 (1009), 1- 10.

[0109] In block 260, the resulting user’s body fat volume, distribution, and/or type is transmitted back to the user via the network 105. In example embodiments, the resulting user information is stored on the data storage unit 137. In example embodiments, the resulting user information is immediately transmitted to the user’s device. In example embodiments, the resulting user information is transmitted across the network 105 to the data acquisition system for subsequent access by the user associated device 100 or machine learning system 130.

[0110] The amount and distribution of adipose tissue in the body is affected by various physiological and clinical factors. Abdominal obesity, characterized as increased adipose tissue surrounding the intra-abdominal organs, has been linked to diabetes associated problems such as impaired glucose and lipid metabolism and insulin resistance. Furthermore, abdominal obesity has also been linked to increased predisposition colon, breast, and prostate cancer as well as prolonged hospital stays, increased incidence of infections and non-infectious complications, and increased mortality in hospital. In particular, high amounts of VAT, resulting in decreased adiponectin, is associated with Type 2 diabetes, elevated glucose levels, hypertension, and cardiovascular disease. In example embodiments, the user’s fat composition are used to diagnosis the risk for disease or inclination for health complications. Accordingly, in some embodiments, the body fat analysis returned by the diagnosis engine 250, may further include a disease risk assessment based on the determined body fat analysis. [oni] The ladder diagrams, scenarios, flowcharts, and block diagrams in the figures and discussed herein illustrate architecture, functionality, and operation of example embodiments and various aspects of systems, methods, and computer program products of the present invention. Each block in the flowchart or block diagrams may represent the processing of information and/or transmission of information corresponding to circuitry that can be configured to execute the logical functions of the present techniques. Each block in the flowchart or block diagrams may represent a module, segment, or portion of one or more executable instructions for implementing the specified operation or step. In example embodiments, the functions/acts in a block may occur out of the order shown in the figures and nothing requires that the operations be performed in the order illustrated. For example, two blocks shown in succession may be executed concurrently or essentially concurrently. In another example, blocks may be executed in the reverse order. Furthermore, variations, modifications, substitutions, additions, or reduction in blocks and/or functions may be used with any of the ladder diagrams, scenarios, flow charts and block diagrams discussed herein, all of which are explicitly contemplated herein.

[0112] The ladder diagrams, scenarios, flow charts and block diagrams may be combined with one another, in part or in whole. Coordination will depend upon the required functionality. Each block of the block diagrams and/or flowchart illustration as well as combinations of blocks in the block diagrams and/or flowchart illustrations may be implemented by special purpose hardware-based systems that perform the aforementioned functions/acts or carry out combinations of special purpose hardware and computer instructions. Moreover, a block may represent one or more information transmissions and may correspond to information transmissions among software and/or hardware modules in the same physical device and/or hardware modules in different physical devices.

[0113] The present techniques can be implemented as a system, a method, a computer program product, digital electronic circuitry, and/or in computer hardware, firmware, software, or in combinations of them. The computer program product can include a program tangibly embodied in an information carrier ( e.g., computer readable storage medium or media) having computer readable program instructions thereon for execution by, or to control the operation of, data processing apparatus (e.g., a processor) to carry out aspects of one or more embodiments of the present invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

[0114] The computer readable program instructions can be performed on general purpose computing device, special purpose computing device, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via, at least partially, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the functions/acts specified in the flowchart and/or block diagram block or blocks. The processors, either: temporarily or permanently; or partially configured, may comprise processor-implemented modules. The present techniques referred to herein may, in example embodiments, comprise processor-implemented modules. Functions/acts of the processor-implemented modules may be distributed among the one or more processors. Moreover, the functions/acts of the processor-implements modules may be deployed across a number of machines, where the machines may be located in a single geographical location or distributed across a number of geographical locations.

[0115] The computer readable program instructions can also be stored in a computer readable storage medium that can direct one or more computer devices, programmable data processing apparatuses, and/or other devices to carry out the function/acts of the processor- implemented modules. The computer readable storage medium containing all or partial processor-implemented modules stored therein, comprises an article of manufacture including instructions which implement aspects, operations, or steps to be performed of the function/act specified in the flowchart and/or block diagram block or blocks.

[0116] Computer readable program instructions described herein can be downloaded to a computer readable storage medium within a respective computing/processing devices from a computer readable storage medium. Optionally, the computer readable program instructions can be downloaded to an external computer device or external storage device via a network. A network adapter card or network interface in each computing/processing device can receive computer readable program instructions from the network and forward the computer readable program instructions for permanent or temporary storage in a computer readable storage medium within the respective computing/processing device.

[0117] Computer readable program instructions described herein can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code. The computer readable program instructions can be written in any programming language such as compiled or interpreted languages. In addition, the programming language can be object-oriented programming language (e.g. “C++”) or conventional procedural programming languages (e.g. “C”) or any combination thereof may be used to as computer readable program instructions. The computer readable program instructions can be distributed in any form, for example as a stand-alone program, module, subroutine, or other unit suitable for use in a computing environment. The computer readable program instructions can execute entirely on one computer or on multiple computers at one site or across multiple sites connected by a communication network, for example on user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on a remote computer or server. If the computer readable program instructions are executed entirely remote, then the remote computer can be connected to the user's computer through any type of network or the connection can be made to an external computer. In examples embodiments, electronic circuitry including, but not limited to, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions. Electronic circuitry can utilize state information of the computer readable program instructions to personalize the electronic circuitry, to execute functions/acts of one or more embodiments of the present invention.

[0118] Example embodiments described herein include logic or a number of components, modules, or mechanisms. Modules may comprise either software modules or hardware- implemented modules. A software module may be code embodied on a non-transitory machine-readable medium or in a transmission signal. A hardware-implemented module is a tangible unit capable of performing certain operations and may be configured or arranged in a certain manner. In example embodiments, one or more computer systems (e.g., a standalone, client or server computer system) or one or more processors may be configured by software (e.g., an application or application portion) as a hardware-implemented module that operates to perform certain operations as described herein.

[0119] In example embodiments, a hardware-implemented module may be implemented mechanically or electronically. In example embodiments, hardware-implemented modules may comprise permanently configured dedicated circuitry or logic to execute certain functions/acts such as a special-purpose processor or logic circuitry (e.g., a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)). In example embodiments, hardware-implemented modules may comprise temporary programmable logic or circuitry to perform certain functions/acts. For example, a general-purpose processor or other programmable processor.

[0120] The term “hardware-implemented module” encompasses a tangible entity. A tangible entity may be physically constructed, permanently configured, or temporarily or transitorily configured to operate in a certain manner and/or to perform certain functions/acts described herein. Hardware-implemented modules that are temporarily configured need not be configured or instantiated at any one time. For example, if the hardware-implemented modules comprise a general-purpose processor configured using software, then the general-purpose processor may be configured as different hardware-implemented modules at different times. [0121] Hardware-implemented modules can provide, receive, and/or exchange information from/with other hardware-implemented modules. The hardware-implemented modules herein may be communicatively coupled. Multiple hardware-implemented modules operating concurrently, may communicate through signal transmission, for instance appropriate circuits and buses that connect the hardware-implemented modules. Multiple hardware-implemented modules configured or instantiated at different times may communicate through temporarily or permanently archived information, for instance the storage and retrieval of information in memory structures to which the multiple hardware-implemented modules have access. For example, one hardware-implemented module may perform an operation, and store the output of that operation in a memory device to which it is communicatively coupled. Consequently, another hardware-implemented module may, at some time later, access the memory device to retrieve and process the stored information. Hardware-implemented modules may also initiate communications with input or output devices, and can operate on information from the input or output devices.

[0122] In example embodiments, the present techniques can be at least partially implemented in a cloud or virtual machine environment.

Other Example Computing Machines

[0123] FIG. 3 depicts a block diagram of a computing machine 2000 and a module 2050 in accordance with certain examples. The computing machine 2000 may comprise, but is not limited to, remote devices, work stations, servers, computers, general purpose computers, Intemet/web appliances, hand-held devices, wireless devices, portable devices, wearable computers, cellular or mobile phones, personal digital assistants (PDAs), smart phones, smart watches, tablets, ultrabooks, netbooks, laptops, desktops, multi-processor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, network PCs, mini-computers, and any machine capable of executing the instructions.. The module 2050 may comprise one or more hardware or software elements configured to facilitate the computing machine 2000 in performing the various methods and processing functions presented herein. The computing machine 2000 may include various internal or attached components such as a processor 2010, system bus 2020, system memory 2030, storage media 2040, input/output interface 2060, and a network interface 2070 for communicating with a network 2080.

[0124] The computing machine 2000 may be implemented as a conventional computer system, an embedded controller, a laptop, a server, a mobile device, a smartphone, a set-top box, a kiosk, a router or other network node, a vehicular information system, one or more processors associated with a television, a customized machine, any other hardware platform, or any combination or multiplicity thereof. The computing machine 2000 may be a distributed system configured to function using multiple computing machines interconnected via a data network or bus system.

[0125] The one or more processor 2010 may be configured to execute code or instructions to perform the operations and functionality described herein, manage request flow and address mappings, and to perform calculations and generate commands. Such code or instructions could include, but is not limited to, firmware, resident software, microcode, and the like. The processor 2010 may be configured to monitor and control the operation of the components in the computing machine 2000. The processor 2010 may be a general purpose processor, a processor core, a multiprocessor, a reconfigurable processor, a microcontroller, a digital signal processor (“DSP”), an application specific integrated circuit (“ASIC”), tensor processing units (TPUs), a graphics processing unit (“GPU”), a field programmable gate array (“FPGA”), a programmable logic device (“PLD”), a radio-frequency integrated circuit (RFIC), a controller, a state machine, gated logic, discrete hardware components, any other processing unit, or any combination or multiplicity thereof. In example embodiments, each processor 2010 can include a reduced instruction set computer (RISC) microprocessor. The processor 2010 may be a single processing unit, multiple processing units, a single processing core, multiple processing cores, special purpose processing cores, co-processors, or any combination thereof. According to certain examples, the processor 2010 along with other components of the computing machine 2000 may be a virtualized computing machine executing within one or more other computing machines. Processors 2010 are coupled to system memory and various other components via a system bus 2020.

[0126] The system memory 2030 may include non-volatile memories such as read-only memory (“ROM”), programmable read-only memory (“PROM”), erasable programmable read-only memory (“EPROM”), flash memory, or any other device capable of storing program instructions or data with or without applied power. The system memory 2030 may also include volatile memories such as random-access memory (“RAM”), static random-access memory (“SRAM”), dynamic random-access memory (“DRAM”), and synchronous dynamic randomaccess memory (“SDRAM”). Other types of RAM also may be used to implement the system memory 2030. The system memory 2030 may be implemented using a single memory module or multiple memory modules. While the system memory 2030 is depicted as being part of the computing machine 2000, one skilled in the art will recognize that the system memory 2030 may be separate from the computing machine 2000 without departing from the scope of the subject technology. It should also be appreciated that the system memory 2030 is coupled to system bus 2020 and can include a basic input/output system (BIOS), which controls certain basic functions of the processor 2010 and/or operate in conjunction with, a non-volatile storage device such as the storage media 2040.

[0127] In example embodiments, the computing device 2000 includes a graphics processing unit (GPU) 2090. Graphics processing unit 2090 is a specialized electronic circuit designed to manipulate and alter memory to accelerate the creation of images in a frame buffer intended for output to a display. In general, a graphics processing unit 2090 is very efficient at manipulating computer graphics and image processing and has a highly parallel structure that makes it more effective than general-purpose CPUs for algorithms where processing of large blocks of data is done in parallel.

[0128] The storage media 2040 may include a hard disk, a floppy disk, a compact disc read only memory (“CD-ROM”), a digital versatile disc (“DVD”), a Blu-ray disc, a magnetic tape, a flash memory, other non-volatile memory device, a solid state drive (“SSD”), any magnetic storage device, any optical storage device, any electrical storage device, any electromagnetic storage device, any semiconductor storage device, any physical-based storage device, any removable and non-removable media, any other data storage device, or any combination or multiplicity thereof. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any other data storage device, or any combination or multiplicity thereof. The storage media 2040 may store one or more operating systems, application programs and program modules such as module 2050, data, or any other information. The storage media 2040 may be part of, or connected to, the computing machine 2000. The storage media 2040 may also be part of one or more other computing machines that are in communication with the computing machine 2000 such as servers, database servers, cloud storage, network attached storage, and so forth. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0129] The module 2050 may comprise one or more hardware or software elements, as well as an operating system, configured to facilitate the computing machine 2000 with performing the various methods and processing functions presented herein. The module 2050 may include one or more sequences of instructions stored as software or firmware in association with the system memory 2030, the storage media 2040, or both. The storage media 2040 may therefore represent examples of machine or computer readable media on which instructions or code may be stored for execution by the processor 2010. Machine or computer readable media may generally refer to any medium or media used to provide instructions to the processor 2010. Such machine or computer readable media associated with the module 2050 may comprise a computer software product. Each of the operating system, one or more application programs, other program modules, and program data or some combination thereof, may include an implementation of a networking environment. It should be appreciated that a computer software product comprising the module 2050 may also be associated with one or more processes or methods for delivering the module 2050 to the computing machine 2000 via the network 2080, any signal-bearing medium, or any other communication or delivery technology. The module 2050 may also comprise hardware circuits or information for configuring hardware circuits such as microcode or configuration information for an FPGA or other PLD.

[0130] The input/output (“I/O”) interface 2060 may be configured to couple to one or more external devices, to receive data from the one or more external devices, and to send data to the one or more external devices. Such external devices along with the various internal devices may also be known as peripheral devices. The I/O interface 2060 may include both electrical and physical connections for coupling in operation the various peripheral devices to the computing machine 2000 or the processor 2010. The I/O interface 2060 may be configured to communicate data, addresses, and control signals between the peripheral devices, the computing machine 2000, or the processor 2010. The I/O interface 2060 may be configured to implement any standard interface, such as small computer system interface (“SCSI”), serial- attached SCSI (“SAS”), fiber channel, peripheral component interconnect (“PCI”), PCI express (PCIe), serial bus, parallel bus, advanced technology attached (“ATA”), serial ATA (“SATA”), universal serial bus (“USB”), Thunderbolt, FireWire, various video buses, and the like. The I/O interface 2060 may be configured to implement only one interface or bus technology. Alternatively, the VO interface 2060 may be configured to implement multiple interfaces or bus technologies. The I/O interface 2060 may be configured as part of, all of, or to operate in conjunction with, the system bus 2020. The I/O interface 2060 may include one or more buffers for buffering transmissions between one or more external devices, internal devices, the computing machine 2000, or the processor 2010.

[0131] The I/O interface 2060 may couple the computing machine 2000 to various input devices including cursor control devices, touch-screens, scanners, electronic digitizers, sensors, receivers, touchpads, trackballs, cameras, microphones, alphanumeric input devices, any other pointing devices, or any combinations thereof. The I/O interface 2060 may couple the computing machine 2000 to various output devices including video displays (The computing device 2000 may further include a graphics display, for example, a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, a cathode ray tube (CRT), or any other display capable of displaying graphics or video), audio generation device, printers, projectors, tactile feedback devices, automation control, robotic components, actuators, motors, fans, solenoids, valves, pumps, transmitters, signal emitters, lights, and so forth. The I/O interface 2060 may couple the computing device 2000 to various devices capable of input and out, such as a storage unit. The devices can be interconnected to the system bus 2020 via a user interface adapter, which can include, for example, a Super I/O chip integrating multiple device adapters into a single integrated circuit.

[0132] The computing machine 2000 may operate in a networked environment using logical connections through the network interface 2070 to one or more other systems or computing machines across the network 2080. The network 2080 may include a local area network (“LAN”), a wide area network (“WAN”), an intranet, an Internet, a mobile telephone network, storage area network (“SAN”), personal area network (“PAN”), a metropolitan area network (“MAN”), a wireless network (“WiFi;”), wireless access networks, a wireless local area network (“WLAN”), a virtual private network (“VPN”), a cellular or other mobile communication network, Bluetooth, near field communication (“NFC”), ultra-wideband, wired networks, telephone networks, optical networks, copper transmission cables, or, or combinations thereof or any other appropriate architecture or system that facilitates the communication of signals and data. The network 2080 may be packet switched, circuit switched, of any topology, and may use any communication protocol. The network 2080 may comprise routers, firewalls, switches, gateway computers and/or edge servers. Communication links within the network 2080 may involve various digital or analog communication media such as fiber optic cables, free-space optics, waveguides, electrical conductors, wireless links, antennas, radio-frequency communications, and so forth.

[0133] Information for facilitating reliable communications can be provided, for example, as packet/message sequencing information, encapsulation headers and/or footers, size/time information, and transmission verification information such as cyclic redundancy check (CRC) and/or parity check values. Communications can be made encoded/encrypted, or otherwise made secure, and/or decry pted/decoded using one or more cryptographic protocols and/or algorithms, such as, but not limited to, Data Encryption Standard (DES), Advanced Encryption Standard (AES), a Rivest-Shamir-Adelman (RSA) algorithm, a Diffie-Hellman algorithm, a secure sockets protocol such as Secure Sockets Layer (SSL) or Transport Layer Security (TLS), and/or Digital Signature Algorithm (DSA). Other cryptographic protocols and/or algorithms can be used as well or in addition to those listed herein to secure and then decrypt/decode communications.

[0134] The processor 2010 may be connected to the other elements of the computing machine 2000 or the various peripherals discussed herein through the system bus 2020. The system bus 2020 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. For example, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus. It should be appreciated that the system bus 2020 may be within the processor 2010, outside the processor 2010, or both. According to certain examples, any of the processor 2010, the other elements of the computing machine 2000, or the various peripherals discussed herein may be integrated into a single device such as a system on chip (“SOC”), system on package (“SOP”), or ASIC device.

[0135] Examples may comprise a computer program that embodies the functions described and illustrated herein, wherein the computer program is implemented in a computer system that comprises instructions stored in a machine-readable medium and a processor that executes the instructions. However, it should be apparent that there could be many different ways of implementing examples in computer programming, and the examples should not be construed as limited to any one set of computer program instructions. Further, a skilled programmer would be able to write such a computer program to implement an example of the disclosed examples based on the appended flow charts and associated description in the application text. Therefore, disclosure of a particular set of program code instructions is not considered necessary for an adequate understanding of how to make and use examples. Further, those ordinarily skilled in the art will appreciate that one or more aspects of examples described herein may be performed by hardware, software, or a combination thereof, as may be embodied in one or more computing systems. Moreover, any reference to an act being performed by a computer should not be construed as being performed by a single computer as more than one computer may perform the act.

[0136] The examples described herein can be used with computer hardware and software that perform the methods and processing functions described herein. The systems, methods, and procedures described herein can be embodied in a programmable computer, computerexecutable software, or digital circuitry. The software can be stored on computer-readable media. For example, computer-readable media can include a floppy disk, RAM, ROM, hard disk, removable media, flash memory, memory stick, optical media, magneto-optical media, CD-ROM, etc. Digital circuitry can include integrated circuits, gate arrays, building block logic, field programmable gate arrays (FPGA), etc.

[0137] A “server” may comprise a physical data processing system (for example, the computing device 2000 as shown in FIG. 3) running a server program. A physical server may or may not include a display and keyboard. A physical server may be connected, for example by a network, to other computing devices. Servers connected via a network may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a distributed (e.g., peer-to-peer) network environment. The computing device 2000 can include clients’ servers. For example, a client and server can be remote from each other and interact through a network. The relationship of client and server arises by virtue of computer programs in communication with each other, running on the respective computers. [0138] The example systems, methods, and acts described in the examples and described in the figures presented previously are illustrative, not intended to be exhaustive, and not meant to be limiting. In alternative examples, certain acts can be performed in a different order, in parallel with one another, omitted entirely, and/or combined between different examples, and/or certain additional acts can be performed, without departing from the scope and spirit of various examples. Plural instances may implement components, operations, or structures described as a single instance. Structures and functionality that may appear as separate in example embodiments may be implemented as a combined structure or component. Similarly, structures and functionality that may appear as a single component may be implemented as separate components. Accordingly, such alternative examples are included in the scope of the following claims, which are to be accorded the broadest interpretation to encompass such alternate examples. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

[0139] Further embodiments are illustrated in the following Examples which are given for illustrative purposes only and are not intended to limit the scope of the invention. EXAMPLES

Example 1 - Estimating local fat depot volumes using silhouettes in 40,000 people

Introduction

[0140] Obesity, defined within clinical practice solely on the basis of body mass index (BMI), is a leading preventable risk factor for disease and death, with afflicted individuals at increased risk of cardiovascular events, type 2 diabetes, cancer, and severe COVID-19 infection. [1-4] Recent projections suggest that, as early as 2030, 1 in 2 U.S. adults will have obesity and 1 in 4 will have severe obesity (BMI > 35). [5]

[0141] While increased BMI identifies a subpopulation at higher risk of adverse outcomes on average, prior studies have demonstrated considerable heterogeneity within BMI categories. [6-9] This heterogeneity is partly attributable to variation in fat distribution at any given BMI. [10,11] Indeed, prior studies have suggested that different fat depots quantified by medical imaging lead to unique metabolic consequences - visceral adipose tissue (VAT) is strongly associated with increased cardiometabolic risk, while gluteofemoral adipose tissue (GF AT) may have a protective role. [12-15] In an effort to capture some information about fat distribution in clinical practice, calls for the routine measurement of waist circumference have recently been renewed. [16] This is supported by the observation that waist circumference is correlated with VAT volume as well as a robust literature demonstrating the association of waist circumference with cardiovascular and all-cause mortality. [14, 17-19]

[0142] Despite these advances, a large gap remains between BMI and waist circumference - easily measured in clinical practice, but providing limited resolution of fat distribution, and imaging-derived fat depot volumes - enabling complete characterization of fat distribution, but unlikely to be clinically feasible in the near future. An individual’s body shape outline, such as one obtained from a photograph, could represent one solution for closing the gap between these two extremes, enabling clinically significant estimation of fat distribution. Prior seminal studies have suggested that this may be feasible, but have several limitations. [20-28] First, most studies in this area have had the primary goal of predicting overall fat and fat-free mass rather than specific fat depots, which represents a more challenging problem. [20-24] Second, no prior study has aimed to predict MRI-quantified gluteofemoral fat or ratios between fat depots, which are less correlated with overall size and hence may be more challenging to predict. [25-28] Finally, all studies in this area have been limited by sample size up to several hundred participants, limiting the ability to perform robust cross-validation. Results

[0143] In this study, Applicants constructed front- and side-facing silhouettes for 40,032 participants of the UK Biobank from raw MRI imaging data and developed deep learning models to predict adipose tissue volumes. Applicants then investigated the clinical utility of the VAT/ASAT ratio and its relationship to type 2 diabetes and coronary artery disease.

Constructing silhouettes from whole-body magnetic resonance images

[0144] Silhouettes were created from whole-body magnetic resonance images by (1) segmenting the body outline in axial acquisitions using image processing, (2) projecting this 3 -dimensional volume into 2-dimensional images in the coronal (front-to-back orientation) and sagittal (side-by-side orientation) as previously described (ref), and (3) binarizing pixel intensities to either zeros for background or ones for body (Figure 4a and Online Methods). Using these silhouettes as inputs, Applicants trained a deep learning model to jointly predict fat depot volumes for VAT, ASAT, and GF AT (see Online Methods).

[0145] To avoid overfitting, deep learning models should be trained and predicted on nonoverlapping sets of subjects. However, Applicants desired predictions for all subjects in order to maximize data available for downstream analyses. Satisfying these requirements, Applicants employed a nested cross-validation approach where the cohort is split into 5 partitions and trained models using data from three partitions with testing and validation using the remaining partitions (Supplemental Figure X and Online Methods).

[0146] Using this approach, Applicants’ deep learning model using silhouettes achieved remarkably high performance for the fat depot volumes VAT (coefficient of determination R² = 0.885), ASAT (R² = 0.937), and GF AT (R² = 0.933) when compared to truth values measured by MRI (Figure 4b). There were no notable differences in performance across the sexes (Supplemental Figure X-X and Supplemental Tables X-X) across BMI subgroups (Supplemental Table X) or across age groups (Supplemental Table X). Since age and sex are not included in the model during training, Applicants investigated the potential benefit of including these in a linear model but found no significant benefit (Supplemental Table X-X).

Silhouettes outperform anthropometric and non-linear models

[0147] Next, Applicants compared Applicants proposed silhouette-based model to a variety of basic models using the anthropometric traits including hip and waist circumference, height, weight, and BMI, and bioelectric impedance measurements, when combined with age and sex (Figure 4c and Supplemental Tables X-X). As expected, a model comprising only of age and sex is essentially uninformative when it comes to estimating fat depot volumes (females R² range 0.001-0.052 and males 0.001-0.028, Supplemental Table X) as these traits does not describe body morphology.

[0148] A single anthropometric traits could moderately explain fat depot volumes (across- sex R² range for ASAT: 0.72-0.81, VAT: 0.67-0.71, GF AT: 0.57-0.71, Supplemental Table X- X) in combination with age and sex, and could be further improved when including all anthropometric traits: VAT (sex-stratified R² range 0.77-0.87), ASAT (R² range 0.72-0.82), and GF AT (R² range 0.69-0.70)(Supplemental Table X-X). In addition to anthropometric traits, Applicants also investigated the utility of bioelectric impedance measurements which have garnered renewed interest in recent years because of its low cost and its ease to acquire (refs). Notably, a model including five impedance measurements (left and right arm and leg and trunk)(ref) together with age and sex explained very little (female R² range 0.068-0.188 and males 0.048-0.141, Supplemental Table X) raising the utility of this modality into question for estimating distinct fat depot volumes. Collectively, Applicants found that it is not possible to predict local fat depot volumes using anthropometric traits either alone or in combination with bioelectric impedance measurements to high accuracy.

[0149] Applicants additionally compared their silhouette-based model to more sophisticated non-linear and sex-specific models constructed for estimating VAT volumes, including several bioelectric impedance measurements, which was recently proposed by Karlsson and colleagues (ref). Applicants observed good correlations between predicted and MRI-measured VAT volumes using these models (females R² range: X and males R² range: X) but with a poor overall fit (Supplemental Table X-X) most likely because their formulas were modelled using dual-energy X-ray absorptiometry (DXA) measurements compared to MRI used in this study. In addition, Applicants were interested in evaluating these models for the other local fat depots in addition to VAT. Because of this, Applicants fitted new models to each individual fat depot for the MRI data using their sex-specific formulas (Figure 4c and Supplemental Table X) and observed a moderate improvement in performance compared to the best basic anthropometric model for each fat depot (Supplemental Table X-X).

[0150] Taken together, Applicants find that silhouettes outperform other traditional approaches in estimating local fat depot volumes with improvements in R² by 9.4% for VAT, 4.4% for ASAT, 9.1% for GF AT, compared to the next best model for males. For females, Applicants observed improved correlations for VAT 11.4%, 5.6% ASAT, and 9.2% GF AT. Silhouettes to predict VAT/ASAT

[0151] Waist circumference is often considered a proxy for VAT, but the parameter it estimates, central obesity, could be driven by elevated AS AT rather than elevated VAT. [16] A representative pair of age, sex, BMI, and waist circumference-matched participants are shown in Figure 5a with highly discordant abdominal fat distribution. One participant had significantly greater VAT (VAT: 9.2 L, AS AT: 4.5 L), while the other has much more ASAT (VAT: 3.7 L, ASAT: 9.3 L).

[0152] Applicants hypothesized that a deep learning model trained on silhouettes could predict VAT/ASAT ratio considerably better than any combination of anthropometric measures, despite the fact that no information about the anatomical boundary between VAT and ASAT can be directly ascertained from a silhouette. Silhouettes performed well for prediction of VAT/ASAT ratio with sex-combined R² of 0.81 (R² male 0.55; R² female 0.49) (Figure 5b). This was a marked improvement over VAT/ASAT ratio prediction with waist circumference (R² male 0.07; R² female 0.17) and a non-linear model combining anthropometric and impedance measures (R² male 0.19; R² female 0.26).

[0153] Applicants additionally confirmed that waist circumference was strongly correlated with silhouette-predicted VAT (R² male 0.72; R² female 0.76) and silhouette-predicted ASAT (R² male 0.73; R² female 0.74), but nearly independent of silhouette-predicted VAT/ASAT (R² male 0.07, R² female 0.20) (Figure 5c, Supplemental Figures X-X — correlograms).

Silhouette-predicted VAT/ASAT associates with cardiometabolic diseases

[0154] Given the relative independence of silhouette-predicted VAT/ASAT with respect to BMI and waist circumference, and prior work demonstrating that VAT, but not necessarily ASAT, increases cardiometabolic risk, Applicants investigated disease associations of silhouette-predicted VAT/ASAT. In logistic regression models adjusted for age, sex, and imaging center, silhouette-predicted VAT/ASAT was associated with increased prevalence of type 2 diabetes (OR/SD 1.86; 95% CI 1.79-1.93), with smaller effect size than waist circumference (OR/SD 2.04; 95% CI: 1.95-2.13) (Figure 6a). When each model included BMI as a covariate, silhouette-predicted VAT/ASAT (OR/SD 1.78; 95% CI 1.71-1.86) demonstrated a comparable effect size to waist circumference (OR/SD 1.76; 95% CI 1.61- 1.93). Finally, when both measurements and BMI were included in a single model, silhouette- predicted VAT/ASAT (OR/SD 1.73; 95% CI 1.66-1.80) was more strongly associated with type 2 diabetes than waist circumference (OR/SD 1.33; 95% CI 1.21-1.47). Similar trends were observed with coronary artery disease: in a mutually adjusted model with BMI, silhouette- predicted VAT/ASAT positively associated with disease (OR/SD 1.26; 95% CI 1.21-1.32), while waist circumference had a trend towards a protective effect (OR/SD 0.89; 95% CI 0.80- 0.98). In models that included MRI-measured VAT/ASAT in lieu of silhouette-predicted VAT/ASAT, a similar pattern was observed with the effect size of VAT/ASAT being only mildly attenuated by BMI and waist circumference adjustment (Supplemental Figures X-X - forest plot).

[0155] Applicants next set out to understand the gradients of absolute prevalence rates according to silhouette-predicted VAT/ASAT using sex-specific standardized estimates for the lowest quintile, quintiles 2-4, and the highest quintile. Applicants estimated prevalence rates for males and females separately across clinical BMI categories of normal, overweight, obese, and severely obese participants with either normal or elevated waist circumference (cite data for BMI specific waist cutoffs). This analysis revealed substantial gradients in prevalence of cardiometabolic diseases according to silhouette-predicted VAT/ASAT quintiles within BMI and waist circumference bins (Figure 6b). For example, men with overweight BMI and nonelevated waist circumference with silhouette-predicted VAT/ASAT in the top quintile had a higher probability of type 2 diabetes 9.5% 95% CI 8.6-10.4%) compared to both (a) men with overweight BMI and elevated waist circumference with silhouette-predicted VAT/ASAT in the bottom quintile (3.7% 95% CI 3.0-4.5%) and (b) men with obese BMI and non-elevated waist circumference with silhouette-predicted VAT/ASAT in the bottom quintile (4.2% 95% CI 3.4-5.1%). Similar trends with attenuated risk gradients were observed with coronary artery disease (Supplementary Figure X).

[0156] Over a median follow-up of 1.9 years, 165 (0.4%) and 393 (1.1%) participants had a new diagnosis of Type 2 diabetes or coronary artery disease recorded in the electronic health record. Silhouette-predicted VAT/ASAT and waist circumference associations with incident disease were consistent with prevalent analyses in mutually adjusted logistic regressions including BMI. For incident type 2 diabetes, silhouette-predicted VAT/ASAT carried a greater effect size (HR/SD 1.43; 95% CI 1.26-1.63) than waist circumference (HR/SD 1.19; 95% CI 0.88-1.60) (Supplementary Table X). Effect sizes for incident coronary artery disease were HR/SD 1.17; 95% CI 1.06-1.28 for silhouette-predicted VAT/ASAT and HR/SD 1.13; 95% CI 0.92-1.39) for waist circumference. Discussion

[0157] In this study, Applicants used a deep learning model trained on simple silhouettes to estimate VAT, ASAT, GF AT, and VAT/ASAT ratio in 40,032 individuals. Applicants demonstrated that silhouette-based predictions outperformed any combination of anthropometric and body fat impedance measures, with the greatest performance improvements observed with VAT/ASAT ratio, a parameter independent of BMI. Silhouette- predicted VAT/ASAT ratio was associated with increased risk of Type 2 diabetes and coronary artery disease independent of BMI and waist circumference.

[0158] These results have at least three implications. First, machine learning models trained on easy-to-acquire, less data-rich imaging modalities can be used to close the gap between MRI-derived phenotypes and clinical practice. In this study, simple silhouettes as inputs into a deep learning model, despite substantial information loss compared to MRI, enabled prediction of VAT, ASAT, GF AT, and VAT/ASAT ratio to a standard that significantly outperformed any combination of anthropometric and bioimpedance measurements. Several recent studies have applied machine learning to define image-derived phenotypes from cardiac and liver MRI to enable new biological insights (ref aortic size, liver fat, cardiac trabecular structure). For the subset of these image-derived phenotypes with clinical significance, this work highlights one strategy for moving closer to clinical implementation . For example, certain cardiac MRI measurements could conceivably be estimated using deep learning models trained on ECG (ref?).

[0159] Second, these results support an important distinction between measurements that are correlated with overall body size, or general adiposity, and those that are relatively independent of overall body size, reflecting local adiposity. VAT, ASAT, and GF AT volumes are all highly correlated with BMI. Hence, a predictive model based on BMI will offer apparently good predictive performance for any of these fat depots, but the fat depot variance explained by such a model will primarily be the component related to the participant’s overall size. Waist circumference is more strongly correlated with central obesity than BMI, but remains strongly correlated with BMI and is unable to distinguish between VAT and ASAT, which are known to have depot-specific and divergent effects on cardiometabolic risk (ref). Consistent with these limitations, a model combining all anthropometric and bioimpedance measures offered little predictive power for VAT/ASAT ratio, a local adiposity measure that is nearly independent of BMI. In contrast, Applicants’ deep learning model trained on silhouettes showed modest to good predictive performance for VAT/ASAT ratio, suggesting that information about local adiposity - orthogonal to information obtainable from BMI and waist circumference - is learned. This is supported by Applicants’ observations that disease associations with silhouette-predicted VAT/ASAT ratio are only mildly attenuated when adjusted for BMI and waist circumference. In light of renewed calls for the routine measurement of waist circumference to better stratify cardiometabolic risk associated with body habitus, Applicants’ work suggests that silhouette-predicted VAT/ASAT ratio could serve as a third adiposity-related cardiometabolic predictor alongside BMI and waist circumference in clinical practice.

[0160] Third, the methodology outlined here could be leveraged to measure change in local adiposity in response to lifestyle or pharmacotherapy changes at scale. Most studies for obesity treatments to date have focused on BMI reduction as a primary outcome, but at least two classes of drugs appear to have a selective VAT reduction effect in clinical trials: thiazolidinediones and a synthetic form of growth hormone releasing hormone (ref). Silhouette-estimated VAT and VAT/ASAT ratio could offer a cheap, scalable way to measure changes in local adiposity in response to these and other interventions in future clinical trials. Of note, several products that are marketed as offering precise information about body habitus are available to the consumer but, to Applicants knowledge, none of these products are able to estimate specific fat depot volumes (refs Amazon, Nakedlabs). - Screening for lipodystrophy-like physiology contributing to insulin resistance (has been suggested that FPLD1 may be a polygenic trait; PhysRev lipodystrophy review, Lotta 53 SNP PRS for IR)

[0161] This study is subject to several limitations. First, the majority of participants in the UK Biobank are white, and the imaged subcohort studied here is of mean age 65. Additional studies are needed to assess the generalizability of these results in ancestrally diverse populations and younger individuals. Second, silhouettes in this study were derived by taking the outline of whole-body MRI, rather than from a more cost-effective modality such as a photograph (ref). An analogous study that utilizes silhouettes obtained from photographs taken with mobile phones will likely require more complex modeling to account for heterogeneity in photography technique. Third, Applicants were unable to assess the accuracy of silhouettes in estimating fat depot volumes over time due to short follow-up in the UK Biobank after the first date of imaging. Future work will investigate the utility of obtaining multiple silhouette- predicted fat depot estimates over time. [0162] In conclusion, Applicants demonstrate that simple whole-body silhouettes are sufficiently informative to predict VAT, ASAT, GF AT, and the VAT/ASAT ratio and that silhouette-predicted VAT/ASAT ratio is associated with cardiometabolic disease independent of BMI and waist circumference. These observations pave the way for the first direct-to- consumer smartphone applications with clinically significant prediction of fat depot volumes.

Online Methods

[0163] Here Applicants provide an overview of the methods used in this manuscript that are explained in more detail below. Applicants converted whole-body magnetic resonance images from the UK Biobank into binary silhouettes in two projected views: (1) coronal (frontfacing), and (2) sagittal (side-to-side facing). Applicants then trained deep learning models predicting previously MRI-estimated fat depot volumes of ASAT, VAT, GFAT, and VAT/ASAT using Applicants binary silhouettes. Applicants analyzed the predictive relationships between these “gold standard”-estimated quantitative measurements of localized fat depots and silhouettes. Applicants also analyzed their relationships with cardiovascular disease and diabetes.

[0164] Statistical analyses were conducted with R version 3.6 (R Foundation for Statistical Computing, Vienna, Austria).

Study population

[0165] All analyses were conducted in the UK Biobank, a richly phenotyped, prospective, population-based cohort that recruited 500,000 individuals aged 40-69 in the UK from 2006- 2010. Applicants analyzed 40,032 participants with previously estimated local fat depot volumes by whole-body MRI (ref). This analysis of data from the UK Biobank was approved by the Mass General Brigham institutional review board and was performed under UK Biobank application #7089.

Preparing silhouettes from whole-body magnetic resonance images

[0166] Whole-body MRI data was preprocessed as previously described (ref). In short, whole-body MRIs were acquired in 6 separate stages with varying resolutions which require preprocessing before merging into 3D volumes. Resolutions ranged from 2.232 x 2.232 x 4.5 mm³ (stages 2-4) to 2.232 x 2.232 x 3.0 mm³ (stage 1). Applicants resampled each series to the highest available resolution (voxel = 2.232 x 2.232 x 3.0 mm³) followed by deduping overlapping acquisitions and merging into 3D volumes. There were four available phase acquisitions, (1) in-phase, (2) out-of-phase, (3) water phase, and (4) fat phase. For this study, Applicants used only the fat-phase acquisition to segment axial slices.

[0167] Silhouettes were then computed from the resampled axial slices in the merged 3D volume as follows. First, the contrast of axial images were enhanced with contrast limited adaptive histogram equalization (CLAHE) with the parameters limit = 2 and tile size = (8, 8). Next, contrast-enhanced images were thresholded with Otsu's method (ref) and all connected components were identified and sorted by their area. There are several situations where multiple segmentations should be returned, for example in axial images involving the legs. In contrast, there are situations where multiple segmentations are undesirable, for example, when the arms are visible in the axial slices or artefacts are inadvertently segmented. In order to distinguish between these two cases, Applicants employed a heuristic such that if the ratio between the top two largest segmentations are >= 0.25 they were both kept and returned. In all other cases, only the largest segmentation was returned. Next, returned segmentations were flood filled such that their interior was completely white. Stacking all these segmentations result in a 3D volume of equal size as the input 3D volume of MRI images.

[0168] Coronal and sagittal two-dimensional projections were generated by computing the mean intensity projection in each orientation of the segmented 3D volume. For example, a given pixel on a coronal two-dimensional projection represents the mean intensity across all pixels making up a line oriented in the anterior-posterior direction perpendicular to the coronal plane. This procedure results in a surface map of each participant. Pixel intensities were rounded and then converted into zeros for background and ones for body. The final coronal and sagittal silhouettes were then concatenated side-by-side and resized to 237 x 256 pixels for downstream applications.

Deep learning to predict fat depot volumes using silhouettes

[0169] For regressing the target local fat depot volumes, Applicants employed the DenseNet-121 architecture (ref) as the base model. In short, DenseNets are constructed with two principal building blocks: (1) dense blocks comprising of batch normalization, the nonlinear ReLU activation function, and 3x3 convolutions of increasing number of channels that are propagated from previous layers to enable efficient gradient flow; and (2) transition blocks that compress the number of channels by half using channel-wise convolutions (1x1), and performs a spatial reduction by a factor of 2 by using an average pooling layer of stride 2 and pool size 2. In Applicants model, the channel output of the last dense block convolution was flattened using a global average pooling (ref) layer and then fed into a 512-dimensional fully connected layer that then split into three arms, one for each fat depot (VAT, ASAT, and GF AT). Each arm comprised of two fully connected layers of size 128 and 32, with the ReLu non-linearity as their activation functions, followed by a single-dimensional linear output layer. For VAT/ASAT, the 32-dimensional fully connected layers from the VAT and ASAT arms were concatenated resulting in a 64-dimensional latent space that was processed as above with two fully connected layers of size 128 and 32 followed by a linear regression output. In summary, Applicants constructed a hierarchical multi-task model that jointly predict VAT, ASAT, GF AT, and VAT/ASAT.

[0170] The input for this model were the coronal and sagittal binary silhouettes placed side- by-side with the shape 237 x 256 x 1. The models were trained with the Adam (ref) optimizer with a learning rate set to a cosine decay policy decaying from 0.0001 to 0 over 100 epochs, weight decay of 0.0001, shrinkage loss (ref) with the hyperparameters a = 10.0 and c = 0.2 as the loss function, and a batch size of 32. No additional hyperparameter search or ablation studies were performed.

[0171] For all training data, the following augmentations (random permutations of the training images) were applied: random shifts in the XY-plane by up to ±16 pixels and rotations by up to ±5 degrees around its center axis.

Model predictions

[0172] To avoid overfitting, deep learning models should be trained and predicted on nonoverlapping sets of subjects. However, all subjects should receive predictions. Satisfying these requirements, Applicants employed a nested cross-validation approach where the cohort is split into 5 partitions and train models using data from three partitions and testing and validation using the remaining partitions. For each current fold, cross-validation was performed within the four partitions that can be used for training and testing. This resulted in four models for each fold for a total of 20 models. Final performance metrics were reported as the meanensemble of the four inner models in each outer partition.

[0173] In order to compare performance of the deep learning models using this approach to that of the simpler linear and non-linear models, Applicants report performance using the same nested cross-validation approach across all models. For the non-deep learning models, errors were estimated by repeating this procedure 100 times. Outcome Definitions

[0174] The primary outcomes were prevalent and incident type-2 diabetes and coronary artery disease. Type-2 diabetes was defined on the basis of ICD-10 codes, self-report during a verbal interview with a trained nurse, use of diabetes medication, or a hemoglobin A1C greater than 6.5% before the date of imaging (Supplemental Table X). Coronary artery disease was defined as myocardial infarction, angina, revascularization (percutaneous coronary intervention and/or coronary artery bypass grafting), or death from CAD as determined on the basis of ICD-10 codes, ICD-9 codes, OPCS-4 surgical codes, nurse interview, and national death registries (Supplemental Table X).

Data availability

[0175] UK Biobank data are made available to researchers from research institutions with genuine research inquiries, following IRB and UK Biobank approval. All other data are contained within the article and its supplementary information, or are available upon reasonable request to the corresponding author.

Code availability

[0176] Code used to ingest silhouettes from whole-body Dixon MRIs and train deep learning models are available at github.com/broadinstitute/ml4h under an open-source BSD license.

References

[0177] 1. Kivimaki M, Kuosma E, Ferrie JE, et al. Overweight, obesity, and risk of cardiometabolic multimorbidity: pooled analysis of individual-level data for 120 813 adults from 16 cohort studies from the USA and Europe. Lancet Public Health 2017;2(6):e277-85.

[0178] 2. Calle EE. Overweight, Obesity, and Mortality from Cancer in a Prospectively

Studied Cohort of U.S. Adults. N Engl J Med 2003; 14.

[0179] 3. Anderson MR, Geleris J, Anderson DR, et al. Body Mass Index and Risk for

Intubation or Death in SARS-CoV-2 Infection : A Retrospective Cohort Study. Ann Intern Med 2020;173(10):782-90.

[0180] 4. Gonzalez-Muniesa P, Martinez-Gonzalez M-A, Hu FB, et al. Obesity. Nat Rev

Dis Primer 20I7;3(l): 1-18.

[0181] 5. Ward ZJ, Bleich SN, Cradock AL, et al. Projected U.S. State-Level Prevalence of Adult Obesity and Severe Obesity. N Engl J Med 2019;381(25):2440-50. [0182] 6. Karelis AD, St-Pierre DH, Conus F, Rabasa-Lhoret R, Poehlman ET. Metabolic and Body Composition Factors in Subgroups of Obesity: What Do We Know? J Clin Endocrinol Metab 2004;89(6):2569-75.

[0183] 7. McLaughlin T, Abbasi F, Lamendola C, Reaven G. Heterogeneity in the prevalence of risk factors for cardiovascular disease and type 2 diabetes mellitus in obese individuals: effect of differences in insulin sensitivity. Arch Intern Med 2007;167(7):642-8.

[0184] 8. Wildman RP, Muntner P, Reynolds K, et al. The obese without cardiometabolic risk factor clustering and the normal weight with cardiometabolic risk factor clustering: prevalence and correlates of 2 phenotypes among the US population (NHANES 1999-2004). Arch Intern Med 2008; 168(15): 1617-24.

[0185] 9. Mathew H, Farr OM, Mantzoros CS. Metabolic health and weight:

Understanding metabolically unhealthy normal weight or metabolically healthy obese patients. Metabolism 2016;65(l):73-80.

[0186] 10. Stefan N, Schick F, Haring H-U. Causes, Characteristics, and Consequences of

Metabolically Unhealthy Normal Weight in Humans. Cell Metab 2017;26(2):292-300.

[0187] 11. Stefan N. Causes, consequences, and treatment of metabolically unhealthy fat distribution. Lancet Diabetes Endocrinol 2020;8(7):616-27.

[0188] 12. Ashwell M, Cole TJ, Dixon AK. Obesity: new insight into the anthropometric classification of fat distribution shown by computed tomography. Br Med J Clin Res Ed 1985;290(6483): 1692-4.

[0189] 13. Tchernof A, Despres J-P. Pathophysiology of Human Visceral Obesity: An

Update. Physiol Rev 2013;93(l):359-404.

[0190] 14. Neeland IJ, Ross R, Despres J-P, et al. Visceral and ectopic fat, atherosclerosis, and cardiometabolic disease: a position statement. Lancet Diabetes Endocrinol 2019;7(9):715- 25.

[0191] 15. Agrawal S, Klarqvist MDR, DiamantN, et al. Association of machine learning- derived measures of body fat distribution in >40,000 individuals with cardiometabolic diseases. medRxiv 2021;2021.05.07.21256854.

[0192] 16. Ross R, Neeland IJ, Yamashita S, et al. Waist circumference as a vital sign in clinical practice: a Consensus Statement from the IAS and ICCR Working Group on Visceral Obesity. Nat Rev Endocrinol 2020; 16(3): 177-89. [0193] 17. Song X, Jousilahti P, Stehouwer CD A, et al. Comparison of various surrogate obesity indicators as predictors of cardiovascular mortality in four European populations. Eur J Clin Nutr 2013;67(12): 1298-302.

[0194] 18. Pischon T, Boeing H, Hoffmann K, et al. General and abdominal adiposity and risk of death in Europe. N Engl J Med 2008;359(20):2105-20.

[0195] 19. Jacobs EJ, Newton CC, Wang Y, et al. Waist circumference and all-cause mortality in a large US cohort. Arch Intern Med 2010;170(15): 1293-301.

[0196] 20. Xie B, Avila JI, Ng BK, et al. Accurate body composition measures from wholebody silhouettes. Med Phys 2015;42(8):4668-77.

[0197] 21. Tian IY, Ng BK, Wong MC, et al. Predicting 3D body shape and body composition from conventional 2D photography. Med Phys 2020;47(12):6232-45.

[0198] 22. Affuso O, Pradhan L, Zhang C, et al. A method for measuring human body composition using digital images. PloS One 2018; 13(1 l):e0206430.

[0199] 23. Kennedy S, Hwaung P, Kelly N, et al. Optical imaging technology for body size and shape analysis: evaluation of a system designed for personal use. Eur J Clin Nutr 2020;74(6):920-9.

[0200] 24. Ng BK, Hinton BJ, Fan B, Kanaya AM, Shepherd JA. Clinical anthropometries and body composition from 3D whole-body surface scans. Eur J Clin Nutr 2016;70(l 1): 1265- 70.

[0201] 25. Ng BK, Sommer MJ, Wong MC, et al. Detailed 3-dimensional body shape features predict body composition, blood metabolites, and functional strength: the Shape Up! studies. Am J Clin Nutr 2019; 110(6): 1316-26.

[0202] 26. Sun J, Xu B, Lee J, Freeland-Graves JH. Novel Body Shape Descriptors for

Abdominal Adiposity Prediction Using Magnetic Resonance Images and Stereovision Body Images. Obesity 2017;25(10): 1795-801.

[0203] 27. Lee JJ, Freeland-Graves JH, Pepper MR, Yu W, Xu B. Efficacy of thigh volume ratios assessed via stereovision body imaging as a predictor of visceral adipose tissue measured by magnetic resonance imaging. Am J Hum Biol Off J Hum Biol Counc 2015;27(4):445-57.

[0204] 28. Wang Q, Lu Y, Zhang X, Hahn JK. A Novel Hybrid Model for Visceral Adipose

Tissue Prediction using Shape Descriptors. Annu Int Conf IEEE Eng Med Biol Soc IEEE Eng Med Biol Soc Annu Int Conf 2019;2019: 1729-32.

*** [0205] Various modifications and variations of the described methods, pharmaceutical compositions, and kits of the invention will be apparent to those skilled in the art without departing from the scope and spirit of the invention. Although the invention has been described in connection with specific embodiments, it will be understood that it is capable of further modifications and that the invention as claimed should not be unduly limited to such specific embodiments. Indeed, various modifications of the described modes for carrying out the invention that are obvious to those skilled in the art are intended to be within the scope of the invention. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the invention and including such departures from the present disclosure come within known customary practice within the art to which the invention pertains and may be applied to the essential features herein before set forth.

Claims

57 CLAIMS

1. A computer-implemented method to determine body fat composition from imaging data, comprising: a) receiving, by a user device, one or more images of a subject from an imaging device, the user device communicatively coupled with an acquisition engine; b) transferring, by the acquisition engine, the one or more images to a deployed machine learning network communicatively coupled to the acquisition engine; c) processing the one or more images with the deployed machine learning network, the deployed machine learning network generated and deployed from a training machine learning network; and d) transferring, by the deployed machine learning network, the processed one or more images as output to a diagnosis engine communicatively coupled to the deployed machine learning network; e) generating, by the diagnosis engine, a body fat composition analysis; and f) transmitting, by the diagnosis engine, the body fat composition analysis to a user device associated with a user, the diagnosis engine being communicatively coupled to the user device.

2. The computer-implemented method of claim 1, further comprising transferring, by the acquisition engine, the one or more images to a reconstruction engine communicatively coupled to the acquisition engine, before transferring the one or more images to the machine learning network, and reconstructing the one or more images with the reconstruction engine.

3. The computer-implemented method of claim 1, wherein the one or more images comprise water or fat phases of MRI image, grayscale DEXA image, CT image, or ultrasound images.

4. The computer-implemented method of claim 2, wherein the images are silhouettes or converted to silhouettes.

5. The computer-implemented method of claim 4, wherein the images are coronal silhouettes and/or sagittal silhouettes. 58

6. The computer-implemented method of claim 1, wherein the machine learning comprises unsupervised learning, supervised learning, semi-supervised learning, reinforcement learning, transfer learning, incremental learning, curriculum learning, and learning to learn.

7. The computer-implemented method of claim 6, herein the machine learning method further comprises linear classifiers, logistic classifiers, Bayesian networks, random forest, neural networks, matrix factorization, hidden Markov model, support vector machine, K- means clustering, or K-nearest neighbor.

8. The computer-implemented method of claim 7, herein the neural network method is a deep learning method.

9. A computer-implemented method of training a machine learning process, wherein the images of any one of claims 1-5 are used to train the machine learning methods of any one of claims 6-8.

10. The method of claim 1, wherein the fat composition comprises adipose tissue volume, adipose tissue distribution, adipose tissue type, and/or BMI.

11. The method of claim 10, wherein the adipose tissue comprises one or more of visceral adipose tissue (VAT), dermal adipose tissue (DAT), and/or subcutaneous adipose tissue (SAT) depots, or any combination thereof, such as ratios of these parameters or values adjusted for BMI or similar characteristics.

12. The method of claim 11, wherein the VAT comprises one or more of epicardial VAT (EV AT), omental VAT (OVAT), perirenal VAT (PVAT), retroperitoneal VAT (RVAT), mesenteric VAT (MV AT), gonadal (GVAT).

13. The method of claim 11, wherein the SAT comprises cranial SAT (CSAT), upper body SAT (USAT), abdominal SAT (ASAT), gluteal SAT (GSAT), femoral SAT (FSAT) 59

14. A system to measure body fat composition, comprising: a storage device; and a processor communicatively coupled to the storage device, wherein the processor executes application code instructions that are stored in the storage device to cause the system to: a) receive one or more images of a subject from a user device by an acquisition engine communicatively coupled to the user device; b) transfer the one or more images with the acquisition engine communicatively coupled to a deployed machine learning network; c) process the one or more images with a deployed machine learning network, the deployed machine learning network generated and deployed from a training machine learning network; d) transfer the processed one or more images as output to a diagnosis engine communicatively coupled to the deployed machine learning network; e) generate a body fat composition analysis with the diagnosis engine communicatively coupled to the deployed machine learning network; and f) transmit the body fat composition analysis to a device associated with a user.

15. The system of claim 14, wherein the one or more images are first transferred by the acquisition engine to the reconstruction engine communicatively coupled to the acquisition engine and reconstructing the one or more images with the reconstruction engine.

16. The system of claim 14, wherein the one or more images comprise water or fat phases of MRI images, grayscale DEXA image, CT image, or ultrasound image.

17. The system of claim 15, wherein the images are silhouettes or converted to silhouettes.

18. The system of claim 17, wherein the images are coronal silhouettes and/or sagittal silhouettes.

19. The system of claim 14, wherein the machine learning comprises unsupervised learning, supervised learning, semi-supervised learning, reinforcement learning, transfer learning, incremental learning, curriculum learning, and learning to learn. 60

20. The system of claim 19, wherein the machine learning method further comprises linear classifiers, , logistic classifiers, Bayesian networks, random forest, neural networks, matrix factorization, hidden Markov model, support vector machine, K-means clustering, or K-nearest neighbor.

21. The system of claim 20, wherein the neural network method comprises a deep learning method.

22. The system of claim 14, wherein the fat composition comprises adipose tissue volume, adipose tissue distribution, adipose tissue type, and/or BMI.

23. The system of claim 22, wherein the adipose tissue comprises one or more of visceral adipose tissue (VAT), dermal adipose tissue (DAT), and/or subcutaneous adipose tissue (SAT) depots, or any combination thereof, such as ratios of these parameters or values adjusted for BMI or similar characteristics.

24. The system of claim 23, wherein the VAT comprises one or more of epicardial VAT (EV AT), omental VAT (OVAT), perirenal VAT (PVAT), retroperitoneal VAT (RVAT), mesenteric VAT (MVAT), gonadal (GV AT).

25. The system of claim 23, wherein the SAT comprises cranial SAT (CSAT), upper body SAT (US AT), abdominal SAT (AS AT), gluteal SAT (GSAT), femoral SAT (FSAT).

61

26. A computer program product, comprising: a non-transitory computer-readable storage device having computer-executable program instructions embodied thereon that when executed by a computer cause the computer to measure body fat composition of a subject, the computer-executable program instructions comprising: a) computer-executable program instructions to receive one or more images of a subject from a user device; b) computer-executable program instructions to transfer the one or more images with an acquisition engine communicatively coupled to the user device to a deployed machine learning network; c) computer-executable program instructions to process the one or more images with the deployed machine learning network, the deployed machine learning network generated and deployed from a training machine learning network and communicatively coupled to the acquisition engine; g) computer-executable program instructions to transfer the processed one or more images as output to a diagnosis engine communicatively coupled to the deployed machine learning network; d) computer-executable program instructions to generate a fat composition analysis with a diagnosis engine communicatively coupled to the deep learning network; and e) computer-executable program instructions to transmit the body fat composition analysis to the user.

27. The computer program of claim 26, wherein the one or more images are first transferred by the acquisition engine to the reconstruction engine communicatively coupled to the acquisition engine and reconstructing the one or more images with the reconstruction engine;

28. The computer program product of claim 26, wherein the one or more images comprise water or fat phases of MRI images, grayscale DEXA image, CT image, or ultrasound image.

29. The computer program product of claim 27, wherein the images are silhouettes or are converted to silhouettes.

30. The computer program product medium of claim 29, wherein the images are coronal silhouettes and/or sagittal silhouettes.

31. The computer program product of claim 26, wherein the machine learning comprises unsupervised learning, supervised learning, semi-supervised learning, reinforcement learning, transfer learning, incremental learning, curriculum learning, and learning to learn.

32. The computer program product of claim 27, wherein the machine learning method further comprises linear classifiers, logistic classifiers, Bayesian networks, random forest, neural networks, matrix factorization, hidden Markov model, support vector machine, K- means clustering, or K-nearest neighbor.

33. The computer program product of claim 32, wherein the neural network method is a deep learning method.

34. The computer program product of claim 26, wherein the fat composition comprises adipose tissue volume, adipose tissue distribution, adipose tissue type, and/or BMI.

35. The computer program product of claim 34, wherein the adipose tissue comprises one or more of visceral adipose tissue (VAT), dermal adipose tissue (DAT), and/or subcutaneous adipose tissue (SAT) depots, or any combination thereof, such as ratios of these parameters or values adjusted for BMI or similar characteristics.

36. The computer program product of claim 35, wherein the VAT comprises one or more of epicardial VAT (EV AT), omental VAT (OVAT), perirenal VAT (PVAT), retroperitoneal VAT (RVAT), mesenteric VAT (MVAT), gonadal (GVAT).

37. The computer program product of claim 35, wherein the SAT comprises cranial SAT (CSAT), upper body SAT (USAT), abdominal SAT (ASAT), gluteal SAT (GSAT), femoral SAT (FSAT).