EP4677546A1 - Image based reconstruction of 3d landmarks for use in generation of personalized head-related transfer functions - Google Patents

Abstract

Techniques for 3D reconstruction of anthropometric features from images of a subject, comprising obtaining a set of images of the subject and for each image in the set detecting 2D locations of a plurality of anthropometric landmarks, and estimating a head pose based on the 2D locations. The techniques may further include generating relative 3D coordinates corresponding to at least some of the anthropometric features and generating scaled 3D coordinates from the relative 3D coordinates based on properties associated with the image capturing device and an estimated distance between the subject and the image capturing device. Such techniques provide an efficient way to obtain scaled 3D anthropometric features using a series of images obtained by simple means, e.g. using a mobile camera device such as a smartphone. The anthropometric features are obtained by the detection of ear landmarks and/or face landmarks.

Description

IMAGE BASED RECONSTRUCTION OF 3D LANDMARKS FOR USE IN GENERATION OF

PERSONALIZED HEAD-RELATED TRANSFER FUNCTIONS

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority from PCT Application No. PCT/CN2023/080861, filed on March 10, 2023, PCT Application No. PCT/CN2023/082495, filed on March 20, 2023, U.S. Provisional Application No. 63/499,759, filed on May 3, 2023 and U.S. Provisional Application No. 63/507,498, filed on June 12, 2023, each of which is incorporated by reference in its entirety.

TECHNICAL FIELD

[0002] The present invention relates to image-based 3D reconstruction of anthropometric features, in particular head, face and ear features. Such anthropometric features may be used for generation of personalized head-related transfer functions (pHRTFs).

BACKGROUND

[0003] Head Related Transfer Functions (HRTFs) are a set of functions describing how human ears receive sound from sources at varying directions of arrival. The functions typically describe linear filtering processes that reflect the acoustic effect of the ears, head and torso on incoming sound waves. Rendering audio using HRTFs is referred to as “binaural rendering”, and the resulting audio is referred to as “binaural audio”.

[0004] HRTFs may be defined in many ways, including as time-domain impulse responses or as frequency-domain responses. HRTFs are typically grouped in pairs, to provide a response for each ear. An HRTF filter pair may be used to provide a listener with an experience that mimics the sound (at each ear) that would occur when the audio signal was presented from a particular direction of arrival. Different HRTF filter pairs will produce the illusion of differing sound-source directions, timber, and extemalization.

[0005] The user’s anatomy, in particular features relating to the head and ears, will have an impact on how the user perceives sound, including sound timber, localization, and extemalization. There is therefore a general desire to tailor HRTFs to a particular individual. Such HRTFs are herein referred to as personalized HRTFs (pHRTFs).

[0006] Personalized HRTFs can be obtained through experimental measurement procedures, or modelled using personalized information pertaining to the user. Dimensions and orientation of ear and head features are examples of anthropometric information that may be used to generate personalized pRHTFs, Document US2021/0211825 discloses an example of such techniques.

SUMMARY

[0007] From the above follows that accurately and reliably obtaining user specific anthropometric features (e.g., landmarks and distances between landmarks) is critical to pHRTF determination and quality. Existing methods for obtaining anthropometric features for pHRTF generation are complicated, often require specialize hardware, and their implementation is challenging on devices with limited sensing and/or processing capabilities, such as smartphones. [0008] It is an object of the present invention to facilitate reconstruction of anthropometric features from images of a user. Such anthropometric features can then be used for generation of pHRTFs.

[0009] A first aspect of the present invention relates to a method for 3D reconstruction of anthropometric features from images of a subject, comprising obtaining a set of images of the subject from a feed of images captured with an image capturing device from a plurality of perspectives and for each image in the set detecting 2D locations of a plurality of anthropometric landmarks, and estimating a head pose based on the 2D locations. The method further includes generating, based on the detected 2D locations and associated head poses for at least some of the images in the set of images, relative 3D coordinates corresponding to at least some of the anthropometric features captured in the set of images, and generating scaled 3D coordinates from the relative 3D coordinates based on properties associated with the image capturing device and an estimated distance between the subject and the image capturing device.

[0010] This approach provides an efficient way to obtain scaled 3D anthropometric features using a series of images obtained by simple means, e.g. using a mobile camera device such as a smartphone. The anthropometric features are obtained by the detection of ear landmarks and/or face landmarks.

[0011] The set of images may be selected based on a desired uniform angular sampling frequency, which means that the images in the set have been acquired from a set of viewing angles with respect the subject (e.g. the subjects face) and that these viewing angles are uniformly distributed over relevant viewing angle ranges. As an example, relevant yaw angle ranges may be 20-55 degrees on either side.

[0012] As a consequence, if the subject is moving relatively fast, the time between consecutive images in the set should be relatively shorter, while if the subject is moving relatively slow, the time between consecutive images in the set should be relatively longer. [0013] One way to achieve a uniform angular sampling frequency is to monitor the angular velocity of the subject (e.g., relative to the image capturing device) and adjust the sampling from the feed of images. A motion metric representing the angular velocity may be based on data from sensors worn by the subject (e.g., IMU, accelerometers, gyroscopes) but may alternatively be provided by software in the image capturing device.

[0014] Another way to achieve a uniform angular sampling frequency is to simply use the head pose obtained for each image. In that case, a head pose may be obtained for a larger set of images, and a smaller set of images would then be selected for further processing.

[0015] In some embodiments, a subset of anthropometric landmarks is detected only for a subset of images for which the head pose is within a pre-determined head pose range (i.e., for viewing angles with respect to the head within a certain range). As a specific example, ear landmarks may be detected only for a subset of images for which the head pose is within a predetermined azimuth (yaw) range. For azimuth (yaw) angles substantially in front of the subject, the ears are not clearly visible, and detection of landmarks may be difficult or unreliable.

[0016] In some embodiments, head pose estimation involves estimating a rotation transform (yaw, pitch, roll) that transforms the 2D anthropometric landmark locations into a generic 3D face mesh centered in the image. The matching may be done in the least square sense.

[0017] Alternatively, head pose estimation may be achieved by making a request (call) to software in the image capturing device and obtaining a response with including head pose data. Such a response may also include 2D locations for at least a subset of the anthropometric landmarks.

[0018] The estimated distance between the subject and the image capturing device may be detected using any appropriate distance detection technology, including structured light imaging or time-of- flight imaging, or any other hardware-based distance/depth sensor. Alternatively, it is typically possible to obtain a satisfactory distance estimate using statistical information about anthropometric features of the subject, such as iris size or inter-pupillary distance. Processing to obtain the estimated distance may be performed on the set of images, or on a different set of images obtained in a separate process. For example, it may be beneficial to obtain a subset of images of the subject looking straight ahead, and estimate the distance based on this subset of images.

[0019] The scaled 3D coordinates may be used to obtain personalized HRTFs. In one embodiment, a default set of head related transfer functions, HRTFs, are modifying based on the scaled 3D coordinates. In another embodiment, the set of scaled 3D coordinates are applied to a generative model to generate data corresponding to one or more pHRTFs adapted to the subject. [0020] A further aspect of the present invention relates to a method for determining a head radius, r, of a subject, comprising acquiring at least one image of the subject acquired using an image capturing device, estimating a distance from the image capturing device to a face plane of the subject, identifying 2D coordinates of the subject’s ear tragi in the at least one image, projecting the 2D coordinates onto an image plane of the image capture device, and estimating the head radius based on a distance between the projections of the ear tragi in the image plane and an estimated ratio between the head radius, r, and a distance between a face plane, to which the estimated distance relates, and an ear plane in which the ear tragi are located.

[0021] This method has specific advantages in connection with the first aspect, but may also be advantageous in other contexts.

BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The present invention will be described in more detail with reference to the appended drawings, showing currently preferred embodiments of the invention.

[0023] Figures la-lc illustrate a video capturing process for acquiring a set of images using a handheld device according to an embodiment of the present invention.

[0024] Figure 2 is a block diagram of a system according to an embodiment of the present invention.

[0025] Figure 3 is a flow chart of a method according to an embodiment of the present invention.

[0026] Figure 4 shows a set of ear landmarks on a head of a subject according to some embodiments of the present invention.

[0027] Figure 5 shows multi- angulation of 3D coordinates according to some embodiments of the present invention.

[0028] Figure 6 is a schematic illustration of imaging of face and ears of a subject.

[0029] Figure 7 is a schematic block diagram of an example device or architecture that may be used to implement embodiments of the invention.

DETAILED DESCRIPTION

[0030] The present disclosure describes techniques enabling reliable and accurate extraction of 3D locations and dimensions of anthropometric landmarks and features (e.g., relating to the ear and head) from image data (e.g., 2-dimensional or 2D images) captured by a wide variety of camera systems (e.g., a mobile phone, tablet, gaming device, laptop, etc. including a camera) using a variety of capture modalities. While these techniques are described with respect to anthropometric landmarks and features most relevant to pHRTF generation, it is understood that such techniques are generally useful for efficiently obtaining 3D dimensional and/or positional data corresponding to objects captured in 2D images.

[0031] The techniques described herein include strategies to infer 3D coordinates of anthropometric landmarks (e.g., head, ear, or torso landmarks) using a set of images showing a subject’s head and/or torso (e.g., a sequence of images or frames, which may or may not be obtained from video data). The 3D coordinates are subsequently used as input information to generate personalized PHRTFs, which can be used to customized audio playback for a specific user.

[0032] The language describing the following terms is exemplary, and only provided to aid in understanding the techniques described; it should not be interpreted as limiting:

• Camera pose - may refer to camera position in the world coordinate system, expressed as (yaw, pitch, roll), for a frame/image (e.g., from a video).

• Head pose - may refer to head orientation of the user’s head related to the camera, expressed as (yaw, pitch, roll).

• Image Plane - may refer to the plane where the image is formed.

Ovemew/Sitmmary

[0033] In some embodiments, a 3D reconstruction process is achieved by a technique including the following steps:

1. Detecting 2D coordinates (x,y) of upper body anthropometric landmarks in each image/frame in a set of images. a. In some embodiments, upper body anthropometric landmark detection includes, for example, head landmark detection, torso landmark detection, facial landmark detection, ear landmark detection, etc.) b. In some embodiments, an additional step of detecting presence and location of a face or an ear prior to 2D anthropometric landmark detection is performed. In some embodiments, ear landmark detection is limited to images/frames where an ear has been detected (e.g., each image/frame of the set of images/frames includes a detected ear). In some embodiments, facial landmark detection is limited to images/frames where a face has been detected (e.g., each image/frame in the set of images/frames includes a detected face). c. In some embodiments, detection of ear landmark coordinates is limited to images/frames corresponding to head pose values that are within a fixed range (e.g., [-10°, -45° ] azimuth range, or the [10°, 45° ] azimuth range). d. In some embodiments, the set of images/frames of the video to which landmark detection is applied is selected such that it uniformly samples the azimuth space. In other words, the angular separation between images in the set is roughly equal, so that the images in the set are evenly distributed around the face. This may be beneficial, as processing of very similar images (with little additional information) is avoided. e. In some embodiments, ear landmark detection includes a determination of data corresponding to ear landmarks that describe the anatomic characteristics of ear pinna, helix, anti-helix, tragi, fossa, and concha. Estimating the camera pose for each image/frame of the set of images/frames of the video by leveraging the head pose of the user’s head relative to the camera in each respective image/frame (e.g., 3D reconstruction assuming that the camera moves and the head remains still or the head rotates and the camera is still). a. In some embodiments, head pose information is obtained via an API call to external software. b. In some embodiments, head pose information is estimated by computing an instantaneous face mesh of the user’s face, and estimating the roto-translation (rotational transform) (yaw, pitch, roll) that roto-translates (transforms) the instantaneous user’ s face mesh points into a generic 3D face mesh centered in the camera frame (yaw == pitch == roll == 0). An instantaneous face mesh is a face mesh computed for each frame in a feed, e.g. a video feed. c. In some embodiments, the instantaneous face mesh information is used to detect the face or ear location to improve accuracy of 2D landmark detection. Projecting 2D anthropometric landmarks from multiple images/frames of the set of images/frames of the video into the 3D world and performing multi-angulation, to obtain 3D face landmark coordinates (x,y,z). In some embodiments, a 3D point is identified as the point that minimizes the error in intersecting the projection lines. In some embodiments, in accordance with a determination that projection error is above a threshold value, a different image is selected and steps 1-3, are repeated. In some embodiments, in accordance with a determination that projection error is above a threshold value, additional video data is captured, and steps 1-3 are repeated for a set of images from the additional video data. 4. Estimating the distance of the face from the camera (e.g., a camera distance) for each image/frame of the set of images/frames, and scaling distances using this absolute distance. a. In some embodiments, the camera distance is estimated using a hardware depth sensor (e.g., a time-of-flight sensor, etc.). In some embodiments, the camera distance is estimated without a hardware depth sensor (e.g., using a depth from motion algorithm, multiple non-depth cameras, infrared camera/dot projector, etc.). b. In some embodiments, the camera distance is estimated by measuring user’s Iris diameter, assuming that iris’s size is constant over the human population, and scaling the world relative distances using the iris dimension as absolute scaling factor. c. In some embodiments, the camera distance is estimated by combining different strategies for distance scaling (e.g., iris diameter, pupil distance, tragi length, reference object, etc.). In some embodiments, results from multiple distance estimation strategies are combined using Bayesian inference.

5. Optionally, estimating the head radius for each image/frame of the set of images/frames based on the projection of 2D face mesh information onto the image plane and the estimated distance of the head from the camera (e.g., camera distance). a. In some embodiments, the head radius is estimated based on the distance between the ear plane (e.g., a plane that includes the two tragi, coronal to the head) and a face plane (e.g., a plane parallel to the ear plane, minimizing the distance between the facial plane and facial mesh landmarks).

6. Extracting features (of the ears, face, and head) from 3D anthropometric landmarks (e.g., distances, angles, areas) and using the extracted features to generate one or more PHRTFs adapted for the specific user.

[0034] A first part of the process relates to acquiring the set of images of the subject (user) 1 including one or several anatomical features, such as ear(s) eye(s), to enable determination of anthropometric features. The images may in principle be acquired as separate images, but may otherwise be selected from a video feed. The image capture device may be a mobile capture device 1 such as a phone, tablet, or a stationary device such as a gaming device, workstation, etc.. [0035] Figures la-lc illustrate a video capturing process acquiring a feed of images of a subject (user) 1 using a hand-held image capturing device 2. The process would be similar if a stationary device was used. For purposes of the present disclosure, the acquired images should include at least the head 3 of the subject 1.

[0036] First, in figure la, the video capture is initialized with the subject 1 holding the camera 2 in front of his/her face and looking straight into the camera. Then, in figure lb, with the video acquisition in progress, the subject turns his/her head to the right (or left) and then back to the starting position. Finally, in figure 1c, the subject turns his/her head to the left (right) and then back to the starting position.

[0037] By acquiring a video feed of images in this manner, the feed will include images over a wide azimuth range (e.g., 180 degree, 110 degree, 90 degrees etc.). Further, by starting in a straight-ahead pose, the process can start by identifying anthropometric features in a front view, where many relevant features are visible. Further, by stopping at the starting position between the left and right turns, a “reset” of any offsets caused by the rotational motion can be ensured.

[0038] Figure 2 illustrates a block diagram of a system 10 according to some embodiments. The system 10 is configured to process a set of input image frames 11 of a person’s upper body to extract 3D head, face, and ear landmark coordinates.

[0039] As shown in figure 2, the set of images 11 are provided to three parallel modules - a 2D anthropometric landmark detection module 12, a head pose detection module 13 and a face distance estimation module 14. It is noted that although the modules 12, 13, 14 are illustrated in parallel, the processing of these modules may at least in part be sequential. For example, as discussed below, the head pose may be acquired by a request to an external software (e.g. available in the image capturing device) and such a request may also provide a face mesh useful for the landmark detection in module 12.

[0040] The results from the modules 12 and 13 are provided to a multi- angulation module 15 outputting relative 3D coordinates of anthropometric features. These coordinates are then provided to a scaling module 16, which generates scaled 3D coordinates using an estimated distance D from module 14 and parameters P relating to the image capturing device (such as focal length).

[0041] With reference to the flow chart in figure 3, the operation of the system (e.g., 10) will now be briefly outlined.

[0042] First, in step S 11 (obtain images) a set of images of the subject are obtained from a feed of images captured with an image capturing device from a plurality of perspectives. The processing then continues to step S12, which, together with steps S13 and S14 for a loop which is executed for each image in the set. Specifically, in step S 13 (detect 2D locations) 2D locations of a plurality of anthropometric landmarks are detected. The landmarks may be e.g., face landmarks or ear landmarks. In step S14 (estimate head pose) a head pose is estimated based on the 2D locations.

[0043] Based on the estimated head poses for all images in the set, a further selection of a subset of images may take place. Such selection may serve to ensure a subset of images with a uniform angular sampling frequency, i.e., images acquired with viewing angles with respect to the head of the subject evenly distributed in angular space. Also, the anthropometric landmarks detected in a specific image may depend on the estimated head pose. For example, for a head pose which is substantially facing straight ahead (into the camera) it may be impossible (or unreliable) to detect ear landmarks.

[0044] In step S15 (generate relative 3D coordinates), relative 3D coordinates corresponding to a set of the anthropometric landmarks, based on the detected 2D locations of corresponding landmarks and associated head poses for the set of images. It is noted that the set of anthropometric landmarks for which 3D coordinates are determined may include all, or only some, of the 2D locations detected in each image. For example, as mentioned above, depending on the estimated head pose, some landmarks may be less relevant.

[0045] In step S16 (generate scaled 3D coordinates) scaled 3D coordinates are generated from the relative 3D coordinates based on properties associated with the image capturing device and an estimated distance between the subject and the image capturing device. Relevant properties may be focal length, pixel density, filed of view, etc. The estimated distance may be obtained by appropriate distance/depth detection hardware, or by relating some of the detected anthropometric landmarks with known statistical data for certain anatomical features, such as pupil diameter or inter-pupillary distance.

2D landmark detection

[0046] In module 12, 2D anthropometric landmark detection is performed on image frames (e.g., individual images or images constituting a frame). Landmark detection can be achieved using different strategies. In some embodiments, 2D landmark detection is performed using a deep neural network (DNN). In some embodiments, a DNN includes a MobileNet VI or V2 backbone, with a fully connected network on top and a regression activation function. The DNN is trained using the mean Euclidean error between estimated (x,y) coordinates and annotated coordinates as cost function. A large dataset of annotated images characterized by human head, torso, and ear landmarks is used for training. Strategies such as image augmentation, regularization, dropout, and pooling can be used to improve the performance on the test data. In some embodiments, traditional computer vision strategies for landmark detection can be used in conjunction with or in place of the exemplary strategies described above.

[0047] In certain applications, optional steps are performed. For example, in accordance with some embodiments, specific feature detectors (e.g., an ear or face detector) are used to identify the presence of the relevant features (e.g., an ear or face) in the image/frame. In some embodiments, a region relating to a specific relevant feature (e.g., an ear or face) can be estimated with instantaneous faceMesh and head pose information obtained from external sources (e.g., via API call or otherwise from external software), e.g. software available in the image capturing device. This step is typically performed before ear and face landmarks detection, and may be used to reduce complexity and/or improve accuracy of subsequent processes.

[0048] Figure 4 illustrates exemplary results of 2D ear landmark detection in accordance with some embodiments. In figure 4, a plurality of ear landmarks 18 are indicated in an image of a head 3. Similar anthropometric landmark annotation can be applied to other upper body parts, such as head, face, or torso.

Camera pose estimation

[0049] In module 13, a head pose of each image in the set is determined. The head pose can be considered a coordinate transform between the camera coordinate system and the head coordinate system. For practical purposes, it may be sufficient to consider a rotational transform in two or three degrees of freedom, e.g. yaw, pitch and roll. In some embodiments, camera pose information (the camera yaw, pitch, and roll in the world reference system) is obtained (e.g., determined, estimated, accessed, received, etc.) by measuring the head pose information (the yaw, pitch, and roll position of the head in relation to the image frame).

[0050] In some embodiments, camera pose information is obtained by calling a head-pose API, e.g. available in the image capturing device, which returns information about head pose and this information used in the algorithms described herein.

[0051] In some embodiments, head pose information is measured directly. To achieve that, known strategies and techniques for facial landmark detection, such as the open-source libraries or other DNNs trained for this purpose, are leveraged.

[0052] By detecting the facial landmarks, we obtain the matrix Y = (x_Y, y_Y^, where x_Y and y_Y are the N -dimensional arrays of x and y coordinates of the instantaneous face landmarks detected on the images, and N is the number of landmarks. [0053] We also create a face mesh of a generic human face Z = (%_z, y , z_z), where x_z, y and z_z are selected to make sure the face is perfectly centered in the image frame (yaw == pitch == roll == 0°). Instantaneous head pose (head to camera transform) as measured as the (rotational) transform that brings the instantaneous face landmarks on the generic face mesh, while minimizing the projection error, e.g. using the Perspective-n-Point algorithm (see Other existing algorithms can be used at this purpose.

[0054] This is done for images in the video, obtaining yaw, pitch and roll for each image/frame. In some embodiments, camera pose estimation is performed on a subset of the images/frames of the video. For example, in accordance with some embodiments, frames where the estimated camera pose is determined to be missing, unreliable, or wrong, the camera pose values are corrected or re-estimated using machine learning or signal processing algorithms, such as regression or Kalman filtering.

Multi -angulation of landmark projections

[0055] In module 15, relative 3D coordinates for a set of anthropometric features are determined. Specifically, using the head pose (rotational transform) expressed as yaw, pitch, and roll, and the 2D (x,y) coordinates of the landmarks in the image plane, the anthropometric landmarks can be projected in the world reference system and the 3D point in space where the projections approximately intersect can be identified as illustrated in figure 5.

[0056] Figure 5 illustrates using head pose information for a set of images acquired from different viewing angles (k-1, k, k+1) to project the 2D (x,y) coordinates Pj k-i, Pj, k, Pj, k+i of a landmark in the images onto the 3D world reference system. As mentioned above, the approximate intersection of the projections identifies the 3D (x, y, z) coordinates Pj of the landmark in the world reference system. Note that, in figure 5, several cameras are illustrated to represent images of the subject acquired from different viewing angles.

[0057] Several algorithmic strategies can be used to approximate the intersection of the projection lines. In some embodiments, strategies include identifying the point p the minimizes the average squared distance between p and the projection lines, in least square terms. In some embodiments, strategies include identifying the point p that minimizes the mean Euclidean error between the original 2D landmarks (x,y) and the (x,y) coordinates obtained by re-projecting p back into the image frames, averaged over the image perspectives.

[0058] Multiple strategies can be used to remove noisy projections or outliers. For example, we can remove projections whose distance from p is in the tails of the distribution describing the distances between the projection lines and p (e.g., below the 5th percentile, or above the 95th percentile), and recompute p recursively.

[0059] Other strategies can be used to improve the multi-angulation accuracy. For example, we can estimate the head angular velocity, and under-sample the available image perspectives if the angular velocity is below a certain threshold, or up-sample them if the angular velocity is above a certain threshold. By adjusting the sampling frequency (i.e. selection of images from the feed to the set) a consistent angular spread between images in the set can be achieved. As mentioned above, a consistent angular spread (uniform angular sampling frequency).

[0060] In addition, angular velocity could be used to infer the accuracy of the multiangulation process. Higher values of angular velocity are associated to fast head rotation, which might be associated to blurred images (e.g., images not suitable performing one or more processing steps). Slower values of angular velocity are instead associated to less blur and higher image definition. As such, in some embodiments, angular velocity is used to efficiently select a subset of images/frames for 2D/3D landmark recognition that are more likely to provide subtle data for feature extraction and ultimately pHRTF generation. Such selection could be made when obtaining the set of images in step S 11 , or could take place before the processing in step S 15 in figure 3, to select a subset of the set of images.

[0061] Multi-angulation error may be used as a surrogate for multi- angulation accuracy. In some embodiments, different metrics of multi- angulation error are used for this purpose. For example, we can measure the multi-angulation error as the mean distance of p from the projection lines, or as the mean Euclidean distance between the original (x,y) landmarks in the image planes and the corresponding (x,y) coordinates obtained by retro-projecting p into the image planes, over all the image perspectives.

[0062] In some embodiments, multi-angulation accuracy is used to steer subsequent steps in pHRTF generation processes. For example, in accordance with determining that multiangulation accuracy is low (e.g., below a threshold) for one ear, additional data may be obtained (e.g., a user may be prompted to recapture images corresponding to that side of their head) and the overall landmark detection and extraction process is repeated with the newly obtain data. In some embodiments, in accordance with determining that multi- angulation accuracy is low (e.g., below a threshold or remains low after a second capture (e.g., subsequent recapture), data corresponding to the contra-lateral ear (i.e., the other ear) is used to generate a corresponding pHRTF, (i.e., by assuming symmetry in user’s ears size and shape).

Scaling the world reference system [0063] The multi- angulation performed in module 15 will obtain 3D coordinates in relative terms. To convert relative world coordinates into absolute world coordinates (e.g., mm or inches), the reference system is scaled to absolute coordinates in module 16.

[0064] In some embodiments, depth information of images/frames (distance D) is obtained (e.g., via API call) and used along with known camera and sensor intrinsic parameters P (e.g., focal length, AOV, FOV, etc.) to determine absolute scaling. In some embodiments, the module 14 obtains depth information which describes the distance of each object in the image from the camera image plane, computed on a pixel-by-pixel basis.

[0065] In some embodiments, the module 14 determines the iris size in an image and infers the distance D of the head from the camera image plane using that measure. In some embodiments, iris size is determined based on one or more images/frames corresponding to the subject facing directly at the camera (e.g., based on a detected head pose angle). Using iris enables measuring/estimating distance of a face from the camera with less than 10% error, without requiring any specialized hardware. This relies on the property that the horizontal iris diameter of the human eye remains roughly constant at 11.7+0.5 mm across a wide population. [0066] Once the distance D of the face from the camera is determined, the relative reference system is scaled using intrinsic camera parameters. For example, by knowing the camera angle of view (aov), we can estimate the camera field of view (fov) with the following equation: f ov = tan — . Camera focal ratio can be obtained as f oe = — .

[0067] The scaling factor that converts any pixel distance into mm is obtained as px2mm = dept /(max (imsize) ■ foe), where depth is the distance of the point from the camera plane (obtained through depth map or alternative strategies), and max imsize) represent the maximum dimension of the image.

Estimating head radius

[0068] Once we know the distance of the face from the camera, we could also use it to estimate the head radius, i.e. a distance between the edges of the ears, also known as ear tragi. A challenge is that the ear tragi are located in an ear plane, which is slightly further away from the camera than the face plane (to which a distance has been estimated).

[0069] Figure 6 illustrates a technique for estimating head radius in accordance with some embodiments.

[0070] The head bitragion is the distance from two ear tragi, the small pointed eminence of the external ear, situated in front of the concha, and projecting backward over the meatus. The head bitragion is assumed as twice the length of the head radius, r. The distance between a face plane and an ear plane (e.g., respective planes as discussed in the overview section above) is correlated to the head radius according to a pre-determined correlation coefficient a. In some embodiments, correlation coefficient a is ~0.6. Thus, the distance between camera and ear plane is D + ar. By projecting 2D locations of the ear tragi onto the image plane, the distance x between the ear tragi ( as they appear on the face plane) are determined. Using the focal length f (i.e., one of the camera intrinsic parameters P), the head radius is estimated by these parameters along with some simple geometric arguments.

[0071] Specifically, considering the two congruent triangles with bases x and 2r and heights f and D+ar, respectively, we have:

Generating PHRTFs

[0072] Once 3D coordinates in absolute terms are obtained (e.g., estimated), derived features (distances, angles, and areas) describing anthropometric features that affect pHRTFs are computed. In some embodiments, the derived features are used to generate pHRTFs using a generative model (e.g., a model training to generate pHRTF data from data describing anthropometric features). Another approach is to use techniques described in PCT/US2024/016014, hereby incorporated by reference.

Implementation Details

[0073] FIG. 7 shows a schematic block diagram of an example electronic device or architecture 200 (e.g., an apparatus 200) suitable for implementing example embodiments of the present disclosure. The architecture 200 may form part of a mobile device such as a smartphone 2, but may also be a stand-alone piece of equipment. Architecture 200 may embody, but is not limited to, the system as described in relation to FIG. 2.

[0074] As shown, the architecture 200 includes central processing unit (CPU) 201 which is capable of performing various processes in accordance with a program stored in, for example, read only memory (ROM) 202 or a program loaded from, for example, storage unit 208 to random access memory (RAM) 203. The CPU 201 may be, for example, an electronic processor 201, which may include one or more processor cores, and in some examples the processor 201 may be multiple processors. In RAM 203, the data required when CPU 201 performs the various processes is also stored, as required. CPU 201, ROM 202 and RAM 203 are connected to one another via bus 204. Input/output (I/O) interface 205 is also connected to bus 204. [0075] The following components are connected to I/O interface 205: input unit 206, that may include a keyboard, a mouse, or the like; output unit 207 that may include a display such as a liquid crystal display (LCD) and one or more speakers; storage unit 208 including a hard disk, or another suitable storage device; and communication unit 209 which may include a network interface card such as a network card (e.g., wired or wireless).

[0076] The input unit 206 also includes a camera enabling capture of a feed of images.

[0077] In some implementations, output unit 207 includes systems with various number of speakers. Output unit 207 can render audio signals in various formats (e.g., mono, stereo, immersive, binaural, and other suitable formats).

[0078] In some embodiments, communication unit 209 is configured to communicate with other devices (e.g., via a network). Drive 210 is also connected to I/O interface 205, as required. Removable medium 211 , such as a magnetic disk, an optical disk, a magneto-optical disk, a flash drive or another suitable removable medium is mounted on drive 210, so that a computer program read therefrom is installed into storage unit 208, as required. A person skilled in the art would understand that although apparatus 200 is described as including the above-described components, in real applications, it is possible to add, remove, and/or replace some of these components and all these modifications or alteration all fall within the scope of the present disclosure.

[0079] In accordance with example embodiments of the present disclosure, the processes described above may be implemented as computer software programs or on a computer-readable storage medium. For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine readable medium, the computer program including program code for performing methods. In such embodiments, the computer program may be downloaded and mounted from the network via the communication unit 209, and/or installed from the removable medium 211, as shown in FIG. 7. [0080] Generally, various example embodiments of the present disclosure may be implemented in hardware or special purpose circuits (e.g., control circuitry), software, logic or any combination thereof. For example, the various steps of FIG. 3 can be executed by control circuitry (e.g., CPU 201 in combination with other components of FIG. 7), thus, the control circuitry may be performing the actions described in this disclosure.

[0081] Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, a processor and/or other computing device(s), which may include control circuitry. While various aspects of the example embodiments of the present disclosure are illustrated and described as block diagrams, flowcharts, or using some other pictorial representation, it will be appreciated that the blocks, apparatus, systems, techniques, or methods described herein may be implemented in, as nonlimiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof. [0082] Additionally, various blocks shown in the flowcharts may be viewed as method steps, and/or as operations that result from operation of computer program code, and/or as a plurality of coupled logic circuit elements constructed to carry out the associated function(s). For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine readable medium, the computer program containing program codes configured to carry out the methods as described above. [0083] Computer program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These computer program codes may be provided to one or more processors of a general-purpose computer, special purpose computer, or other programmable data processing apparatus that has control circuitry, such that the program codes, when executed by one or more processors of the computer or other programmable data processing apparatus, cause the functions/operations specified in the flowcharts and/or block diagrams to be implemented. The program code may execute entirely on a computer, partly on the computer, as a stand-alone software package, partly on the computer and partly on a remote computer or entirely on the remote computer or server or distributed over one or more remote computers and/or servers.

[0084] The one or more processors may operate as a standalone device or may be connected, e.g., networked to other processor(s). Such a network may be built on various different network protocols, and may be the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), or any combination thereof.

[0085] The software may be distributed on computer readable media, which may comprise computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to a person skilled in the art, the term computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, physical (non-transitory) storage media in various forms, such as ROM, PROM, EPROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Further, it is well known to the skilled person that communication media (transitory) typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.

[0086] The implementation of the technologies disclosed in the figures are merely illustrative examples, and the invention is not so limited. For example, the illustrated partitions such as blocks in FIG. 2 are merely illustrative logical partitions for ease of discussion, where such partitions may be split into additional partitions, combined into fewer partitions, supplemented with additional partitions, or reduced by eliminating partitions, without departing from the spirit of the present invention. For the illustrated flow chart of FIG. 3, the partitions of the operational steps, which may be also referred to as functions, steps, operations, processes, or acts, may be combined into fewer steps or split into additional steps, where steps may be reordered or eliminated, in whole or in part, without departing from the spirit of this disclosure. [0087] Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the disclosure discussions utilizing terms such as “processing”, “computing”, “calculating”, “determining”, “analyzing” or the like, may refer to the function, action, steps and/or processes of a computer hardware or computing system, or similar electronic computing devices, that manipulate and/or transform data represented as physical, such as electronic, quantities into other data similarly represented as physical quantities.

[0088] It should be appreciated that in the above description of example embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention. Furthermore, while some embodiments described herein include some, but not other, features included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the invention, and form different embodiments, as would be understood by those skilled in the art. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0089] Furthermore, some of the embodiments are described herein as a method or combination of elements of a method that can be implemented by a processor of a computer system or by other means of carrying out the function. Thus, a processor with instructions for carrying out such a method or element of a method forms a means for carrying out the method or element of a method. Note that when the method includes several elements, e.g., several steps, no ordering of such elements is implied, unless specifically stated. Furthermore, an element described herein of an apparatus embodiment is an example of a means for carrying out the function performed by the element for the purpose of carrying out the embodiments of the invention. In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

[0090] The person skilled in the art realizes that the present invention by no means is limited to the preferred embodiments described above. On the contrary, many modifications and variations are possible within the scope of the appended claims.

[0091] Various aspects and implementations of the present disclosure may also be appreciated from the following enumerated example embodiments (EEEs), which are not claims. EEE1. 1. A method comprising: at an electronic device: obtaining a plurality of images capturing a subject (e.g., person, object) from a plurality of perspectives (e.g., multiple images including a subject and an anatomical feature of the subject (e.g., an ear), a sequence of images, a video, etc.); for at least two images of the plurality of images: performing landmark detection to detect locations (e.g., 2D coordinates) corresponding to a plurality of landmarks; estimating head pose data from the locations corresponding to the landmarks; and generating camera pose data from the head pose data; generating, from camera pose data corresponding to the at least 2 images of the plurality of images, relative 3D coordinate data corresponding to locations (e.g., 3D relative coordinates) of a set of anatomical features captured in the plurality of images; and generating scaled 3D coordinate data from the relative 3D coordinate data using at least one of: properties (e.g., aov, fov, focal length, resolution, etc.) associated with the camera used to capture the plurality of images; and depth data associated with the plurality of images.

EEE2. The method of EEE1 further comprising: generating data corresponding to a head related transfer function adapted to the subject based at least in part on the scaled 3D coordinate data. EEE3. The method of any of EEEs 1-2, further comprising: determining a motion metric (e.g., velocity or acceleration, angular or linear) associated the plurality of images; and in accordance with a determination that the motion metric satisfies a first criteria (e.g., is above a first threshold value; is below a first threshold value), the at least 2 images comprise a first quantity of images; and in accordance with a determination that the motion metric satisfies a second criteria (e.g., is below a second threshold value; is above a second threshold value), the at least 2 images comprise a second quantity of images larger than the first quantity of images. In some embodiments, the first and second criteria are thresholds corresponding to a common value (e.g., an angular value related to head or camera orientation). In some embodiments, the first and second criteria are thresholds corresponding to different values. In some embodiments, the motion metric is based on data from sensors of the electronic device (e.g., IMU, accelerometers, gyroscopes, etc.). In some embodiments, the motion metric is based image data of the plurality of images.

EEE4. The method of any of EEEs 1-3, further comprising: determining a frame rate associated the plurality of images (e.g., fps of video associated with the plurality of images); and in accordance with a determination that the frame rate satisfies a third criteria (e.g., is above a third threshold value; is below a third threshold value), the at least 2 images comprise a first quantity of images; and in accordance with a determination that the frame rate satisfies a fourth criteria (e.g., is below a fourth threshold value; is above a fourth threshold value), the at least 2 images comprise a second quantity of images larger than the first quantity of images. In some embodiments, the third and fourth criteria are thresholds corresponding to the same value. In some embodiments, the first and second criteria are thresholds corresponding to different values. In some embodiments, frame rate is obtained from metadata associated with the plurality of images. EEE5. The method of any of EEEs 1-4, wherein the at least 2 images are selected based at least in part on a determination that an estimated head pose angle of a respective image of the plurality of images is within pre-determined head pose range (e.g., an azimuth range).

EEE6. The method of any of EEEs 1 -5, wherein the at least 2 images are selected based at least in part on a desired uniform angular sampling frequency (e.g., N = number of images to sample a predetermined azimuth range at a predetermined frequency).

EEE7. The method of any of EEEs 1-6, wherein detecting a plurality of landmarks includes performing at least one of: ear detection and face detection. EEE8. The method of EEE 7, wherein ear detection, includes a determination of data corresponding to ear landmarks that describe the anatomic characteristics of ear pinna, helix, anti-helix, tragi, fossa, and concha.

EEE9. The method of any of EEEs 1-8, wherein estimating head pose data includes obtaining head pose data via a request (e.g., an API call; transmitting a request for head pose data for a respective image and receiving head pose data for the respective image in response to the request).

EEE10. The method of any of EEEs 1-9, wherein estimating head pose data includes: computing an instantaneous face mesh of the subject’s face; and estimating a roto-translation (yaw, pitch, roll) that roto-translates the instantaneous subject’s face mesh points into a generic 3D face mesh centered in an image/frame.

EEE11 The method of any of EEEs 1-10, landmark detection includes: computing an instantaneous face mesh of the subject’s face; and using the instantaneous face mesh to detect a face location or an ear location in a respective image of at least two images.

EEE12. The method of any of EEEs 1-11, wherein generating relative 3D coordinate data includes: projecting corresponding detected landmarks from each of the at least two images to the 3D world using camera pose data or head pose data associated with each respective image of the at least two images; and performing multi-angulation, to obtain relative 3D face landmark coordinates for each detected landmark.

EEE13. The method of EEE 12, wherein a respective 3D face landmark coordinate is identified as a point that minimizes an error in intersecting the projection lines.

EEE 14. The method of any of EEEs 1-13, wherein generating scaled 3D coordinate data from the relative 3D coordinate data includes estimating a distance from the subject to the camera for at least one of: the at least two images of the plurality of images; and a reference subset of the plurality of images different from the subset formed by the at least two images of the plurality of images (e.g., a set of one or more images of the plurality of images not including the at least two images). In some embodiments, a reference subset corresponds to one or more images of the plurality of images capturing a frontal view of the subject (e.g., images associated with a head pose within one or more central threshold angular ranges (e.g., yaw, pitch, or roll approaching ~ 0° ); one or more images of the plurality of images associated with the subject directly facing a camera that captured the respective image; or one or more images of the plurality of images of the subject facing toward a camera (e.g., normal to the surface of the camera lens of the camera capturing the image). In some embodiments, the reference subset of the plurality of images is associated with a specific ear of the subject (Left or Right) captured in each of the at least two images of the plurality of images.

EEE15. The method of EEE 14, wherein estimating a distance from the subject to the camera is based on at least one of the set of: an iris size; an inter pupillary distance; a reference object size; data from a hardware depth sensor; and a depth from motion algorithm.

EEE16. The method of any of EEEs 14-15, wherein estimating a distance from the subject to the camera includes using Bayesian inference to combine the results from more than one distance estimation strategies (e.g., different estimation modalities).

EEE17. The method of any of EEEs 1-16, further comprising: estimating a head radius for each image of the at least two images by projecting 2D face mesh information onto an image plane. EEE18. The method of EEE 17, wherein head radius is estimated based on a distance between an ear plane or axis (e.g., a plane that includes the two tragi, coronal to the head of the subject; an axis between the two tragi ) and a face plane or axis (e.g., a plane parallel to the ear plane, that minimizes the distance between facial mesh landmarks; an axis connecting symmetric facial landmarks).

EEE19. The method of any of EEEs 2-18, wherein generating head related transfer data corresponding to a head related transfer function data includes: extracting features (e.g., related to an ear, face, and/or head) from scaled 3D coordinate data corresponding to anthropometric landmarks (e.g., distances, angles, areas); and applying the extracted features to a generative model to generate data corresponding to one or more PHRTFs adapted to the subject.

EEE20. The method of any of EEEs 1-19, further comprising: calculating a 2D to 3D projection error; and in accordance a determination that the projection error meets first error criteria (e.g., including a determination that the error value is greater than a first threshold), initiating a process to capture additional image data of the subject; and using the additional image data to generate scaled 3D coordinate data (e.g., according to the method of EEE 1).

EEE21. The method of any of EEEs 1-19, further comprising: calculating a 2D to 3D projection error; and in accordance a determination that the projection error meets a second error criteria (e.g., including a determination that the error value is greater than a second threshold and/or a determination that a process to capture additional image data of the subject has previously been initiated), using image data associated with a first ear (e.g. left ear, right ear), to generate scaled 3D coordinate data corresponding to a second ear (e.g. right ear, left ear).

EEE22. A non-transitory computer-readable storage medium including instructions, which when executed by one or more processors of an electronic device, cause the electronic device to perform the methods of any of EEEs 1-21. EEE23. An electronic device including one or more processors and a memory storing instructions, which when executed by the one or more processors, cause the electronic device to perform the methods of any of EEEs 1-21.

EEE24. The electronic device according to EEE 23, the electronic device further comprising one or more cameras, and wherein the plurality of images are captured by the one or more cameras.

Claims

1. A method for 3D reconstruction of anthropometric features from images of a subject, comprising: obtaining a set of images of the subject from a feed of images captured with an image capturing device from a plurality of perspectives; for each image in the set: detecting 2D locations of a plurality of anthropometric landmarks, estimating a head pose based on the 2D locations, and based on said detected 2D locations and associated head poses for at least some of the images in the set of images, generating relative 3D coordinates corresponding to at least some of the anthropometric features captured in the set of images; and generating scaled 3D coordinates from the relative 3D coordinates based on: properties associated with the image capturing device; and an estimated distance between the subject and the image capturing device.

2. The method of claims 1, wherein the set of images are selected based on a desired uniform angular sampling frequency.

3. The method of claim 1 or 2, wherein the anthropometric landmarks include ear landmarks and face landmarks.

4. The method of claim 3, wherein 2D locations of a subset of anthropometric landmarks are detected only for a subset of images for which the head pose is within a pre-determined head pose range.

5. The method according to claim 4, wherein the 2D location of ear landmarks are detected only for a subset of images for which the head pose is within a pre-determined azimuth range.

6. The method of claim 4 or 5, wherein detection of ear landmarks includes a determination of data describing the anatomic characteristics of at least one of ear pinna, helix, anti-helix, tragi, fossa, and concha.

7. The method of any one of the preceding claims, wherein estimating a head pose includes estimating a rotation transform (yaw, pitch, roll) that transforms the 2D anthropometric landmark locations into a generic 3D face mesh centered in the image.

8. The method of any of the preceding claims, wherein estimating a head pose includes making a request to software in the image capture device and obtaining a response including head pose data.

9. The method according to claim 8, wherein the response also includes 2D locations for anthropometric landmarks.

10. The method of any one of the preceding claims, wherein generating relative 3D coordinates includes: projecting the 2D locations into a 3D coordinate system using the estimated head pose, and identifying 3D coordinates of a 2D location as an intersection of projections of this 2D location from multiple images.

11. The method of claim 10, wherein 3D coordinates of a 2D location are identified as a point that minimizes an error with respect to projections of this 2D location from multiple images.

12. The method of any of the preceding claims, wherein the estimated distance between the subject and the image capturing device is based on the set of images or other images from the feed of images.

13. The method of any one of the preceding claims, wherein estimating a distance between the subject and the image capturing device is based on at least one of: an iris size; an inter pupillary distance; a reference object size; data from a hardware depth sensor; and a depth from motion algorithm.

14. The method of any of claims 12 or 13, wherein estimating a distance from the subject to the image capturing device includes using Bayesian inference to combine the results from more than one distance estimation strategies.

15. The method of any one of claims 12 - 14, further comprising projecting 2D landmarks extracted from the images onto an image plane of the image capture device, and estimating a head radius, r, based on the projected 2D landmarks and propertied of the image capturing device.

16. The method of claim 15, wherein the 2D landmarks relate to ear tragi of the subject, and wherein the head radius, r, is determined based on a distance between the projections of the ear tragi in the image plane and an estimated ratio between the head radius, r, and a distance between a face plane, to which said estimated distance relates, and an ear plane in which the ear tragi are located.

17. The method of claim 16, wherein the head radius, r, is determined according to: where D is the estimated distance to the face plane, x is the distance between the ear tragi in the image plane of the image capture device, /is the focal length of the image capture device, and a is the ratio between the head radius, r, and a distance between the face plane and the ear plane.

18. The method of any one of the preceding claims, further comprising: calculating a 2D to 3D projection error; and in accordance a determination that the projection error is greater than an error threshold, initiating a process to include additional images the set of images; and using the additional image data to generate scaled 3D coordinate data.

19. The method of any one of the preceding claims, further comprising: using image data associated with a first ear to generate scaled 3D coordinate data corresponding to a second ear.

20. The method of any one of the preceding claims, further comprising: determining a motion metric of the subject based on the feed of images; and including in the set of images only images for which said motion metric is within a predefined range.

21. A method for generating personalized head related transfer functions, pHRTFs, comprising: obtaining a default set of head related transfer functions, HRTFs; obtaining a set of scaled 3D coordinates of anthropometric features using the method of any one of claims 1 - 20; and modifying the default set of head related transfer functions, HRTFs, based on the scaled 3D coordinates.

22. A method for generating personalized head related transfer functions, pHRTFs, comprising: obtaining a set of scaled 3D coordinates of anthropometric features using the method of any one of claims 1 - 20; and applying the set of scaled 3D coordinates to a generative model to generate data corresponding to one or more PHRTFs adapted to the subject.

23. An electronic device including one or more processors and a memory storing instructions, which when executed by the one or more processors, cause the electronic device to perform the methods of any of claims 1 -22.

24. The electronic device according to claim 23, the electronic device further comprising one or more image capturing device, and wherein the set of images is captured by the one or more image capturing device.

25. A non-transitory computer-readable storage medium including instructions, which when executed by one or more processors of an electronic device, cause the electronic device to perform the methods of any of claims 1-22.

26. A method for determining a head radius, r, of a subject, comprising: acquiring at least one image of the subject acquired using an image capturing device; estimating a distance from the image capturing device to a face plane of the subject; identifying 2D coordinates of the subject’s ear tragi in the at least one image; projecting the 2D coordinates onto an image plane of the image capture device; and estimating the head radius based on a distance between the projections of the ear tragi in the image plane and an estimated ratio between the head radius, r, and a distance between a face plane, to which said estimated distance relates, and an ear plane in which the ear tragi are located.

27. The method of claim 26, wherein the head radius, r, is determined according to: where D is the estimated distance to the face plane, x is the distance between the ear tragi in the image plane of the image capture device, /is the focal length of the image capture device, and a is the ratio between the head radius, r, and a distance between the face plane and the ear plane.

28. The method of claim 27, wherein estimating a distance is based on at least one of: an iris size; an inter pupillary distance; a reference object size; data from a hardware depth sensor; and a depth from motion algorithm.

29. The method of any one of claims 26 - 28, wherein estimating a distance from the subject to the image capturing device includes using Bayesian inference to combine the results from more than one distance estimation strategies.

30. An electronic device including one or more processors and a memory storing instructions, which when executed by the one or more processors, cause the electronic device to perform the methods of any of claims 26 - 29.

31. The electronic device according to claim 30, the electronic device further comprising one or more image capturing device, and wherein the set of images is captured by the one or more image capturing device.

32. A non-transitory computer-readable storage medium including instructions, which when executed by one or more processors of an electronic device, cause the electronic device to perform the methods of any of claims 26 - 29.