GB2559977A

GB2559977A - Systems and methods for obtaining information about the face and eyes of a subject

Info

Publication number: GB2559977A
Application number: GB1702871.3A
Authority: GB
Inventors: William Joy Thomas; Henry John Larkins Andrew
Original assignee: Fuel 3d Tech Ltd
Current assignee: Fuel 3d Tech Ltd
Priority date: 2017-02-22
Filing date: 2017-02-22
Publication date: 2018-08-29
Also published as: GB201702871D0; WO2018154272A1

Abstract

A computer-implemented method and an apparatus for obtaining information about the face of a subject, the method comprising illuminating the face of the subject 101 and capturing one or more images 102 of the face during each of a plurality of time periods. Then for each time period carrying out the steps of: using the images to form a respective three-dimensional skin model of the skin of the subject 103 and obtaining eye data indicative of the position of at least a portion of an eye of the subject during the time period 104. Having completed the steps for each time period, the eye data for the time periods is converted to a common reference frame using the respective skin models 108 and from the converted eye data an eye model comprising an estimate of the position of the centre of eye rotation CER of the eye of the subject is obtained 109. Typically, such an eye model is obtained for each of the subject's eyes. The eye model is used in a process for designing an item of eyewear such as vision correcting glasses or spectacles, or used in an augmented-reality or virtual-reality system.

Description

(71) Applicant(s):

Fuel 3D Technologies Limited (Incorporated in the United Kingdom)

Unit 2 Douglas Court, Seymour Business Park, Station Road, Chinnor, Oxfordshire, 0X39 4HA, United Kingdom (72) Inventor(s):

Thomas William Joy Andrew Henry John Larkins (74) Agent and/or Address for Service:

Marks & Clerk LLP

Fletcher House (2nd Floor), Heatley Road,

The Oxford Science Park, OXFORD, OX4 4GE, United Kingdom (51) INT CL:

A61B 3/113 (2006.01) G02C 13/00 (2006.01) G06T7/00 (2017.01) (56) Documents Cited:

GB 2544460 A WO 2015/177459 A

US 20090040460 A1 (58) Field of Search:

INT CLA61B, G02C, G06T

Other: EPODOC, WPI, MEDLINE, BIOSIS, XPESP,

SPRINGER (54) Title of the Invention: Systems and methods for obtaining information about the face and eyes of a subject Abstract Title: Apparatus and method for obtaining information about the face and eyes of a subject.

(57) A computer-implemented method and an apparatus for obtaining information about the face of a subject, the method comprising illuminating the face of the subject 101 and capturing one or more images 102 of the face during each of a plurality of time periods. Then for each time period carrying out the steps of: using the images to form a respective three-dimensional skin model of the skin of the subject 103 and obtaining eye data indicative of the position of at least a portion of an eye of the subject during the time period 104. Having completed the steps for each time period, the eye data for the time periods is converted to a common reference frame using the respective skin models 108 and from the converted eye data an eye model comprising an estimate of the position of the centre of eye rotation CER of the eye of the subject is obtained 109. Typically, such an eye model is obtained for each of the subject's eyes. The eye model is used in a process for designing an item of eyewear such as vision correcting glasses or spectacles, or used in an augmented-reality or virtual-reality system.

101 100

FIG.2

At least one drawing originally filed was informal and the print reproduced here is taken from a later filed formal copy.

1/7

Fig. 1

2/7

101 100

FIG.2

3/7

07 17

Fig. 3

34164190-1-TW ATKIN

4/7

07 17

Fig. 4

34164190-1-TW ATKIN

5/7

07 17

Fig. 5

34164190-1-TW ATKIN

6/7

07 17

FIG. 6

34164190-1-TW ATKIN

7/7

200

FIG. 7

Application No. GB1702871.3

RTM

Date :27 June 2017

Intellectual

Property

Office

The following terms are registered trade marks and should be read as such wherever they occur in this document:

Xenon, P.g 8

Intellectual Property Office is an operating name of the Patent Office www.gov.uk/ipo

Systems and Methods For Obtaining information About the Face and Eves Of A Subject

Summary of the invention

The present invention relates to systems and methods for obtaining a numerical eye model of each of the eyes of a subject. The eye models of a subject’s eyes may be used for the design and production of eyewear item, to be used in proximity with the subject’s face. The eye models may also be used a part of an augmented reality (AR) or virtual reality (VR) system.

Background of the invention

A conventional process for providing a subject with eyewear such as glasses (a term which is used here to include both vision correction glasses (spectacles) and sunglasses) involves the subject trying on series of dummy frames, and examining his or her reflection in a mirror. Once a frame has been selected, an optician conventionally makes a number of manual measurements of the subject’s face, to obtain distance parameters of the subject’s face, which are used to produce an item of eyewear including a modified version of the frames. The measurement process is subject to various errors. Furthermore, the modification of the frames is carried out when the subject is not present, so that the resulting glasses may be unsuitable, for example because the lower edge of the fitted lenses impacts on the subject’s cheek. Also, the modification of the frame varies the distance of the lens from the eye of the subject, which may be highly disadvantageous for glasses which perform visual correction. It has been estimated that a 2mm variation of the spacing of the eye and the lens can result in a 10% difference in the resulting field of vision. Additional problems are that the modification of the frame changes the position of the optical centre of the lens in the up/down and left/right directions (relative to the face of the subject) which may also have undesirable optical effects.

Various proposals have been made to obtain the distance parameters automatically while the subject looks in successive directions, but this faces the difficulty that the subject’s head often moves during the procedure. Various proposals have been made to address this problem by requiring the subject to wear a specially designed item of headwear which moves with the subject’s head and thus allows compensation for movements of the head. The item of headwear includes a position sensor (for example, in the case of US 2009/0040460), or marker elements positioned on spectacles which can be identified in successive photographs of the subject’s head (for example, in the case of US 8,220,922 and US 8,360,580). US 2009/0040460 aims to obtain the position of the centre of eye rotation (CER) in a reference frame which moves with the subject’s head, so that ophthalmic lenses can be formed which are made to measure and provide better performance.

Summary of the invention

The present invention aims to provide new and useful methods and systems for obtaining an eye model of the face of a subject.

It also aims to provide new and useful methods and systems for using the eye models for the design and production of an eyewear item for placement in proximity to the subject’s face. The item of eyewear typically includes refractive lenses for vision correction.

In general terms, the invention proposes that at each of a series of successive time periods in which a subject is looking in different respective directions, an imaging system captures a plurality of images of the subject’s face. Using the images for each time period, the system forms a respective three-dimensional skin model of a skin portion of the subject’s face. The system also obtains eye data indicative of the position of at least a portion of at least one of the subject’s eyes. For example, the eye data may be data characterizing specular reflections in the images from at least one eye of the subject. The system uses the skin models to convert the eye data for the multiple time periods into a common frame of reference, and then uses the eye data to obtain a numerical eye model indicative at least of the position of the centre of the eye’s rotation (CER). Typically, such an eye model is obtained for each of the subject’s eyes.

In this way, the system can form an eye model without using any elements of the system placed in a fixed position relative to the face of the subject. Indeed the subject’s face may be imaged at a time when the subject is not wearing any objects.

Forming the eye model using the eye data from time periods in which the subject is looking in different respective directions provides more accuracy in measuring the centre of rotation than only using eye data from a time when the subject is looking in a single direction.

Specific expressions of the invention are given in the appended claims.

As noted above, in the different time periods the subject looks in different respective directions. These respective directions may be directions relative to the face or the subject and/or relative to a static reference frame (e.g. a building in which the subject is located).

Specifically, in one possibility the head of the subject may be substantially stationary, but the subject may move his or her eyes, i.e. in the different time periods the subject looks in different respective directions relative to a static frame of reference (e.g. a building in which the subject is located) and relative to the face of the subject.

Alternatively, the head of the subject may be permitted to move too. For example, in one possibility the subject may turn his head and gaze direction substantially in register with each other, so that the direction in which the subject looks does not change relative to the face of the subject, though it does change relative to the static reference frame. In another example, the subject may continue looking in the same direction relative to the static frame of reference while turning his or her head. Thus, in the different time periods, the subject looks in different respective directions relative to the face of the subject but not relative to the static reference frame.

The imaging system may employ any imaging technique(s) for forming the skin model(s), such as stereoscopy (discussed below), laser triangulation, time or flight measurement, phase structured fringe pattern imaging and/or photometry (the science of measuring the brightness of light; also discussed below).

Preferably, the imaging system comprises at least one directional energy source (e.g. a light source such as a visible light source) for illuminating the face (preferably successively) in at least three directions. In this case, the model of the skin may be formed using photometry, assuming that the skin exhibits Lambertian reflection in which the reflected radiation is isotropic with an intensity according to Lambert’s cosine law (an intensity directly proportional to the cosine of the angle between the direction of the incident light and the surface normal). Recently, great progress has been made in imaging three-dimensional surfaces which exhibit Lambertian reflective properties by means of photometry. WO 2009/122200 proposes a system in which a three-dimensional model of a 3-D object is produced in which large-scale features are imaged by stereoscopic imaging (that is, by comparing the respective positions of landmarks in multiple images of a subject captured from respective directions), and fine detail is produced by photometric imaging. A system of that type may be employed also in the present invention.

By contrast, as noted above, the eye data may comprise, or be derived from, data describing specular reflections (“glints”) in at least some of the images. Since many portions of the eye, such as the lens, are transparent, Lambertian reflection is not present, so photometry cannot be used for this purpose. However, detection of specular reflections allows the surface of the eye to be detected accurately. When this is used in combination with photometric modelling of the skin accurate numerical information may be obtained about the CER of the eye in a reference frame defined with reference to the skin of the subject.

The eye data collected over multiple time periods includes at least one of (i) data indicating reflections from portions of the surface of the eye other than the cornea, and/or (ii) data indicating reflections from portions of the cornea of the eye when the cornea is in multiple locations relative to the skin (i.e. the direction in which the subject looks varies other than by movements of the subject’s head).

Note that in principle other eye data may be collected in each of the time periods in addition to, or instead of, the data characterizing specular reflections. For example, the eye data may comprise eye data obtained from any other gaze-tracking technique, such as techniques including iris tracking.

Preferably the eye data is obtained from images captured by image-capture devices which are also used to capture images used to obtain the three-dimensional skin models. As a result the eye data and the three-dimensional skin models are obtained in the same reference frame. This eliminates a source of errors in a system in which the eye data and the skin models are obtained from different respective cameras which have a (possibly unknown) offset in position and/or orientation.

The system may comprise a display device for viewing by the subject and arranged to display a time-varying image in different ones of the time periods. The time-varying image is typically designed to include an attention-attracting element which is in different locations in the display in different ones of the time periods, so that the direction in which the subject looks varies between the time periods. It is not necessary that the element has the same appearance in different ones of the time periods. The time-varying image may, in one example, be produced by activating in successive time periods light-emitting elements (“lights”) in different respective locations in the display area of the display device, so that the lights function as the attentionattracting elements.

The eye model of each eye may include a sclera portion representing the sclera, and a cornea portion representing the cornea. The sclera portion may be portion of the surface of a first sphere centred on the centre of rotation of the eye, and the cornea portion may a portion of the surface of a second sphere having a smaller radius of curvature than the first sphere. The centers of the two spheres are spaced apart, and the line joining them intersects with the center of the cornea portion of the model, at a position which may be taken as the center of the pupil.

Note that in different embodiments of the invention, the generation of the eye data and the eye model are performed in different ways. For example, in one option the data describing the specular reflections may be subject to little or no processing to form the eye data which is converted to the common reference frame. Alternatively, the data describing the specular reflections in each time period may be significantly pre-processed prior to converting it into the common reference frame. For example, the eye data may be in the form of a respective provisional model of the eye for each of the time periods, e.g. defined by at least part of a sphere representing the sclera, and/or at least part of a second sphere representing the cornea. The provisional eye models for each of the respective time periods are converted to a common reference frame using the skin model, and combined by any of several possible processes to form the (final) eye model, e.g. choosing the CER of the final eye model as the average of the centre positions of the respective spheres representing the sclera in the provisional eye models in the common reference frame.

The combination of the eye data for the different time periods to form the eye model, may include using the eye data to estimate a respective gaze direction for each of the time periods. For example, the converted eye data may be data indicating the respective position and orientation of the cornea of the eye in each of the time periods in the common reference frame. In this case, the CER may be obtained from these positions and orientations, e.g. as the location which would allow (or be most likely to allow) the cornea of a rotating eye to reach these positions and orientations. Optionally, this process may employ an additional mechanism for estimating the gaze direction, such as one based on the position of the respective portion of the display device which is designed to attract the attention of the subject in each of the respective time periods.

The common frame of reference may be defined based on the position of the head of the subject in one of the time periods. In this case, the step of converting the eye data into the common frame of reference only includes modifying the eye data for the other time periods.

The conversion of the eye data into a common frame of reference may be performed by forming positional (geometrical) mappings between corresponding points of the skin models, and then exploiting a pre-known geometrical relationship between the skin models and the respective eye data. The pre-known relationship may be based on a pre-known geometrical relationship between the set of camera(s) which captured the set of images used to produce the skin models, and the set of camera(s) which captured the set of images used to produce the eye data. Note that these two sets of cameras, and two sets of images may overlap.

The positional mappings may be defined based on landmarks of the skin models corresponding to landmarks of the subject’s face, such as the tip of the subject’s nose and/or a plane of symmetry. These landmarks are recognized in each of the skin models, and used to define positional transforms which map the skin models into the common frame of reference. As noted above, this may be a frame of reference in which one of the skin models was generated, so that for that skin model the positional transform is trivial (i.e. it does not move or re-orient the skin model).

Optionally, there may be a step of selecting which landmarks are used to map the skin models together, for example by estimating how well each of a set of landmarks are defined in each of the skin models, and selecting those landmarks which are best defined as the ones to use for forming positional transforms between the skin models and the common reference frame. Thus, if the subject has an unusual feature (e.g. a broken nose) which makes a certain landmark unsuitable, other landmarks will tend to be used. Optionally, different landmarks may be used for converting different ones of the skin models into the common reference frame.

Preferably, the skin model includes two portions corresponding to portions of the skin of the subject’s face proximate each of the subject’s eyes. Alternatively or additionally, it may include at least one portion corresponding respectively to at least one of the subject’s ears.

The eye model may be employed as part of a composite model including the eye model for each of the subject’s eyes and a skin model portion which indicates the contours of the subject’s skin. The skin model portion of the composite model may be all or part of one of the skin models obtained for the respective time periods, or a skin model obtained by combining multiple ones of those skin models. Optionally, the skin model portion may not include portions of the face of the subject which are distal from the portions of the face which are proximate to, or come into contact with the item of eyewear.

The eye model, or composite model, may be employed in an automatic process for designing a personalized item of eyewear for use in proximity with the subject’s face (the term “proximity” is used here to include also the possibility that the object is in contact with the face). The term “designing” is used here to include a process of selecting from a plurality of pre-defined designs for eyewear items, and/or modifying one of more parameters (typically distance parameters) of a pre-defined design of eyewear items.

The eyewear typically includes at least one lens for each eye, and a frame for supporting the lens(es) in relation to the subject’s face. For example, the item of eyewear may be a set of glasses, of a type having any one of more of the following functions: vision correction, eye protection (including goggles or sunglasses) and/or cosmetic reasons.

At least one lens may be a refractive lens for vision correction. The shape of the lens may be selected based on the centre of rotation of the corresponding eye, optionally including selecting the refractive power of one or more portions of the lens. This has a significant effect on the field of vision of the subject. For example, the refractive power of different portions of the lens may be adjusted to compensate for the different distances of those portions of the lens from the CER. Alternatively or additionally, the overall shape of the lens may be varied to reduce the distance of different portions of the lens from the CER.

This design process may assume that the glasses are positioned on the face of the subject in contact with one or more portions of the face model portion of the composite model. Thus, the face model may be used as part of the procedure to work out how far the lenses are from the CER and/or the lenses. For this purpose the design process may employ a template representing an item of eyewear.

The design of the eyewear may also include varying dimensions of the frame based on the composite model. If the template representing the item of eyewear is defined by one or more parameters, the design process may include varying those parameters according to the composite model. For example the modification of the frame may be to select a distance between a certain portion of the lens (e.g. the optical centre of the lens) in accordance with the spacing of the CERs of the subject’s eyes according to the two eye models, and/or to place the certain portion of the lenses at a desired distance from the CERs of the respective eyes.

Optionally, the selection of the respective refractive power of different portion(s) of the lens(es) and the selection of the dimensions of the frame may be conducted together, to produce a design of the item of eyewear which may be optimal in terms of both vision correction and comfort.

At least one component of the item of eyewear (e.g. the arms of the glasses, or the nose pads) may be fabricated (e.g. by molding or 3D printing) according to the designed eyewear. This would provide the item of personalized eyewear in a comfortable form, and with high performance.

Note that it is not essential that the steps of designing or constructing the eyewear are performed by the same individuals who carry out the formation of the eye model or composite model. For example, a first organization may produce an eye model or composite model, which is transmitted to a second organization to produce an item of eyewear consistent with the eye model or composite model.

As mentioned above, the subject is preferably illuminated successively in individual ones of at least three directions. If this is done, the energy sources may emit light of the same frequency spectrum (e.g. if the energy is visible light, the directional light sources may each emit white light and the captured images may be color images). However, in principle, the subject could alternatively be illuminated in at least three directions by energy sources which emit energy with different respective frequency spectra (e.g. in the case of visible light, the directional light sources may respectively emit red, white and blue light). In this case, the directional energy sources could be activated simultaneously, if the energy sensors are able to distinguish the energy spectra. For example, the energy sensors might be adapted to record received red, green and blue light separately. That is, the red, green and blue light channels of the captured images would be captured simultaneously, and would respectively constitute the images in which the object is illuminated in a single direction. However, this second possibility is not preferred, because coloration of the object may lead to incorrect photometric imaging.

Various forms of directional energy source may be used in embodiments of the invention. For example, a standard photographic flash, a high brightness LED cluster, or Xenon flash bulb or a 'ring flash'. It will be appreciated that the energy need not be in the visible light spectrum.

In principle, there could be only one directional energy source which moves so as to successively illuminate the subject from successive directions.

However, more typically, at least three energy sources are provided. It would be possible for these sources to be provided as at least three energy outlets from an illumination system in which there are fewer than three elements which generate the energy. For example, there could be a single energy generation unit (light generating unit) and a switching unit which successively transmits energy generated by the single energy generation unit to respective input ends of at least three energy transmission channels (e.g. optical fibers). The energy would be output at the other ends of the energy transmission channels, which would be at three respective spatially separately locations. Thus the output ends of the energy transmission channels would constitute respective energy sources. The light would propagate from the energy sources in different respective directions.

Where visible-light directional energy is applied, then the energy sensors may be two or more standard digital cameras, or video cameras, or CMOS sensors and lenses appropriately mounted. In the case of other types of directional energy, sensors appropriate for the directional energy used are adopted. A discrete sensor may be placed at each viewpoint, or in another alternative a single sensor may be located behind a split lens or in combination with a mirror arrangement.

The energy sources and viewpoints preferably have a known positional relationship, which is typically fixed. The energy sensor(s) and energy sources may be incorporated in a portable, hand-held instrument. Alternatively, particularly in the application described below involving eyewear, the energy sensor(s) and energy sources may be incorporated in an apparatus which is mounted in a building, e.g. at the premises of an optician or retailer of eyewear. In a further application, as discussed below, the apparatus may be adapted to be worn by a subject, e.g. as part of a helmet.

Although at least three directions of illumination are required for photometric imaging, the number of illumination directions may be higher than this. The energy sources may be operated to produce a substantially constant total intensity over a certain time period (e.g. by firing them in close succession), which has the advantage that the subject is less likely to blink.

Alternatively, the energy sources may be controlled to be turned on by processor (a term which is used here in a very general sense to include for example, a field-programmable gate array (FGPA) or other circuitry) which also controls the timing of the image capture devices. For example, the processor could control the a different subset of the energy sources to produce light in respective successive time periods, and for each of the image capture device to capture a respective image during these periods. This has the advantage that the processor would be able to determine easily which of the energy sources was the cause of each specular reflection.

Specular reflections may preserve polarization in the incident light, while Lambertian reflections remove it. To make use of this fact, some or all of the light sources may be provided with a filter to generate light with a predefined linear polarization direction, and some or all of the image capture devices may be provided with a filter to remove incident light which is polarized in the same direction (thus emphasizing Lambertian reflections) or the transverse direction (thus emphasizing specular reflections).

One possibility, if the energy sources include one or more energy sources of relatively high intensity and one or energy sources which are of relatively lower intensity, is to provide polarization for the one of more of the energy sources of high intensity, and no polarization for the one or more energy sources which are of relatively lower intensity. For example, the specular reflections may only be captured using only the high intensity energy sources, in which case (e.g. only) those energy sources would be provided with a polarizer producing a polarization which is parallel to a polarization of the energy sensors used to observe the specular reflections.

One or more of the energy sources may be configured to generate light in the infrared (IR) spectrum (wavelengths from 700nm to 1mm) or part of the near infrared spectrum (wavelengths from 700nm to 1100nm). These wavelength ranges have several advantages. Since the subject is substantially not sensitive to IR or near-IR radiation, it can be used in situations in which it is not desirable for the subject to react to the imaging process. For example, IR or near-IR radiation would not cause the subject to blink. Also, IR and near-IR radiation may be used in applications as discussed below in which the subject is presented with other images during the imaging process.

Brief description of the drawings

An embodiment of the invention will now be described for the sake of example only with reference to the following figures in which:

Fig. 1 is a schematic view of an imaging assembly for use in an embodiment of the present invention;

Fig. 2 is a flow diagram of a method performed by an embodiment of the invention

Fig. 3 shows an eye model for use in the embodiment;

Fig. 4 illustrates schematically how specular reflections from the eye are used by the embodiment to find the parameters of a provisional eye model of the form shown in Fig. 2;

Fig. 5 illustrates schematically how specular reflections from the eye are used by a variation of the embodiment to find the parameters of a provisional eye model of the form shown in Fig.2;

Fig. 6 illustrates the use of an eye model in designing a lens of an item of eyewear; and

Fig. 7 illustrates an embodiment of the invention incorporating the imaging assembly of Fig. 1 and a processor.

Detailed description of the embodiments

Referring firstly to Fig. 1, an imaging assembly is shown which is a portion of an embodiment of the invention. The embodiment includes energy sources 1,2,3. It further includes energy sensors 4, 5 in form of image capturing devices (cameras). The energy sensors 4, 5 and energy sources 1,2, 3 are fixedly mounted to each other by struts 6. The exact form of the mechanical connection between the energy sources 1,2, 3 and the energy sensors 4, 5 is different in other forms of the invention, but it is preferable if it maintains the energy sources 1,2, 3 and the energy sensors 4, 5 not only at fixed distances from each other but at fixed relative orientations. The positional relationship between the energy sources 1,2,3 and the energy sensors 4, 5 is pre-known. The energy sources 1,2, 3 and image capturing devices 4, 5 may be incorporated in a portable, hand-held instrument. In addition to the assembly shown in Fig. 1, the embodiment includes a processor which is in electronic communication with the energy sources 1,2, 3 and image capturing devices 4, 5. This is described below in detail with reference to Fig. 7.

The energy sources 1,2, 3 are each adapted to generate electromagnetic radiation, such as visible light or infra-red radiation. The energy sources 1,2, 3 are all controlled by the processor. The output of the image capturing devices 4, 5 is transmitted to the processor.

Each of the image capturing devices 4, 5 is arranged to capture an image of the face of a subject 7 positioned in both the respective fields of view of the image capturing devices 4, 5.

The image capturing devices 4, 5 are spatially separated, and preferably also arranged with converging fields of view, so the apparatus is capable of providing two separated viewpoints of the subject 7, so that stereoscopic imaging of the subject 7 is possible. The case of two viewpoints is often referred to as a “stereo pair” of images, although it will be appreciated that in variations of the embodiment more than two spatially-separated image capturing devices may be provided, so that the subject 7 is imaged from more than two viewpoints. This may increase the precision and/or visible range of the apparatus. The words “stereo” and “stereoscopic” as used herein are intend to encompass, in addition to the possibility of the subject being imaged from two viewpoints, the possibility of the subject being imaged from more than two viewpoints.

Note that the images captured are typically color images, having a separate intensity for each pixel each of three color channels. In this case, the three channels may be treated separately in the process described below (e.g. such that the stereo pair of images also has two channels).

The system comprises a display device 8 having a plurality of lights 9. The imaging system is operative to illuminate the lights 9 in successive time periods (which are spaced apart as described below), so that the subject, who looks towards each light 9 as it is illuminated, successively changes his or her viewing direction (i.e. the direction in which he or she is looking). Note that in a variation of the embodiment, the subject might simply be asked to shift his or her viewing direction successively, for example to look in successive time periods at a respective ones of a plurality of portions of a static display.

A natural human reaction when a subject changes his or her viewing direction is for the subject to slightly move his or her head. However, as described in more detail below, in each of the time periods, the imaging system forms a respective three-dimensional model of the skin of the subject, and collects a respective set of eye data indicative of specular reflections from the subject’s eyes. The skin models are used to reference the eye data into a common reference frame based on landmarks defined on the subject’s face, so that motion of the subject’s head is compensated for.

Suitable image capture devices for use in the invention include the 1/3-lnch CMOS Digital Image Sensor (AR0330) provided by ON Semiconductor of Arizona, US. All the images used for the modelling are preferably captured for a given time period are preferably captured within a duration of no more than 0.2s, and more preferably no more than 0.1s. Note that the time is preferably less than a blink reaction time, so that imaging is unaffected if the subject closes his or her eyes in respect to the illumination of the energy sources 1,2, 3. However, it is possible to envisage embodiments in which the images are captured over a longer duration, such as up to about 1 second or even longer. This may be appropriate for example if the electromagnetic radiation generated by the energy sources 1,2, 3 is not bright enough to cause blinking, and/or does not include electromagnetic radiation in the visible spectrum.

The skin of the subject 7 will typically reflect electromagnetic radiation generated by the energy sources 1,2, 3 by a Lambertian reflection, so the skin portion of the subject’s face may be imaged in the manner described in detail in WO 2009/122200, to form a skin model. The skin model may optionally also include a portion of the subject’s hair, although since a subject’s hair may move relative to the subject’s face as the subject’s head moves preferably the landmarks in the skin model discussed below are landmarks of the subject’s skin rather than the subject’s hair.

In brief, two acquisition techniques for acquiring 3D information are used to construct the skin model. One is photometric reconstruction, in which surface orientation is calculated from the observed variation in reflected energy against the known angle of incidence of the directional source. This provides a relatively high-resolution surface normal map alongside a map of relative surface reflectance (or illumination-free colour), which may be integrated to provide depth, or range, information which specifies the 3D shape of the object surface. Inherent to this method of acquisition is output of good high-frequency detail, but there is also the introduction of low-frequency drift, or curvature, rather than absolute metric geometry because of the nature of the noise present in the imaging process. The other technique of acquisition is passive stereoscopic reconstruction, which calculates surface depth based on optical triangulation. This is based around known principles of optical parallax. This technique generally provides good unbiased low-frequency information (the coarse underlying shape of the surface of the object), but is noisy or lacks high frequency detail. Thus the two methods can be seen to be complementary. The skin model may be formed by forming an initial model of the shape of the skin using stereoscopic reconstruction, and then refining the initial model using the photometric data to form the skin model.

The photometric reconstruction requires an approximating model of the surface material reflectivity properties. In the general case this may be modelled (at a single point on the surface) by the Bidirectional Reflectance Distribution Function (BRDF). A simplified model is typically used in order to render the problem tractable. One example is the Lambertian Cosine Law model. In this simple model the intensity of the surface as observed by the camera depends only on the quantity of incoming irradiant energy from the energy source and foreshortening effects due to surface geometry on the object. This may be expressed as:

l=P_PL-N (Eqn 1) where I represents the intensity observed by the image capture devices 4, 5 at a single point on the object, Rthe incoming irradiant light energy at that point, Λ/the object-relative surface normal vector, L the normalized object-relative direction of the incoming lighting and p the Lambertian reflectivity of the object at that point. Typically, variation in Land L is pre-known from a prior calibration step (e.g. using the localization template 8), or from knowledge of the position of the energy sources 1, 2, 3, and this (plus the knowledge that N is normalized) makes it possible to recover both N and p at each pixel. Since there are three degrees of freedom (two for Λ/and one for p), intensity values / are needed for at least three directions L in order to uniquely determine both N and p. This is why three energy sources 1, 2, 3 are provided.

The stereoscopic reconstruction uses optical triangulation, by geometrically correlating the positions in the images captured by the image capture devices 4, 5 of the respective pixels representing the same point on the face (e.g. a feature such as a nostril or facial mole which can be readily identified on both images). The pair of images is referred to as a “stereo pair”. This is done for multiple points on the face to produce the initial model of the surface of the face.

The data obtained by the photometric and stereoscopic reconstructions is fused by treating the stereoscopic reconstruction as a low-resolution skeleton providing a gross-scale shape of the face, and using the photometric data to provide high-frequency geometric detail and material reflectance characteristics.

The process 100 performed by the embodiment is illustrated in Fig. 2.

In the first step 101, the system is initiated, and one of the lights 9 is illuminated.

In the second step 102, the energy sources 1,2, 3 are illuminated (e.g. one by one successively), and the image capture devices 4, 5 triggered to capture multiple images at respective times when different respective one(s) of the energy sources 1,2, 3 are in operation. As noted above, this procedure is carried out during a time period which is typically preferably no more than 0.1s, or no more than 0.2s.

In step 103, an initial version of a three-dimensional model of the face (typically including the skin and eye regions, and usually the ear regions) is formed stereoscopically. Note that in an alternative form of the embodiment, the initial 3D model may be formed in other ways, for example using a depth camera. Known types of depth camera include those using sheet-of-light triangulation, structured light (that is, light having a specially designed light pattern), time-offlight or interferometry.

In step 104, the initial 3D model is refined using the images and the photometric techniques described above. The resulting 3D model is referred to here as a skin model, since it includes an accurate model of the skin of the subject’s face. However, it may also include the subject’s hair and also a portion corresponding to (though not accurately representing) the eye regions of the subject’s face.

In step 105, the specular reflections in the images are identified, and for each eye the specular reflections are used to form a set of eye data.

This eye data may be in the form of a provisional eye model for each eye. A simple form for the provisional eye model which can be used is shown in Fig. 3. It consists of a sclera portion 10 representing the sclera (the outer white part of the eye), and a cornea portion 11 intersecting with the sclera portion 10. The sclera portion 10 may be frusto-spherical (i.e. a sphere minus a segment of the sphere which is to one side of a plane which intersects with the sphere). However, since only the front of the eyeball can cause reflections, the sclera portion 10 of the provisional eye model may omit portions of the spherical surface which are angularly spaced from the cornea portion about the centre of the sphere by more than a predetermined angle.

The centre of the sphere of which the sclera portion 10 forms a part is the centre of rotation (CER) of the eye.

The cornea portion 11 of the model is a segment of a sphere with a smaller radius of curvature than then sclera portion 10; the cornea portion 11 too is frusto-spherical, being less than half of the sphere having smaller radius of curvature. The cornea portion 11 is provided upstanding from the outer surface of the sclera portion 10 of the model, and the line of intersection between the sclera portion 10 and the cornea portion 11 is a circle. The center of the cornea portion 11 is taken as the center of the pupil. It lies on the line which passes through the center of the sphere used to define the sclera portion 10, and the center of the sphere used to define the cornea portion 11.

The provisional eye model of Fig. 3 is defined by 8 parameters (numerical values): the coordinates of the CER in a 3-D space defined in relation to the position of the imaging assembly (3 numerical values); the radius of the sclera portion 10; the direction of the gaze of the subject (2 numerical values defining the orientation of the eye); the radius of curvature of the cornea portion 11; and the degree to which the cornea portion 11 stands up from the sclera portion 10. These values are estimated from the specular reflections to form a provisional eye model. Optionally, additional knowledge may be used in this process. For example, the eyeballs of individuals (especially adult individuals) tend to be of about the same size, and this knowledge may be used to pre-set certain dimensions of the provisional eye model. Furthermore, on the assumption that subject is looking at the light 9, the orientation of the eye is pre-known.

Suppose that each of the energy sources 1,2, 3 is fired in turn, and that when each of the energy sources 1,2, 3 is fired each of the image capturing devices 4, 5 captures an image. The electromagnetic radiation produced by each energy source is reflected by each of the eyes of the subject in a specular reflection. Thus, each image captured by one of the devices 4, 5 will include at least one very bright region for each eye, and the position in that image of the very bright region is a function of the translational position and orientation of the eye. In total six images of the face are captured, and if each of them contains (in the eye) a very bright region (“glint”) with a two dimensional position in the image, then in total 12 data values can be obtained.

Using all 12 data values, it is possible for the 8 parameters of the provisional eye model to be estimated (“fitted” to the data values). This can include computationally searching for values of the desired parameters of the provisional eye model which are most closely consistent with the observed positions of the specular reflections within the images.

This is illustrated schematically in Fig. 4, which shows by crosses 12a, 12b, 12c specular reflections captured by the image capturing device 4, and by crosses 13a, 13b, 13c the specular reflections captured by the image capturing device 5. The crosses are shown in relation with the provisional eye model following the process of fitting the parameters of the provisional eye model to the observed positions of the specular reflections in the image.

As mentioned above, the number of energy sources may be increased. Suppose for example that there are six energy sources. In this case, each of the imaging devices 4, 5 could capture up to six images, each showing the specular reflection when a corresponding one of the energy sources is generating electromagnetic radiation. Again the specular reflection would cause a bright spot in the corresponding two-dimensional image, so in total, having identified in each the two-dimensional image the two-dimensional position of the bright spot, the processor would then have twenty-four data values. These twenty-four values could then be used to estimate the six numerical parameters defining the provisional eye model. This is illustrated in Fig. 5, where the six specular reflections captured by the imaging device 4 are labelled 22a, 22b, 22c, 22d, 22e and 22f. The six specular reflections captured by the imaging device 5 are shown in Fig. 5 but not labelled.

The processor expresses the provisional eye model in a coordinate system defined relative to the pre-known fixed relative positions of the energy sensors 4, 5 and the energy sources 1,2,

3. Thus, the skin model and the provisional eye model are in the same coordinate system.

In step 106 it is determined whether all the lights 9 in the display have been illuminated. If not, in step 107 the light 9 illuminated in step 101 is turned off and another of the lights 9 is illuminated,

In step 108 there is a delay (typically of 3 to 5 seconds), to allow the eye(s) of the subject to recover from any blink caused by the flashes in step 102, and for the subject’s eye(s) to stabilize at the newly illuminated light 9. Note that the delay in step 108 is typically at least a factor of 10 greater than the time taken to perform step 102. Then the process returns to step 102. Thus, another skin model and another provisional eye model are formed. In total, in the respective time period in which each of the lights 9 is illuminated, a respective skin model and respective provisional eye model are formed. The time periods are spaced apart by an amount of time substantially equal to the delay of step 108.

In step 109, the skin models for each of the time periods are brought into a common reference frame. Typically, this step employs recognizable landmarks on the skin of the skin model. The process may be carried out by identifying a number of landmarks on each skin model (typically corresponding to pre-determined landmark features, such as the tip of the nose or the plane of mirror symmetry of the face). Then a respective positional mapping (including a translational component and a rotational component) is derived between a first of the skin models and the other skin models. Each positional mapping brings the landmarks of the first skin model into register with corresponding ones of the landmarks of the respective other skin model, and represents the movement of the subject’s head between the capture time of the images used to form the first skin model and the capture time of the images used to form the respective other skin model.

The provisional eye model corresponding to the first skin model is already in the reference frame of the first skin model. The positional mapping is used to bring the other provisional eye models into the reference frame of the first skin model. Thus all the provisional eye models are converted into the common reference frame.

In step 110, the provisional eye models in the common reference frame are combined to form a (final) eye model of each of the eyes in the common reference frame. The final eye model may have the form shown in Fig. 3, including a (e.g. frusto-spherical) cornea portion 11, and a (e.g. frusto-spherical) sclera portion 10 with a centre which is the centre of eye rotation (CER). For example, the positions of the respective centres of the sclera portion 10 of each provisional eye model may be combined (e.g. their mean calculated), to derive the centre of rotation of the final eye model.

In step 111a composite model is derived of the face of the subject. This includes the eye models for each eye derived in step 109, and a skin portion derived from one or more of the skin models (in locations away from the respective eye models).

In step 112 the processor measures one or more dimensions of the composite model (the skin portion of the composite model and/or the eye model(s) of the composite model), such as the inter-pupil distance, and the distances between locations on the nose where the eyewear will be supported and the ears.

The processor stores in a data-storage device a 3D model of at least part of an item of eyewear intended to be placed in proximity of the face. The item of eyewear may be a pair of glasses (which may be glasses for vision correction, sunglasses or glasses for eye protection, or a headset for AR or VR). In step 113, the processor uses the measured dimensions of the composite model to modify at least one dimension of the 3D model of the eyewear. For example, the configuration of a nose-rest component of the object model (which determines the position of a lens relative to the nose) may be modified according to the inter-pupil distance, and/or to ensure that the lenses are positioned at a desired spatial location relative to the subject’s eyes when the eyes face in a certain direction. Furthermore, if the item of eyewear has arms to contact the subject’s ears, the length of the arms may be modified in the eyewear model based on the skin portion of the composite model to make this a comfortable fit. If the face model is accurate to within 250 microns, this will meet or exceed the requirements for well-fitting glasses.

Furthermore, at least one dimension of at least one lens of the eyewear may be modified based on the composite model. For example, as illustrated in Fig. 6, the lens 30 of the item of eyewear includes a portion 32 which is relatively close to the CER 31 of the eye model portion of the composite model, and a portion 33 which is relatively far from the CER 31. The refractive power of the lens 30 may be controlled to be different in the region 33 from that in the region 32 according to this difference in distances, so that the subject’s vision is corrected irrespective of whether the subject is looking towards the portion 32 or the portion 33. Note that this control of the refractive power may be performed in combination with any control of the refractive power which is used to make the lens 30 is into a bifocal, multi-focal or varifocal lens.

In step 114, the system uses the modified eyewear model to produce at least one component of an item of eyewear according to the model (e.g. the arms and/or the nose-rest component, and/or at least one of the two lenses). This can be done for example by three-dimensional printing. Note that if the eyewear is an item such as varifocal glasses, great precision in producing them is essential, and a precision level of the order of the 250 microns, which is possible in preferred embodiments of the invention, may be essential for high technical performance.

A number of variations are possible to the process of Fig. 2 within the scope of the invention. Firstly, the order of steps may be different (e.g. computational steps 103-105 may be performed after the steps 102, 106 and 107 have been completed. Also computational steps 103, 104 may be performed after, or in parallel with, step 105.

In another variation step 105 may not include forming a respective provisional eye model for each of the time periods. Instead, the eye data used in step 109 may be eye data describing the specular reflections, and this “raw” reflection data may be processed in step 110. In this case, the generation of the eye model may include identifying as many as possible of the specular reflections which lie substantially on a common sphere. These reflections are identified as being made by the sclera, and are used to construct the sclera portion 10 of an eye model as shown in Fig. 2, which is centred on the CER. The other reflections may be assumed to be reflections from the cornea, and are used to construct the cornea portion 11 of the eye model.

The energy sources 1,2, 3 may be designed and controlled in several ways. First, as mentioned above, it may be advantageous for the processor to control the timing of the operation of the energy sources, for example to ensure that only a selected subset of the energy sources 1,2, 3 are operating when a certain image is captured, e.g. such that only one of the energy sources is operating when any corresponding image is captured; this is usual for photometry. If the energy sources (at least, those which produce the same level of light intensity) are activated successively with no significant gaps between then during this period the total level of light would be substantially constant; this would minimize the risk of the subject blinking. Optionally, an additional image may be captured with all the light sources firing.

Secondly, the illumination system may employ polarization of the electromagnetic radiation. As described above, the processor forms the skin model using Lambertian reflections, and fits the parameters of each eye model using the specular reflections. In fact, however, the skin is not a perfect Lambertian reflector, and an eye is not a perfect specular reflector. To address this, the imaging process may use polarization to help the processor distinguish Lambertian reflection from specular reflection, since Lambertian reflection tends to destroy any polarization in the incident light, whereas specular reflection preserves polarization.

In one possibility, the energy sources 1,2, 3 would comprise polarization filters (e.g. linear polarization filters), and the image capturing devices 4, 5 would be provided with a respective constant input polarization filter, to preferentially remove electromagnetic radiation polarized in a certain direction. The choice of that direction, relative to the polarization direction of the electromagnetic radiation emitted by the energy sources 1,2, 3, would determine whether the filter causes the image capturing devices 4, 5 to preferentially capture electromagnetic radiation due to Lambertian reflection, or conversely preferentially capture electromagnetic radiation due to specular reflection. A suitable linear polarizer would be the XP42 polarizer sheet provided by ITOS Gesellschaft fur Technische Optik mbH of Mainz, Germany. Note that this polarizer sheet does not work for IR light (for example, with wavelength 850nm), so should not be used if that choice is made for the energy sources.

A further possibility would be for the imaging apparatus to include a first set of image capturing devices for capturing the Lambertian reflections, and a second set of image capturing devices for capturing the specular reflections. The first image capturing devices would be provided with a filter for preferentially removing light polarized in the direction parallel to the polarization direction of the electromagnetic radiation before the reflection and/or the second image capturing devices would be provided with a filter for preferentially removing light polarized in the direction transverse to the polarization direction of the electromagnetic radiation before the reflection. The processor would use the images generated by the first set of image capturing devices to form the skin model, and the images generated by the second set of image capturing devices for fit the parameters of the eye model.

Alternatively, each of the image capturing devices 4, 5 may be provided with a respective electronically-controllable filter, which filters light propagating towards the image capturing device to preferentially remove electromagnetic radiation polarized in a certain direction. The image capturing device may capture two images at times when a given one of the energy sources 1,2, 3 is illuminated: one image at a time when the filter is active to remove the electromagnetic radiation with the certain polarization, and one when the filter is not active. The relative proportions of Lambertian reflection and specular reflection in the two images will differ, so that by comparing the two images, the processor is able to distinguish the Lambertian reflection from the specular reflection, so that only light intensity due to the appropriate form of reflection is used to form the skin model and/or the eye model.

Thirdly, some of all of the energy sources 1,2, 3 may generate IR or near4R light. This is particularly desirable if it is not desirable for the subject to see the directional energy (e.g. because it is not desirable to make him or her blink; or because the embodiment is used at a time when the subject is looking at other things).

In a further variation of the method, the subject may be requested to keep his or her direction of vision constant, but to turn only his or her head. In this case, the display 8 would require only a single light 9, and step 107 is replaced with a step in which the subject moves his or her head.

An alternative way to use the composite model obtained in step 110 is within an augmented reality (AR) or virtual reality (VR) system. In this case, steps 112-114 of the method 100 are omitted. Instead, a respective 3-D skin model of the skin of the subject may be formed at each of a series of successive times. For example, this may be done using a process employing one or more energy sources and one more image capturing devices at known positions in a certain reference frame. Steps corresponding to steps 101 to 104 may be performed at each of the successive times.

At each one of the successive times (referred to as the “current time”), the respective skin model (“current skin model”) is compared to the skin portion of the composite model. In this way, the composite model is brought into the reference frame of the energy source(s) and image capturing device(s). Using the eye model portion of the composite model in this reference frame, the processor may calculate at least one image, which is displayed to the subject using at least one respective display device at a known respective position in the common reference frame. There may be respective images and display devices for each of the subject’s eyes. The images may such as to give the subject the experience of AR or VR in a realistic way, since the eye model of the composite model gives valuable information for generating the images.

In a further variation, the eye data collected in each time period which is indicative of the position of at least a portion of the subject’s eye, may not be derived from specular reflections. For example, it may be data obtained by tracking the movement of the subject’s iris in the images.

Yet further possibilities exist. Azuma & Bishop (“Improving static and dynamic registration in an optical see-through HMD”, Proceedings of SIGGRAPH 1994) use a two vector method to find the centre of an eye by aligning the subject’s eyepoint with two intersecting vectors. This is based on the principle that when a user aligns a central axis of his or her eye with a “vector” (i.e. a line having a certain direction and passing through a certain point), that vector passes through the centre of the eye. A similar method is used in Caudell and Mizell (“Augmented Reality: an application of heads-up display technology to manual manufacturing processes”, 1992). In the present system too, the subject might in a given time period align his or her eye with a vector (possibly one generated by a display device), so that the eye data might be data characterizing the position of the vector in a reference frame of the imaging devices used to obtain the 3-D skin model for the corresponding time period, and thereby indicating that the centre of the eye lies on this vector.

It would even be possible in principle for the eye data not to be obtained from the images. For example, E. Whitmire et al (“Eyecontact: Scleral coil eye tracking for virtual reality”, ISWC 2016) describe a system in which the position of an eye is obtained from eye data characterizing magnetic interactions with magnetic elements mounted into an element (e.g. a silicone annulus) worn by the user on the sclera of the eye. In principle, eye data obtained in this way could be used in the present invention also.

Fig. 7 is a block diagram showing a technical architecture of the overall system 200 for performing the method.

The technical architecture includes a processor 322 (which may be referred to as a central processor unit or CPU) that is in communication with the cameras 4, 5, for controlling when they capture images and receiving the images. The processor 322 is further in communication with, and able to control the energy sources 1,2, 3, and the display 8.

The processor 322 is also in communication with memory devices including secondary storage 324 (such as disk drives or memory cards), read only memory (ROM) 326, random access memory (RAM) 328. The processor 322 may be implemented as one or more CPU chips.

The system 200 includes a user interface (Ul) 330 for controlling the processor 322. The Ul 330 may comprise a touch screen, keyboard, keypad or other known input device. If the Ul 330 comprises a touch screen, the processor 322 is operative to generate an image on the touch screen. Alternatively, the system may include a separate screen (not shown) for displaying images under the control of the processor 322. Note that the Ul 330 is separate from the display 8, since the Ul 330 is typically used by an operator to control the system, whereas the display is used for the subject to look at.

The system 200 optionally further includes a unit 332 for forming 3D objects designed by the processor 322; for example the unit 332 may take the form of a 3D printer. Alternatively, the system 200 may include a network interface for transmitting instructions for production of the objects to an external production device.

The secondary storage 324 is typically comprised of a memory card or other storage device and is used for non-volatile storage of data and as an over-flow data storage device if RAM 328 is not large enough to hold all working data. Secondary storage 324 may be used to store programs which are loaded into RAM 328 when such programs are selected for execution.

In this embodiment, the secondary storage 324 has an order generation component 324a, comprising non-transitory instructions operative by the processor 322 to perform various operations of the method of the present disclosure. The ROM 326 is used to store instructions and perhaps data which are read during program execution. The secondary storage 324, the RAM 328, and/or the ROM 326 may be referred to in some contexts as computer readable storage media and/or non-transitory computer readable media.

The processor 322 executes instructions, codes, computer programs, scripts which it accesses from hard disk, floppy disk, optical disk (these various disk based systems may all be considered secondary storage 324), flash drive, ROM 326, RAM 328, or the network connectivity devices 332. In one possibility the processor 322 may be provided as a FPGA (field-programmable gate array), configured after its manufacturing process, for use in the system of Fig. 7.

While only one processor 322 is shown, multiple processors may be present. Thus, while instructions may be discussed as executed by a processor, the instructions may be executed simultaneously, serially, or otherwise executed by one or multiple processors.

Whilst the foregoing description has described exemplary embodiments, it will be understood by 5 those skilled in the art that many variations of the embodiment can be made within the scope of the attached claims.

The project leading to this application has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 710384.

Claims

1. An imaging apparatus comprising:

at least one energy source;

an imaging assembly having at least one energy sensor arranged to capture at least one 5 image of the face of a subject when the face is illuminated, the imaging assembly being operative to capture images during each of a plurality of time periods;

a processor arranged to analyze the images, by:

(a) for each of the time periods:

(i) using the images to form a respective three-dimensional skin model of the

10 skin of the subject, and (ii) obtaining eye data indicative of the position of at least a portion of an eye of the subject during the time period;

(b) converting the eye data for the time periods to a common reference frame using the respective skin models; and

15 (c) from the converted eye data obtaining an eye model comprising an estimate of the position of the centre of eye rotation (CER) of the eye of the subject.

2. An apparatus according to claim 1 in which the at least one directional energy source is a directional energy source arranged to directionally illuminate the face of the subject in at least three directions during each time period to generate photometric data, and

20 the processor is arranged to generate the skin model by:

generating an initial three-dimensional model by stereoscopic reconstruction using optical triangulation; and refining the initial three-dimensional model using the photometric data.

3. An apparatus according to claim 1 or claim 2 in which the eye model comprises a sclera

25 portion representing a sclera of the eye, and a cornea portion representing a cornea of the eye.

4. An apparatus according to claim 3 in which the sclera portion of the eye model is a portion of the surface of a first sphere centred on the centre of eye rotation, and the cornea portion is a portion of the surface of a second sphere having a smaller radius of curvature than the first sphere, the centers of the two spheres being spaced apart.

5. An apparatus according to any preceding claim which comprises a display device, the apparatus being arranged to display a different image in each of the time periods.

5

6. An apparatus according to any of preceding claim in which the processor is operative to form a composite model including the eye model for each of the subject’s eyes and a model of at least a portion of the skin of the subject.

7. An apparatus according to claim 6 further including modifying at least one parameter of a model of an item of eyewear based on a distance measurement of the composite model, and

10 to transmit instructions to cause the item to be fabricated, whereby the item is fabricated with at least one dimension dependent on the distance measurement.

8. An apparatus according to claim 7 in which the at least one parameter includes respective refractive power of each of multiple portions of a lens of the item of eyewear.

9. An apparatus according to claim 7 or 8 further comprising a 3D printer for receiving the

15 instructions from the processor and fabricating the item according to the at least one parameter.

10. An apparatus according to any of claims 7 to 9 in which the item of eyewear is a pair of glasses including reflective lenses.

11. An apparatus according to claim 6 in which the processor is operative at each of a series of successive times:

20 to generate a respective skin model, to register the respective skin model with a skin portion of the composite model, based on the registration, use the eye model to generate at least one augmented reality or virtual reality image, and display the image to the subject.

25

12. An apparatus according to any preceding claim in which processor is arranged to obtain the eye data for each time period from at least one of the images captured during the corresponding time period.

13. An apparatus according to claim 12 in which the processor is arranged to obtain the eye data at least partly from specular reflections within at least one of the images captured during the time period.

14. An apparatus according to claim 12 or claim 13 in which the processor is arranged, for

5 each time period, to use at least one image captured by at least one said energy sensor both to form the skin model and to form the eye data.

15. A computer-implemented method for obtaining information about the face of a subject, the method comprising:

(a) illuminating the face of the subject;

10 (b) capturing one or more images of the face during each of a plurality of time periods;

(c) for each of the time periods:

(i) using the images to form a respective three-dimensional skin model of the skin of the subject; and (ii) obtaining eye data indicative of the position of at least a portion of an eye of the

15 subject during the time period;

(d) converting the eye data for the time periods to a common reference frame using the respective skin models; and (e) from the converted eye data obtaining an eye model comprising an estimate of the position of the centre of eye rotation (CER) of the eye of the subject.

20

16. A method according to claim 15 in which the at least one directional energy source is a directional energy source arranged to directionally illuminate the face of the subject in at least three directions during each time period to generate photometric data, the method further comprising:

generating an initial three-dimensional model by stereoscopic reconstruction using

25 optical triangulation; and refining the initial three-dimensional model using the photometric data.

17. A method according to claim 15 or claim 16 in which the eye model comprises a sclera portion representing a sclera of the eye, and a cornea portion representing a cornea of the eye.

18. A method according to claim 17 in which the sclera portion of the eye model is a portion of the surface of a first sphere centred on the centre of eye rotation, and the cornea portion is a

5 portion of the surface of a second sphere having a smaller radius of curvature than the first sphere, the centers of the two spheres being spaced apart.

19. A method according to any of claims 15 to 18 which comprises displaying a different image to the subject in each of the time periods.

20. A method according to any of claims 15 to 19 further comprising generating a composite 10 model including the eye model for each of the subject’s eyes and a model of at least a portion of the skin of the subject.

21. A method according to claim 20 further including modifying at least one parameter of a model of an item of eyewear based on a distance measurement of the composite model, and transmitting instructions to cause the item to be fabricated, whereby the item is fabricated with

15 at least one dimension dependent on the distance measurement.

22. A method according to claim 21 in which the at least one parameter includes respective refractive power of each of multiple portions of a lens of the item of eyewear.

23. A method according to claim 21 or 22 further comprising fabricating the item of eyewear according to the at least one parameter.

20

24. A method according to any of claims 21 to 23 in which the item of eyewear is a pair of glasses including reflective lenses.

25. A method according to claim 20 further comprising at each of a series of successive times:

generating a respective skin model,

25 registering the respective skin model with a skin portion of the composite model, based on the registration, using the eye model to generate at least one augmented reality or virtual reality image, and displaying the image to the subject.

26. A method according to any of claims 15 to 25 in which the eye data for each time period is obtained from at least one of the images captured during the corresponding time period.

27. A method according to claim 26 in which the eye data is at least partly obtained from specular reflections within at least one of the images captured during the time period.

5

28. A method according to claim 26 or 27 in which, in each time period, at least one image captured by at least one said energy sensor is used both to form the skin model and to form the eye data.

Intellectual

Property

Office

Application No: GB1702871.3