WO2019025146A1

WO2019025146A1 - Eye tracking system for head-mounted display

Info

Publication number: WO2019025146A1
Application number: PCT/EP2018/068829
Authority: WO
Inventors: Lionel ANTON; Portes STÉPHANE
Original assignee: Starbreeze Ip Lux Ii S.À.R.L.
Priority date: 2017-08-01
Filing date: 2018-07-11
Publication date: 2019-02-07
Also published as: LU100354B1; TW201910865A

Abstract

The invention is directed to an eye tracking system for a head-mounted display, comprising: at least one infrared LED (6); at least one infrared fluorescent or phosphorescent dot (8) reemitting the light emitted by the LED (6); and at least one sensor (10) that is suitable for detecting the position and/or movement of a glint on an eye (22) generated by the dot (8). The invention is also directed to such an eye tracking system wherein the fluorescent or phosphorescent dots are integrated on or into a contact lens. The invention further relates to a head-mounted device with such an eye-tracking system. The invention further relates to a contact lens with an infrared fluorescent or phosphorescent feature.

Description

EYE TRACKING SYSTEM FOR HEAD-MOUNTED DISPLAY

Description

Technical field

[0001 ] The invention is directed to an eye tracking system for a head mounted device.

Background art

[0002] Head-mounted devices (HMD) such as virtual reality glasses may require the tracking of the eyes movement to broadcast an appropriate image in front of the eyes.

[0003] A known technique for tracking the eye position or movement is to detect the glint or corneal reflection of a light source. To this end, a light source and a sensor must be positioned such that they can emit light to, and detect the glints on, the eye. The head-mounted devices can be complex and the direct access from a light source to the eye and the direct detection of the glint may be difficult. This is particularly the case in large lenses or large field of view HMD. Indeed, it can be challenging to have a good illumination of the eyes by simply positioning illuminators around the lenses as the eyebrow arch, eyelashes and cheekbones become obstacle. Complex mirror arrangements do exist but these are particularly heavy.

[0004] Prior art patent document published US 2014/0198017 A1 discloses an example of such an eye tracking system in accordance with the preamble of claims 1 and 3.

Summary of invention

Technical Problem

[0005] The invention has for technical problem to provide an element of an eye tracking system which enables a better illumination of the eye and allows therefore a more reliable tracking of the eye.

[0006] The invention solves furthermore the problem of providing a more compact eye tracking system which is consequently less occluding in front of the eyes. Technical solution

[0007] The overall general inventive concept of the various embodiments of the invention is to provide IR fluorescent or phosphorescent features aiming at tracking the position or movement of an eye.

[0008] The invention is directed to an eye tracking system for a head-mounted display, the system comprising: at least one infrared LED; at least one infrared fluorescent or phosphorescent dot reemitting the light emitted by the LED; and at least one sensor that is suitable for detecting the position and/or movement of glints on an eye generated by the dot.

[0009] According to a preferred embodiment, the dot is drawn, painted or embedded on/in the surface of an object, preferably the lens of a head- mounted display. The dot can also be on the casing of the head-mounted device around the lens. The dot can be embedded into the lens or any transparent material positioned in front of the eyes and preferably close to the HMD lens. For instance, the dot can be embedded into a transparent coating or a sticker that can be put on the lens of the HMD.

[0010] The above-stated problem is also solved by an eye tracking system for a head-mounted display, the system comprising: at least one infrared LED; at least one infrared fluorescent or phosphorescent dot reemitting the light emitted by the LED, wherein the dots are integrated on or into a contact lens; and at least one sensor suitable for detecting the position and/or movement of the dot.

[001 1 ] According to a preferred embodiment, the contact lens is corrective. In other words, the contact lens can be adapted to the user's optical disability. It plays therefore a combined role of helping the tracking of the eyes and correcting the user's disability.

[0012] According to a preferred embodiment, the contact lens is non- axisymmetric. The contact lens can therefore take a pre-determined angular position on the eye through a non-homogenous weight distribution. In practice, the lens is growing thicker in a predefined angular point. This is generally used to correct astigmatism by ensuring that the lens takes a precise angular position on the eye. In the context of eye tracking, this feature can allow to monitor the local vertical, which can be useful in combination with a head inclination tracking system. The dots pattern can be non-regularly arranged (i.e. not a circular homogenous distribution) to further detect more precisely the vertical.

[0013] According to a preferred embodiment, the sensor is a camera, preferably provided with a wavelength filter that filters out the light that does not correspond to the wavelength of the glint or the dot. This is particularly useful to filter out the light coming from the IR source (i.e. LED). In absence of filter, the sensor can be saturated by the light of the IR source and can have trouble detecting the glint or corneal reflection which is of lowest intensity than the source.

[0014] According to an alternative or complementary embodiment, the at least one dot (8) is IR phosphorescent, the IR LED emits light discontinuously and the acquisition of the sensor is only made at times when the LED does not emit light and when the at least one dot (8) continues to reemit light. The LED may emit light as pulses and the periods of non-emission can be synchronized with the acquisition rate of the sensor, preferably the frame rate of the camera. This way, each acquisition of the sensor is made when the LED is OFF to avoid the saturation of the sensor. This is particularly relevant when the reemitted light of the dots is at a wavelength close to the wavelength of the LED and therefore the efficiency of a filter is not guaranteed.

[0015] According to a preferred embodiment, the LED has a wavelength between 700 and 1050 nm, preferably between 780 and 850 nm.

[0016] According to a preferred embodiment, the system comprises a plurality of IR fluorescent or phosphorescent dots positioned in a substantially circular pattern.

[0017] According to a preferred embodiment, the system comprises an infinity of IR fluorescent or phosphorescent dots arranged to form a line and/or a curve. The pattern can form a unique shape or design (QR-code like). This may be of relevance in a context where the HMD can be used by several users. This way, the user can be identified through the pattern of the contact lens. The content to be displayed on the HMD can be adapted according to the use made of the system. Namely, the displayed information can be adapted to personalized settings or can be restricted to authorized personal. The wavelength of the reemitted light can also be a means to identify a user from another user. This may be used in combination with iris recognition.

[0018] According to a preferred embodiment, the dot is made of ink, paint or phosphorescent powder and/or it reemits light at a wavelength between 700 and 1050 nm, preferably between 850 and 950 nm. For example, the commercially available infrared down-conversion powders or inks provided by the company Llewellyn Data Processing (LDP LLC) could be used.

[0019] The invention also relates to a head-mounted device with at least one and preferably two eye tracking system(s) according to the previously described embodiments. The HMD can be used for entertainment, sport activities (motorbike, motor sport, diving skiing), for shooting a movie, for carrying out scientific studies or marketing research. The HMD can be of regular field of view or larger field of view. Although it is not mandatory, providing a HMD with two eye tracking systems, i.e. one for each eye, can be useful. This can serve to adapt the image displayed to users whose eyes' movement is not synchronized. Such an HMD could be used for orthoptic treatment.

[0020] According to a preferred embodiment, the device is a military helmet, preferably for air fighter pilots, or a helmet for a sport activity or for general entertainment purposes.

[0021 ] According to a preferred embodiment, the device is virtual reality glasses, augmented reality glasses or 3D cinema glasses.

[0022] The above-stated problem is furtherly solved by a contact lens comprising an infrared fluorescent or phosphorescent feature. The feature can be a dot, a line, a pattern, and it can be painted on the lens, or can be integrated into a layer of the contact lens. It can be a geometrical figure, a symbol, a logo, etc. The lens can have the properties exposed above (corrective, non-axisymmetric, uniqueness, etc.). [0023] According to a preferred embodiment, the infrared fluorescent or phosphorescent feature has an excitation wavelength in the visible spectrum and an emission wavelength in the infrared spectrum. For instance, the fluorescent or phosphorescent feature has an excitation wavelength equal to or lower than 700 nm and an emission wavelength equal to or higher than 700 nm. Indeed, the wavelength of the light reemitted by a fluorescent or phosphorescent body is generally higher than the wavelength which excites this body. By choosing near-infrared fluorescent or phosphorescent material for said feature of the contact lens, this makes it possible to track the eye movement without the need of an IR LED. The ambient light excites the fluorescent or phosphorescent feature, which reemits light in the IR spectrum. This makes the eye tracking system of the invention very flexible and usable in many environments. One example of application is the use in combination with a safety system in a car. When the car detects that the driver blinks too often, closes his/her eyes or stares without moving the eyes, the car can warn the driver that signs of tiredness are being detected and that a break is advised.

[0024] The invention further relates to a kit comprising such a contact lens and an infrared light source. Independently of a head-mounted display, such a kit can be used in various situations. For instance, to identify from distance a specific person amongst a crowd, or to track the movement of players in a game or a sport, etc.

[0025] The invention also concerns a kit for manufacturing a lens according to claim 15 or 16, comprising a contact lens, a fluorescent or phosphorescent composition, an IR sensor and optionally an IR LED. The IR LED and IR sensor may be provided with USB cables to connect to a computer. The fluorescent or phosphorescent composition may be provided in a container or a tube with an applicator to apply drops of the composition on the contact lens. Such a kit is particularly relevant as a Do-lt-Yourself eye tracking device. The sensor can be a webcam that may be integrated into a computer device.

[0026] The invention further relates to the use of such a contact lens for tracking the position or movement of an eye in combination with a computer device, in particular a HUD, a smartphone or a smartwatch. Cars are now equipped with head-up display, smartphone or smartwatch can have a camera to recognize the user. In both cases, such devices could be controlled by eye movements. This can be useful for disabled people, laboratory experimentation and many more applications.

Advantages of the invention

[0027] The invention is particularly interesting in that by shining in front of the eye, the illumination of the eye is optimal. Furthermore, painted dots do not require wires to feed them in power. There is an enhanced flexibility as to where the IR LED can be positioned. This technique is furthermore simple to adapt to any existing head mounted devices.

[0028] The use of the contact lens is not cumbersome since some people already wear theirs.

[0029] The present invention is compact and can be adapted to any size form or shape of head-mounted device.

Brief description of the drawings

[0030] Figure 1 is a top view of the right side of a HMD according to the invention; [0031 ] Figures 2 to 4 show various patterns for the dots; [0032] Figures 5 and 6 show a contact lens according to the invention.

Description of an embodiment

[0033] Figure 1 shows (schematically) a top view cross-section of a head- mounted display 1 (HMD) according to the invention. Only the right side of the HMD is shown. The HMD comprises a casing 2 with a display 3 and an optional lens 4. Any kind of lens can be used to appropriately adapt the image displayed to the needs. The HMD is worn in front of the face 20 of the user such that the lens 4 is positioned in the field of view of the user. A source of infrared light 6, preferably an IR LED is provided.

[0034] In this example, the LED 6 is positioned on the side of the casing 2 of the HMD. The LED 6 emits light, represented on figure 1 by the light beams 6.1 . The casing 2 is provided with fluorescent or phosphorescent dots 8. The dots may have a size between a few tenth of millimetre (diameter of the dot) up to 2 mm. In this example, there is a plurality of dots positioned on the lens 4 in front of the eye. The dots 8 reemit light 6.1 emitted by the LED and can generate light at a similar or different wavelength.

[0035] This reemitted light is represented by light beams 8.1 that are drawn thicker than the light beams 6.1 (to depict the possible difference in wavelength). The light 8.1 reaches the cornea of the eye 22 of the user. This creates a glint or a corneal reflection. The light beam 22.1 illustrates the light emitted by the glint. This light is detected by an infrared sensor 10. This sensor 10 can be positioned at an appropriate location (above, below, aside, or behind the lens 4). The sensor 10 is held in the casing 2.

[0036] Depending on the position of the sensor 10, some light 6.2 from the IR source 6 can also reach the sensor 10. It may be appropriate to provide the system with a filter 12 which prevents light 6.2 from the source to reach the sensor 10 but allows light 22.1 from the eye to reach the sensor. If the reemitted light 8.1 and the reflected light 22.1 have the same wavelength as or a close wavelength to the LED light 6.1 , the filter may be combined or replaced with a synchronisation of the sensor acquisition rate and periods of time when the LED does not emit light. When the LED emits light, the phosphorescent dots accumulate energy and when the LED stops emitting, the dots keep on reemitting light.

[0037] Other positions of the LED are possible (above, in the middle, below, etc.), as long as the LED light 6.1 reaches the dots 8, in direct sight with the dots or using any IR reflective material (such as a mirror).

[0038] Figures 2 to 4 show various patterns 30, 31 , 32 for the dots 8. The preferred embodiments of the pattern are here described for the right eye. On figure 2, the pattern 30 is substantially circular. The average diameter of the pattern may be between 5 mm to 8 cm. A few dots 8 on the left side, i.e. near the nose of the user, can be aligned along a line. The pattern 30 can be positioned in the centre of the lens 4 (or field of view), or can be on the left side of the lens 4 (or field of view). On figure 3, the pattern is also substantially circular, but is more extended in the horizontal direction. This is particularly relevant for large field of view HMDs. On figure 4, the dots 8 are positioned at the periphery of the lens 4 (or field of view). All these embodiments allow a good illumination of the eye.

[0039] Needless to say, that the number of dots, or their regularity along the pattern can be varied. The shape, size or form of the pattern can be adapted to the HMD used. The size of the dots themselves can also vary, from a few tenth of millimetres to a few millimetres.

[0040] Figure 5 depicts a contact lens 40. On the left side, an isometric view of the contact lens 40 is represented, whereas the right side shows a front view of the contact lens 40. The weight distribution on the contact lens 40 is not homogenous since the bottom side 40.1 is thicker than the rest of the contact lens 40. When such a contact lens is put on an eye, the heaviest side 40.1 will be at the bottom of the iris. Since the contact lens is not fixed to the eye and can freely rotate around its axis, when the user inclines laterally his head, the contact lens will rotate and the heaviest side 40.1 will always be at the lowest point of the iris. The contact lens 40 is provided with fluorescent or phosphorescent dots 48. The pattern of the dots 48 is irregular. In this example, the two closest dots 48 are at the heaviest side 40.1 and therefore the sensor can detect the vertical direction irrespective of the user's head inclination. The direction of the vertical can be used in combination with a head inclination tracking system (accelerometers, cameras, etc.) to broadcast an image corresponding to the user's head inclination.

[0041 ] Figure 6 depicts another example of a contact lens. On the left side, an isometric view of the contact lens 50 is represented, whereas the right side shows a front view of the lens 50. The contact lens 50 is provided with a layer 51 which comprises a pattern 58. The pattern is made of a line. Any pattern can be used. The pattern can be unique to the contact lens such as a QR-code. It can be a string of characters recognizable with OCR. It has to be noted that the same patterns can also be used on the HMD of the embodiments of figures 1 to 4.

Needless to say, that dots and lines can be combined, painted on the lens and/or integrated in the contact lens.

Claims

1 . Eye tracking system for a head-mounted display (1 ), the system comprising:

- at least one infrared LED (6);

characterized in that it further comprises:

- at least one infrared fluorescent or phosphorescent dot (8) reemitting the light emitted by the LED (6);

- and at least one sensor (10) that is suitable for detecting the position and/or movement of glints on an eye (22) generated by the dot (8),

the dot (8) being drawn, painted or embedded on/in the surface of the lens (4) of a head-mounted display.

2. Eye tracking system for a head-mounted display, the system comprising:

- at least one infrared LED;

characterized in that it further comprises:

- at least one infrared fluorescent or phosphorescent dot (48, 58) reemitting the light emitted by the LED, wherein the dots (48, 58) are integrated on or into a contact lens (40, 50);

- and at least one sensor suitable for detecting the position and/or movement of the dot.

3. System according to claim 2, characterized in that the contact lens (40, 50) is corrective.

4. System according to claim 2 or 3, characterized in that the contact lens (40, 50) is non-axisymmetric.

5. System according to any of the claims 1 to 4, characterized in that the sensor (10) is a camera, preferably provided with a wavelength filter (12) that filters out the light that does not correspond to the wavelength of the glint or the dot.

6. System according to any of the claims 1 to 5, characterized in that the at least one dot (8) is IR phosphorescent, the IR LED (6) emits light discontinuously and the acquisition of the sensor (10) is only made at times when the LED (6) does not emit light and when the at least one dot (8) continues to reemit light.

7. System according to any of the claims 1 to 6, characterized in that the LED (6) has a wavelength between 700 and 1050 nm, preferably between 780 and 850 nm.

8. System according to any of the claims 1 to 7, characterized in that it comprises a plurality of IR fluorescent or phosphorescent dots (8, 48) positioned in a substantially circular pattern.

9. System according to any of the claims 1 to 8, characterized in that it comprises an infinity of IR fluorescent or phosphorescent dots (58) arranged to form a line and/or a curve.

10. System according to any of the claims 1 to 9, characterized in that the dot (8, 48, 58) is made of ink, paint or phosphorescent powder and/or it reemits light at a wavelength between 700 and 1050 nm, preferably between 850 and 950 nm.

1 1 . Head-mounted device (1 ) with at least one and preferably two eye tracking system(s) according to any of the claims 1 to 10.

12. Head-mounted device according to claim 1 1 , characterized in that the device is a military helmet, preferably for air fighter pilots, or a helmet for a sport activity or for general entertainment purposes.

13. Head-mounted device according to claim 1 1 , characterized in that the device is virtual reality glasses, augmented reality glasses or 3D cinema glasses.

14. Contact lens (40, 50) characterized in that it comprises an infrared fluorescent or phosphorescent feature (48, 58).

15. Contact lens (40, 50) according to claim 14, wherein the infrared fluorescent or phosphorescent feature has an excitation wavelength in the visible spectrum and an emission wavelength in the infrared spectrum.

16. Kit comprising a contact lens (40, 50) according to claim 14 and an infrared light source.

17. Kit for manufacturing a lens according to claim 14 or 15, comprising a contact lens, a container with an infrared fluorescent or phosphorescent composition, an infrared sensor and optionally an infrared LED.

18. Use of the contact lens (40, 50) according to claim 14 or claim 15 for tracking the position or movement of an eye in combination with a computer device, in particular a HUD, a smartphone or a smartwatch.