TW201910865A - Eye tracking system for head-mounted display - Google Patents

Abstract

The invention is directed to an eye tracking system for a head-mounted display, comprising: at least one infrared LED (6); at least one infrared fluorescent or phosphorescent dot (8) reemitting the light emitted by the LED (6); and at least one sensor (10) that is suitable for detecting the position and/or movement of a glint on an eye (22) generated by the dot (8). The invention is also directed to such an eye tracking system wherein the fluorescent or phosphorescent dots are integrated on or into a contact lens. The invention further relates to a head-mounted device with such an eye-tracking system. The invention further relates to a contact lens with an infrared fluorescent or phosphorescent feature.

Description

 Eye tracking system for head mounted displays  

The present invention relates to an eyeball tracking system for a head mounted device.

Head mounted devices (HMDs) such as virtual reality glasses may need to track eye movements to broadcast appropriate images in front of the eyes.

A known technique for tracking the position or movement of an eye is to detect the flash or corneal reflection of the source. For this purpose, the light source and sensor must be positioned such that they can emit light to the eyeball and detect flashing light on the eyeball. Head-mounted devices can be complex and can be difficult to access directly from the light source to the eyeball and directly detect the flash. This is especially the case with large lenses or HMDs with large fields of view. Rather, because the eyebrows, eyelashes, and shins become obstacles, it can be challenging to obtain good eyeball illumination only by placing an illuminator around the lens. Complex mirror configurations do exist, but these are particularly heavy.

An example of such an eye tracking system is disclosed in the prior art of US 2014/0198017 A1, in accordance with claims 1 and 3.

SUMMARY OF THE INVENTION The present invention is directed to a feature of an eyeball tracking system that provides better illumination of the eyeball and thus allows for more reliable tracking of the eyeball.

Furthermore, the present invention solves the problem of providing a smaller eyeball tracking system that necessarily has less blocking in front of the eye.

The overall inventive concept of various embodiments of the present invention provides infrared (IR) fluorescent or phosphorescent features for tracking the position or motion of the eye.

The present invention relates to an eyeball tracking system for a head mounted display, the system comprising: at least one infrared LED; at least one infrared fluorescent or phosphorescent spot that re-emits light emitted by the LED; and at least one sense A detector adapted to detect the position and/or motion of the flash produced by the point on the eyeball.

According to a preferred embodiment, the point is drawn, painted or embedded on/in the surface of an object, preferably a lens of a head mounted display. This point can also be on the housing of the head mounted display around the lens. This point can be embedded in the lens or in any transparent material in front of the eyeball and preferably adjacent to the HMD lens. For example, the point can be embedded in a clear coating or a sticker that can be placed on the lens of the HMD.

The above problem can also be solved by an eyeball tracking system for a head mounted display, the system comprising: at least one infrared LED; at least one infrared fluorescent or phosphorescent spot, which re-transmits the light emitted by the LED, wherein The points are integrated on or in a contact lens; and at least one sensor adapted to detect the position and/or motion of the point.

According to a preferred embodiment, the contact lens is for correction. In other words, the contact lens can be adapted to the visual impairment of the user. Therefore, it is accompanied by a combined role of assisting in eye catching and correcting visual impairment in users.

According to a preferred embodiment, the contact lens is non-axisymmetric. Thus, the contact lens can achieve a pre-determined angular position on the eyeball via a non-uniform weight distribution. In fact, the contact lens gradually thickens toward a predetermined angular point. This is typically used to correct astigmatism by ensuring that the contact lens achieves a precise angular position on the eyeball. In the context of eye tracking, this feature allows for the monitoring of local verticals, which can be useful in conjunction with a head tilt tracking system. The patterns of the dots may be arranged irregularly (i.e., not uniformly distributed in a circle) to further detect the perpendiculars more accurately.

According to a preferred embodiment, the sensor camera has a wavelength filter that filters out light that does not correspond to the flash or wavelength of the point. This is particularly useful for filtering out light systems from the IR source (i.e., LED). If there is no filter, the sensor will fill the light of the IR source, and thus there is a problem in detecting the minimum intensity of flash or corneal reflection compared to the IR source.

According to an alternative or supplementary embodiment, the at least one point (8) is IR phosphorescence, the IR LED intermittently emits light, and the sensor is collected only when the LED does not emit light and at the point (8) Only when the light is continuously re-emitted. The LED can emit light like a pulse, and can synchronize the period of non-emission with the acquisition rate of the sensor (preferably, the frame rate of the camera). In this manner, each acquisition of the sensor is performed while the LED is off to avoid saturation of the sensor. The wavelength of the re-emitted light at these points is close to the wavelength of the LED, and thus the performance of the filter is not guaranteed to be particularly appropriate.

According to a preferred embodiment, the LED has a wavelength between 700 nm and 1050 nm, preferably between 780 nm and 850 nm.

According to a preferred embodiment, the system includes a plurality of IR fluorescent or phosphorescent spots positioned in a substantially circular pattern.

According to a preferred embodiment, the system includes an infinite number of IR fluorescent or phosphorescent spots arranged in a line and/or a curve. The pattern can constitute a unique shape or design (like QR-code). In the case where the HMD can be used by several users, the above may be suitable. In this manner, the user can perform identity recognition via the pattern of the contact lens. The content displayed on the HMD can be adjusted according to the purpose of the system. That is, the displayed information may be suitable for personalization or may be limited to authorized personnel. The wavelength of the re-emitted light can also be a means of identifying the identity of the user. This can be used in conjunction with iris recognition.

According to a preferred embodiment, the dots are made of ink, paint or phosphorescent powder and/or they re-emit light at a wavelength between 700 nm and 1050 nm, preferably between 850 nm and 950 nm. For example, commercially available infrared down-converting powders or inks available from Llewellyn Data Processing (LDP LLC) can be used.

The invention also relates to a head mounted device having at least one and preferably two eye tracking systems in accordance with the foregoing specific examples. The HMD can be used for entertainment, sports activities (motorcycles, motorsports, diving, skiing), filming, scientific research or market research. The HMD can have a conventional field of view or a large field of view. Although not mandatory, it may be useful to provide an HMD with two eye tracking systems (ie, one system per eye). This allows the displayed image to be suitable for users whose movements of the two eyes are not synchronized. Such an HMD can be used to correct the treatment of squinting.

According to a preferred embodiment, the device is preferably used in a military helmet for an air fighter pilot or a helmet for athletic activities or general entertainment purposes.

According to a preferred embodiment, the device is virtual reality glasses, augmented reality glasses or 3D movie glasses.

The above problems are further solved by a contact lens comprising infrared fluorescent or phosphorescent features. The feature can be a dot, a line, a pattern, and it can be applied to the contact lens or can be integrated into one layer of the contact lens. It can be geometry, symbols, logos, etc. The contact lens can have the characteristics (corrected, non-axisymmetric, unique, etc.) disclosed above.

According to a preferred embodiment, the infrared fluorescent or phosphorescent feature has an excitation wavelength in the visible spectrum and an emission wavelength in the infrared spectrum. For example, the fluorescent or phosphorescent feature has an excitation wavelength equal to or less than 700 nm and an emission wavelength equal to or greater than 700 nm. Rather, the wavelength of light re-emitted by the phosphor or phosphor is typically greater than the wavelength at which the body is excited. By selecting a near-infrared fluorescent or phosphorescent material for the features of the contact lens, eye movement can be tracked without the need for an IR LED. Ambient light excites this fluorescent or phosphorescent feature, which re-emits light in the IR spectral range. This makes the eye tracking system of the present invention very flexible and useful in this multi-environment. An application example is used in conjunction with the safety system of the car. When the car detects that the driver blinks too often, closes his/her eyes or gazes without moving the eye, the car alerts the driver that he has detected signs of fatigue and recommends taking a break.

The invention further relates to a kit comprising such a contact lens and an infrared source. Independent of the head-mounted display, such a kit can be used in a variety of situations. For example, identifying a particular person in a group from a distance, or tracking the movement of an athlete during a game or sport.

The invention also relates to a kit for making the lens of claim 15 or 16, comprising a contact lens, a fluorescent or phosphorescent composition, an IR sensor and an optional IR LED. The IR LED and the IR sensor can have a USB cable for connection to a computer. The fluorescent or phosphorescent composition can be placed in a container or tube and a few drops of the composition applied to the contact lens in an applicator. Such a kit is particularly suitable for use as a do-it-yourself eye tracking device. The sensor can be a webcam that can be integrated into a computer device.

The invention further relates to the use of a contact lens for use in conjunction with a computer device, particularly a HUD, smart phone or smart watch, to track the position or movement of the eye. Current vehicles are equipped with head-up displays, while smart phones or smart watches can have cameras to identify users. In both cases, such a device can be controlled by eye movement. This can be useful for people with disabilities, laboratory experiments, and more.

Of particular interest to the present invention is that the illumination of the eye is optimal by illuminating in front of the eye. Again, the painted points do not require wires to power them. Improve where the elasticity of the IR LED can be placed. Moreover, this technique can be easily applied to any existing head mounted device.

The use of the contact lens is not troublesome because some people have worn contact lenses.

The present invention is small and thus can be adapted to a head mounted device of any size or shape.

1‧‧‧ head mounted display

2‧‧‧Shell

3‧‧‧ display

4‧‧‧ optical lens

6‧‧‧Infrared source

6.1‧‧‧ Beam

6.2‧‧‧Light

8‧‧‧Fluorescent or phosphorescent spots

8.1‧‧‧ Beam

10‧‧‧Infrared sensor

12‧‧‧ Filters

20‧‧‧ Faces

22‧‧‧ eyeballs

22.1‧‧‧ Beam

30‧‧‧ pattern

31‧‧‧ pattern

32‧‧‧ patterns

40‧‧‧Contact lenses

40.1‧‧‧ bottom side

48‧‧‧Fluorescent or phosphorescent spots

50‧‧‧Contact lenses

51‧‧‧ layer

58‧‧‧ pattern

1 is a top view of the right side of the HMD in accordance with the present invention; FIGS. 2 through 4 show various patterns of dots; and FIGS. 5 and 6 show contact lenses in accordance with the present invention.

Figure 1 (schematically) shows a top cross section of a head mounted display 1 (HMD) in accordance with the present invention. Only the right side of the HMD is displayed. The HMD includes a housing 2 having a display 3 and an optical lens 4. Any type of lens can be used to properly tailor the displayed image to the desired one. The HMD is worn in front of the user's face 20 such that the lens 4 is within the user's field of view. An infrared source 6, preferably an IR LED, is provided.

In this example, the LED 6 is located on one side of the housing 2 of the HMD. LED 6 emits light, which is represented in Figure 1 by beam 6.1. The housing 2 is provided with a fluorescent or phosphorescent spot 8. These points can have dimensions ranging from a few tenths of a millimeter (diameter of dots) to between two millimeters. In this example, there are a plurality of points on the lens 4 in front of the eye. These points 8 re-emit light 6.1 emitted by the LEDs and can produce light of similar or different wavelengths.

The re-emitted light is represented by beam 8.1, which is drawn thicker than beam 6.1 (to depict potentially different wavelengths). Light 8.1 reaches the cornea of the user's eye 22. This produces a flash or corneal reflection. Beam 22.1 represents the light emitted by the flash. This light is detected by the infrared sensor 10. This sensor 10 can be in place (above, below, beside or behind the lens 4). The sensor 10 is fixed in the housing 2.

Depending on the position of the sensor 10, light 6.2 from the IR source 6 may also reach the sensor 10. It is suitable to provide a filter 12 for the system that prevents light 6.2 from the IR source from reaching the sensor 10, but allows light 22.1 from the eye to reach the sensor. If the re-emitted light 8.1 and the reflected light 22.1 have the same or near the wavelength of the LED light 6.1, the filter can be combined with or replaced by the synchronization of the sensor acquisition rate with the time period when the LED does not emit light. When the LED emits light, the phosphor dots accumulate energy, and when the LED stops emitting, these points continue to re-emit.

LEDs in other locations are possible (above, in the middle, below, etc.) as long as the LED light 6.1 is looking directly at these points or using any IR reflective material (eg, a mirror) to reach these points 8.

Figures 2 to 4 show various patterns 30, 31, 32 of these points 8. A preferred embodiment of the pattern is described herein for the right eye. In Figure 2, the pattern 30 is generally circular. The average diameter of the pattern can be between 5 mm and 8 cm. A few points 8 on the left side (i.e., near the nose of the user) can be aligned. The pattern 30 can be located at the center of the lens 4 (or field of view) or on the left side of the lens 4 (or field of view). In Fig. 3, the pattern is also substantially circular, but extends more in the horizontal direction. This is especially suitable for large field of view HMD. In Figure 4, these points 8 are located around the lens 4 (or field of view). All of these specific examples allow for good eyeball illumination.

Needless to say, the number of dots or their regularity along the pattern is variable. The shape, size or form of the pattern can be adapted to the HMD used. The dimensions of these points themselves are also variable, ranging from a few tenths of a millimeter to a few millimeters.

FIG. 5 depicts a contact lens 40. On the left side, an isometric view of the contact lens 40 is shown, while the right side shows a front view of the contact lens 40. The weight distribution on the contact lens 40 is not uniform because the bottom side 40.1 is thicker than the remainder of the contact lens 40. When such a contact lens is worn on the eyeball, the heaviest side 40.1 will be at the bottom of the iris. Because the contact lens is not fixed to the eyeball and is free to rotate about its axis, when the user tilts his head laterally, the contact lens will rotate and the heaviest side 40.1 will always be at the lowest point of the iris. The contact lens 40 has a fluorescent or phosphorescent spot 48. The pattern of these points 48 is irregular. In this example, the two closest points 48 are at the heaviest side 40.1, so the sensor can detect the vertical direction regardless of the user's head tilt. The vertical direction can be used in conjunction with a head tilt tracking system (accelerometer, camera, etc.) to broadcast an image corresponding to the user's head tilt.

Figure 6 depicts another example of a contact lens. On the left side, an isometric view of the contact lens 50 is shown, while the right side shows a front view of the contact lens 50. The contact lens 50 has a layer 51 comprising a pattern 58. The pattern is made up of lines. Any pattern can be used. The pattern can be unique to the contact lens like QR-code. It can be a string of characters that can be recognized by OCR. It must be noted that the same pattern can also be used on the HMD of the specific examples of FIGS. 1 to 4.

Needless to say, the dots and lines can be combined, applied to the lens and/or integrated into the contact lens.

Claims (19)

 An eyeball tracking system for a head mounted display (1), the system comprising: at least one infrared LED (6); characterized in that the system further comprises: at least one infrared fluorescent or phosphorescent spot (8) that re-emits Light emitted by the LED (6); and at least one sensor (10) adapted to detect the position and/or motion of the flash produced by the point (8) on an eyeball (22).    The eye tracking system of claim 1, wherein the point (8) is drawn, painted or embedded on/in the surface of an object, preferably a lens (4) of a head mounted display.    An eyeball tracking system for a head mounted display, the system comprising: at least one infrared LED; wherein the system further comprises: at least one infrared fluorescent or phosphorescent spot (48, 58) re-emitted by the LED Emitted light, wherein the points (48, 58) are integrated on or in a contact lens (40, 50); and at least one sensor adapted to detect the position and/or motion of the point.    The eyeball tracking system of claim 3, wherein the contact lens (40, 50) is for correction.    The eye tracking system of claim 3 or 4, wherein the contact lens (40, 50) is non-axisymmetric.    The eye tracking system of any one of claims 1 to 5, wherein the sensor (10) is a camera, preferably having a wavelength that filters out light that does not correspond to the flash or the wavelength of the point Filter (12).    The eye tracking system of any one of claims 1 to 6, wherein the at least one point (8) is IR phosphorescence, the IR LED (6) intermittently emits light, and the sensor (10) is collected only in This LED (6) does not occur when no light is emitted and when the at least one point (8) continues to re-emit light.    The eye tracking system of any one of claims 1 to 7, wherein the LED (6) has a wavelength between 700 nm and 1050 nm, preferably between 780 nm and 850 nm.    The eye tracking system of any of claims 1 to 8, wherein the system comprises a plurality of IR fluorescent or phosphorescent spots (8, 48) positioned in a substantially circular pattern.    The eye tracking system of any of claims 1 to 9, wherein the system comprises an infinite number of IR fluorescent or phosphorescent spots (58) arranged in a line and/or a curve.    The eye tracking system of any one of claims 1 to 10, wherein the point (8, 48, 58) is made of ink, paint or phosphor powder and/or it is between 700 nm and 1050 nm, preferably 850 nm The light is re-emitted at a wavelength between 950 nm.    A head mounted device (1) having at least one and preferably two eye tracking systems of any one of claims 1 to 11.    A head mounted device according to claim 12, wherein the device is preferably used for a military helmet of an air fighter pilot or a helmet for athletic activities or general entertainment purposes.    A head mounted device as claimed in claim 12, wherein the device is a virtual reality eyewear, augmented reality eyewear or 3D cinema eyewear.    A contact lens (40, 50) characterized in that the contact lens comprises an infrared fluorescent or phosphorescent feature (48, 58).    The contact lens (40, 50) of claim 15 wherein the infrared fluorescent or phosphorescent feature has an excitation wavelength in the visible spectrum and an emission wavelength in the infrared spectrum.    A kit comprising the contact lens (40, 50) of claim 15 and an infrared source.    A kit for making a contact lens of claim 15 or 16, comprising a contact lens, a container having an infrared fluorescent or phosphorescent composition, an infrared sensor, and an optional infrared LED.    A use of a contact lens (40, 50) of claim 15 or 16 for use in conjunction with a computer device, particularly a HUD, smart phone or smart watch, to track the position or movement of the eye.