OA20513A - A contact lens system - Google Patents

Abstract

A contact lens system (110) for placing in an eye for augmented reality systems is disclosed. The contact lens system (110) comprises a display (111) comprising a matrix of display elements; a driver unit (112) configured to receive data from a host (120) and to present

Description

O.A.P.I. - B.P. 887, YAOUNDE (Cameroun)-Tel. (237) 222 20 57 00-Site web: http:/www.oapi.int - Email: oapi@oapi.int

A CONTACT LENS SYSTEM

TECHNICAL FIELD

Embodiments herein relate to a contact lens System and method therein. In particular, they relate to a contact lens System with dynamic active area display for augmented reality Systems.

BACKGROUND

Display s in augmented reality Systems are evolving fast. Research has been done in the fields of adding electronics on a contact lens. Lens electronîcs hâve been used for multiple applications, such as dynamic vision correction or displaying images. In R. Bîum et.al, “Enhanced electro-active lens System,” 2007, electro-active vision correction has been proposed as means of adjusting lens correction depending on different conditions, such as distance from objects being viewed, ambient light or size of the pupil. In B. A. PARVIZ, “Augmented Reality in a Contact Lens,” IEEE, 2010, researchers at University of Washington made a contact lens with 64 pixels that was tested on rabbits.

The pupil dilates, i.e., hâve different sîzes, at different illumination, state of mind, distance to the object in focus, etc. Pupil size does not directly affect the field of vision, but it does affect the perceived depth-of-fie!d, such that objects will appear blurrier in the edges ofthe vision with smaller pupil size, as described in S. Marcos, eLaL, “The depthof-field of the human eye from objective and subjective measurements,” Vision Res., 1999

Things are different for objects displayed very close to the eyes, e.g., in a contact lens display. if the pupil size is smaller than the display size. then light from the edges of the dispîay will not be perceived. That means that a display on a contact lens may be too small for the pupil at one scénario and too large in another.

US2014/0240665A1 discioses an eye-facing pupil diameter sensing System for an ophthalmic lens comprising an electronic System. The eye-facing pupil diameter sensing System is utilized to détermine pupil diameter and use this information to controi various aspects of the ophthalmic lens. The pupil diameter sensor is implemented as an array of smaller sensors placed at various locations in the contact lens to sam pie various points on the iris. Sensors may détermine pupil diameter and changes thereof by detecting light reflection, impédance, electromagnetic field, neural activity, muscle activity, and other parameters as are known in the ophthalmic art. The possibility of incorporating an image display into the lens is briefly mentioned.

SUMMARY (t is therefore an object of embodiments herein to provide an improved contact lens System and method to counter act the problem described above.

According to one aspect of embodiments herein, the object is achieved by a contact lens system for placing in an eye. The contact lens System comprises a dispiay comprising a matrix of display éléments; a driver unit configured to receive data from a remote host and to présent the data on the display; and a set of sensors integrated on the display for measuring pupil size. The driver unit is further configured to read outputs from the set of sensors to détermine the pupil size and adjust a size of an active area of the display based on the pupil size by actîvating and deactivating the display éléments.

According to one aspect of embodiments herein, the object is achieved by a method performed in a contact lens system for placing in an eye. The contact Sens System comprises a display comprising a matrix of display éléments; a driver unit configured to receive data from a remote hosi and io présent the data on the display, and a set of sensors integrated on the display for measuring pupil size. The driver unit reads outputs from the set of sensors to détermine the pupil size and adjust a size of an active area of the display based on the pupil size by activating and deactivating the display éléments.

In other words, the contact lens system and method therein according to embodiments herein enables the size of the display being dynamically adjusted according to the pupil size. By measuring the pupil size, the dispiay may dynamicaily change in size so that maximum amount of pixels that can be perceived by the user, can be used to display content.

Some advantages the contact lens system according to embodiments herein are that the user will always be able to see the maximum size of the dispiay and that the data to be presented on the display may be dynamically adjusted based on the size of active area ofthe display.

Therefore, the embodiments herein provide an improved contact lens system and method foraugmented reahty Systems.

BRIEF DESCRIPTION OF THE DRAWINGS

Examples of embodiments herein are described in more detail with reference to attached drawings in which:

Figure 1 is a schematic block diagram iîiusîrating an augmented reality system in which a contact lens system according to embodiments herein may be implemented;

Figure 2 is a schematic showing a contact lens system wth different display sizes and photo sensors according to one embodiment herein;

Figure 3 is a schematic showing a contact lens System with different display sizes and photo sensors according to another embodiment herein;

Figure 4 is a schematic showing a contact lens System where sensors are positioned by each sensor replacing one or more display pixels;

Figure 5 is a schematic showing a contact lens system where sensors are positioned in between rows and columns of the dispïay éléments;

Figure 6 is a flow chart illusirating a method performed in a contact lens System according to embodiments herein;

Figure 7 is a diagram iltustrating responses of photo sensor on an LTPS substrate; and Figure 8 is a diagram illustrating activation curves for a contact lens System with different numbers of sensors according to embodiments herein.

DETAILED DESCRIPTION

Figuré 1 depicts a block diagram of an augmented reality system 100 in which a contact lens system 110 according to embodiments herein may be implemented. The augmented reality System 100 comprises a contact lens system 110 for placing in an eye and a host 120 that sends information and interacts with the contact lens system 110. The contact lens System 110 comprises a display 111 comprising a matrix of display éléments, a driver unit 112 configured to receive data from the remote host 120 and to présent the data on the display 111, a set of sensors 113 for measuring the pupil size. The set of sensors 113 may be integrated on the display 111.The driver unit 112 Controls what is outputted on the display 111 and handles the input from the set of sensors 113. The Host 120 interacts with the contact lens System 110 via the driver unit 112. The display 111 with the matrix of display éléments is positioned on the contact lens and is oKIa tn amii lïnKt rn a etniMnrâd iV VIHIL 11^1 IL il i U U Xj W ITVÎS J .

The contact lens system may comprise a pair of contact lenses, one lens for each eye of the user, The lenses may be identical and display the same or different data.

The set of sensors 113 may be photodiodes that measures the light reflected from the eye. Other types of pupil size or diameter sensors may also be used. For example, as disclosed in US2014/0240665A1, the set of sensors may be a single- or multi-tum coil antenna. Such an antenna may receive eiectromagnetic radiation from the eye as the muscles controlling the iris contract and relax. It is well-known in the relevant art that muscle and neural activity of the eye may be detected through changes in eiectromagnetic émissions, for example with contact électrodes, capacitive sensors, and antennas. In this manner, a pupil diameter sensor based on a muscle sensor may be implemented. The pupil diameter sensor may also be implemented as one or more contact- or capacitive électrodes designed to measure impédance across the eye. Impédance may be used to detect changes in pupil diameter. For example, the impédance measured across the iris and pupil may change appreclably depending on pupil diameter. A pupil diameter sensor piaced at the appropriate location on the eye and properly coupled to the eye could detect these changes in impédance and hence the pupil diameter or size. However, not ali of these sensors can be încorporated into the display. Some of these may be piaced outside the display, as shown in Figure 1 sensors 113 with dotted line,

The display 111 may hâve at least two display sizes defined by a matrix with different numbers of rows and columns of the display éléments. Each display size is herein referred to as a valid active area setting of the display. So an active area of the display may be adjusted to hâve different sizes, where the numbers of rows and columns of the display éléments which are active for different dispîay sizes are different. The display éléments may herein also be referred to as pixels. The amount of available display sizes varies from minimum of two, i.e. minimum and maximum, and up to the number of pixels on the diagonal of the display divided by two, if ail pixel row-coiumn interceptions hâve a sensor. Sut there is no reason of having a smaller active area than the smallest pupil dimension. Normally the pupil ranges from 2mm-8mm in diameter. The display 111 and its active areas may be in different shapes, where a rectangular shape and a round shape are most common for a contact lens.

Figure 2 shows one example of a contact lens 110 with different display sizes in rectangular shapes and a set of sensors 113. As shown in Figure 2, the set of sensors may be positioned diagonaüy in a corner within the display 111 at an edge of each display size. The set of photo sensors may be any format that is supported by the display 111,

Figure 3 shows another example of a contact lens 110 with different display sizes in a concentric pixel arrangement and a set of sensors 113. As shown in Figure 3, the set of sensors may be positioned at different places within the display at an edge of each display size.

For both figure 2 and 3, al! four “corners” of possible activa areas may be used. Then it will be less obvious where the sensors are positioned from the useris perspective. This wil! give a better user expérience that is less perceived disturbance for the user.

The sensors 113 shown in Figures 2 and 3 may be photo sensors. Each photo sensor is designed to look into the eye and are shielded from light comin g from the outside, thus only activated by light that is reflected in the eye. Measurements are performed when the display éléments or pixels are in off State to avoid crosstalk the measurement from the display.

The set of photo sensors may be positioned at intersections of the rows and columns of the display éléments matrix, as shown in Figure 4 That is, they may be positioned by each sensor replacing one or more display éléments or pixels. Then the places occupjed by the sensors will appear as dead pixels, When replacing one or more display éléments or pixels, the photo sensors may be distributed in different arrangements, as long as they are in an intersection of a row and column, for example in the corners of each display size.

It is also conceivable to position the set of sensors in between rows and columns of the display éléments, as shown in Figure 5. Then the rows and colurnns of the display éléments will stay intact and places occupied by the sensors will not appear as dead pixels, but the sensors still might be perceived as such if they are large enough.

According to the embodiments herein, the driver unit 112 is further configured to read outputs from the set of sensors 113 to détermine the pupil size and adjust a size of an active area of the display 111 based on the pupil size by activaiing and deactivating the display éléments.

According to some embodiments herein, the contact lens system 110 may further comprise one or more iR diodes placed close to a sensor to provide light towards the eye to reflect.

In the following a method performed in the contact lens system 110 for adjusting a size of an active area of the display 111 will be described in detail with reference to Figure 6. As described above, the contact lens system 110 comprises a display 111 comprising a matrix of display éléments, a driver unit 112 configured to receive data from a host 120 and to présent the data on the display 111 and a set of sensors 113 integrated on the display 111 for measunny pupil size. The method comprises the following actions, which actions may be performed in any suitable order.

Action 610

Before each new frame to be presented on the display 111 or at regular intervals, e.g. every tenth frame, a few times per second or every second, the driver unit 112 checks the set of sensors 113 in order to detect the pupil size. Even if the processing and reading of the sensor signais only consume low power, a less frequent check will save some power, The driver unit 112 reads outputs from the set of sensors 113, The photo sensors are directed into the eye to be able to distinguish the edge between the pupil and the iris. Depending on the distance between the photo sensors and the number of photo sensors, the response curves when reading out the photo sensors will differ. See description of Figure 9 below.

In some cases, the light from the display éléments may be used as the light source for réflectance measurements, i.e. in dark environments. This is to get the best possible signal to noise ration (SNR) to be able to find a threshold for the reflected light for the set of sensors.

In one embodiment, the driver unit 112 may activate the relevant display éléments to provide light towards the eye to reflect. For example, the display éléments surrounding one photo sensor may be lighted up and the reflected rays from the display éléments will be measured by the photo diode. Suitably the display éléments outside the current pupil size should be activated to be able to find the edge. Further the display éléments outside the current active area of the display may also be activated to provide light.

The driver unit 112 may activate the relevant display éléments one at a time or ail at once, i.e. the relevant display éléments may be flashed quickly at once or may be activated one by one in a sweeping manner.

In one embodiment, one or more infrared (IR) diodes may be placed close to a sensor to provide light towards the eye to reflect. One or several additional IR diodes may be plaœd next to or close to any one or ail of the photo sensors. The direction of the light should be towards the eye. By adding the IR diode there is no need for surrounding illumination to be able to detect the pupil size. The reflected IR rays from the IR diodes are measured instead. The benefit is that it results a more stable measurement as it is not dépendent on enough light from the surroundings. The down part is if at least one more pixel area for the display is exchanged for an IR diode, this leaves less space for pixels on the display.

Action 620

ΤηΘ υΠνθΓ unit 112 υθιθΠΤϊίΠΘ» ΐπβ püpil Size based 0Π thê OütpüiS ffOrTi the set Of se n sors.

Depending on the pu pii diameter, sensors at various distances from the c-enter of the iris will detect different amounts of reflected light. For example, when the iris is dilated most of the sensors may detect little light because of the large, dark pu pii. Con verse ly, when the iris is constricted most sensors may detect higher light because of reflection of the iris.

Figure 7 shows a simplified mechanism how to use photo diodes to detect the pupil size. The driver unit 112 may detect the edge between the pupil and iris by comparing the detected light from the set of sensors. In the pupil area, there is aîmost no light reflection, so photo diodes 701, 702 will not detect any reflected light and no output signal. Photo diodes 703, 704, 705... wsll detect reflected light and hâve output signal. By comparing the detected light from the set of photo diodes 701, 702, 703, 704... the edge between the pupil and iris may be found and the display size is selected based on this.

The driver unit 112 may détermine the size of pupil by comparing the reflected light with a threshold. The granularity of the sensors is predefmed or configured, i.e. the position, the distance between each sensor and how many sensors per area. The number of photo sensors that get reflected light enough to hâve an output signal over the threshold is counted and from that the pupil size is decided and the display size is selected based on this.

The différence in signai level from the photo sensors will be dépendent on whether the light that is reflected by the iris or by the inner area surrounded by the iris, i.e. the pupil, with hardly any reflection. Différences may also a ri se with an iris of different color and resulting different réflectance at different areas of the iris and between different users.

So the light reflected will dépend on the color of the iris and of course on the amount of light emitted from the environment and from the display. Therefore the threshold should be adjusted or calibration may be needed to take account these variations. The threshold for reflected light may be adjusted depending on the réflectance or brightness ofthe iris.

A calibration process may be implemented where the host 120 or the driver unit 112 may control the light output of the contact lens display 111. The gain ofthe photo sensors may also be controlled to find a suitable level of sensitîvity. If no edge is detected between the pupil and iris, threshold can be lowered, while if multiple edges are detected, threshold can be increased.

There might be problems if the iris is dark brown and the light conditions of the environment and the display are low. Then the light reflected might be too little. Turning on display éléments shortly before measuring or using IR diodes may solve this problem.

Photo sensors on a Low-temperature polycrystalline Silicon (LTPS) substrate may be used which hâve the wavelength sensitivity as shown in Figure 8, according to the article of W.J. Chiang et. al., “Silicon nanocrystal-based photosensor on low-temperature polycrystalline-silicon panels, Applied Physics Letter, 91, 051120, 2007, Figure 8 shows light absorption spectrum of samples with Silicon nanocrystal layers of 100, 200, and 300 nm thickness monochromatically illuminated by a constant optical power of 35 W/cm² and a bias voltage of 2.5 V. Other substrates that can be used may be Amorphous Silicon (ASi), Indium Gallium Zinc Oxide (IGZO) or similar technologies.

Action 630

The driver unit 112 détermines if the pupil size is the same as from the previous measurement. If the pupil has not changed size, the driver unit 112 will check and read outputs from the set of sensors 113 again. If the pupil has changed size, the driver unit 112 wiil adjust the size of an active area of the dispiay based on the pupil size by activating and deactivating the display éléments.

So once the pupil size is determined based on the threshold and the granularity of the photo sensors, the active area of the display may need to be decided and adjusted. That is appropriate number of rows and columns of the display éléments may be activated based on the pupil size. Depending on the number of photo sensors placed in the display 111, the précision of the active area size will differ. Figure 9 shows an example of activation curves where 3, 6 and 18 photo sensors are used, where the y axis is the pupil size ranging from 2mm to 8mm, and the x axis is the number of photo sensors included in the set of sensors within the active area that ranges from 2mm to 8mm, indicated as the amount of possible display sizes. It can be seen from Figure 9, that less photo sensors will give a jagged activation curve, i.e. when using fewer photo sensors, the adjusting ofthe display active area wiîi hâve larger discrète steps.

Action 640

Once the size of the active area is decided based on the threshold and the granularity of the photo sensors, the new active area size needs to be communicated. So according to some embodiments herein, the driver unit 112 may send the size of the active area of the display to the host 120.

There are at least two different ways to handie the different active area sizes of the display.

According ίο some embodiments herein, the driver unit 112 either transmits the new display size to the host 120 or the host 120 asks if there is a new size and retrieves the new size from registers in the driver unit 112. The host 120 then adjusts its internai display buffers and settings so that a correct size of image is sent to the driver unit 112 in the contact lens 110. If the host can adjust the image size, power will be saved in the whole system 100 as the image réduction is done aiready on the host.

Action 650

According to some embodiments herein, the driver unit 112 may always receive the same size of image and depending on the active area size that is decided based on the photo sensors, the driver unit 112 will reduce the size of the image accordingly. This is a more flexible solution which will give the host 120 more freedom. The amount of data handled in the host 120 and that is transferred to the driver unit 112 is the same (maximum) size independent on the active area size of the display, thus consuming more power.

So according to some embodiments herein, the driver unit 112 may be configured to adjust frame buffer size of an image received from a host based on the size of active area of the display.

Action 660

The pupii size will be affected by changes in the amount of îight entering the eye. In order not to change the pupii size when the luminance is changed on the lens display due to the change of the display size or other reasons, there is a control that keeps the same amount of lumen output from the display. So when the active area of the display gets larger the luminance of the display will drop so that the same amount of lumen will hit the retina. The opposite will happen when active display area gets smaller, the luminance of the display will increase to keep a consistent lumen.

So the total luminance out of ali active dispiay éléments shouid be kept at or not exceeding a defined value, e.g. a maximum lumen, max_lumen, as expressed in Eq.1 :

Σ Îumen_pixel = max Jumen Eq. 1

That is the maximum lumen out of the display shouid always be the same independent on how many active dispiay éléments. For A R applications, the ssze of the Virtual objects is important to consider. Displaying large Virtual objects might produce lots □f lumens, while displaying short texts and bounding boxes might require less. Therefore the maximum lumen output shouid dépend on that too.The defined lumen value may be a preferred value or automatic adjusted according to likings of individual users. This is done to hâve a more comfortable usage and the user shouid expérience the same îight intensity to the retina.

Therefore accordiny to some emuodiments herein, the driver unit 112 may be configured to adjust luminance of the active display éléments to keep the total amount of lumen output from the display at a defined value. The driver unit 112 may be further configured to adjust the defined value according to preferred settings of individual users.

To summarize, the contact iens System 110 and method therein according to the embodiments herein provide an improved Visual System for augmented reaiity Systems with at least the foilowing advaniages:

Eye-mounted displays bring significant advantages in companson to non-eye10 mounted dispiays.

From the usabilîty perspective, contact lenses could be promising as wearabie dispiays, since they are naturally discrète, and frie form factor is widely accepted, Moreover, the amount of energy required to drive the dispiays would be significantly less than dispiays that are farther away.

By measuring the pupit sîze, the display can dynamically be changed in size so that maximum amount of pixels can be used to display content.

The user will always be abie to see the maximum size of lhe display and that the data needed is dynamically adjusted based on the active display size.

integrating the display and the sensors in a unique way by manufacturing the 20 sensors in the same process steps as the display éléments simplifies the manufacture.

The system always provides a display with maximum field of view, regardless of changes to the pupil size, and without wasting power on pixels that are outside the field of view. The system saves power by reducing the display size and turn off pixels in the periphery, which would not be able to be seen anyway since they would be out of the field 25 of view of the eye. When the pupil size increase, then the display size can get larger again to fil! the field of view.

The system increases the user expérience by maintaining perceived brightness of the display constant or at a preferred/comfcrtable 'evel by adjusting the display luminance based on the display size. This can also be seen as an optîmization on power usage.

There is no need for additional hardware to adjust the size of the active area of the display. The contact iens system itself indudes ail needed hardware to accompîish this.

When using the word comprise” or “comprising” it shall be interpreted as nonlimiting, î.e. meaning consist at least of.

The embodiments herein are not limited to the above described preferred embodiments. Various alternatives, modifications and équivalents may be used. Therefore, the above embodiments should not be taken as limiting the scope of the invention, which is defined by the appended daims.

Claims (18)

1. A contact Sens System (110) for placing in an eye comprising:

a display (111) comprising a matrix of display éléments;

a driver unit (112) configured to receive data from a host (120) and to present the data on the display (111);

a set of sensors (113) integrated on the display (111) for measuring pupsl size; and wherein the driver unit (112) is further configured to read outputs from the set of sensors to détermine the pupil size and adjust a size of an active area of the display (111) based on the pupil size by activating and deactivating the display éléments,

2. The contact lens system (110) according to claim 1, wherein the display has at least two display sizes defined by different numbers of the rows and columns of the display éléments and the set of sensors are photodiodes to detect light refiected from the iris of the eye.

3. The contact lens system (110) according to claim 2, wherein the set of sensors are positioned diagonaiiy within the display at an edge of each display size.

4. The contact Sens System (110) according to claim 2, wherein the set of sensors are positioned at different places within the display at an edge of each display size.

5. The contact lens System (110) according to any one of daims 1-4, wherein the set of sensors are positioned at intersections of the rows and columns of the display éléments matrix.

6. The contact lens system (110) according to any one of daims 1-4, wherein the set of sensors are positioned in between of the display éléments.

7. The contact lens system (110) according to any one of daims 1-6, wherein the driver unit is further configured to détermine the size of pupil by comparing the refiected light with a threshold.

8. The contact lens system (110) according to claim 7, wherein the driver unit is further configured to adjust the threshold based on the iris réflectance.

S. The contact lens System (110) according to any one of claims 1-8, wherein the driver unit is further configured to adjust luminance of the active display éléments to keep the total amount of lumen output from the display at a defined value.

10. The contact lens System (110) according to claim 9, wherein the driver unit is further configured to adjust the defined value according to variable user settings.

11. The contact lens System (110) according to any one of claims 1-10, wherein the driver unit is further configured to activate relevant display éléments to provide light towards the eye.

12. The contact lens system (110) according to claim 11, wherein the driver unit is further configured to activate the relevant display éléments one at a time or ail at once.

13. The contact lens system (110) according to any one of claims 1-12 further comprising one or more IR diodes placed close to a sensor to provide light towards the eye.

14. The contact lens system (110) according to any one of claims 1-13, wherein the driver unit is further configured to send the size of the active area of the display to the host.

15. The contact lens System (110) according to any one of claims 1-14, wherein the driver unit is further configured to adjust frame buffer size of an image received from the host based on the size of active area of the display.

16. A method for adjusting a size of an active area of a display în a contact lens system placed in an eye, wherein the display comprises a matrix of display éléments, the method comprising:

reading (610) outputs from a set of sensors integrated on the display; determining (620) pupil size based on the outputs from the set of sensors;

adjusting (630) a size of an active area of the display based on the pupil size by activating and deactivating the display éléments.

17. The method according to claim 16, further comprising sending (640) the size of the active area of the display to a host.

18. The method according to any one of claims 16-17, further comprising adjusting (650) frame buffer size of an image received from a host based on the size of active area of the display.

19. The method according to any one of daims 16-18, further comprising adjusting (660) luminance of the active display éléments to keep the total amount of lumen output from the display at a defined value.