EP4654882A1 - Apparatus and method for touchless monitoring of rapid dynamics in pupil size and gaze direction through closed eyes - Google Patents

Abstract

A system and method for combining short-wave infrared (SWIR) imaging with image processing algorithms to measure pupil size dynamics in closed eye conditions, either in response to an external stimulus (such as light-induced PLR) or occurring spontaneously during sleep or anesthesia. The system, with deep learning-based analysis, can further estimate gaze direction during eye movements and achieve robust pupillometry. The system has at least one electromagnetic radiation source for radiating electromagnetic energy in a spectral range of 700 nm to 2,000 nm, and an imaging device for detecting electromagnetic energy and sending image data to a processor which performs algorithmic and deep- learning based analysis of image data to continuously track changes in pupil diameter and gaze direction over time.

Description

APPARATUS AND METHOD FOR TOUCHLESS MONITORING OF RAPID DYNAMICS IN PUPIL SIZE AND GAZE DIRECTION THROUGH CLOSED EYES.

Field of the Invention.

The present invention relates generally to the field of pupillometry for neurological examinations for use in clinical settings ranging from critical care, through sleep, anesthesia, endocrinology and drug addiction, to cardiology and psychiatry.

Background of the Invention.

Evaluation of gaze, pupil size, and pupillary light reflex (PLR) play a crucial role in neurological examination and in other clinical domains, as well as in neuroscience. Pupillometry refers to measuring dynamics in pupil size, which is continually adjusted according to levels of ambient light and arousal. Dark settings and high arousal lead to mydriasis (dilation) whereas illumination and low arousal lead to miosis (contraction). The PLR adjusts pupil size to variations in light intensity. PLR is typically evaluated by measuring pupil size in response to a brief flash of light. In healthy individuals, PLR is symmetric across the two eyes even when stimulating one eye, and exhibits a stereotypical ‘impulse response’ timecourse where multiple parameters (e.g. latency, constriction time, return to baseline) carry important clinical information.

Currently, tracking of gaze patterns and pupillometry are limited to situations where eyes are open and the eye and pupil are visible to the human observer, or standard video imaging. Existing techniques are based on manual subjective assessment, or on few video-based devices that identify the pupil in open eye conditions. However, an unmet clinical need exists for performing gaze monitoring and pupillometry through closed eyes. Such a device would enable important applications such as identifying changes in continuously monitored patients at Intensive Care Units (ICUs), monitoring the depth of anesthesia during surgery, detection and diagnosis of sleep disorders, and other applications.

The human eye can be described as an optical system. The eye focuses and processes incoming light through a lens and a retina. A pupil, which is an opening in the ring-shaped iris, optimizes retinal illumination. Pupil size is modulated by two factors: the level of ambient light, and arousal. Mydriasis (dilation) occurs in conditions of low light intensity and high arousal and is caused by the dilator pupillae muscle. Miosis (contraction) occurs in illuminated conditions and low arousal and is caused by the sphincter pupillae muscle. Pupil diameter varies from 1 to 8 mm and is mostly symmetric between two eyes in healthy individuals. It is known that pupil functional abnormalities occur in psychiatric disorders, neurodegenerative disorders, drug overdose, and autonomic neuropathies in diabetes. Furthermore, pupillometry can be used to monitor anesthesia depth and analgesia, head injuries, and clinical status following cardiac arrest.

The PLR has stereotypical dynamics that allow for real-time detection of any abnormalities as clinical indications. When light is presented to the eye, the pupil automatically constricts, whereas stimulus termination leads to pupil re -dilation. The PLR response is mediated by concerted action of the sympathetic and parasympathetic systems. An important PLR feature for clinical diagnosis is that light stimulation in front of one eye causes a symmetric reaction in both eyes whenever the brain pathway is intact. PLR dynamics follow a pattern consisting of four phases: response latency, maximum constriction, pupil escape, and re-dilation. When light stimulation is brief, pupil changes are characterized by a sharp, "impulse response" -like profile.

In clinic, pupillometry finds applications in numerous domains related to the autonomic system including: neurology and head trauma, critical care and emergency medicine, neurosurgery, endocrinology, neurodegeneration, drug addiction, psychiatry, pain, and cardiology. Typically, clinical pupillometry is sporadic and performed manually using a penlight and ruler. Therefore, it is time-consuming, inaccurate, subjective, and lacks continuity. Most restricting, it is limited to situations where the eyes are open. Thus, despite its potential in detecting arousal and brain state, pupillometry is not employed during surgical anesthesia or sleep due to the absence of reliable technology for monitoring pupil dynamics behind closed eyelids. Current bedside pupillometry is typically intermittent, qualitative, manual, and limited to openeye situations, restricting its use in sleep medicine, anesthesia, and intensive care.

Ideally, a medical tool would be available that could perform touchless quantitative pupillometry continuously to enable novel applications such as monitoring pain, awareness, or abnormal arousal during anesthesia or sleep.

It is an aim of the present invention to provide an apparatus and method for monitoring pupil size dynamics in closed eye conditions to overcome this major unmet clinical need.

Summary of the Invention.

According to one aspect of the present invention there is provided a system for touchless monitoring of eye activity through closed eyelids, the system comprising at least one imaging device for detecting eye activity through closed eyelids.

Preferably, the system further comprises at least one electromagnetic radiation source for radiating electromagnetic energy in a spectral range of 700-2000nm towards the closed eyelid. In a preferred embodiment of the present invention, the system includes at least one illumination source for providing at least infrared radiation. The illumination source may be, for example, an infrared projector or an optical fiber coupled to a laser/LED system. One, or more preferably, an array of radiation sources may be arranged to emit radiation through the closed eyelid, for example the system may include an array of light emitting diodes arranged in a particular geometric configuration.

Preferably, the system includes an imaging device for capturing electromagnetic energy in a spectral range of between approximately 700-3000nm, more preferably 900-1700nm. Ideally, such a system includes at least one electromagnetic radiation source for radiating electromagnetic energy in a spectral range of 700-2000nm towards the closed eyelid.

Additionally or alternatively, the system may include an imaging camera for detecting electromagnetic energy in a spectral range of between approximately 8000-12000nm. Such a system may or may not include at least one electromagnetic radiation source for radiating electromagnetic energy in a spectral range of 700-2000nm towards the closed eyelid.

Optionally, the system may further comprise a light polarization source.

The system may further comprise at least one sensory stimulus for inducing a change in pupil size or gaze direction. The at least sensory stimulus may be selected from a light stimulus and/or an auditory stimulus. For example, the system may include a light source for delivering short visible light flashes towards the closed eye or sound may be delivered via headphones or a speaker. Alternatively or additionally, other sensory stimuli may be provided, such as a tactile, touch, pain stimulus.

Thus, according to some embodiments, the system can either measure spontaneous changes in pupil diameter over time, and/or the system may include at least one sensory stimulus, such as a visible light source to stimulate the closed eye, for example delivering brief controlled light stimulation to measure stimulus-evoked changes in pupil diameter (pupillary light reflex, PLR). The visible light source may comprise a monitor with a controller to deliver controlled light stimulation to the eye but is not limited thereto.

According to some embodiments, the system includes multiple electromagnetic radiation sources which are configured to radiate electromagnetic energy at different wavelengths.

Alternatively, or additionally, the at least one radiation source may be oriented at different angles to the eye. Preferably, the at least one radiation source contains more than one electromagnetic source, the electromagnetic sources configured to radiate electromagnetic energy at different angles. It is to be appreciated that the system may include any type of imaging device known in the art that is capable of capturing images emitted through the closed eyelid in respect of the desired spectral imaging range Preferably, the at least one imaging device of the system includes at least one continuous imaging device, such as a camera or video camera.

In some embodiments, the continuous imaging device is configured to detect electromagnetic energy and capture and record an image or images, such as in the case of video. In some embodiments, the continuous imaging device is a SWIR camera, preferably detecting electromagnetic energy in a spectral range of between approximately 700-3000nm, more preferably 900-1700nm.

Additionally or alternatively, the system may include a thermal imaging camera, preferably detecting electromagnetic energy in a spectral range of between approximately 8000-12000nm.

Preferably, the at least one of the radiation source is angled approximately 3° -7° with respect to an axis between the eyelid and the imaging device, such as a SWIR camera, more preferably at an angle of 5°.

In preferred embodiments, the system further includes a computerized module comprising a processor and a non-transitory storage medium storing computer readable instructions wherein the processor is configured to process images of an eye to continuously detect changes in at least one of pupil diameter and gaze direction, and optionally other diagnostic qualities derivable from such images. The processor of the computerized module is preferably configured to process at least one of (i) image data of an eye to continuously track changes in pupil diameter over time and (ii) image data of an eye to continuously track changes in gaze direction over time.

The computerized module with processor preferably includes dedicated algorithms to detect the pupil along with eye gaze orientation to estimate pupil area/radius at any given time.

According to some embodiments, the apparatus can either measure spontaneous changes in pupil diameter over time, and/or the apparatus may include an external stimulus, such as a light source to deliver brief controlled light stimulation to measure stimulus-evoked changes in pupil diameter (pupillary light reflex, PLR). For example, the light source may comprise a monitor with controller to deliver controlled light stimulation to the eye.

Preferably, the processor processes image data of an eye to continuously track changes in both pupil diameter and in gaze direction over time.

More preferably, the system combines infrared imaging, more preferably short-wave infrared (SWIR) imaging with dedicated image processing algorithms to compare detected images with reference images thereby enabling measurement of stimulus-evoked PLR, capturing pupil size dynamics in closed eye conditions. Deep learning-based analysis may be applied to the data to estimate gaze direction during eye movements and achieve robust pupillometry beyond laboratory visual fixation conditions.

Thus the system of the present invention may provide touchless short-wave infrared imaging combined with dedicated data analysis to reliably monitor rapid (< 100ms) dynamics in gaze and pupil size through closed eyes.

A second aspect of the present invention provides a method for providing touchless monitoring of eye activity through closed eyelids, the method comprising: monitoring any eye activity through the closed eyelid by capturing an image of at least part of the eye through the closed eyelid.

Preferably, the method further comprises directing electromagnetic energy in a spectral range of 700nm to 2,000nm from at least one electromagnetic radiation source towards a closed eyelid and monitoring a response of the eye to the applied electromagnetic energy by the capture of an image of at least part of the eye through the closed eyelid.

The at least one electromagnetic radiation may be directed at the closed eyelid at different angles and/or an array of radiation sources may be directed at the eyelid at different angles.

The method may capture spontaneous changes in at least one, preferably both, of pupil size and eye gaze direction. Preferably, the method continuously captures changes in the closed eye. Optionally, the method may further comprise providing at least one sensory stimulus to induce a change in at least one of pupil size or eye gaze direction.

The sensory stimulus may comprise at least one or a combination of a light stimulus, an audible stimulus and a tactile stimulus.

Preferably, capturing the image of the response of the eye is provided by at least one camera for detecting electromagnetic energy, preferably at least one of mid IR imaging in the spectral range 8000-12000nm or SWIR imaging in the spectral range 700-3000nm, more preferably 900-1700nm. Optionally, the method may further comprise a polarization option at the imaging device.

In embodiments wherein the method comprises capturing the image of the eye activity by at least one camera for detecting electromagnetic energy in the spectral range 8000-12000nm, the method may or may not include a step of directing electromagnetic energy in a spectral range of 700nm to 2,000nm from at least one electromagnetic radiation source towards a closed eyelid. However, in embodiments wherein SWIR imaging is used in the method, electromagnetic energy in the spectral range of 700nm to 2000nm is required to be directed towards the closed eyelid.

Preferably, the method further comprises sending image data to a processor, the processor configured to process at least one of (i) image data of the eye to continuously track changes in pupil diameter over time and (ii) image data of the eye to continuously track changes in gaze direction over time. The method preferably includes performing deep learning-based image processing to identify at least one of pupil dynamics and gaze direction in closed eye.

Preferred embodiments of the present invention provide a system and method for combining short-wave infrared (SWIR) imaging and/or thermal imaging with dedicated image processing algorithms which can successfully measure stimulus-evoked PLR, capturing pupil size dynamics in closed eye conditions. Moreover, deep learning-based analysis successfully estimates gaze direction during eye movements.

Brief Description of the Drawings.

For a better understanding of the present invention and to show more clearly how it may be carried into effect, reference will now be made by way of example only to the accompanying drawings in which:

Figure 1 is a typical pupillogram and PLR parameters for open-eye PLR as measured with SWIR camera and pupil size extracted with DeepLabCut (N=18). A time course of pupil size (change from baseline in mm, y-axis) as a function of time (sec, x-axis) around brief light stimulation (starting at time zero) reveals typical PLR dynamics;

Figure 2 is a schematic diagram of a side perspective view of an experimental setup for assessing PLR in open eye and closed eye conditions;

Figure 3 is a pupillogram obtained from during a sample experimental setup;

Figure 4A is a schematic diagram of a ‘fixed circle’ analysis approach;

Figure 4B is a plot of pixel intensity within a fixed circle over time, showing a representative 120 sec segment from one experiment depicting five experimentally-induced PLR trials;

Figure 4C shows the mean PLR response averaged across 20 trials in four recording sessions, where the y axis represents percent relative to each trial’s pre-trial ([-1000] baseline values;

Figure 5A (i) illustrates static (still) mid-IR (‘thermal’) imaging of an eye area which reveals robust differences in temperature across center of eye and Figure 5A (ii) provide quantification of temperature across a horizontal eye section revealing distinct temperature values across an eye after thermal imaging;

Figure 5B is a schematic diagram of a front plan view of another experimental set up for SWIR imaging; Figure 5C demonstrates that SWIR imaging allows monitoring of gaze direction through closed eyelids;

Figures 5D(i) and (ii) illustrate respectively schematic and actual experimental set-ups for polarized light imaging according to another embodiment of the present invention and Fig. 5D(iii) provides sample polarization images obtained;

Figure 6 shows SWIR imaging of (i) open eye, (ii) closed eye and (iii) control region, where y-axis denotes ‘ 1- pixel intensity in fixed circle that serves as proxy for pupil area;

Figure 7A is a representative trace of estimated pupil dynamics from closed eye SWIR imaging (trace (ii)) compared with open eye (trace (i)) and a control (trace (iii) on forehead);

Figure 7B is a grand average of PLR dynamics across a dataset of 30 participants;

Figures 7C and 7D respectively illustrate two examples of average PLR dynamics for participants with different iris colours (26 y.o. female green eyes and 24 y.o. female light brown eyes);

Figure 8 is a pupillogram representing a 40s data of one participant to show dynamics in closed eye conditions as estimated using U-Net deep learning-based analysis around two PLR events (7s intervals), showing trace (i) open eye ground truth data, (ii) models output estimation when trained on closed eye data and (iii) when trained on control region (forehead);

Figure 9 is a plot of Pearson correlation coefficients (y-axis) for all participants (N=27) for of pupil size time-courses in closed eye vs. open eye ground-truth data (left, median of 0.85 ± 0.48) or in control region data vs. open eye ground-truth data (right, median of 0.21 ± 0.44); and

Figure 10 is a plot illustrating estimation accuracy of gaze direction across entire dataset (N=25) showing typical variability (SD) over time of 13 degrees of visual angle.

Detailed Description.

The present invention relates to a validated system and method that allows touchless non-invasive monitoring with millisecond precision of pupil and gaze through closed eyes. Certain embodiments are based on short-wave infrared (SWIR) imaging, and can be further optimized with a combination of thermal and/or polarization imaging, as well as additional image processing techniques. The invention greatly enhances the use of pupillometry as a powerful diagnostic tool.

It is widely known that light enters the human eye through the cornea, through the aqueous humor, through the pupil (an opening in the ring-shaped iris), through the lens (which modulates illumination and focus and is controlled by ciliary muscles), and through the vitreous humour. The retina on the interior back of the eye transduces light to electrical activity that is then relayed through axons of ganglion cells via the optic nerve to further processing in the brain. Pupillary light reflex (PLR) is an autonomic reflex that constricts the pupil in response to light via iris sphincter muscles. After light impinges on the retina, ganglion cells send impulses through the optic nerve and the optic chiasm, and through the optic tracts (where nasal fibers cross to the contralateral side, and temporal fibers continue on the ipsilateral side). Signals travel further to the pretectal nucleus in the midbrain, where they send signals to the parasympathetic Edinger -Westphal nucleus. Efferent (output) parasympathetic preganglionic fibers travel on the oculomotor nerve and synapse with the ciliary ganglion, which sends postganglionic axons to directly innervate the iris sphincter muscles. The contraction of the iris sphincter muscles leads to pupil constriction (miosis). The structural and functional integrity of this pathway is typically tested by a light shined on the eyes.

A typical pupillogram and PLR parameters is illustrated in Fig. 1 of the accompanying drawings, where the PLR is in response to brief (e.g. 100ms) light stimulation. The figure illustrates typical PLR dynamics and demonstrates a time course of pupil size (area in mm, y-axis) as a function of time (seconds, x-axis) around brief light stimulation. The PLR consists of: (a) a fast (lasting ~60ms) constriction shortly after the eye has been exposed to light stimuli, (b) an early fast re -dilation of the pupil (lasting ~ls), and (c) an additional slow pupil dilation to original baseline size. The difference between baseline diameter and the maximum constriction is termed the constriction amplitude (vertical arrow), and the time interval between constriction onset to maximum constriction is termed construction time.

These conventional pupillograms are generally carried out with eyes open using light stimulation. In contrast, the present invention provides new methods and systems for monitoring pupil dynamics with eyes closed, based on short-wave infrared (SWIR) imaging, and can be further optimized with a combination of thermal and/or polarization imaging, as well as additional image processing techniques. This development opens completely new avenues for eye and pupil tracking in many new domains including anesthesia, critical care, sleep studies (at the lab and ambulatory monitoring at home) and autonomic disorders that have long posed challenges to clinical and research communities.

According to preferred embodiments, the device includes a possible illumination source such as an infrared projector or an optical fiber coupled to a laser/LED system, a continuous imaging device, and a computerized module with dedicated algorithms to detect the pupil along with eye gaze orientation to estimate pupil area/radius at any given time. According to some embodiments, the apparatus can either measure spontaneous changes in pupil dimeter over time, and/or deliver brief controlled light stimulation to measure stimulus-evoked changes in pupil diameter (pupillary light reflex, PLR). The novel use of SWIR imaging at the 900-1500 mm spectral range captures the radiation reflected, back scattered, or re-emitted from tissue shined by light sources in the camera’s spectral range. It represents a safe, tolerable, non-invasive non-contact modality to collect information using an NIR camera in real time. It is not affected by ambient light and is therefore well suited for measurements at the near patient’s setting.

Another imaging modality that could improve sensitivity and applicability of pupillometry is polarization imaging. It builds upon differences between two polarization modes of light that are back scattered from tissue after penetrating through it. The polarized image reveals any inclusion that breaks the structure of the surrounding tissue (e.g., melanoma in comparison to benign nevus).

The present invention has the potential to transform neurocritical care, prevent intra-operative awareness and pain during surgery, and change sleep medicine. The present invention enables the continuous, automatic assessment of a patient’s pupil size and reactivity through closed eyes and is capable of being performed at the patient’s bedside. Thus, the present invention can transform neurocritical care by significantly improving monitoring of patients in the ICU and during anesthesia, and allowing practitioners to quickly identify cases in need of immediate attention and improving patient care.

Experimental Data.

An experimental setup and proof-of-concept for the potential of IR imaging and polarized light imaging were developed to first obtain static pupil information, following which the experimental methods were further developed to capture dynamic pupil information and gaze direction information and extended to 40 participants for data acquisition and further analysis.

1. Validation of experimental setup obtaining robust PLR with visible light video imaging.

In order to evaluate and validate the results of the presently disclosed new methodologies, results were compared to a ‘gold-standard’ EyeLink™ system for pupillometry in lab settings. The experimental setup relied on a computer monitor as a light stimulus whose brightness could be controlled via software.

Figure 2 of the accompanying drawings illustrate the experimental setup used. The participants sat 60 cm (distance x) from the presentation screen 100, the participant's head 20 placed on a dedicated chinrest 10a. The participant was requested to sit motionless and keep eyes open and minimize blinking as best as possible. The experiment (changes in computer monitor brightness, programmed in Python 3.6) consisted of six “light”- visual stimuli (uniform white pixels) for a duration of 200 ms each, with inter-stimulus intervals of 10,000 ms (black screen). During this entire time, the face and eyes were illuminated with a near infra-red (0.75-1.4pm) LED array 30 that is commonly used when performing pupillometry in lab settings. Its main advantage is that it creates reflections on the cornea and reduces the interference of visible light during pupil tracking. The LED array 30 and camera 40 were placed a distance y of 18 cm from the participant’s head.

Offline data analysis used the commercial’s system tools for (a) image segmentation to separate pupil from surrounding pixels in the image, and (b) estimation of a circle that best fits this area, to generate a time-course of pupil diameter as a function of time during the experiment. The results fit the expected PLR response as illustrated in Figure 1. As can be seen in Figure 3, after a short delay, brief (200ms) light stimulation induces a rapid pupil constriction (a), which then gradually returns to baseline levels (b) and (c). Thus, the presently disclosed setup can recapitulate a robust PER response in line with literature.

2. Development of data analysis pipeline that does not require segmentation of pupil pixels.

Next, an experiment was conducted to measure PER with visible light video imaging but without pupil segmentation in the data analysis. It was thought that this approach may be more suitable as a flexible tool in closed eye situations, where the contract/difference between pixel intensity within and around the pupil may be more modest. Accordingly, video was collected with a widely-used Samsung Galaxy 8 (16 megapixels) smartphone camera. Periodic visual stimulation (screen turning bright) was identical to the first experiment.

A ‘fixed circle’ analysis approach was applied, as illustrated in Figure 4A. In this scheme, when the pupil dilates, dark pixels (pupil) comprise a larger percent of the circle, yielding lower average pixel intensity. Conversely, upon light illumination and associated PER, a pupil constriction was expected where dark pixels (pupil) comprise a smaller percent of the circle, yielding higher average pixel intensity within the fixed circle. In other words, it was expected that shortly after light stimulation the time-course of pixel intensity within a fixed circle would exhibit a sharp increase and a gradual decrease back to baseline. To simplify visual comparison to standard PLR plots, the calculated values were inverted to present “percent change” in pixel darkness relative to baseline after the light stimulus had been presented. The results (Fig. 4B, C) establish that with the ‘fixed circle’ analysis approach, without performing pupil segmentation in data analysis, a robust PER response can be recapitulated.

Fig. 4A provides a schematic illustration of a ‘fixed circle’ analysis approach. As shown in Fig.4A, to avoid segmentation of the image to pupil and surrounding pixels, an alternative approach is put forward where the average pixel intensity (brightness) is averaged within a circle with fixed size (contour C) that roughly captures both pupil and surrounding iris. When the pupil is dilated (top), dark pixels (pupil) comprise a larger percent of the circle, yielding lower average pixel intensity. Upon light illumination and associated PLR (bottom), the pupil constricts, and therefore dark pixels (pupil) comprise a smaller percent of the circle, yielding higher average pixel intensity within the fixed circle.

As shown in Fig. 4B, novel analysis pipeline applied to data recorded from visible light video system reveals PLR. Fig. 4B shows a representative 120 sec segment from one experiment depicting five experimentally-induced trials. Top line d shown in Fig. 4B shows 20 sec epochs of experimental “trials” (screen turning bright, white bars). Middle line e in Fig. 4B shows actual luminance changes measured via infrared photodiode placed on the computer monitor to confirm synchronization with recorded data. Bottom line f in Fig. 4B shows observed dynamics of pixel intensity within circular region-of-interest around pupil and iris. For convenience and visual similarity with typical PLR plots, the average pixel intensity was inverted to obtain ‘negative PLR’. Note the negative peaks in bottom line f, corresponding to PLR, around each trial of screen turning bright.

Fig. 4C provides the mean PLR response averaged across 20 trials in 4 recording sessions, where (y-axis) represent percent relative to each trial’s pre-trial ([-1000 0]) baseline values. PLR in this experimental setup (with visible light video hardware placed ~60cm from eye), and without dedicated segmentation of pupil from surroundings, yields a response greater than 2.5 -fold increase from baseline.

3. IR imaging enables static pupil and gaze monitoring (in still images).

Next, visible light imaging incorporating NIR imaging was investigated. In static (‘still’) images, it was established that mid-IR imaging (thermal imaging) could detect the pupil. Thermal imaging of an eye area revealed detectable differences in temperature across the eye, with quantification of temperature across a horizontal eye section demonstrating distinct temperature values for sclera, iris and pupil (see Figure 5A(i) and (ii)). Moreover, with SWIR imaging (Widy Sens 640V-ST SWIR camera), gaze direction was also readily detectable through closed eyes.

Figure 5B is a schematic illustration of the experimental setup, including: a chinrest and forehead post 10a, 10b used for placing subject face 20 in fixed position, distance, and angle from illumination sources and cameras, a screen 100 used to present bright stimuli for PLR measurements, measuring instruments, a 940nm LED 30 and a Widy Sens 640V-ST SWIR camera 40. Fig. 5C demonstrates that SWIR imaging allows monitoring of gaze direction (marked with bold dots) through closed eyelids.

Polarized light imaging may also be used in the present invention., as illustrated in Figures 5D(i) to (iii). For example, using a broadband light source 404, hens 402, polarizers 406, 408, an ocular phantom 410 and mechanical aperture with CCD camera 400 for conversion of the light scattered into digital signals for further processing. Fig. 5D(i) provides a schematic of a suggested setup for polarized light imaging with a broadband light source, polarizers, an ocular phantom and a mechanical aperture. Fig. 5D(ii) shows actual implementation of experimental setup for polarized light imaging and Fig. 5D(iii) shows gray levels of polarization images which reveals a central aperture serving as a mock pupil.

4. NIR imaging enables dynamic PLR and pupillometry (in video streams) with millisecond resolution.

The PLR experiment (intermittent visual stimuli of screen turning bright) was then combined with SWIR imaging. A subject was sat 75 cm from the presentation screen with her head was placed on the dedicated chinrest. She sat motionless and kept one eye open and the other closed (one finger was placed on her left eye eyelashes). The experiment (changes in computer monitor brightness, programmed in Python 3.6) consisted of five “light”- visual stimuli (uniform white pixels) for a duration of 2 sec each, with interstimulus intervals of ~20 sec (black screen). During this entire time, the face and eyes were illuminated with a 940 nm LED placed in front of the subject's closed eye, 21 cm from the chinrest (Vishay Semiconductor Opto Division (VA), EMITTER IR 940NM 1A 3SMD). The camera that was used is an SWIR camera, located 60 cm from the chinrest (NIT, WiDy SenS 640V-ST), 31Hz. Measurements were collected in the dark (fixed ambient light) and at a fixed temperature of 24°C.

A ‘fixed circle’ analysis approach was applied as presented above. The results (Fig. 6 and later in 7D) demonstrated a simultaneous PLR response in the open eye (i), (note sharp vertical transients corresponding to blinks) and the closed eye (ii). When inspecting the average response (across 5 repetitions of visual stimulation) in open and closed eyes, as well as in a control region-of-interest (iii), it was established that SWIR imaging can capture PLR response behind closed eyelids including: (a) a fast constriction shortly after the eye has been exposed to light stimuli, (b) an early fast re-dilation of the pupil (lasting —Is), and (c) an additional slow pupil dilation to original baseline size. In further detail, Figure 6 provide SWIR imaging results. SWIR enables dynamic PLR and pupillometry (in video streams) with millisecond resolution. Figure 6 shows 120sec of SWIR imaging during intermittent visual stimuli (screen turning bright) simultaneously performed for open eye around pupil area (top trace (i)) with sharp transients corresponding to blinks, for closed eyes around pupil area (bottom trace (ii)), and for a control region (middle trace (iii)) at the side of the eye away from pupil. The Y-axis denotes T - pixel intensity in fixed circle' that serves as proxy for pupil area (smaller pupil corresponds with fewer dark pixels in the fixed circle and higher average pixel intensity). Note that brief flashes of illumination from screen, marked by vertical black lines, are associated with dips in pupil size (PLR, pupil constriction) in both open and closed eyes, but not in the control region of interest

5. Extension of Methods of the Invention to 40 Participants for Data Acquisition.

PLR was then assessed in closed eye conditions using SWIR imaging for a larger cohort to increase statistics. Participants (n=40, ages 19-44, 24 female) held one eye closed throughout the experiment while the other eye was open. Leveraging symmetry of pupil dynamics across eyes in healthy individuals, closed eye SWIR imaging was compared to the open eye data used as ground-truth. A representative trace of estimated pupil dynamics from closed eye SWIR data (using ‘fixed circle darkness’ approach, as above) is shown in Fig. 7 A where the top trace (i) is for open eye, the middle (iii) a control and the bottom trace (ii) for closed eye. Deflections could be readily observed around each PLR event. Comparing closed eye SWIR data estimates against the ground truth open eye measurements (Fig. 7 A) revealed that SWIR imaging can accurately capture PLR dynamics through closed eyes (see Fig. 7B) and perform robustly in participants with different iris colors (Fig. 7C). Across the entire dataset, with this setup and parameters, and after exclusion criteria, a robust PLR could be revealed in 27/30 (90%) of individuals (Fig. 7C).

The experimental approach was based on artificially inducing changes in pupil size and pupil position (reflecting gaze direction), measuring the pupil parameters in a closed -eye setting using methods described above, and validating these measurements by comparing them to concurrent open-eye measurement used as the ground truth.

A gaze direction was conducted in the same session immediately after the first one, focusing on tracking changes in gaze direction. After a 15sec baseline period, a series of eight crosshair fixation targets were shown on the screen, one at a time. Each target appeared on the screen for 5sec and the sequence was repeated twice. Subjects were instructed to fixate on these targets, as is customary when calibrating commercial eye trackers before cognitive experiments. The eight positions represented a 3*3 grid apart from the bottom center position (this area of the screen was partially obscured by the camera). The second part lasted a total of 90 sec.

The experimental set up was similar to that previously used, including a chinrest, LED illumination, a SWIR camera (NIT, WiDy SenS 640V-ST, 31Hz), and a computer screen (see Fig. 5B). Participants sat 50 cm from the presentation screen while their head was positioned on a dedicated chinrest ((Eyelink®, SR). The participant’s face and eyes were illuminated with a 1100 nm LED (Thorlabs™ M1100D1, 168mW) placed 18cm from the chinrest at an approximate height of the eyes, with a slight angle (~5 degrees) with respect to the axis between the participant and center of screen. Video was captured via a SWIR camera equipped with 16mm lens (LM16HC, Kowa™), placed 18cm from the chinrest directly in front of the subjects at an approximate height of the eyes. Experiments were collected in a dark room with fixed ambient light and a fixed temperature of 24°C.

In total, the same 40 individuals participated in the two experiments. In the first PLR experiment, 30 individuals adequately followed procedure guidelines so that their data could be used for later analysis. In the second ‘gaze direction’ (crosshair fixation) experiment, DLC analysis was performed on 25 participants where data could be used. Deep learning U-NET-based analysis was then performed on 30 participants for the first PLR experiment (see further details below), where the data was suitable for neural-net image analysis (e.g. eye was away from frame edges).

To verify capture of PLR dynamics with the setup and when using a computer screen as the light stimulus, DeepLabCut™ was used to track changes in pupil size, using SWIR video images of the open eye. 18 participants were focused on whose data in the first PLR experiment allowed continuous DLC tracking of pupil throughout the experiment (i.e. without excessive blinking or movements that precluded DLC to run smoothly). Pupil diameter was expressed in mm changes relative to the pre-stimulus baseline.

To quantify PLR dynamics in closed eye SWIR data via the ‘fixed circle’ analysis approach previously defined, for each individual separately, a circle whose position and size approximated the area of pupil and iris in the closed eye just before it was closed. A similar second circle with equal size defined a region-of-interest in the open eye, and a third circle with equal size defined a control region-of-interest on the forehead center (like a Hindu bindi). For each circle and time point separately, the pixel darkness (1 - brightness value) was averaged to create time-courses as shown in Fig. 7A. Baseline pupil diameter was defined as the mean pupil diameter during 150 ms before light stimulation onset. Average pixel intensity that exhibited changes larger than 20% between consecutive video frames was marked as blinks, and trials in which the data included more than 10% blinks were rejected from further analysis. Average pixel intensity data were forward-backward filtered, using a 5 Hz low pass filter (“Filtfilt”-digital filter forward and backward). After this preprocessing, time-courses were averaged per participant to show average dynamics per subject (Fig. &C and 7D). A grand-average (Fig. 7B) was calculated across all participants, representing 260 trials for all regions-of-interest.

6. U-NET Neural Net Analysis

To complement the intuitive ‘fixed circle brightness’ approach used above with more advanced data analysis, deep learning-based image processing was employed to identify both pupil dynamics and gaze direction in closed eye SWIR data. A neural net model was trained using U-NET architecture with pairs of images of open and closed eyes. The model’s output was a set of images each representing the estimated image of the eye if it were open, from which pupil size could be extracted. Model estimates of pupil size derived from closed eye data (or a control forehead region) were compared with the ground truth open eye data. Fig. 8 shows representative examples of the model’s successful estimation of pupil size based on closed eye SWIR data. Quantitative analysis confirmed successful estimation around PLR events: the median and standard deviation of Pearson correlation coefficients with ground truth open-eye data were 0.85±0.48 based on closed eye data vs. 0.21±0.44 for the control region (N=27, p<0.001 via paired one-tailed t-test after fisher transformation).

In addition, a separate experiment examined gaze direction estimation when participants fixated on eight screen positions (Methods, N=25). Ground truth information about pupil position (coordinates obtained from DLC trained on open eye data) was compared with pupil position coordinates based on closed-eye SWIR data (DLC trained only on closed eye data). Gaze direction could be successfully extracted from closed eye SWIR data. The accuracy of gaze direction estimation was quantified by comparing pupil positions in the two DLC models. With the experimental conditions, it was found that typical variability (SD) over time in gaze direction estimation was 13 degrees of visual angle (median across N=25 participants, Fig. 10).

The U-net neural net analysis discussed above was initiated in Python, 'Tensorflow' and 'Keras' libraries. Implementation of skip connections in the fully convolutional encoder-decoder neural network facilitated capturing different latent representations within various network layers. A separate model was trained for each participant. The training was conducted per frame for very short (180 sec) videos, using a highly imbalanced dataset due to the short temporal nature of PLR events, comprising only a short (-55%) part of the experiment’s duration. The data were divided such that the first 20% were used as the test set (i.e. 2 PLR events), while the remaining (i.e. 8 PLR events) were used for training. During training, 20% of the data was randomly selected as a validation set, and the rest constituted the training set. Two different models were trained for each trial, one based on data from the closed eye, and another based on data from the control forehead region. Prior to insertion into the network, each image was normalized to have a mean of zero and a variance between -1 and 1. The Mean Absolute Error (MAE) loss was employed for training the network for 200 epochs across all participants. Epoch is a parameter that defines the number times that the learning algorithm will work through the entire training dataset.

After training, predictions for the test set from all participants were used to train a DLC model, which marked the boundaries of the pupil within the eye. A time-course of the dynamics of the radius of this circle over time was calculated for the ground truth open eye, for the predicted open eye using closed eye data, and for the predicted open eye using control region data. Blinks in the predictions were filtered out by removing high frequency components in the Fourier Transform of the traces.

Performance evaluation focused on similarity of time -courses around PLR events. First, 7sec intervals-of- interest were identified around each PLR event (±3.5s), centered around local minima (<0.7 see purple shaded areas in Fig. 8). For each region-of-interest separately (open eye, closed eye, control forehead region), time-courses in these intervals were averaged across the two PER test trials per participant, and two Pearson correlations were calculated: open eye vs. closed eye (Fig. 9 left), or open eye vs. control region (Fig.9 right).

To track changes in the gaze direction with closed eyes, two separate DEC models were trained, one receiving images cropped to only include the open eye, and the other receiving images cropped to only include the data of the closed eye. Pupil position was calculated as the (x,y) location center of eight points marked on the circumference of the pupil (not shown). Next, changes in 2D (pixel) coordinates in the image were transformed to changes in degrees visual angle (DVA) relative to the gaze direction towards the central target location. For each participant, a time-course was calculated representing the dynamics of the difference between the DVA of the closed eye and the DVA of the open eye. Next, the variability in this time-course was characterized by computing the SD across time, separately for each participant. The distribution of SD values are shown in Fig. 10.

The results demonstrated that the method and system according to the present invention can achieve touchless tracking of rapid changes in pupil size and gaze through closed eyes, initiating novel avenues for research and clinical care advancements. The present disclosure presents several significant benefits over existing technology. These advantages encompass (i) illumination positioned away from the face, essential for uninterrupted use without inducing heat, (ii) touchless functionality, vital for tracking natural sleep, and (iii) enhanced performance due to the deeper penetration capabilities of SWIR, crucial for subsecond temporal resolution. Further optimization of imaging and data analysis will enable the extension of this approach to monitor continuous variations in pupil size, beyond the PER. The present invention provides enhanced methods and systems for monitoring eye movement through closed eye lids which may be employed in a wide range of applications, including monitoring depth of anesthesia and pain during surgery; automated pupillometry in neuro-critical care; detection and diagnosis of sleep disorders; autonomic neuroscience; defense and space industries; assessing drug metabolism, drug impairment, and addiction; endocrinology/diabetes (e.g. detecting hypo/hyper glycemia during sleep); marketing/advertising industry; otolaryngology and psychiatry. It is to be appreciated that carrying out further measurements on a larger cohort and diverse population (gender, age, eye and skin color) will enable further extension of the data base and improve statistics. In addition, ongoing optimizations to improve SNR, resolution, and user experience, include optimizing light source (i.e. diode) illumination (e.g. wavelength, spatial position, Light diffusers), adding structured illumination and polarization imaging, together with further investigations in midlR (‘thermal’) imaging to complement SWIR imaging or can be used separately.

The present invention not only provides methods and systems than allow for tracking closed eye pupil response to stimulus such as light stimulus (PLR) or auditory stimulus but, just as importantly allows tracking of spontaneous changes, not induced by any external stimulus, in both pupil size and in eye gaze direction, that may occur for example during natural sleep, anesthesia, or any other condition with closed eyes. It is this second context that could allow monitoring wide array of medical conditions such as events during sleep (PTSD/nightmares/seizures/cardiac events) or during anesthesia (pain or intraoperative awareness).

Further modifications to the system and method of the present invention may be made without departing from the principles embodied in the examples described and illustrated herein.

Claims

1. A system for touchless monitoring of eye activity through closed eyelids, the system comprising at least one imaging device for detecting eye activity through closed eyelids.

2. The system as claimed in claim 1 , further comprising at least one electromagnetic radiation source for radiating electromagnetic energy in a spectral range of 700nm to 2000nm towards the closed eyelid.

3. The system as claimed in claim 2, wherein the imaging device captures short wave infra-red radiation in the range 700-3000nm, preferably 900-1700nm.

4. The system as claimed in claim 1, claim 2 or claim 3, wherein the imaging device captures mid infrared radiation in the range of 8000-12000nm.

5. The system as claimed in claim 2, 3 or 4, wherein the electromagnetic radiation source is selected from at least one projector and an optical fiber coupled to the source.

6. The system as claimed in claim 5, wherein the source is at least one of a light emitting diode or a diode laser.

7. The system as claimed in claim 6, wherein the source comprises an array of light emitting diodes in a predefined geometric configuration.

8. The system as claimed in any one of the preceding claims, further comprising a polarizer.

9. The system as claimed in any one of the preceding claims, further comprising at least one sensory stimulus for inducing a change in pupil size and/or gaze direction.

10. The system as claimed in claim 9, wherein the at least one sensory stimulus is selected from one or a combination of a light stimulus and an auditory or tactile stimulus.

11. The system as claimed in any one of claims 2 to 10, wherein the at least one electromagnetic radiation source contains more than one electromagnetic source, the multiple sources being configured to radiate electromagnetic energy at different wavelengths and/or at different angles.

12. The system as claimed in any one of claims 2 to 11, wherein the angle of the at least one electromagnetic radiation source to the eye is adjustable.

13. The system as claimed in any one of the preceding claims wherein the at least one imaging device includes at least one continuous imaging device selected from a camera or video camera.

14. The system as claimed in claim 13, wherein the continuous imaging device is configured to detect electromagnetic energy and capture and record an image or images.

15. The system as claimed in claim 13 or claim 14, wherein the continuous imaging device is a SWIR camera.

16. The system as claimed in claim 13 or claim 14, wherein the continuous imaging device is a thermal imaging camera.

17. The system as claimed in any one of the preceding claims, further comprising a computerized module comprising a processor and a non-transitory storage medium storing computer readable instructions wherein the processor is configured to process image data of an eye to detect changes in at least one of pupil diameter and gaze direction

18. The system as claimed in claim 17, wherein the processor of the computerized module is configured to process at least one of (i) image data of an eye to continuously track changes in pupil diameter over time and (ii) image data of an eye to continuously track changes in gaze direction over time.

19. The system as claimed in claim 17 or claim 18, wherein the computerized module with processor includes algorithms to detect a pupil of the eye and eye gaze orientation to estimate pupil size at any given time.

20. The system as claimed in any one of the preceding claims further comprising a visible light source to deliver light stimulation to measure stimulus-evoked changes in pupil diameter.

21. The system as claimed in claim 20, wherein the light source comprises a monitor with controller to deliver controlled light stimulation to the eye.

22. The system as claimed in any one of claims 17 to 21 wherein the processor is configured to process image data of an eye to continuously track changes in both pupil diameter and in gaze direction over time.

23. A method for providing touchless monitoring of eye activity through closed eyelids, the method comprising: monitoring any eye activity through the closed eyelid by capturing an image of at least part of the eye through the closed eyelid.

24. The method according to claim 23, further comprising directing electromagnetic energy in the spectral range of 700nm to 2000nm from at least one electromagnetic radiation source towards a closed eyelid and monitoring a response of the eye to the applied electromagnetic radiation by capturing at least one image of the eye through the closed eyelid.

25. The method according to claim 24, wherein the imaging of electromagnetic radiation is in a spectral range of 700-3000nm, preferably 900nm to 1700nm.

26. The method according to claim 23 or claim 24, wherein the imaging of electromagnetic radiation is in a spectral range of 8000nm to 12000nm.

27. The method according to any one of claims 24 to 26, wherein the at least one electromagnetic radiation is directed at the closed eyelid at different angles.

28. The method according to any one of claims 23 to 27, wherein capturing the image of the response of the eye is provided by at least one camera for detecting electromagnetic energy.

29. The method according to claim 28, wherein capturing the image is carried out by a SWIR camera.

30. The method according to claim 28 or claim 29, wherein capturing the image is carried out by a thermal camera.

31. The method according to any one of claims 23 to 30 further comprising polarization of the electromagnetic radiation.

32. The method according to any one of claims 23 to 31 wherein spontaneous changes in at least one of pupil size and eye gaze direction are captured by the image.

33. The method according to any one of claims 23 to 31 further comprising applying at least one sensory stimulus to induce a change in at least one of pupil size and eye gaze direction.

34. The method according to claim 33, further comprising delivering controlled visible light stimulation to the closed eyelid to measure stimulus -evoked changes in pupil diameter.

35. The method according to any one of claims 23 to 34 further comprising sending image data to a processor, the processor configured to process at least one of (i) image data of the eye to continuously track changes in pupil diameter over time and (ii) image data of the eye to continuously track changes in gaze direction over time.

36. The method according to claim 35 further comprising performing deep learning-based image processing to identify at least one of pupil dynamics and gaze direction in the closed eye.

37. A system for providing touchless monitoring of eye activity through closed eyelids comprising: at least one electromagnetic radiation source for radiating electromagnetic energy in a spectral range of 700nm to 2000nm towards a closed eyelid; at least one imaging device for detecting electromagnetic energy from the closed eye and sending image data to a processor; and a computerized module comprising the processor, the processor configured to process at least one of image data of the closed eye to continuously track changes in pupil size over time and image data to continuously track changes in gaze direction over time.

38. The system as claimed in claim 37, wherein the imaging device is a shortwave infrared camera that detects electromagnetic energy in a spectral range of 700-3000nm, preferably 900-1700nm.

39. The system as claimed in claim 37 or claim 38, wherein the imaging device is a thermal camera that detects electromagnetic energy in a spectral range of 8000-12000nm.

40. The system as claimed in claim 37, claim 38 or claim 39, wherein the processor is configured to perform algorithmic and deep learning based analysis of image date to identify at least one pupil dynamics and gaze direction in the closed eye.