WO1998049028A1

WO1998049028A1 - An alertness monitor

Info

Publication number: WO1998049028A1
Application number: PCT/US1998/008185
Authority: WO
Inventors: Joshua D. Borah; Robert I. Sinclair
Original assignee: Applied Science Group, Inc.
Priority date: 1997-04-25
Filing date: 1998-04-23
Publication date: 1998-11-05

Abstract

A method of determining alertness of a subject including the steps of non-invasively monitoring the subject's eye dynamics; generating from monitored eye dynamics a first output which is a measure of the subject's gaze patterns; generating from monitored eye dynamics a second output which is a measure of the subject's saccadic movements; generating from monitored eye dynamics a third output which is a measure of the subject's pupil dynamics; generating from monitored eye dynamics a fourth output which is a measure of the subject's blink patterns, and using the first, second, third and fourth outputs to compute a measure of alertness of the subject.

Description

AN ALERTNESS MONITOR Background of the Invention The invention relates to a system and method for monitoring the alertness of a person, such as a driver of a vehicle.

Diminished driver attention was found in various Department of Transportation (DOT) sponsored studies to be responsible for up to 50% of all accidents on US highways. The segment of these accidents most carefully studied is that involving heavy trucks. Total annual cost of all accidents involving trucks is estimated by the DOT as being nearly $9.0 billion. Inattention generally is implicated in 30% to 50% of truck accidents (DOT figures) at a cost of $3.0 to 4.5 billion/year in the US alone. Diminished attention may be due to a variety of causes: emotional stress, boredom, environment, sleep history, food history, drugs, alcohol, and fatigue. However, the portion of truck accidents attributable to driver fatigue has received the most concentrated attention since this is the cause most easily isolated in police accident reports. Fatigue has been directly implicated in the various studies as accounting for between 5% and 10% of all truck accidents with an annual cost to the US approximately $0.5 to $1.0 billion. Since 1990 a significant number of studies and research programs have been completed by existing commercial and academic organizations. These programs have concentrated on identifying driver fatigue, whilst accepting that this represents only one part of the whole driver attention problem. Also, they have tended to use a variety of indirect measurements tied back to fatigue (e.g. steering wheel movement, lane position, and lateral acceleration), since no practical way of measuring diver alertness in the real world of a commercial truck cabin or an operators environment was known. People have also developed comprehensive strategies for dealing with fatigue. Some of those strategies are based, for example, upon initially using the vehicle air conditioning and sound systems to alter the conditions of the environment. If these techniques fail to generate a response, a second step utilizes induced vibration of the operator' s seat and steering wheel and possibly reduction in the engine throttle setting.

There is, however, still a need for an alertness monitor that signals the deterioration of alertness soon enough to produce reliable corrective actions and that measures not merely fatigue but the broader and more important question of inattention.

Summary of the Invention The invention is an alertness monitor which combines the measurement of blink patterns used in past programs with direct measurement of changes in saccadic movements (i.e., autonomous movement of the eye) , in pupil characteristics (e.g. diameter and hippus) , and in viewing strategies (e.g. the alert driver pans from windshield to mirrors to side view to instruments and back, the less alert the driver tends to tunnel vision of the road ahead) .

In general, in one aspect, the invention is a method of determining alertness of a subject. The method includes the steps of non-invasively monitoring the subject's eye dynamics; generating from monitored eye dynamics a first output which is a measure of the subject's gaze patterns; generating from monitored eye dynamics a second output which is a measure the subject's saccadic movements; generating from monitored eye dynamics a third output which is a measure of the subject's pupil dynamics; generating from monitored eye dynamics a fourth output which is a measure of the subject's blink patterns; and using the first, second, third and fourth outputs to compute a measure of alertness of the subject.

In preferred embodiments, the method also includes the step of using the measure of alertness to generate an alarm.

In general, in another aspect, the invention is an apparatus for determining alertness of a subject. The apparatus includes an eye tracker module which during operation non-invasively monitors the subjects' eye dynamics; and a processor module which analyzes the monitored eye dynamics and generates therefrom a measure of the subject's alertness. The processor module is programmed to implement the following functions : generate from monitored eye dynamics a first output which is a measure of the subject's gaze patterns; generate from monitored eye dynamics a second output which is a measure the subject's saccadic movements; generate from monitored eye dynamics a third output which is a measure of the subject's pupil dynamics; generate from monitored eye dynamics a fourth output which is a measure of the subject's blink patterns; and use the first, second, third and fourth outputs to compute said measure of alertness of the subject.

In general, in yet another aspect, the invention is a method of determining alertness of a subject. The method includes the steps of non-invasively monitoring the subject's eye dynamics; generating from monitored eye dynamics three measures of the subject's eye dynamics, wherein the three measures are selected from the group consisting of the subject's gaze patterns, the subject's saccadic movements, the subject's pupil dynamics, and the subject's blink patterns; and using all three measures to compute a measure of alertness of the subject. Other advantages and features will become apparent from the following description of the preferred embodiment and from the claims .

Description of the Preferred Embodiments Fig. 1 is a block diagram of the alertness monitoring system.

Structure and Operation

Referring to Fig. 1, a system which embodies the invention includes a remotely mounted eye tracker 10, a computer or data processor 12 which analyzes the output of the eye tracker to generate a measure of alertness, and an alarm unit 13 which produces an alarm to notify the driver. Eye tracker 10, which includes a video camera 14 and an eye gaze monitor 16, is directed toward the location where the drivers head will be so that during operation it obtains an image of the driver's head and eyes 15. Typically, eye tracker 10, or at least the video camera component of it, is mounted on the dashboard in front of the driver and below the level of the driver's head. It is positioned so as to the view the driver's eyes and not be obstructed by any objects, such as the steering wheel or the driver's arms. The output of video camera 14 passes to eye gaze monitor 16, which continuously analyzes the video image of the driver' s eye, extracts relevant information relating to eye dynamics, computes a measure of alertness from that information, and generates an alarm signal which causes alarm unit 13 to take corrective action, e.g. alert the driver, when the alertness measure indicates that the driver's alertness has dropped below a predetermined level . Eye gaze monitor 16 is programmed to extract from the image of the driver's eye measures relating to the following four characteristics: gaze pattern, saccadic movements, pupil dynamics, and blinking characteristics. Measurable changes in each of these characteristics are linked to increased fatigue and diminished alertness and collectively they provide a good indicator of the driver's alertness.

As the subject's alertness diminishes, his or her gaze strategies reflect this in the following ways. The area of the field covered by the subject's gaze pattern decreases - a phenomenon referred to as "gaze tunneling" .

Also, the duration of the subject's fixations on objects or locations in the subject's visual field increases and the frequency of those fixations decreases. In addition, slow rolling eye movements begin to occur. Saccadic movements refer to the naturally occurring, involuntary, rapid, small movements of the eye which result in changing the point of fixation on a visualized object. As the subject's alertness decrease, the velocity profile of these rapid eye movements becomes skewed (i.e., the shape of the velocity profile changes) . Also, the average distance moved and the peak velocities which occur during these saccadic movements changes and the frequency of the saccades decreases, i.e., there are fewer fixations and saccades corresponding to less active scanning. Pupil dynamics reflect decreases in alertness by a reduction in the mean pupil diameter and by significant changes in the Hippus movements, which are the abnormally exaggerated, rhythmic contractions and dilations of the pupil which are independent of changes in illumination or in fixation of the eyes.

Finally, the blink characteristics show the following changes in response to reduced alertness. The frequency of closure (i.e., blinks) increases during periods of rapid blinking. The intervals between periods of rapid blinking increases. The 80% perclose values (i.e., the percentage of time that the eyelids are at least 80% closed) increases. The eyelid velocity profile during blinking changes . And the percentage or amount of pupil area obscured by the eyelids between blinks increases.

Processor 12 executes an algorithm 17 which is stored in local computer readable memory. The algorithm combines the measures provided by monitor 16, preferably all four of them, to compute the measure of alertness. The algorithm can be implemented in several alternative ways. For example, one approach is to combine the four parameters in a regression equation to generate an output value that is the measure of alertness. In one embodiment, the regression equation would take the following form:

Z = b₀ + Σ^X, where Z is the combined measure of alertness

X_x for l≤i≤4, are the four previously described measures of eye dynamics b₀ is an intercept coefficient; and b_x is the coefficient of the i^ch independent variable X_x ; and

The coefficients b₀ and b_t can be readily determined through empirical studies of alertness using another well recognized and acknowledged indicator of alertness, such as those produced by electroencephalography (EEG) , electrocardiography (ECG) , electromyography (EOM) , or electro-oculography (EOG) . In other words, the coefficients are computed so that the four measured variables when weighted by the coefficients produce an output that corresponds to the output obtained from one of these other sources, preferably EEG.

Alternative approaches involve using the parameters sequentially. That is, a partial set of parameters is first used to produce a preliminary measure of alertness and the remaining parameters are then used to produce a more precise measure of alertness when the preliminary measure exceeds a first threshold. In that case, the eye monitor can also be instructed to produce only the measures that are needed. When the generated measures indicate that the full set of parameters are required, then the eye monitor can be instructed to provide the additional parameters. These approaches have the advantage of imposing a reduced computational load on the processor and on the eye monitor during periods when the driver is fully sufficiently alert. In the described embodiment, the alarm unit generates an audible alarm. Alternatively, it could notify the driver in any of a number of different ways, e.g. through light, sound, or vibration, to just name a few. Also, the alarm could be programmed to be progressively more intrusive if the driver does not react as desired, e.g. if the monitoring equipment does not show an improvement in alertness. In addition, different alarms could be used depending on the success on the first or earlier alarms. For the eye tracker and components thereof, one can use commercially available off-the-shelf tracking equipment such as the Model ETS (Eye Tracker System) , the Series 5000 eye trackers, and the Model 210 Eye Movement Monitor, all of which are commercially available from Applied Science Laboratories, Inc., the assignee of the present invention.

Of course, the equipment which is currently available through commercial source is typically not optimally designed for use in automobiles or trucks, so it is preferable to modify it by, for example, reducing its size and configuration so that it fits more conveniently and unobtrusively in the environment in which the alertness monitor is intended to be used. In the case of the Model ETS from Applied Science

Laboratories as well as other commercially available eye trackers, the unit illuminates the eye with an infra-red beam, or near-infra red beam, and views the eye with a solid state video camera. Since the near-infra red (IR) light is outside of the visible spectrum, it does not annoy the subject (e.g. the driver) but is in a high sensitivity region for most solid state light sensors. The video information is sent to a computer or special purpose processor for feature recognition and computation tasks. Line of gaze is computed, independently of head translations, by detecting the relative position of two features, namely, the pupil and the reflection of the IR light source on the corneal surface. The latter feature is commonly called the corneal reflection, or CR. A known mapping and/or curve fitting technique is then used to transform the pupil-CR vector to a point of gaze on the scene space. Pupil diameter is generally a byproduct of this pupil recognition task.

Some systems illuminate the eye with a near IR beam that is coaxial with the camera optical axis. In that case, the camera receives a retro reflection from the retina, and the pupil appears as a bright circle with respect to a generally darker background. Other systems use a light beam that is off axis from the camera yielding a pupil with the familiar appearance of a black circle. These systems look for a pupil that is generally darker than any similar shape in the video field. In order to achieve sufficient image resolution and to simplify the pattern recognition task, many eye tracking systems use a telephoto image of the eye with a field of view having roughly a one inch diameter at the distance of the subject's face. The image area thus includes only the eye and a small portion of the surrounding eye lid structure. The resulting image is relatively uncomplicated, and features of interest cover a large number of pixels. Head motion is handled by moving the entire optics module, on a servo controlled platform, to keep the camera aimed at the eye or, alternatively, by bending the optical path with a servo controlled mirror to accomplish the same purpose.

For driver alertness applications, it may be desirable, for reasons of dependability and cost, to use a wide angle camera and then to use known relatively simple pattern recognition processing to locate the subjects eye within the image field.

Other embodiments are within the following claims. For example, an effective altertness monitor can also be constructed using a subset of the four parameters described above. A workable subset includes gaze pattern, saccadic movements, and blinking characteristics. Thus, in an alternative embodiment, the alertness measure is derived from such a subset of measured eye dynamics . What is claimed is:

Claims

Claims :

1. A method of determining alertness of a subject, said method comprising: non-invasively monitoring the subject's eye dynamics ; generating from monitored eye dynamics a first output which is a measure of the subject's gaze patterns; generating from monitored eye dynamics a second output which is a measure the subject's saccadic movements; generating from monitored eye dynamics a third output which is a measure of the subject's pupil dynamics; generating from monitored eye dynamics a fourth output which is a measure of the subject's blink patterns; and using the first, second, third and fourth outputs to compute a measure of alertness of the subject.

2. The method of claim 1, further comprising using the measure of alertness to generate an alarm.

3. An apparatus for determining alertness of a subject, said apparatus comprising: an eye tracker module which during operation non- invasively monitors the subjects' eye dynamics; and a processor module which analyzes the monitored eye dynamics and generates therefrom a measure of the subject's alertness, said processor module programmed to implement the following functions: generate from monitored eye dynamics a first output which is a measure of the subject's gaze patterns; generate from monitored eye dynamics a second output which is a measure the subject's saccadic movements; generate from monitored eye dynamics a third output which is a measure of the subject's pupil dynamics; generate from monitored eye dynamics a fourth output which is a measure of the subject's blink patterns; and use the first, second, third and fourth outputs to compute said measure of alertness of the subject.

4. A method of determining alertness of a subject, said method comprising: non-invasively monitoring the subject's eye dynamics ; generating from monitored eye dynamics three measures of the subject's eye dynamics, wherein the three measures are selected from the group consisting of the subject's gaze patterns, the subject's saccadic movements, the subject's pupil dynamics, and the subject's blink patterns; and using all three measures to compute a measure of alertness of the subject.