USRE42471E1 - System and method for monitoring eye movement - Google Patents
System and method for monitoring eye movement Download PDFInfo
- Publication number
- USRE42471E1 USRE42471E1 US12/199,693 US19969308A USRE42471E US RE42471 E1 USRE42471 E1 US RE42471E1 US 19969308 A US19969308 A US 19969308A US RE42471 E USRE42471 E US RE42471E
- Authority
- US
- United States
- Prior art keywords
- eye
- person
- emitters
- frame
- array
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Lifetime
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 37
- 238000000034 method Methods 0.000 title claims description 27
- 230000004424 eye movement Effects 0.000 title description 13
- 210000000744 eyelid Anatomy 0.000 claims abstract description 87
- 210000003128 head Anatomy 0.000 claims abstract description 26
- 206010041349 Somnolence Diseases 0.000 claims abstract description 23
- 238000001514 detection method Methods 0.000 claims description 37
- 238000012545 processing Methods 0.000 claims description 32
- 210000001747 pupil Anatomy 0.000 claims description 24
- 230000005540 biological transmission Effects 0.000 claims description 7
- 230000036626 alertness Effects 0.000 claims description 6
- 239000007787 solid Substances 0.000 claims description 5
- 210000005069 ears Anatomy 0.000 claims description 2
- 238000002496 oximetry Methods 0.000 claims description 2
- 230000002596 correlated effect Effects 0.000 abstract description 4
- 210000001508 eye Anatomy 0.000 description 118
- 238000004891 communication Methods 0.000 description 32
- 230000000193 eyeblink Effects 0.000 description 23
- 230000001276 controlling effect Effects 0.000 description 13
- 230000007958 sleep Effects 0.000 description 11
- 230000006870 function Effects 0.000 description 9
- 230000000875 corresponding effect Effects 0.000 description 8
- 230000000694 effects Effects 0.000 description 8
- 230000011514 reflex Effects 0.000 description 8
- 230000004399 eye closure Effects 0.000 description 7
- 230000001133 acceleration Effects 0.000 description 6
- 230000004397 blinking Effects 0.000 description 6
- 238000005259 measurement Methods 0.000 description 6
- 230000004044 response Effects 0.000 description 6
- 206010062519 Poor quality sleep Diseases 0.000 description 5
- 238000003491 array Methods 0.000 description 5
- 206010010904 Convulsion Diseases 0.000 description 4
- 230000004913 activation Effects 0.000 description 4
- 239000003814 drug Substances 0.000 description 4
- 229940079593 drug Drugs 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000001179 pupillary effect Effects 0.000 description 4
- 210000001525 retina Anatomy 0.000 description 4
- 230000002747 voluntary effect Effects 0.000 description 4
- 230000003213 activating effect Effects 0.000 description 3
- QVGXLLKOCUKJST-UHFFFAOYSA-N atomic oxygen Chemical compound [O] QVGXLLKOCUKJST-UHFFFAOYSA-N 0.000 description 3
- 230000003190 augmentative effect Effects 0.000 description 3
- 206010016256 fatigue Diseases 0.000 description 3
- 239000000835 fiber Substances 0.000 description 3
- 239000001301 oxygen Substances 0.000 description 3
- 229910052760 oxygen Inorganic materials 0.000 description 3
- 238000002310 reflectometry Methods 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 230000035945 sensitivity Effects 0.000 description 3
- 206010021143 Hypoxia Diseases 0.000 description 2
- WHXSMMKQMYFTQS-UHFFFAOYSA-N Lithium Chemical compound [Li] WHXSMMKQMYFTQS-UHFFFAOYSA-N 0.000 description 2
- 206010034960 Photophobia Diseases 0.000 description 2
- 208000006011 Stroke Diseases 0.000 description 2
- 206010002026 amyotrophic lateral sclerosis Diseases 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 2
- 230000007177 brain activity Effects 0.000 description 2
- 210000000133 brain stem Anatomy 0.000 description 2
- 210000005252 bulbus oculi Anatomy 0.000 description 2
- 239000006071 cream Substances 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000002996 emotional effect Effects 0.000 description 2
- 206010015037 epilepsy Diseases 0.000 description 2
- 230000004886 head movement Effects 0.000 description 2
- 230000000147 hypnotic effect Effects 0.000 description 2
- 229910052744 lithium Inorganic materials 0.000 description 2
- 230000005226 mechanical processes and functions Effects 0.000 description 2
- 238000002483 medication Methods 0.000 description 2
- 230000003340 mental effect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 210000003205 muscle Anatomy 0.000 description 2
- 201000006938 muscular dystrophy Diseases 0.000 description 2
- 230000036961 partial effect Effects 0.000 description 2
- 230000001681 protective effect Effects 0.000 description 2
- 230000004620 sleep latency Effects 0.000 description 2
- 238000009423 ventilation Methods 0.000 description 2
- 208000024827 Alzheimer disease Diseases 0.000 description 1
- 238000012935 Averaging Methods 0.000 description 1
- 208000006373 Bell palsy Diseases 0.000 description 1
- 208000003508 Botulism Diseases 0.000 description 1
- 208000014644 Brain disease Diseases 0.000 description 1
- 208000019736 Cranial nerve disease Diseases 0.000 description 1
- 206010012289 Dementia Diseases 0.000 description 1
- 235000002673 Dioscorea communis Nutrition 0.000 description 1
- 241000544230 Dioscorea communis Species 0.000 description 1
- 208000032274 Encephalopathy Diseases 0.000 description 1
- LFQSCWFLJHTTHZ-UHFFFAOYSA-N Ethanol Chemical compound CCO LFQSCWFLJHTTHZ-UHFFFAOYSA-N 0.000 description 1
- 206010015995 Eyelid ptosis Diseases 0.000 description 1
- 208000034308 Grand mal convulsion Diseases 0.000 description 1
- 206010057410 Hippus Diseases 0.000 description 1
- 241000282412 Homo Species 0.000 description 1
- 206010020850 Hyperthyroidism Diseases 0.000 description 1
- 208000004619 Inert Gas Narcosis Diseases 0.000 description 1
- WTDRDQBEARUVNC-LURJTMIESA-N L-DOPA Chemical compound OC(=O)[C@@H](N)CC1=CC=C(O)C(O)=C1 WTDRDQBEARUVNC-LURJTMIESA-N 0.000 description 1
- WTDRDQBEARUVNC-UHFFFAOYSA-N L-Dopa Natural products OC(=O)C(N)CC1=CC=C(O)C(O)=C1 WTDRDQBEARUVNC-UHFFFAOYSA-N 0.000 description 1
- 201000000251 Locked-in syndrome Diseases 0.000 description 1
- 241001465754 Metazoa Species 0.000 description 1
- 239000004909 Moisturizer Substances 0.000 description 1
- 208000029578 Muscle disease Diseases 0.000 description 1
- 206010061533 Myotonia Diseases 0.000 description 1
- 206010028980 Neoplasm Diseases 0.000 description 1
- 206010033799 Paralysis Diseases 0.000 description 1
- 208000007542 Paresis Diseases 0.000 description 1
- 208000018737 Parkinson disease Diseases 0.000 description 1
- 208000035753 Periorbital contusion Diseases 0.000 description 1
- 206010034759 Petit mal epilepsy Diseases 0.000 description 1
- 208000012641 Pigmentation disease Diseases 0.000 description 1
- 206010036105 Polyneuropathy Diseases 0.000 description 1
- 241001282135 Poromitra oscitans Species 0.000 description 1
- 241001085205 Prenanthella exigua Species 0.000 description 1
- 208000032140 Sleepiness Diseases 0.000 description 1
- 208000004350 Strabismus Diseases 0.000 description 1
- 206010043118 Tardive Dyskinesia Diseases 0.000 description 1
- 206010043376 Tetanus Diseases 0.000 description 1
- 208000003217 Tetany Diseases 0.000 description 1
- 208000000323 Tourette Syndrome Diseases 0.000 description 1
- 208000016620 Tourette disease Diseases 0.000 description 1
- 208000036826 VIIth nerve paralysis Diseases 0.000 description 1
- 206010048232 Yawning Diseases 0.000 description 1
- 208000003554 absence epilepsy Diseases 0.000 description 1
- 238000004378 air conditioning Methods 0.000 description 1
- 230000001773 anti-convulsant effect Effects 0.000 description 1
- 230000000648 anti-parkinson Effects 0.000 description 1
- 229940125681 anticonvulsant agent Drugs 0.000 description 1
- 239000001961 anticonvulsive agent Substances 0.000 description 1
- 239000000939 antiparkinson agent Substances 0.000 description 1
- 239000003124 biologic agent Substances 0.000 description 1
- 206010005159 blepharospasm Diseases 0.000 description 1
- 230000000744 blepharospasm Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000000747 cardiac effect Effects 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 239000004020 conductor Substances 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 201000005937 cranial nerve palsy Diseases 0.000 description 1
- 230000003412 degenerative effect Effects 0.000 description 1
- 230000010339 dilation Effects 0.000 description 1
- 230000008034 disappearance Effects 0.000 description 1
- 230000002526 effect on cardiovascular system Effects 0.000 description 1
- 206010014599 encephalitis Diseases 0.000 description 1
- 238000005265 energy consumption Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 208000028329 epileptic seizure Diseases 0.000 description 1
- 208000001936 exophthalmos Diseases 0.000 description 1
- 210000004709 eyebrow Anatomy 0.000 description 1
- 210000000720 eyelash Anatomy 0.000 description 1
- 230000001815 facial effect Effects 0.000 description 1
- 230000008921 facial expression Effects 0.000 description 1
- 239000012530 fluid Substances 0.000 description 1
- 239000011888 foil Substances 0.000 description 1
- 239000007789 gas Substances 0.000 description 1
- 230000001339 gustatory effect Effects 0.000 description 1
- 208000018875 hypoxemia Diseases 0.000 description 1
- 230000007954 hypoxia Effects 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 208000015181 infectious disease Diseases 0.000 description 1
- 208000014674 injury Diseases 0.000 description 1
- 230000030214 innervation Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000002045 lasting effect Effects 0.000 description 1
- 230000003902 lesion Effects 0.000 description 1
- 208000013469 light sensitivity Diseases 0.000 description 1
- 230000006996 mental state Effects 0.000 description 1
- 208000030159 metabolic disease Diseases 0.000 description 1
- 230000001333 moisturizer Effects 0.000 description 1
- 201000006417 multiple sclerosis Diseases 0.000 description 1
- 230000003274 myotonic effect Effects 0.000 description 1
- 201000003631 narcolepsy Diseases 0.000 description 1
- 230000000926 neurological effect Effects 0.000 description 1
- 230000000474 nursing effect Effects 0.000 description 1
- 206010029864 nystagmus Diseases 0.000 description 1
- 210000002589 oculomotor nerve Anatomy 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000005693 optoelectronics Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000007824 polyneuropathy Effects 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
- 201000003004 ptosis Diseases 0.000 description 1
- 230000010349 pulsation Effects 0.000 description 1
- 230000000241 respiratory effect Effects 0.000 description 1
- 230000029058 respiratory gaseous exchange Effects 0.000 description 1
- 230000036387 respiratory rate Effects 0.000 description 1
- 230000000284 resting effect Effects 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 230000035939 shock Effects 0.000 description 1
- 230000011664 signaling Effects 0.000 description 1
- 230000037321 sleepiness Effects 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 210000004243 sweat Anatomy 0.000 description 1
- 230000009182 swimming Effects 0.000 description 1
- 206010042772 syncope Diseases 0.000 description 1
- 230000001225 therapeutic effect Effects 0.000 description 1
- 238000002560 therapeutic procedure Methods 0.000 description 1
- 210000001685 thyroid gland Anatomy 0.000 description 1
- 208000005057 thyrotoxicosis Diseases 0.000 description 1
- 208000016686 tic disease Diseases 0.000 description 1
- 230000036962 time dependent Effects 0.000 description 1
- 230000003867 tiredness Effects 0.000 description 1
- 208000016255 tiredness Diseases 0.000 description 1
- 239000003053 toxin Substances 0.000 description 1
- 231100000765 toxin Toxicity 0.000 description 1
- 108700012359 toxins Proteins 0.000 description 1
- 208000009174 transverse myelitis Diseases 0.000 description 1
- 230000008733 trauma Effects 0.000 description 1
- 238000001429 visible spectrum Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 230000002618 waking effect Effects 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
- A61B5/1103—Detecting eye twinkling
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
- A61B3/0016—Operational features thereof
- A61B3/0066—Operational features thereof with identification means for the apparatus
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
- A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
- A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
- A61B3/113—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for determining or recording eye movement
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A61B5/163—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state by tracking eye movement, gaze, or pupil change
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61M—DEVICES FOR INTRODUCING MEDIA INTO, OR ONTO, THE BODY; DEVICES FOR TRANSDUCING BODY MEDIA OR FOR TAKING MEDIA FROM THE BODY; DEVICES FOR PRODUCING OR ENDING SLEEP OR STUPOR
- A61M21/00—Other devices or methods to cause a change in the state of consciousness; Devices for producing or ending sleep by mechanical, optical, or acoustical means, e.g. for hypnosis
- A61M21/02—Other devices or methods to cause a change in the state of consciousness; Devices for producing or ending sleep by mechanical, optical, or acoustical means, e.g. for hypnosis for inducing sleep or relaxation, e.g. by direct nerve stimulation, hypnosis, analgesia
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
- G06V20/597—Recognising the driver's state or behaviour, e.g. attention or drowsiness
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
- G06V40/193—Preprocessing; Feature extraction
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B21/00—Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
- G08B21/02—Alarms for ensuring the safety of persons
- G08B21/06—Alarms for ensuring the safety of persons indicating a condition of sleep, e.g. anti-dozing alarms
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B25/00—Alarm systems in which the location of the alarm condition is signalled to a central station, e.g. fire or police telegraphic systems
- G08B25/01—Alarm systems in which the location of the alarm condition is signalled to a central station, e.g. fire or police telegraphic systems characterised by the transmission medium
- G08B25/016—Personal emergency signalling and security systems
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B2503/00—Evaluating a particular growth phase or type of persons or animals
- A61B2503/20—Workers
- A61B2503/22—Motor vehicles operators, e.g. drivers, pilots, captains
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B2562/00—Details of sensors; Constructional details of sensor housings or probes; Accessories for sensors
- A61B2562/04—Arrangements of multiple sensors of the same type
- A61B2562/043—Arrangements of multiple sensors of the same type in a linear array
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/40—Detecting, measuring or recording for evaluating the nervous system
- A61B5/4076—Diagnosing or monitoring particular conditions of the nervous system
- A61B5/4094—Diagnosing or monitoring seizure diseases, e.g. epilepsy
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/48—Other medical applications
- A61B5/486—Bio-feedback
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/68—Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient
- A61B5/6801—Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient specially adapted to be attached to or worn on the body surface
- A61B5/6802—Sensor mounted on worn items
- A61B5/6803—Head-worn items, e.g. helmets, masks, headphones or goggles
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/68—Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient
- A61B5/6801—Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient specially adapted to be attached to or worn on the body surface
- A61B5/6813—Specially adapted to be attached to a specific body part
- A61B5/6814—Head
- A61B5/6821—Eye
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61M—DEVICES FOR INTRODUCING MEDIA INTO, OR ONTO, THE BODY; DEVICES FOR TRANSDUCING BODY MEDIA OR FOR TAKING MEDIA FROM THE BODY; DEVICES FOR PRODUCING OR ENDING SLEEP OR STUPOR
- A61M21/00—Other devices or methods to cause a change in the state of consciousness; Devices for producing or ending sleep by mechanical, optical, or acoustical means, e.g. for hypnosis
- A61M2021/0005—Other devices or methods to cause a change in the state of consciousness; Devices for producing or ending sleep by mechanical, optical, or acoustical means, e.g. for hypnosis by the use of a particular sense, or stimulus
- A61M2021/0044—Other devices or methods to cause a change in the state of consciousness; Devices for producing or ending sleep by mechanical, optical, or acoustical means, e.g. for hypnosis by the use of a particular sense, or stimulus by the sight sense
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61M—DEVICES FOR INTRODUCING MEDIA INTO, OR ONTO, THE BODY; DEVICES FOR TRANSDUCING BODY MEDIA OR FOR TAKING MEDIA FROM THE BODY; DEVICES FOR PRODUCING OR ENDING SLEEP OR STUPOR
- A61M21/00—Other devices or methods to cause a change in the state of consciousness; Devices for producing or ending sleep by mechanical, optical, or acoustical means, e.g. for hypnosis
- A61M2021/0005—Other devices or methods to cause a change in the state of consciousness; Devices for producing or ending sleep by mechanical, optical, or acoustical means, e.g. for hypnosis by the use of a particular sense, or stimulus
- A61M2021/0083—Other devices or methods to cause a change in the state of consciousness; Devices for producing or ending sleep by mechanical, optical, or acoustical means, e.g. for hypnosis by the use of a particular sense, or stimulus especially for waking up
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61M—DEVICES FOR INTRODUCING MEDIA INTO, OR ONTO, THE BODY; DEVICES FOR TRANSDUCING BODY MEDIA OR FOR TAKING MEDIA FROM THE BODY; DEVICES FOR PRODUCING OR ENDING SLEEP OR STUPOR
- A61M2205/00—General characteristics of the apparatus
- A61M2205/35—Communication
- A61M2205/3546—Range
- A61M2205/3569—Range sublocal, e.g. between console and disposable
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61M—DEVICES FOR INTRODUCING MEDIA INTO, OR ONTO, THE BODY; DEVICES FOR TRANSDUCING BODY MEDIA OR FOR TAKING MEDIA FROM THE BODY; DEVICES FOR PRODUCING OR ENDING SLEEP OR STUPOR
- A61M2205/00—General characteristics of the apparatus
- A61M2205/35—Communication
- A61M2205/3576—Communication with non implanted data transmission devices, e.g. using external transmitter or receiver
- A61M2205/3592—Communication with non implanted data transmission devices, e.g. using external transmitter or receiver using telemetric means, e.g. radio or optical transmission
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61M—DEVICES FOR INTRODUCING MEDIA INTO, OR ONTO, THE BODY; DEVICES FOR TRANSDUCING BODY MEDIA OR FOR TAKING MEDIA FROM THE BODY; DEVICES FOR PRODUCING OR ENDING SLEEP OR STUPOR
- A61M2230/00—Measuring parameters of the user
- A61M2230/04—Heartbeat characteristics, e.g. ECG, blood pressure modulation
- A61M2230/06—Heartbeat rate only
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61M—DEVICES FOR INTRODUCING MEDIA INTO, OR ONTO, THE BODY; DEVICES FOR TRANSDUCING BODY MEDIA OR FOR TAKING MEDIA FROM THE BODY; DEVICES FOR PRODUCING OR ENDING SLEEP OR STUPOR
- A61M2230/00—Measuring parameters of the user
- A61M2230/08—Other bio-electrical signals
- A61M2230/10—Electroencephalographic signals
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61M—DEVICES FOR INTRODUCING MEDIA INTO, OR ONTO, THE BODY; DEVICES FOR TRANSDUCING BODY MEDIA OR FOR TAKING MEDIA FROM THE BODY; DEVICES FOR PRODUCING OR ENDING SLEEP OR STUPOR
- A61M2230/00—Measuring parameters of the user
- A61M2230/40—Respiratory characteristics
- A61M2230/42—Rate
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
- G01S7/481—Constructional features, e.g. arrangements of optical elements
Definitions
- the present invention relates generally to systems and methods for monitoring movement of a human eye, and more particularly to systems and methods for real-time monitoring of fatigue and other states of mind in individuals, purposeful communication, and/or controlling devices based upon movement of their eye, eyelid, and/or other components of their eye.
- U.S. Pat. No. 3,863,243 discloses a device that sounds an alarm to warn a person using the device that they are beginning to fall asleep.
- the device includes a frame similar to a set of eyeglasses onto which is mounted a fiber optic bundle and a photocell that are directed towards the user's eye when the frame is worn.
- the fiber optic bundle is coupled to a source of light and a pulse generator to emit light towards the user's eye.
- the photocell detects the intensity of light reflected off of the user's eye, i.e., either by the eyelid when the eye is closed or the eye surface when the eye is open.
- Circuitry receives a signal from the photocell, and uses a timer to distinguish between regular blinks, and an extended time period during which the eye is closed, i.e., a time period that may indicate that the person is falling asleep. When a threshold time elapses, an alarm is sounded to notify and/or wake the user.
- This device requires running wires and fiber optic bundles from the frame to external components, e.g., the pulse generator and the required circuitry, and for this reason, the device may be awkward or inconvenient to use.
- some devices may detect eye movement, but may not be able to distinguish when the eye is opened or closed. For example, it may be desirable to measure the percentage of total time that the eyelids are closed as a function of time or the area of the palpebral fissure that is covered by the eyelid as the eye is opened or closed, commonly known as “PERCLOS,” for example during medical research or when monitoring driver alertness.
- PERCLOS the percentage of total time that the eyelids are closed as a function of time or the area of the palpebral fissure that is covered by the eyelid as the eye is opened or closed
- Devices that merely detect eye muscle movement or eyelash movement may not be able to distinguish when the eye is open or closed, and consequently may not be able to measure PERCLOS.
- such devices may not measure other parameters, such as velocity of eyelid closing or opening, acceleration or deceleration characteristics, duration of open or closed eye states, intervals between eye blinks and/or partial versus full eye blinks or eye closures.
- infrared cameras or other devices may be used to monitor a driver's awareness, which are typically mounted on the dashboard, roof or other fixed mounting within the user's vehicle. Such devices, however, require that the user maintain constant eye contact with the camera. In addition, they do not monitor eyelid movement if the user looks sideways or downwards, turns around, exits the vehicle or compartment in which he or she is being monitored, or if the camera moves relative to the individual. Further, such cameras may have problems seeing through eyeglasses, sunglasses, or even contact lenses, and may not operate effectively in sunlight.
- the present invention is directed to systems and methods for monitoring eye movement.
- humans blink at least about 5-30 times per minute, or about 7,000-43,000 times per day.
- Each involuntary-reflexive blink lasts about 200-300 milliseconds, and generally averaging about 250 milliseconds, amounting to about 1,750-10,800 seconds per day of eye closure due to involuntary blinking.
- the eye blink gets longer and slower until the eyes begin to close for short term “microsleeps,” i.e., sleep conditions that last for about 3-5 seconds or longer, or for prolonged sleep.
- the present invention provides systems and methods for monitoring, measuring, and/or responding to eye movement, e.g., nonpurposeful reflexive eyeblinks.
- the system includes an emitter and a sensor in a predetermined relationship with an eye such that the emitter emits light and the sensor detects light from the emitter, the sensor producing a light intensity signal indicating when the eye is open or closed. More preferably, the emitter is directed or aimed at the eyelid and eye, while the sensor detects eyelid-reflected light, since, unlike the eyelid, the eye ball (except the retina, which may cause a “red reflex” under white light conditions or “white pupil” under infrared light) does not reflect substantial light back to the sensor. Circuitry is coupled to the sensor for converting sequential light intensity signals corresponding to eyelid movement received from the sensor into a stream of data, and a processor converts the stream of data into an understandable message.
- the circuitry for converting sequential light intensity signals may compare the sequential light intensity signals with a predetermined time threshold to detect voluntary-intentional or unintentional-involuntary sequences of eyelid movements, corresponding, for example, to a predetermined binary code.
- Memory circuitry may be coupled to the processor for storing the stream of data and/or a communication device, such as a video monitor or synthesized voice module, may be coupled to the processor for communicating the understandable message.
- a control system may be coupled to the processor, and the understandable message may include a command for controlling equipment, including electrical or electronic equipment, machinery, or a computer or computer accessory devices coupled to the control system.
- the system preferably also includes a transmitter, preferably a radio frequency transmitter, for wireless transmission of the stream of data to a remote location.
- a transmitter preferably a radio frequency transmitter
- other forms of wireless transmission e.g. infrared, as well as hard-wire connections may be used.
- the processor, as well as the memory circuitry, communication device, and/or control system, may be located at the remote location, and a receiver may be coupled to the processor for receiving the stream of data from the transmitter.
- the system includes a detection device having a frame adapted to be worn on a person's head, e.g., with the frame resting on the bridge of the user's nose and/or ears.
- the frame has the emitter and sensor thereon such that the emitter and sensor are oriented towards the person's eye when the frame is worn on the person's head.
- the emitter and sensor are a single solid state device, such as a biosensor device, that emits light within a predetermined frequency range, for example infrared light, towards the eye and detects the emitted light reflected off of the eyelid, respectively.
- a system for monitoring a blinking cycle of a person from a remote location includes an emitter for directing light towards an eye, and a sensor in a predetermined relationship with the emitter for detecting the emitted light reflected off of the eye, the sensor producing an output signal indicating when the eye is open or closed.
- the emitter light may be reflected off of the eyelid back to the sensor, or diffused by the surface of the eyeball.
- a transmitter is coupled to the sensor for wireless transmission of the output signal, and a processor is provided for comparing the output signal to a predetermined threshold to detect when the eyelid is closed for a minimum predetermined duration.
- a warning indicator may be coupled to the processor, the warning indicator being activated when the processor detects that the eyelid is closed for the minimum predetermined duration.
- the warning indicator may be an audible buzzer, a visible warning light, a vibrating device, an electrical shock device, a gustatory smell device, or other device that may act as a stimulus to any sensory modality.
- a receiver may be provided at the remote location coupled to the processor for receiving the wireless transmission from the transmitter.
- Memory circuitry may be provided for storing the output signal and/or a processor may be provided for converting the output signal into an understandable message.
- a communication device may be coupled to the processor for communicating the understandable message.
- a self-contained device for detecting movement of a person's eyelid includes a frame adapted to be worn on the person's head, an emitter on the frame for directing light towards an eye of the person when the frame is worn, and a sensor on the frame for detecting light from the emitter.
- the sensor produces an output signal indicating when the eye is open or closed, and a transmitter on the frame is coupled to the sensor for wireless transmission of the output signal to a remote location.
- the frame may also include a processor for comparing the output signal to a predetermined threshold to detect drowsiness-induced eyelid movement.
- the emitter and sensor are preferably a solid state biosensor device for emitting and detecting infrared light, or alternatively an array of emitters and/or sensors in a predetermined configuration on the frame, e.g., in a vertical, horizontal, diagonal, or other linear or other geometric array of more than one emitter and/or sensor oriented towards one or both eyes.
- an array of emitters and/or sensors allows measurement of eyelid velocity, acceleration and deceleration, and calculation of “PERCLOS.”
- the emitter and/or sensors may be affixed to any number of points on the frame, e.g., around the lens and preferably in the nose bridge, or alternatively anywhere along the frame, including near or on the nasal portion of the frame, the attachment of a temple piece of the frame, and/or surface mounted on the lens of an eyeglass.
- the emitter and/or sensor may be embedded in the lens of an eyeglass, or otherwise such that they operate through the lens.
- the emitter(s) and/or sensor(s) are fixed on an eye-frame such that they move with the wearer's head movements, and continuously focus on the user's eyes, whether the user is in a vehicle, outdoors or in any other environment.
- a system in accordance with the present invention may detect eyelid movement of the user, distinguish normal blinks from other voluntary or involuntary eyelid movement, and produce a stream of data.
- the stream of data may be converted into an understandable message, such as a binary code, a command for controlling a piece of equipment, or an indicator of the user's physical, mental or emotional state.
- the system may provide a convenient and/or effective method for voluntary or involuntary communication based simply upon movement of the user's eye.
- a system for monitoring movement of a person's eye.
- the system includes a device configured to be worn on a person's head and an array of emitters on the device for directing light towards an eye of the person when the device is worn.
- the array of emitters is configured for projecting a reference frame towards the eye.
- a camera is oriented towards the eye for monitoring movement of the eye relative to the reference frame.
- the camera may be provided on the device or may be provided remote from the device, but in relatively close proximity to the user.
- the array of emitters includes a plurality of emitters disposed in a substantially vertical arrangement on the device, and a plurality of emitters disposed in a substantially horizontal arrangement on the device.
- the array of emitters may project a focused set of crossed bands towards the eye for dividing a region including the eye into four quadrants.
- the system preferably includes one or more scanning or nonscanning sensors on the device for detecting light from the array of emitters.
- the one or more sensors produce an output signal indicating when the eye is open or closed, similar to the embodiments described above.
- the sensors include an array of focused sensors in a predetermined relationship with the array of focused emitters for detecting light from the array of emitters that is reflected off of respective portions of the eye or its eyelid.
- the emitters because of their fixed position, produce a fixed reflection off of the surface of the eye and eyelid, appearing as a “glint,” i.e., a spot or band of light.
- Each sensor produces an output signal indicating when the respective portion of the eye is covered or not covered by the eyelid.
- the system may also include a processor for correlating the output signal from the one or more sensors with a video signal from the camera for determining the person's level of alertness.
- the system may also include a warning indicator on the device, the warning indicator being activated when the processor determines a predetermined level of drowsiness has occurred.
- Light from the array of emitters may be emitted towards the eye of a user wearing the device to project a reference frame onto the eye.
- the camera is capable of imaging light produced by the emitters, e.g., in the infrared light range, thereby detecting the projected light as a spot of light, band of light or other “glint.” Movement of the eye relative to the reference frame may be monitored with the camera. A graphical output of the movement monitored by the camera relative to the reference frame may be monitored. For example, infrared light from the emitters may be reflected off of the retina as a “red reflex” under white light, as a “white pupil” under infrared light, or as a dark pupil under subtraction, using methods known to those skilled in the art.
- the processor using these methods, may detect movement of the eye's pupil may be measured relative to the reference frame. This movement may be graphically displayed, showing the movement of the eye's pupil relative to the reference frame.
- the output signal from the one or more sensors may be correlated with video signals produced by the camera monitoring movement of the eye relative to the reference frame, thereby determining the person's level of drowsiness.
- FIG. 1 is a perspective view of a patient in a hospital wearing a system for communication using eyelid movement in accordance with the present invention.
- FIG. 2 is an enlarged perspective view of a preferred embodiment of the system for communication using eyelid movement, shown in FIG. 1 , including a detection device and a processing box.
- FIG. 3 is a schematic drawing of a preferred embodiment of circuitry for transmitting an output signal corresponding to a sequence of eyelid movements.
- FIG. 4 is a schematic drawing of a preferred embodiment of circuitry for controlling equipment in response to an output signal corresponding to a sequence of eyelid movements.
- FIG. 5 is a schematic drawing of a preferred embodiment of circuitry for detecting eyelid movement.
- FIGS. 6A-6C are sectional and front views of alternate embodiments of a device for emitting light towards and detecting light reflected from a surface of an open eye.
- FIGS. 7A-7C are sectional and front views of the devices of FIGS. 6A-6C , respectively, emitting light towards and detecting light reflected from a closed eyelid.
- FIG. 8 is a perspective view and block diagram of another preferred embodiment of a system for communication using eyelid movement.
- FIG. 9 is a block diagram of the components of a system for communication in accordance with the present invention.
- FIG. 10A is a perspective view of still another preferred embodiment of a system for communication using eyelid movement.
- FIG. 10B is a schematic detail of a portion of the system of FIG. 10A .
- FIG. 10C is a detail of a preferred embodiment of an array of emitters and sensors that may be provided on a nose bridge of an eye frame, such as that of FIG. 10A .
- FIG. 10D is a sectional view of the array of emitters and sensors of FIG. 10C emitting light and detecting light reflected from an eye.
- FIG. 11A is a schematic view of a system for selectively controlling a number of devices from a remote location based upon eyelid movement.
- FIG. 11B is a schematic view of additional devices that may be controlled by the system of FIG. 11B .
- FIG. 12A is a table showing the relationship between the activation of an array of sensors, such as that shown in FIGS. 10A-10D and an eye being monitored by the array, as the eye progresses between open and closed conditions.
- FIG. 12B is a graph showing a stream of data provided by an array of sensors, such as that shown in FIGS. 10A-10D , indicating the percentage of eye coverage as a function of time (“PERCLOS”).
- FIG. 12C is a graphical display of a number of physiological parameters, including PERCLOS, of a person being monitored by a system including a device such as that shown in FIGS. 10A-10D .
- FIG. 12D is a table showing the relationship between the activation of two-dimensional arrays of sensors and an eye being monitored, as the eye progresses between open and closed conditions.
- FIG. 13 is a perspective view of another system for monitoring eye movement, in accordance with the present invention.
- FIG. 14 is a detail of a camera on the frame of FIG. 13 .
- FIGS. 15A-15I are graphical displays of several parameters that may be monitored with the system of FIG. 13 .
- FIG. 16 is a detail of video output from a camera on the frame of FIG. 13 .
- FIG. 17 is a schematic showing circuitry for processing signals from a five-element sensor array, in accordance with an exemplary embodiment of the present invention.
- FIG. 1 shows a patient 10 in a bed 12 wearing a detection device 30 for detecting eyelid movement of the patient 10 to provide voluntary-purposeful and/or involuntary-nonpurposeful communication.
- the detection device 30 is coupled to a processing box 130 which converts the detected eyelid movement into a stream of data, an understandable message and/or into information, which may be communicated, for example, using a video display 50 , to a medical care provider 40 .
- the detection device 30 and processing box 130 together provide a system for communication 14 in accordance with one aspect of the present invention.
- FIGS. 2 , 6 A and 7 A a preferred embodiment of a system for communication 14 is shown that includes an aimable and focusable detection device 30 that is attachable to a conventional pair of eyeglasses 20 .
- the eyeglasses 20 include a pair of lenses 21 attached to a frame 22 , which includes bridgework 24 extending between the lenses 21 , and side members or temple pieces 25 carrying ear pieces 26 , all of which are conventional.
- the frame 22 may also be provided without the lenses 21 .
- the detection device 30 includes a clamp 27 for attaching to one of the side members 25 and an adjustable arm 31 onto which is mounted an emitter 32 and a sensor 33 .
- the emitter 32 and sensor 33 are mounted in a predetermined relationship such that the emitter 32 may emit a signal towards an eye 300 of a person wearing the eyeglasses 20 and the sensor 33 may detect the signal reflected from the surface of the eye 300 and eyelid 302 .
- the emitter 32 and sensor 33 may be mounted adjacent one another.
- the emitter 32 ′ and sensor 33 ′ may be mounted on the frame separately away from one another, preferably such that the emitter 32 ′ and sensor 33 ′ are disposed substantially laterally with respect to each other.
- the emitter 32 ′′ and sensor 33 ′′ may be mounted across the eye 300 in axial alignment with another. As the eyelid 302 closes, it may break the beam 340 being detected by the sensor 33 ′′.
- the emitter 32 and sensor 33 produce and detect continuous or pulsed light, respectively, preferably within the infrared range to minimize distraction or interference with the wearer's normal vision.
- the emitter 32 emits light in pulses at a predetermined frequency and the sensor 33 is configured to detect light pulses at the predetermined frequency. This pulsed operation may reduce energy consumption by the emitter 32 and/or may minimize interference with other light sources.
- other predetermined frequency ranges of light beyond or within the visible spectrum such as ultraviolet light, or other forms of energy, such as radio waves, sonic waves and the like, may be used.
- the processing box 130 is coupled to the detection device 30 by a cable 34 including one or more wires therein (not shown).
- the processing box 130 preferably includes a central processing unit (CPU) 140 and/or other circuitry, such as the exemplary circuitry shown in FIGS. 3-5 , for receiving and/or processing an output signal 142 , such as a light intensity signal, from the sensor 33 .
- the processing box 130 may also include control circuitry 141 for controlling the emitter 32 and/or the sensor 33 , or the CPU 140 may include internal control circuitry.
- the control circuitry 141 controls the emitter 32 to produce a flickering infrared signal pulsed at a predetermined frequency, as high as thousands of pulses per second to as little as about 4-5 pulses per second, and preferably at least about 5-20 pulses per second, thereby facilitating detection of nonpurposeful or purposeful eye-blinks as short as about 200 milliseconds per blink.
- the sensor 33 may be controlled to detect light pulses only at the predetermined frequency specific to the flicker frequency of the emitter 32 .
- the system 10 may be used under a variety of ambient conditions without the output signal 142 being substantially affected by, for example, bright sun light, total darkness, ambient infrared light backgrounds, or other emitters operating at different flicker frequencies.
- the flicker frequency may be adjusted to maximize the efficient measurement of the number of eye blinks per unit time (e.g. about ten to about twenty eye blinks per minute), the duration of each eye blink (e.g. about 200 milliseconds to about 300 milliseconds), and/or PERCLOS (i.e., the percentage of time that the eyelid is completely or partially closed), or to maximize efficiency of the system, while keeping power consumption to a minimum.
- the control circuitry 141 and/or processing box 130 may include manual controls (not shown) for adjusting the frequency, focus, or intensity of the light emitted by the emitter 32 , to turn the emitter 32 off and on, to adjust the threshold sensitivity of the sensor 33 , and/or to allow for self-focusing with maximal infrared reflection off of a closed eyelid, as will be appreciated by those skilled in the art.
- the processing box 130 also preferably includes a power source 160 for providing power to the emitter 32 , the sensor 33 , the CPU 144 , and/or other components in the processing box 130 .
- the processor box 130 may be powered by a conventional DC battery, e.g., a nine volt battery or a lithium battery.
- an adapter (not shown) may be connected to the processor box 130 , such as a conventional AC adapter or a twelve volt automobile lighter adapter.
- the CPU 140 includes timer circuitry 146 for comparing the length of individual elements of the output signal 142 to a predetermined threshold to distinguish between normal blinks and other eyelid movement.
- the timer circuitry 146 may be separate discrete components or may be provided internally within the CPU 140 , as will be appreciated by those skilled in the art.
- the CPU 140 converts the output signal 142 into a stream of data 144 which may be used to communicate to other persons or equipment.
- the stream of data 144 produced by the CPU 140 may be a binary signal, such as Morse code or ASCI code.
- the CPU 140 may be capable of producing a synthesized voice signal, a control signal for a piece of equipment, or even a pictorial representation.
- the processing box 130 may include a variety of output devices for using the stream of data 144 .
- an internal speaker 150 may be provided, which may produce an alarm sound or a synthesized voice.
- An output port 148 may be provided to which a variety of equipment, such as the video display 50 shown in FIG. 1 , may be directly coupled by hard-wire connections.
- the processing box 130 may also include a transmitter 152 coupled to the CPU 144 for wireless communication of the stream of data 144 to a remote location.
- the system for communication 14 may also include a receiving and processing unit 154 , such as a computer or other control or display system.
- the transmitter 152 is preferably a radio frequency transmitter capable of producing a short range signal, for example, reaching as far as about one hundred feet or more, and preferably about forty five feet to fifty feet, even through walls or obstacles, although alternatively an infrared transmitter may also be effective.
- the transmitter 152 may also be coupled to an amplifier (not shown) to allow the stream of data to be transmitted thousands of feet or more.
- the amplifier and transmitter 152 may communicate via telephone communication lines, satellites and the like, to transmit the stream of data to a remote location miles away from the system.
- the system may include, or may be coupled to a global positioning system (GPS) for monitoring the location, movement, and state of wakefulness and safety of an individual wearing the detection device 30 .
- GPS global positioning system
- the receiving and processing unit 154 includes a receiver 156 , preferably a radio frequency receiver, for receiving a signal 153 , including the stream of data, transmitted by the transmitter 152 .
- a processor 158 is coupled to the receiver 156 for translating, storing and/or using the information in the stream of data, the processor 158 being coupled to memory circuitry 160 , a communication device 162 , and/or a control system 164 .
- the receiving and processing unit 154 may include the memory circuitry 160 therein into which the processor 158 may simply store the stream of data for subsequent retrieval and analysis.
- the processor 158 may interpret the stream of data, for example, by converting a binary code in the stream of data into an understandable message, i.e., a series of letters, words and/or commands, and/or may use augmentative communication devices or software (such as KE:NX or Words Plus) to facilitate communication.
- the resulting message may be displayed on the communication device 162 , which may include a video display for displaying text, pictures and/or symbols, a synthesized voice module for providing electronic speech, and the like.
- the stream of data may be displayed graphically on a computer of video screen or other electronic display device as a “real time” message signal or numerically (e.g., displaying blink rate, blink duration, PERCLOS, etc.), or displayed graphically similar to an EKG or EEG tracing.
- the stream of data may be displayed along with other physiological data (e.g. heart rate, respiratory rate, other sleep polysomnographic (PSG) or electroencephalographic (EEG) variables).
- the stream of data may be integrated with controllers which monitor automobile or mechanical functions (e.g.
- vehicle speed, acceleration, braking functions, torque, sway or tilt, engine or motor speed, etc. to make intelligent decisions regarding slowing down or speeding up the vehicle depending upon road and/or vehicle conditions, as well as the state of consciousness, wakefulness or attentiveness of the driver or machine operator.
- the message may be interpreted by the processor 158 for directing the control system 164 to control one or more pieces of machinery or equipment.
- the stream of data may include a command to direct the control system 164 to control relay switches or other devices to turn off and on an electrical device, such as an appliance, electrical wheelchair, engine, light, alarm, telephone, television, computer, a tactile vibrating seat, and the like, or to operate an eye-activated computer mouse or other controller.
- the processor 158 may use the stream of data to control PC, IBM, Macintosh and other computers and compatible computer software and/or hardware, e.g., to interact with a computer similar to a mouse, a “return” key or a “joystick.”
- the stream of data may include commands to activate a series of menus from which sub-menus or individual items may be selected, as are used in commercially available special communications software, such as WORDS-PLUS or Ke:NX.
- the processor 158 may then control, scroll or select items from computer software programs, operate a printer or other peripheral device (e.g., selecting a font, paragraph, tab or other symbol operator, selecting commands, such as “edit,” “find,” “format,” “insert,” “help,” or controlling CD-ROM or disc drive operations, and/or other Windows and non-Windows functions).
- a printer or other peripheral device e.g., selecting a font, paragraph, tab or other symbol operator, selecting commands, such as “edit,” “find,” “format,” “insert,” “help,” or controlling CD-ROM or disc drive operations, and/or other Windows and non-Windows functions.
- the receiver 156 may be coupled directly to a variety of devices (not shown), such as radio or television controls, lamps, fans, heaters, motors, remote control vehicles, vehicle monitoring or controlling devices, computers, printers, telephones, lifeline units, electronic toys, or augmentative communication systems, to provide a direct interface between the user and the devices.
- devices such as radio or television controls, lamps, fans, heaters, motors, remote control vehicles, vehicle monitoring or controlling devices, computers, printers, telephones, lifeline units, electronic toys, or augmentative communication systems, to provide a direct interface between the user and the devices.
- the detection device 30 is placed on a user's head, i.e., by putting the eyeglasses 20 on as shown in FIG. 1 .
- the adjustable arm 31 and/or the clamp 27 may be adjusted to optimally orient the emitter 32 and sensor 33 towards the user's eye 300 (shown in FIGS. 6A-6C and 7 A- 7 C).
- the emitter 32 is activated and a beam of light 340 is directed from the emitter 32 towards the eye 300 .
- the intensity and/or frequency of the emitter 32 and/or the threshold sensitivity of the sensor 33 or other focus may then be adjusted (e.g. manually or automatically using self-adjusting features).
- FIGS. 6A and 6B illustrate an open eye condition, in which a ray of light 340 produced by the emitter 32 strikes the surface of the eye 300 itself and consequently is scattered, as shown by the rays 350 .
- the resulting light intensity detected by the sensor 33 is relatively low, i.e., the sensor 33 may not receive any substantial return signal.
- the eye 300 is shown with the eyelid 302 closed as may occur during normal blinks, moments of drowsiness, intentional blinks, or other eyelid movement. Because the light 340 strikes the eyelid 302 , it is substantially reflected back to the sensor 33 , as shown by the ray 360 , resulting in a relatively high light intensity being detected by the sensor 33 . Alternatively, as shown in 7 C, the beam of light 340 may be broken or cut by the eyelid 302 when the eye 300 is closed.
- the sensor 33 consequently produces a light intensity signal that indicates when the eye 300 is open or closed, i.e., corresponding to the time during which reflected light is not detected or detected, respectively, by the sensor 33 .
- the intensity of the infrared light reflected from the surface of the eyelid is not substantially affected by skin pigmentation. If it is desired to adjust the intensity of light reflected from the eyelid, foil, glitter, reflective moisturizer creams and the like may be applied to increase reflectivity, or black eye liner, absorptive or deflective creams and the like may be applied to reduce reflectivity.
- the light intensity detected by the sensor 33 results in an output signal 142 including a series of time-dependent light intensity signals (as shown, for example, in FIG. 12B ).
- the output signal 142 is received by the CPU 140 coupled to the sensor 33 , which compares the length of time of each light intensity signal 142 , for example, corresponding to a closed eye condition, with a predetermined threshold.
- the timer circuitry 146 may provide a threshold time to the CPU 140 for distinguishing normal blinks from intentional and/or other unintentional eyelid movement, which the CPU 140 may then filter out of the output signal 142 .
- the CPU 140 then produces a stream of data 144 which may be used for voluntary and/or involuntary communication.
- the detection device 30 may be used to detect impending drowsiness or “micro-sleeps” (i.e., sleep intrusions into wakefulness lasting a few seconds) of a user, with the processing box 130 triggering a warning to alert the user, others in his or her presence, or monitoring equipment of the onset of drowsiness.
- the threshold of the timer circuitry 146 may be adjusted such that the CPU 140 detects relatively long periods of eye closure, as may occur when a person is falling asleep.
- the threshold may be set at a time ranging from close to zero seconds up to several seconds, preferably from about 200 milliseconds to about 300 milliseconds, and most preferably about 250 milliseconds, to distinguish normal blinks from drowsiness-induced eyelid movement.
- the CPU 140 may activate a warning device.
- the warning device may be included within the processing box 130 , such as the speaker 150 , or alternatively on the frame, for example, by mounting a warning light (not shown) or an alarm speaker (not shown in FIG. 9 , see FIG. 10C ) on the frame.
- the detection device 30 may be used to unobtrusively record or monitor drowsiness-induced eyelid movement, with the CPU 140 producing a stream of data 144 which the transmitter 152 may transmit to the receiving and processing unit 154 ( FIG. 9 ).
- the device 30 may be used in conjunction with a vehicle safety system to monitor a driver's level of awareness or attentiveness.
- the stream of data 144 may be transmitted to a receiving and processing unit 154 mounted in a vehicle, which may store data on the driver's drowsiness and/or may use the data to make decisions and control the vehicle, e.g., adjust the vehicle's speed or even turn the vehicle's engine off.
- the detection device 30 may be used to monitor truck drivers, taxi drivers, ship or airline pilots, train conductors or engineers, radar or airport control tower operators, operators of heavy equipment or factory machinery, scuba divers, students, astronauts, entertainment participants or observers, and the like.
- the signals may be stored and analyzed in real time for trend changes measured over time to predict drowsiness effects of individuals using the device.
- the detection device 30 and system 14 may also be used in a medical diagnostic, therapeutic, research or professional setting to monitor the wakefulness, sleep patterns and/or the effects of drugs, which may affect blink rate, blink velocity, blink duration, or PERCLOS of a patient or vehicle operator. Similar to the method just described, the CPU 140 produces a stream of data 144 , which the transmitter may send to a remote receiving and processing unit 154 , which may store the stream of data 144 in the memory circuitry 160 for later retrieval and analysis by researchers, medical professionals, or safety personnel (e.g., similar to the way in which flight recorder data may be stored in an aircraft's “black box” recorder).
- the receiving and processing unit 154 may also display the stream of data 144 , for example at a nurse's station, as an additional parameter to continually monitor a patient's physical, mental, or emotional condition.
- the unit 154 may store and/or produce a signal, e.g., by a series of algorithms, that must be responded to within a predetermined time (e.g., performance vigilance monitoring) to prevent false positives and negatives.
- a number of medical conditions may be monitored by the detection device 30 and system 14 , such as petit mal epilepsy, in which the eyes flutter at a rate of about three cycles per second, grand mal or psychometer seizures, where the eyes may stare or close repetitively in a jerky manner, myoclonic seizures, in which the lids may open and close in a jerky manner, or tics, or other eye movements, such as encountered by people with Tourette's syndrome.
- the system may be used to monitor g-lock of pilots caused by g-force effects, hypoxemia of passengers or crew in aircraft due to losses in cabin pressure, nitrogen narcosis or “the bends” in divers, or the effects of gases, chemicals, or biological agents on military personnel or other individuals.
- the system may also be used to monitor psychological situations, for example, to detect when a person lies (e.g., by closing their eyes when lying), during hypnosis, to monitor attentiveness, the effects of medications, e.g., L-dopa and other anti-Parkinsonian medications or anti-convulsants, drugs, alcohol, toxins, or the effects of hypoxia or ventilation, and the like.
- medications e.g., L-dopa and other anti-Parkinsonian medications or anti-convulsants
- drugs e.g., alcohol, toxins, or the effects of hypoxia or ventilation, and the like.
- Neurological conditions may also be monitored where the innervation or mechanical function of the eyelid may be affected, such as in Parkinson's disease, muscle diseases, e.g., myotonia, myotonic muscular dystrophy, blepharospasm, photophobia or light sensitivity, encephalopathy, seizures, Bell's palsy, or where the condition may produce eyelid drooping or ptosis, such as third cranial nerve palsy or paresis, brainstem lesions or stroke, tumors, infection, metabolic diseases, trauma, degenerative conditions, e.g., multiple sclerosis, amyotrophic lateral sclerosis, polyneuropathy, myesthenia gravis, botulism, tetanus, tetany, tardive dyskinesia, brainstem encephalitis, and other primary eyelid conditions, such as exophthalmos, thyrotoxicosis or other thyroid conditions.
- muscle diseases e.g., myotonia, myotonic muscular
- the detector device 30 may be used in biofeedback applications, for example, in biofeedback, hypnosis or psychological therapies of certain conditions (e.g. tic disorders).
- the detector device 30 may produce a stimulus, e.g. activating a light or speaker, and monitor the user's eyelid movement in anticipation of receiving a response, e.g., a specific sequence of blinks, acknowledging the stimulus within a predetermined time. If the user fails to respond, the processor may store the response, e.g. including response time, and/or may automatically transmit a signal, such as an alarm signal.
- the detection device 30 may be used to monitor individuals in non-medical settings, such as during normal activity in a user's home or elsewhere.
- individuals with involuntary medical conditions such as epilepsy or narcolepsy
- other individuals such as, infants and children, prison inmates, demented patients (e.g., with Alzheimer's disease), law enforcement personnel, military personnel, bank tellers, cashiers, casino workers, students, swing or graveyard shift workers, and the like, may be monitored.
- Similar application may be applied in a sleep laboratory for monitoring sleep patients to measure parameters, such as onset of sleep, sleep latency, time of eyelid closing or opening, time of awakening during the night, etc., or to animal research where eye blinking may be a factor to be studied.
- the performance and vigilance abilities of the user may be tested and analyzed as a direct function of, or in relationship to, PERCLOS.
- the CPU 140 When the CPU 140 detects the presence of particular eyelid movement, such as an extensive period of eye closure which may occur, for example, during an epileptic seizure, a syncopal episode, a narcoleptic episode, or when dozing off while driving or working, the CPU 140 may produce an output signal which activates an alarm.
- the transmitter 152 may send an output signal to shut off equipment being used, to notify medical personnel, such as by automatically activating a telephone to dial emergency services, to signal remote sites, such as police stations, ambulances, vehicle control centers, guardians, and the like.
- the system for communication 14 may also find useful application for voluntary communication.
- a user wearing the detection device 30 may intentionally blink in a predetermined pattern, for example, in Morse code or other blinked code, to communicate an understandable message to people or equipment (e.g., to announce an emergency).
- the CPU 140 may convert a light intensity signal 142 received from the sensor 33 and corresponding to the blinked code into a stream of data 144 , or possibly directly into an understandable message including letters, words and/or commands.
- the stream of data 144 may then be displayed on a video 60 display 50 (see FIG. 1 ) coupled to the output port 148 , or emitted as synthesized speech on the internal speaker 150 .
- the stream of data 144 may be transmitted by the transmitter 152 via the signal 153 to the receiving and processing unit 154 for displaying messages, or for controlling equipment, such as household devices, connected to the control system 164 .
- the system 14 may be used by individuals in hospitalized or nursing care, for example by intubated, ventilated, restrained, paralyzed or weakened patients, to communicate to attending medical staff and/or to consciously signal a nurse's station.
- the device may be used in any environment or domain, e.g., through water or other substantially transparent fluids. Further, the device 30 may also be used as an emergency notification and/or discrete security tool.
- a person who may be capable of normal speech may wear the device 30 in the event of circumstances under which normal communication, i.e., speech, is not a viable option. For example, a bank or retail employee who is being robbed or is otherwise present during the commission of a crime may be able to discretely blink out a preprogrammed warning to notify security or to call law enforcement.
- a person with certain medical conditions may wear the device in the event that they are physically incapacitated, i.e., are unable to move to call for emergency medical care, but are still able to voluntarily move their eyes.
- a pre-recorded message or identifying data e.g. name of the user, their location, the nature of the emergency, etc.
- identifying data may be transmitted to a remote location by a specific set of eyeblink codes or preprogrammed message.
- the detection device 30 may be used to monitor patients in an ICU setting, patients on ventilators, prisoners, elderly or disabled persons, heavy equipment operators, truck drivers, motorists, ship and aircraft pilots, train engineers, radar or airport control tower operators, or as a nonverbal or subliminal tool for communication by military guards, police bank tellers, cashiers, taxi-drivers, and the like.
- the detection device 30 may also be used as a recreational device, for example, as a children's toy similar to a walkie-talkie or to operate a remote control toy vehicle.
- additional timer circuitry may be coupled to the CPU 140 such that the CPU 140 may compare the light intensity signals received from the sensor 33 to a second predetermined threshold provided by the timer circuitry.
- the second predetermined threshold corresponds to a time period during which a person would normally blink. If the CPU 140 fails to detect a normal blink within this time period or if the user fails to respond to a predetermined stimulus (e.g. a blinking light or sound), the CPU 140 may produce a signal, activating the speaker 150 or transmitting a warning using the transmitter 152 .
- the detection device 30 is removed by a perpetrator during commission of a crime, falls off because of the onset of a medical episode, as well as to prevent “false alarms,” or to measure the “state of attentiveness” of the user.
- performance vigilance tasks may be required of the user to determine whether the signal transmitted is a purposeful or “false alarm” signal, and also for measuring attention or drowsiness levels for purposes of biofeedback, and also to measure compliance of the user wearing the device.
- the polarity of the output signal 142 may be reversed such that a stream of data is produced only when the eye is opened, for example, when monitoring patients in a sleep lab to measure onset of sleep, sleep latency, time of eyelid closure, etc., or to monitor sleeping prison inmates.
- the CPU 140 may activate an alarm only when an open eye condition is detected, as will be appreciated by those skilled in the art.
- the emitter and sensor are a single solid state light emission and detecting biosensor device 132 which are mounted directly onto the eyeglasses 20 .
- the biosensor device 132 which preferably produces and detects infrared light, may be as small as 2 mm ⁇ 4 mm and weigh only a few grams, thereby enhancing the convenience, comfort and/or discretion of the detection device 30 . Because of the small size, the biosensor device 133 may be mounted directly in the lens 21 , as shown in FIG. 8 , on an outside or inside surface of the lens 21 , in the bridgework 24 or at another location on the frame 22 that may facilitate detection of eye movement.
- the biosensor device 132 may measure less than about five millimeters by five millimeters surface area, and may weigh as little as about one ounce, thereby providing a emitter/sensor combination that may be unobtrusive to vision, portable, and may be conveniently incorporated into a light weight eye frame. Because the entire system may be self-contained on the frame, it moves with the user no matter which direction he or she looks and may operate in a variety of environments or domains, day or night, underwater, etc.
- Multiple element arrays, e.g., linear optical scanning sensor arrays, appropriate for use with the present invention may be available from Texas Advanced Optoelectronic Solutions, Inc. (TAOS) of Plano, Tex, such as Model Nos. TSL 201 (64 pixels ⁇ 1 pixel), TSL 202 (128 ⁇ 1), TSL 208 (512 ⁇ 1), TSL 2301 (102 ⁇ 1).
- TAOS Texas Advanced Optoelectronic Solutions, Inc.
- biosensor devices 132 may be provided on the eyeglasses 20 , for example, a pair of biosensor devices 132 may be provided, as shown in FIG. 8 , for detecting eyelid movement of each eye of the user (not shown).
- a cable 134 extends from each biosensor device 132 to a processing box 130 , similar to the processing box 130 described above.
- the CPU 140 of the processing box 130 may receive and compare the output signal from each biosensor device 132 to further augment distinguishing normal blinks from other eyelid movement.
- the pair of biosensor devices 132 may allow use of more sophisticated codes by the user, e.g., blinking each eye individually or together, for communicating more effectively or conveniently, as will be appreciated by those skilled in the art.
- a blink of one eye could correspond to a “dot,” and the other eye to a “dash” to facilitate use of Morse code.
- the output signals from each eye could then be interpreted by the CPU 140 and converted into an understandable message.
- a right eye blink (or series of blinks) may cause an electric wheelchair to move to the right
- a left eye blink (or series of blinks) may move to the left
- two simultaneous right and left eye blinks may cause the wheelchair to move forward
- four simultaneous right and left eye blinks may cause the wheelchair to move backward.
- Similar combinations or sequences of eye blinks may be used to control the on/off function, or volume or channel control of a television, AM/FM radio, VCR, tape recorder or other electronic or electromechanical device, any augmentative communications or controlling device, or any device operable by simple “on/off” switches (e.g., wireless television remote controls single switch television control units, universal remote controllers, single switch multi-appliance units with AC plug/wall outlet or wall switch modules, computer input adapters, lighted signaling buzzer or vibrating signal boxes, switch modules of all types, video game entertainment controller switch modules and switch-controlled electronic toys).
- simple “on/off” switches e.g., wireless television remote controls single switch television control units, universal remote controllers, single switch multi-appliance units with AC plug/wall outlet or wall switch modules, computer input adapters, lighted signaling buzzer or vibrating signal boxes, switch modules of all types, video game entertainment controller switch modules and switch-controlled electronic toys.
- one or more lenses or filters may be provided for controlling the light emitted and/or detected by the biosensor device, an individual emitter and/or detector.
- the angle of the light emitted may be changed with a prism or other lens, or the light may be columnated or focused through a slit to create a predetermined shaped beam of light directed at the eye or to receive the reflected light by the sensor.
- An array of lenses may be provided that are adjustable to control the shape, e.g. the width, etc., of the beam of light emitted or to adjust the sensitivity of the sensor.
- the lenses may be encased along with the emitter in plastic and the like, or provided as a separate attachment, as will be appreciated by those skilled in the art.
- FIG. 10A another preferred embodiment of a system for communication 414 is shown, that includes a frame 422 including a biosensor device 432 with associated processor and transmitter circuitry 430 provided directly on the frame 422 , for example, to enhance the convenience and discretion of the system for communication 414 .
- the frame 422 may include a bridge piece 424 onto which the biosensor device 432 may be slidably and/or adjustably mounted, and a pair of ear supports 423 , 425 .
- One of the supports 423 may have a larger size compared to the other support 425 , for example, to receive the processor and transmitter circuitry 430 embedded or otherwise mounted thereon.
- a processor 440 similar to the CPU 140 in the processing box 130 previously described, may be provided on the frame 422 , and a power source, such as a lithium battery 460 , may be inserted or affixed to the support 423 .
- a radio frequency or other transmitter 452 is provided on the support 423 , including an antenna 453 , which may be embedded or otherwise fastened along the ear support 423 , in the temple piece or elsewhere in the frame 422 .
- the system 414 may also include manual controls (not shown) on the ear support 423 or elsewhere on the frame 422 , for example to turn the power off and on, or to adjust the intensity and/or threshold of the biosensor device 432 .
- a system for communication 414 may be provided that is substantially self-contained on the frame 422 , which may or may not include lenses (not shown) similar to eyeglasses. External cables or wires may be eliminated, thereby providing a more convenient and comfortable system for communication.
- a linear array 530 of emitters 532 and sensors 533 may be provided, preferably in a vertical arrangement mounted on a nose bridge 524 of an eye frame 522 .
- a CPU 540 , battery 460 , transmitter antenna 543 , and warning indicator 550 may also be provided on the frame 522 , preferably in the temple piece 525 , similar to the previously described embodiment.
- An LED 542 or similar stimulus device may also be provided at a predetermined location on the eye frame 522 to allow routine biofeedback responses from the user.
- a receiver 544 may be provided for receiving the stream of data created by the CPU 540 and transmitted by the transmitter 543 .
- each of the sensors 533 and the emitter 532 are coupled to the CPU 540 or other control circuitry for controlling the emitter 532 and for processing the light intensity signals produced by the sensors 532 .
- the CPU 540 may cycle through the sensors 533 in the array 530 and sequentially process the signal from each of the sensors 533 , similar to the processors previously described.
- the emitter 532 includes a lens 534 to focus a beam of light (indicated by individual rays 360 a, 360 b) onto the eye 300 , preferably towards the pupil 301 .
- the sensors 533 are embedded within the nose bridge 524 and a slit 535 is provided for each, the slits 535 having a predetermined size to control the reflected light detected by each sensor 533 .
- each sensor 535 may detect movement of the eyelid 302 past a particular portion of the eye 300 , e.g., to measure PERCLOS, as shown in FIG. 12A .
- the sensors or emitters may have lenses or columnating devices to focus emitted or reflected light.
- the linear array 530 may facilitate measurement of additional parameters related to eyelid movement in addition to mere eye closure. For example, to measure the velocity of the eyelid opening or closing, i.e., the rate of eye closure, the CPU 540 may compare the time delay between the activation of successive sensors 533 . In addition, the output signals from the sensors 553 may be processed to measure the percentage of pupil coverage of the eyelid 302 , for example, due to partial eye closure, as a function of time, e.g., to monitor when the eye is partially, but not completely, closed, and/or to monitor the percentage of time that the eye is closed (PERCLOS), as shown in FIGS. 12A-12C , e.g., compared to the user's baseline of maximal eye opening.
- PERCLOS percentage of time that the eye is closed
- a two-dimensional array of sensors such as a 5 ⁇ 5 array 633 or a 9 ⁇ 11 array, 733 may be provided.
- Other arrays including any number of elements in the array may be provided, and the invention should not be limited to the exemplary embodiments described herein.
- the sensors 633 , 733 may then be used to measure surface area reflectivity of light from the emitter 632 , i.e., the processor (not shown) may process the signals from each sensor in the array 633 , 733 to create a stream of data indicating the percentage of surface area of the eye 300 covered by the eyelid 302 .
- the sensors in the array 633 , 733 may be sufficiently sensitive or have sufficient resolution such that they may detect “red reflex” or the equivalent infrared “bright pupil” reflection due to the reflection of light off of the retina through the pupil 301 .
- the sensors may produce a light intensity signal that includes a substantially zero value, indicating no red reflex or bright pupil, a low output, indicating red reflex or white pupil reflex, and a high output, indicating reflection off of a closed eyelid 302 .
- the red reflex may appear as a bright white light pupil (resulting from infrared light from the emitter(s) reflecting off of the retina when the eyelid is open, or as a dark or “black pupil” if the processor uses subtraction algorithms, as is known in the art.
- the processor may thereby process the light intensity signals to detect when the pupil 301 is covered by the eyelid 302 , i.e., at which point the user cannot see, even though their eye 300 may not be entirely covered by the eyelid 302 , generally at a PERCLOS value of about 50-75 percent in primary gaze.
- the sensor(s) may detect a red reflex or bright pupil even through the PERCLOS measurement may be as great as 75 -80 percent or more, e.g., where the eye may still see through a narrow slit-like palpebral fissure opening in downward gaze.
- the processor and/or transmitter circuitry may include identification circuitry (not shown), either as a discrete memory chip or other circuit element, or within the CPU itself.
- the identification circuitry may be preprogrammed with a fixed identification code, or may be programmable, for example, to include selected identification information, such as the identity of the user, the user's location, an identification code for the individual detection device, and the like.
- the CPU may selectively add the identification information to the transmitted stream of data 553 , or the identification information may be automatically or periodically included in the stream of data 553 , thereby allowing the stream of data 553 to be associated with a particular detection device, individual user and/or a specific location.
- the identification information may be used by the processor, for example, at a remote location, to distinguish between streams of data received from a number of detection devices, which may then be stored, displayed, etc. as previously described. Thus, the detection device may not require users to consciously communicate certain identification or other standard information when the system is used.
- the receiver 544 may allow the user to control one or more devices coupled to the receiver 544 through a single switch multi-appliance control unit 550 .
- the control unit 550 includes its own transmitter adapted to transmit on/off or other control signals that may be received by individual control modules 552 a- 552 fe.
- the user 10 may blink to create a transmitted stream of data 553 that includes commands to turn off and on, or otherwise control, selected appliances using the control unit 550 and control modules 552 a- 552 fe, such as, a radio 554 , a television 556 , a light 558 a., a light 562 controlled by a wall switch 560 , a fan 566 plugged into a wall socket 564 , and the like.
- a radio 554 such as, a radio 554 , a television 556 , a light 558 a., a light 562 controlled by a wall switch 560 , a fan 566 plugged into a wall socket 564 , and the like.
- the receiver 554 may be coupled to other systems, such as a computer 570 and printer 572 , a vehicle integration system 574 , a lifeline unit 576 , a GPS or other satellite transmitter 578 , and the like.
- the transmitted stream of data 553 may be processed alone or along with additional data, such as other vehicle sensor information 573 , to further enhance monitoring a user, such as a long-distance truck driver.
- the system 810 includes a frame 812 that may include a bridge piece 814 and a pair of ear supports 816 .
- the frame 812 may include a pair of lenses (not shown), such as prescription, shaded, or protective lenses, although they are not necessary for operation of the invention.
- the system may be provided on other devices that may be worn on a user's head, such as a pilot's oxygen mask, protective eye gear, a patient's ventilator, a scuba or swimming mask, a helmet, a hat, a head band, a head visor, and the like (not shown).
- the components of the system may be provided at a variety of locations on the device that generally minimize interference with the user's vision and/or normal use of the device.
- An array of emitters 820 are provided on the frame 812 , preferably in a vertical array 820 a and a horizontal array 820 b.
- the emitters 820 are infrared emitters configured to emit pulses at a predetermined frequency, similar to the embodiments described above.
- the emitters 820 are arranged on the frame such that they project a reference frame 850 onto the region of the user's eye 300.
- the reference frame includes a pair of crossed bands 850 a, 850 b dividing the region into four quadrants.
- intersection of the crossed bands is preferably disposed at a location corresponding substantially to the eye's pupil during primary gaze, i.e., when the user is looking generally straight forward along axis 310 extending directly ahead of the user's eye 300.
- other reference frames may be provided, generally including a vertical component and a horizontal component.
- An array of sensors 822 are also provided on the frame 812 for detecting light from the emitters 820 that is reflected off of the user's eyelid.
- the sensors 822 preferably generate output signals having an intensity identifying whether the eyelid is closed or open, similar to the embodiments described above.
- the sensors 822 are disposed adjacent to respective emitters 820 for detecting light reflected off of respective portions of the eyelid.
- sensors 822 may only be provided in a vertical array, e.g., along the bridge piece 814 , for monitoring the amount of eyelid closure, similar to the embodiments described above.
- the emitters 820 and sensors 822 may be solid state biosensors (not shown) that provide both the emitting and sensing functions in a single device.
- Circuitry may be provided for measuring PERCLOS or other parameters using the signals generated by the array of sensors.
- FIG. 17 shows an exemplary schematic that may be used for processing signals from a five element array, e.g., to obtain PERCLOS measurements or other alertness parameters.
- the system 810 also includes a camera 830 provided on the frame 810 .
- the camera 830 is mounted on or adjacent the bridge piece 814 offset from the axis 310 such that the camera 830 is oriented towards the region surrounding one of the user's eyes 300 while minimizing interference with the user's vision.
- the camera 830 preferably includes a bundle of fiberoptic cables 832 that terminate in a lens 834 , as shown in FIG. 14 , on a first end mounted adjacent the bridge piece 814 and a second end 837 that is connected to a detector 838 , e.g., a CCD or CMOS sensor, such as those used in endoscopes, that may convert an image into a digital video signal.
- a detector 838 e.g., a CCD or CMOS sensor, such as those used in endoscopes, that may convert an image into a digital video signal.
- the camera 830 is configured to detect the frequency of light emitted by the emitters 820 , e.g., infrared light.
- the camera 830 may rely on the light projected by the emitters 820 , or the fiberoptic cables 832 may include emitters 836 for projecting light, e.g., infrared light, onto the user's eyes and/or face.
- the system 810 may include a second camera 840 oriented away from the user's head, e.g., to monitor the user's surroundings.
- One of the ear supports 816 may include a panel 818 for mounting a controller or other processor 842 , a transmitter 844 , an antenna 845 , and a battery 846 .
- the processor 840 842 is coupled to the emitters 820 , the sensors 822 , and/or the camera 830 for controlling their operation.
- the transmitter 844 may be coupled to the processor 842 for receiving the output signals from the sensors 822 and/or the video signals from the camera 830 , e.g., to transmit the signals to a remote location, as described below.
- the transmitter 844 may be coupled directly to output leads from the sensors 822 and the camera 830 .
- the frame 812 may also include manual controls (not shown), e.g., on the ear support 816 , for example, to turn the power off and on, or to adjust the intensity and/or threshold of the emitters 820 , the sensors 822 , and/or the camera 830 .
- manual controls e.g., on the ear support 816 , for example, to turn the power off and on, or to adjust the intensity and/or threshold of the emitters 820 , the sensors 822 , and/or the camera 830 .
- the system 810 may also include one or more additional sensors on the frame 812 .
- the sensors may be coupled to the processor 842 and/or to the transmitter 844 so that the signals from the sensors may be monitored, recorded, and/or transmitted to a remote location.
- one or more position sensors 852 a, 852 b may be provided, e.g., for determining the spatial orientation of the frame 812 , and consequently the user's head.
- actigraphic sensors may be provided to measure tilt or movement of the head, e.g., to monitor whether the user's head is drooping forward or tilting to the side.
- Acoustic sensors, e.g., a microphone 854 may be provided for detecting environmental noise or sounds produced by the user.
- the frame 812 may include one or more sensors for measuring one or more physical characteristics of the user.
- EEG electrodes 856 may be provided on the ear support 816 , above or below the nasion, and/or other region that may contact the patient's skin to measure brain activity, e.g., waking, drowsy, or other sleep-related brain activity.
- An EKG electrode (not shown) may be provided that is capable of measuring cardiac activity through a skin contact site.
- a pulse sensor (not shown) may be used to measure cardiovascular pulsations, or an oximetry sensor 858 may be used to measure oxygen saturation levels.
- a thermistor or other sensor may measure of respiratory air flow, e.g., through the user's nose.
- a thermister, thermocouple, or other temperature sensor may be provided for measuring the user's skin temperature.
- a sweat detector (not shown) may be provided for measuring moisture on the user's skin.
- the system 810 may include one or more feedback devices on the frame 812 . These devices may provide feedback to the user, e.g., to alert and/or wake the user, when a predetermined condition is detected, e.g., a state of drowsiness or lack of consciousness.
- the feedback devices may be coupled to the processor 842 , which may control their activation.
- a mechanical vibrator device 860 may be provided at a location that may contact the user, e.g., on the ear support 816 , that may provide tactile vibrating stimuli through skin contact.
- An electrode (not shown) may be provided that may produce relatively low power electrical stimuli.
- a light emitter such as one or more LED's may be provided at desired locations, e.g., above the bridge piece 814 .
- audio devices 862 such as a buzzer or other alarm, may be provided, similar to the previous embodiments.
- aroma-emitters may be provided on the frame 810 812, e.g., on or adjacent to the bridge piece 814 .
- the feedback devices may be provided separate from the frame, but located in a manner capable of providing a feedback response to the user.
- audio, visual, tactile (e.g., vibrating seat), or olfactory emitters may be provided in the proximity of the user, such as any of the devices described above.
- heat or cold generating devices may be provided that are capable of producing thermal stimuli to the user, e.g., a remotely controlled fan or air conditioning unit.
- the system 810 may also include components that are remote from the frame 812 , similar to the embodiments described above.
- the system 810 may include a receiver, a processor, and/or a display (not shown) at a remote location from the frame 812 , e.g., in the same room, at a nearby monitoring station, or at a more distant location.
- the receiver may receive signals transmitted by the transmitter 842 , including output signals from the sensors 822 or any of the other sensors provided on the frame 812 and/or the video signals from the camera 830 .
- a processor may be coupled to the receiver for analyzing signals from the components on the frame 812 , e.g., to prepare the signals for graphical display.
- the processor may prepare the video signals from the camera 830 for display on a monitor, thereby allowing personal monitoring of the user.
- other parameters may be displayed, either on a single monitor or on separate displays.
- FIG. 15a FIGS. 15A- 15 I shows signals indicating the output of various sensors that may be on the frame 812 , which may be displayed along a common time axis or otherwise correlated, e.g., to movement of the user's eye and/or level of drowsiness.
- the processor may superimpose or otherwise simultaneously display the video signal in conjunction with the other sensed parameters to allow a physician or other individual to monitor and personally correlate these parameters to the user's behavior.
- the processor may automatically process the signals to monitor or study the user's behavior.
- the processor may use the output signals to monitor various parameters related to eye movement, such as eye blink duration (EBD), eye blink frequency, eye blink velocity, eye blink acceleration, interblink duration (IBD), PERCLOS, PEROP (percentage eyelid is open), and the like.
- the video signals from the camera 830 may be processed to monitor various eye parameters, such as pupillary size, location, e.g., within the four quadrant defined by the crossed bands 850 , eye tracking movement, eye gaze distance, and the like.
- various eye parameters such as pupillary size, location, e.g., within the four quadrant defined by the crossed bands 850 , eye tracking movement, eye gaze distance, and the like.
- the camera 830 may detect a reference frame projected onto the region of the user's eye by the emitters.
- FIG. 16 shows an exemplary video output from a camera included in a system having twenty emitters disposed in a vertical arrangement.
- the camera may detect twenty discrete regions of light arranged as a vertical band.
- the camera may also detect a “glint” point, G, and/or a moving bright pupil, P.
- the movement of the pupil may be monitored in relation to the glint point, G, and/or in relation to the vertical band 1-20.
- the processor may determine the location of the pupil in terms of orthogonal coordinates (e.g., x-y or angle-radius) relative to the reference frame 850 .
- the location of the pupil may be determined relative to any stationary “glint” point on the user's eye.
- the camera 830 itself may project a point of light onto the eye that may be reflected and detected by the camera. This “glint” point remains substantially stationary since the camera 830 is fixed to the frame 812 .
- the video signals from a remote camera that may view the user's face from a distance may be used to monitor various facial measures, such as facial expression, yawning frequency, and the like, in addition to or alternatively, the project instead of the projected light reference frame from the emitters.
- the parameters from other sensors may be processed and correlated, such as head orientation, tilt, body movement, physiological parameters, and the like.
- the processor may correlate these parameters to generate a composite fatigue index (CFI) that is a function of two or more of these parameters.
- CFI composite fatigue index
- the system 810 may activate an alarm or other notice to the user and/or to another party at a remote location.
- the system 810 may provide a more effective way to monitor the user's fatigue, drowsiness, alertness, mental state, and the like.
- the system 810 may be used to generate predetermined outputs, e.g., to activate or deactivate equipment, such as a vehicle being operated by the user when a predetermined condition, e.g., CFI value, is determined by the system 810 .
- a predetermined condition e.g., CFI value
- the processor may be provided on the frame 812 , e.g. as part of processor 842 , for monitoring the parameters for a predetermined event, such as a predetermined CFI value, to occur.
- a predetermined event such as a predetermined CFI value
- the eye tracking parameters described above may be monitored by a remote camera, e.g., in a fixed position in front of the user, such as the dashboard of a vehicle and the like.
- the remote camera may be coupled to the processor, either directly or via its own transmitter, as will be appreciated by those skilled in the art.
- a system in accordance with the present invention may monitor or detect one or more parameters, such as those listed below in Table 1.
- EYELID MEASURES EYEGLAZE MEASURES Percentage of time (t) and Eye Tracking Movements (ETM) the amount palpebral including Directional Nystagmus fissure is opened Eye Gaze Distance (EGD) and (PEROP-t, -d, -dt), Direction or closed Eye Movement Distance (PERCLOS-t, -d, -dt), Eye Movement Velocity (EMV) lid droop Eye Movement Acceleration (EMA) Eye Blink Duration (EBD) and Deceleration (EMD) Eye Blink Frequency (EBF) Eye Movement Frequency (EMF) Eye Blink Velocity (EBV) Phoria/eye Drift Measures (PDM) Eye Blink Acceleration HEAD ORIENTATION MEASURES (EBAc) Head Direction or Orientation Decceleration (EBDc) (HDir) Interblink duration (IBD) HEAD MOVEMENT MEASURES Eye blink flurries Head Nodding Freque
Landscapes
- Health & Medical Sciences (AREA)
- Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Animal Behavior & Ethology (AREA)
- Veterinary Medicine (AREA)
- Biomedical Technology (AREA)
- Heart & Thoracic Surgery (AREA)
- General Physics & Mathematics (AREA)
- Biophysics (AREA)
- Medical Informatics (AREA)
- Molecular Biology (AREA)
- Surgery (AREA)
- Ophthalmology & Optometry (AREA)
- Psychology (AREA)
- Pathology (AREA)
- Anesthesiology (AREA)
- Computer Networks & Wireless Communication (AREA)
- Remote Sensing (AREA)
- Radar, Positioning & Navigation (AREA)
- Electromagnetism (AREA)
- Human Computer Interaction (AREA)
- Emergency Management (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Developmental Disabilities (AREA)
- Child & Adolescent Psychology (AREA)
- Educational Technology (AREA)
- Social Psychology (AREA)
- Hospice & Palliative Care (AREA)
- Psychiatry (AREA)
- Physiology (AREA)
- Acoustics & Sound (AREA)
- Hematology (AREA)
- Pain & Pain Management (AREA)
- Computer Security & Cryptography (AREA)
- Dentistry (AREA)
Abstract
Apparatus for monitoring movement of a person's eye, e.g., to monitor drowsiness. The system includes a frame that is worn on a person's head, an array of emitters on the frame for directing light towards the person's eye, and an array of sensors on the frame for detecting light from the array of emitters. The sensors detect light that is reflected off of respective portions of the eye or its eyelid, thereby producing output signals indicating when the respective portions of the eye is covered by the eyelid. The emitters project a reference frame towards the eye, and a camera on the frame monitors movement of the eye relative to the reference frame. This movement may be correlated with the signals from the array of sensors and/or with signals from other sensors on the frame to monitor the person's level of drowsiness.
Description
Notice: More than one reissue application has been filed for the reissue of U.S. Pat. No. 6,163,281. The reissue applications are application Ser. No. 11/097,942, issued as Re 39,539, Ser. No. 11/732,828, which is a continuation reissue of Re 39,539, and Ser. No. 12/199,693 (the present application), which is a divisional reissue of Ser. No. 11/732,828.
This application is a reissue of U.S. Pat. No. 6,542,081 and is a divisional of application Ser. No. 11/732,828, which is a continuation of application Ser. No. 11/097,942, issued as Re 39,539, which are both reissues of application Ser. No. 09/740,738, U.S. Pat. No. 6,542,081, which is a continuation-in-part of application Ser. No. 09/104,258, filed Jun. 24, 1998, issuing as U.S. Pat. No. 6,163,281 on Dec. 19, 2000, which is a continuation-in-part of application Ser. No. 08/978,100, filed Nov. 25, 1997, now U.S. Pat. No. 6,246,344 issued Jun. 12, 2001, which is a continuation-in-part of application Ser. No. 08/699,670, filed Aug. 19, 1996, now U.S. Pat. No. 5,748,113 issued May 5, 1998, the disclosures of which are expressly incorporated herein by reference.
The present invention relates generally to systems and methods for monitoring movement of a human eye, and more particularly to systems and methods for real-time monitoring of fatigue and other states of mind in individuals, purposeful communication, and/or controlling devices based upon movement of their eye, eyelid, and/or other components of their eye.
There have been attempts to use movement of the human eye to monitor involuntary conditions, specifically a person's wakefulness or drowsiness. For example, U.S. Pat. No. 3,863,243 discloses a device that sounds an alarm to warn a person using the device that they are beginning to fall asleep. The device includes a frame similar to a set of eyeglasses onto which is mounted a fiber optic bundle and a photocell that are directed towards the user's eye when the frame is worn. The fiber optic bundle is coupled to a source of light and a pulse generator to emit light towards the user's eye.
The photocell detects the intensity of light reflected off of the user's eye, i.e., either by the eyelid when the eye is closed or the eye surface when the eye is open. Circuitry receives a signal from the photocell, and uses a timer to distinguish between regular blinks, and an extended time period during which the eye is closed, i.e., a time period that may indicate that the person is falling asleep. When a threshold time elapses, an alarm is sounded to notify and/or wake the user. This device, however, requires running wires and fiber optic bundles from the frame to external components, e.g., the pulse generator and the required circuitry, and for this reason, the device may be awkward or inconvenient to use.
Other devices, such as those disclosed in U.S. Pat. Nos. 5,469,143 and 4,359,724, directly engage the eyelid or eyebrow of a user to detect movement of the eye and activate an alarm when a drowsiness condition is detected. These mechanical devices may be mounted directly onto the skin to detect muscle movement or may involve placing a mechanical arm against the eyelid, and consequently may be uncomfortable to wear and use.
In addition, some devices may detect eye movement, but may not be able to distinguish when the eye is opened or closed. For example, it may be desirable to measure the percentage of total time that the eyelids are closed as a function of time or the area of the palpebral fissure that is covered by the eyelid as the eye is opened or closed, commonly known as “PERCLOS,” for example during medical research or when monitoring driver alertness. Devices that merely detect eye muscle movement or eyelash movement may not be able to distinguish when the eye is open or closed, and consequently may not be able to measure PERCLOS. Similarly, such devices may not measure other parameters, such as velocity of eyelid closing or opening, acceleration or deceleration characteristics, duration of open or closed eye states, intervals between eye blinks and/or partial versus full eye blinks or eye closures.
Further, infrared cameras or other devices may be used to monitor a driver's awareness, which are typically mounted on the dashboard, roof or other fixed mounting within the user's vehicle. Such devices, however, require that the user maintain constant eye contact with the camera. In addition, they do not monitor eyelid movement if the user looks sideways or downwards, turns around, exits the vehicle or compartment in which he or she is being monitored, or if the camera moves relative to the individual. Further, such cameras may have problems seeing through eyeglasses, sunglasses, or even contact lenses, and may not operate effectively in sunlight.
Accordingly, it is believed that a more effective system and method for monitoring eye and/or eyelid movement would be considered useful.
The present invention is directed to systems and methods for monitoring eye movement. Generally, humans blink at least about 5-30 times per minute, or about 7,000-43,000 times per day. Each involuntary-reflexive blink lasts about 200-300 milliseconds, and generally averaging about 250 milliseconds, amounting to about 1,750-10,800 seconds per day of eye closure due to involuntary blinking. As tiredness or sleepiness occurs, the eye blink gets longer and slower until the eyes begin to close for short term “microsleeps,” i.e., sleep conditions that last for about 3-5 seconds or longer, or for prolonged sleep. The present invention provides systems and methods for monitoring, measuring, and/or responding to eye movement, e.g., nonpurposeful reflexive eyeblinks.
In a preferred embodiment, the system includes an emitter and a sensor in a predetermined relationship with an eye such that the emitter emits light and the sensor detects light from the emitter, the sensor producing a light intensity signal indicating when the eye is open or closed. More preferably, the emitter is directed or aimed at the eyelid and eye, while the sensor detects eyelid-reflected light, since, unlike the eyelid, the eye ball (except the retina, which may cause a “red reflex” under white light conditions or “white pupil” under infrared light) does not reflect substantial light back to the sensor. Circuitry is coupled to the sensor for converting sequential light intensity signals corresponding to eyelid movement received from the sensor into a stream of data, and a processor converts the stream of data into an understandable message.
The circuitry for converting sequential light intensity signals may compare the sequential light intensity signals with a predetermined time threshold to detect voluntary-intentional or unintentional-involuntary sequences of eyelid movements, corresponding, for example, to a predetermined binary code. Memory circuitry may be coupled to the processor for storing the stream of data and/or a communication device, such as a video monitor or synthesized voice module, may be coupled to the processor for communicating the understandable message. In addition, a control system may be coupled to the processor, and the understandable message may include a command for controlling equipment, including electrical or electronic equipment, machinery, or a computer or computer accessory devices coupled to the control system.
The system preferably also includes a transmitter, preferably a radio frequency transmitter, for wireless transmission of the stream of data to a remote location. Alternatively, other forms of wireless transmission, e.g. infrared, as well as hard-wire connections may be used. The processor, as well as the memory circuitry, communication device, and/or control system, may be located at the remote location, and a receiver may be coupled to the processor for receiving the stream of data from the transmitter.
In a preferred form, the system includes a detection device having a frame adapted to be worn on a person's head, e.g., with the frame resting on the bridge of the user's nose and/or ears. The frame has the emitter and sensor thereon such that the emitter and sensor are oriented towards the person's eye when the frame is worn on the person's head. Preferably, the emitter and sensor are a single solid state device, such as a biosensor device, that emits light within a predetermined frequency range, for example infrared light, towards the eye and detects the emitted light reflected off of the eyelid, respectively.
In another preferred embodiment, a system for monitoring a blinking cycle of a person from a remote location is provided that includes an emitter for directing light towards an eye, and a sensor in a predetermined relationship with the emitter for detecting the emitted light reflected off of the eye, the sensor producing an output signal indicating when the eye is open or closed. Depending upon the relative position of the emitter and sensor with respect to the moving eyelid, the emitter light may be reflected off of the eyelid back to the sensor, or diffused by the surface of the eyeball.
A transmitter is coupled to the sensor for wireless transmission of the output signal, and a processor is provided for comparing the output signal to a predetermined threshold to detect when the eyelid is closed for a minimum predetermined duration. A warning indicator may be coupled to the processor, the warning indicator being activated when the processor detects that the eyelid is closed for the minimum predetermined duration. For example, the warning indicator may be an audible buzzer, a visible warning light, a vibrating device, an electrical shock device, a gustatory smell device, or other device that may act as a stimulus to any sensory modality.
Similar to the previous embodiment, a receiver may be provided at the remote location coupled to the processor for receiving the wireless transmission from the transmitter. Memory circuitry may be provided for storing the output signal and/or a processor may be provided for converting the output signal into an understandable message. A communication device may be coupled to the processor for communicating the understandable message.
In another preferred embodiment, a self-contained device for detecting movement of a person's eyelid is provided that includes a frame adapted to be worn on the person's head, an emitter on the frame for directing light towards an eye of the person when the frame is worn, and a sensor on the frame for detecting light from the emitter. The sensor produces an output signal indicating when the eye is open or closed, and a transmitter on the frame is coupled to the sensor for wireless transmission of the output signal to a remote location. The frame may also include a processor for comparing the output signal to a predetermined threshold to detect drowsiness-induced eyelid movement. Similar to the previous embodiments, the emitter and sensor are preferably a solid state biosensor device for emitting and detecting infrared light, or alternatively an array of emitters and/or sensors in a predetermined configuration on the frame, e.g., in a vertical, horizontal, diagonal, or other linear or other geometric array of more than one emitter and/or sensor oriented towards one or both eyes. In particular, an array of emitters and/or sensors allows measurement of eyelid velocity, acceleration and deceleration, and calculation of “PERCLOS.”
The emitter and/or sensors may be affixed to any number of points on the frame, e.g., around the lens and preferably in the nose bridge, or alternatively anywhere along the frame, including near or on the nasal portion of the frame, the attachment of a temple piece of the frame, and/or surface mounted on the lens of an eyeglass. Alternatively, the emitter and/or sensor may be embedded in the lens of an eyeglass, or otherwise such that they operate through the lens. Thus, the emitter(s) and/or sensor(s) are fixed on an eye-frame such that they move with the wearer's head movements, and continuously focus on the user's eyes, whether the user is in a vehicle, outdoors or in any other environment.
Thus, a system in accordance with the present invention may detect eyelid movement of the user, distinguish normal blinks from other voluntary or involuntary eyelid movement, and produce a stream of data. The stream of data may be converted into an understandable message, such as a binary code, a command for controlling a piece of equipment, or an indicator of the user's physical, mental or emotional state. Thus, the system may provide a convenient and/or effective method for voluntary or involuntary communication based simply upon movement of the user's eye.
In accordance with another aspect of the present invention, a system is provided for monitoring movement of a person's eye. The system includes a device configured to be worn on a person's head and an array of emitters on the device for directing light towards an eye of the person when the device is worn. The array of emitters is configured for projecting a reference frame towards the eye. A camera is oriented towards the eye for monitoring movement of the eye relative to the reference frame. The camera may be provided on the device or may be provided remote from the device, but in relatively close proximity to the user.
Preferably, the array of emitters includes a plurality of emitters disposed in a substantially vertical arrangement on the device, and a plurality of emitters disposed in a substantially horizontal arrangement on the device. Thus, the array of emitters may project a focused set of crossed bands towards the eye for dividing a region including the eye into four quadrants.
In addition, the system preferably includes one or more scanning or nonscanning sensors on the device for detecting light from the array of emitters. The one or more sensors produce an output signal indicating when the eye is open or closed, similar to the embodiments described above. More preferably, the sensors include an array of focused sensors in a predetermined relationship with the array of focused emitters for detecting light from the array of emitters that is reflected off of respective portions of the eye or its eyelid. The emitters, because of their fixed position, produce a fixed reflection off of the surface of the eye and eyelid, appearing as a “glint,” i.e., a spot or band of light. Each sensor produces an output signal indicating when the respective portion of the eye is covered or not covered by the eyelid.
The system may also include a processor for correlating the output signal from the one or more sensors with a video signal from the camera for determining the person's level of alertness. The system may also include a warning indicator on the device, the warning indicator being activated when the processor determines a predetermined level of drowsiness has occurred.
Light from the array of emitters may be emitted towards the eye of a user wearing the device to project a reference frame onto the eye. The camera is capable of imaging light produced by the emitters, e.g., in the infrared light range, thereby detecting the projected light as a spot of light, band of light or other “glint.” Movement of the eye relative to the reference frame may be monitored with the camera. A graphical output of the movement monitored by the camera relative to the reference frame may be monitored. For example, infrared light from the emitters may be reflected off of the retina as a “red reflex” under white light, as a “white pupil” under infrared light, or as a dark pupil under subtraction, using methods known to those skilled in the art. The processor, using these methods, may detect movement of the eye's pupil may be measured relative to the reference frame. This movement may be graphically displayed, showing the movement of the eye's pupil relative to the reference frame.
In addition, the output signal from the one or more sensors may be correlated with video signals produced by the camera monitoring movement of the eye relative to the reference frame, thereby determining the person's level of drowsiness.
Other objects and features of the present invention will become apparent from consideration of the following description taken in conjunction with the accompanying drawings.
Turning now to the drawings, FIG. 1 shows a patient 10 in a bed 12 wearing a detection device 30 for detecting eyelid movement of the patient 10 to provide voluntary-purposeful and/or involuntary-nonpurposeful communication. The detection device 30 is coupled to a processing box 130 which converts the detected eyelid movement into a stream of data, an understandable message and/or into information, which may be communicated, for example, using a video display 50, to a medical care provider 40. The detection device 30 and processing box 130 together provide a system for communication 14 in accordance with one aspect of the present invention.
Turning to FIGS. 2 , 6A and 7A, a preferred embodiment of a system for communication 14 is shown that includes an aimable and focusable detection device 30 that is attachable to a conventional pair of eyeglasses 20. The eyeglasses 20 include a pair of lenses 21 attached to a frame 22, which includes bridgework 24 extending between the lenses 21, and side members or temple pieces 25 carrying ear pieces 26, all of which are conventional. Alternatively, because the lenses 21 are not necessary to the present invention, the frame 22 may also be provided without the lenses 21.
The detection device 30 includes a clamp 27 for attaching to one of the side members 25 and an adjustable arm 31 onto which is mounted an emitter 32 and a sensor 33. Preferably, the emitter 32 and sensor 33 are mounted in a predetermined relationship such that the emitter 32 may emit a signal towards an eye 300 of a person wearing the eyeglasses 20 and the sensor 33 may detect the signal reflected from the surface of the eye 300 and eyelid 302. As shown in FIGS. 6A and 7A, the emitter 32 and sensor 33 may be mounted adjacent one another.
Alternatively, as shown in FIGS. 6B and 7B , the emitter 32′ and sensor 33′ may be mounted on the frame separately away from one another, preferably such that the emitter 32′ and sensor 33′ are disposed substantially laterally with respect to each other. In a further alternative, shown in FIGS. 6C and 7C , the emitter 32″ and sensor 33″ may be mounted across the eye 300 in axial alignment with another. As the eyelid 302 closes, it may break the beam 340 being detected by the sensor 33″.
In a preferred form, the emitter 32 and sensor 33 produce and detect continuous or pulsed light, respectively, preferably within the infrared range to minimize distraction or interference with the wearer's normal vision. Preferably, the emitter 32 emits light in pulses at a predetermined frequency and the sensor 33 is configured to detect light pulses at the predetermined frequency. This pulsed operation may reduce energy consumption by the emitter 32 and/or may minimize interference with other light sources. Alternatively, other predetermined frequency ranges of light beyond or within the visible spectrum, such as ultraviolet light, or other forms of energy, such as radio waves, sonic waves and the like, may be used.
The processing box 130 is coupled to the detection device 30 by a cable 34 including one or more wires therein (not shown). As shown in FIG. 9 , the processing box 130 preferably includes a central processing unit (CPU) 140 and/or other circuitry, such as the exemplary circuitry shown in FIGS. 3-5 , for receiving and/or processing an output signal 142, such as a light intensity signal, from the sensor 33. The processing box 130 may also include control circuitry 141 for controlling the emitter 32 and/or the sensor 33, or the CPU 140 may include internal control circuitry.
For example, in a preferred form, the control circuitry 141 controls the emitter 32 to produce a flickering infrared signal pulsed at a predetermined frequency, as high as thousands of pulses per second to as little as about 4-5 pulses per second, and preferably at least about 5-20 pulses per second, thereby facilitating detection of nonpurposeful or purposeful eye-blinks as short as about 200 milliseconds per blink. The sensor 33 may be controlled to detect light pulses only at the predetermined frequency specific to the flicker frequency of the emitter 32. Thus, by synchronizing the emitter 32 and the sensor 33 to the predetermined frequency, the system 10 may be used under a variety of ambient conditions without the output signal 142 being substantially affected by, for example, bright sun light, total darkness, ambient infrared light backgrounds, or other emitters operating at different flicker frequencies. The flicker frequency may be adjusted to maximize the efficient measurement of the number of eye blinks per unit time (e.g. about ten to about twenty eye blinks per minute), the duration of each eye blink (e.g. about 200 milliseconds to about 300 milliseconds), and/or PERCLOS (i.e., the percentage of time that the eyelid is completely or partially closed), or to maximize efficiency of the system, while keeping power consumption to a minimum.
The control circuitry 141 and/or processing box 130 may include manual controls (not shown) for adjusting the frequency, focus, or intensity of the light emitted by the emitter 32, to turn the emitter 32 off and on, to adjust the threshold sensitivity of the sensor 33, and/or to allow for self-focusing with maximal infrared reflection off of a closed eyelid, as will be appreciated by those skilled in the art.
In addition, the processing box 130 also preferably includes a power source 160 for providing power to the emitter 32, the sensor 33, the CPU 144, and/or other components in the processing box 130. The processor box 130 may be powered by a conventional DC battery, e.g., a nine volt battery or a lithium battery. Alternatively, an adapter (not shown) may be connected to the processor box 130, such as a conventional AC adapter or a twelve volt automobile lighter adapter.
Preferably, the CPU 140 includes timer circuitry 146 for comparing the length of individual elements of the output signal 142 to a predetermined threshold to distinguish between normal blinks and other eyelid movement. The timer circuitry 146 may be separate discrete components or may be provided internally within the CPU 140, as will be appreciated by those skilled in the art. The CPU 140 converts the output signal 142 into a stream of data 144 which may be used to communicate to other persons or equipment. For example, the stream of data 144 produced by the CPU 140 may be a binary signal, such as Morse code or ASCI code. Alternatively, the CPU 140 may be capable of producing a synthesized voice signal, a control signal for a piece of equipment, or even a pictorial representation.
To facilitate communication, the processing box 130 may include a variety of output devices for using the stream of data 144. For example, an internal speaker 150 may be provided, which may produce an alarm sound or a synthesized voice. An output port 148 may be provided to which a variety of equipment, such as the video display 50 shown in FIG. 1 , may be directly coupled by hard-wire connections.
The processing box 130 may also include a transmitter 152 coupled to the CPU 144 for wireless communication of the stream of data 144 to a remote location. For example, as shown in FIG. 9 , the system for communication 14 may also include a receiving and processing unit 154, such as a computer or other control or display system. The transmitter 152 is preferably a radio frequency transmitter capable of producing a short range signal, for example, reaching as far as about one hundred feet or more, and preferably about forty five feet to fifty feet, even through walls or obstacles, although alternatively an infrared transmitter may also be effective.
The transmitter 152 may also be coupled to an amplifier (not shown) to allow the stream of data to be transmitted thousands of feet or more. For example, the amplifier and transmitter 152 may communicate via telephone communication lines, satellites and the like, to transmit the stream of data to a remote location miles away from the system. The system may include, or may be coupled to a global positioning system (GPS) for monitoring the location, movement, and state of wakefulness and safety of an individual wearing the detection device 30.
The receiving and processing unit 154 includes a receiver 156, preferably a radio frequency receiver, for receiving a signal 153, including the stream of data, transmitted by the transmitter 152. A processor 158 is coupled to the receiver 156 for translating, storing and/or using the information in the stream of data, the processor 158 being coupled to memory circuitry 160, a communication device 162, and/or a control system 164. For example, the receiving and processing unit 154 may include the memory circuitry 160 therein into which the processor 158 may simply store the stream of data for subsequent retrieval and analysis.
The processor 158 may interpret the stream of data, for example, by converting a binary code in the stream of data into an understandable message, i.e., a series of letters, words and/or commands, and/or may use augmentative communication devices or software (such as KE:NX or Words Plus) to facilitate communication. The resulting message may be displayed on the communication device 162, which may include a video display for displaying text, pictures and/or symbols, a synthesized voice module for providing electronic speech, and the like.
Alternatively, the stream of data may be displayed graphically on a computer of video screen or other electronic display device as a “real time” message signal or numerically (e.g., displaying blink rate, blink duration, PERCLOS, etc.), or displayed graphically similar to an EKG or EEG tracing. In addition, as shown in FIG. 12C , the stream of data may be displayed along with other physiological data (e.g. heart rate, respiratory rate, other sleep polysomnographic (PSG) or electroencephalographic (EEG) variables). Alternatively, the stream of data may be integrated with controllers which monitor automobile or mechanical functions (e.g. vehicle speed, acceleration, braking functions, torque, sway or tilt, engine or motor speed, etc.) to make intelligent decisions regarding slowing down or speeding up the vehicle depending upon road and/or vehicle conditions, as well as the state of consciousness, wakefulness or attentiveness of the driver or machine operator.
In addition, the message may be interpreted by the processor 158 for directing the control system 164 to control one or more pieces of machinery or equipment. For example, the stream of data may include a command to direct the control system 164 to control relay switches or other devices to turn off and on an electrical device, such as an appliance, electrical wheelchair, engine, light, alarm, telephone, television, computer, a tactile vibrating seat, and the like, or to operate an eye-activated computer mouse or other controller.
Alternatively, the processor 158 may use the stream of data to control PC, IBM, Macintosh and other computers and compatible computer software and/or hardware, e.g., to interact with a computer similar to a mouse, a “return” key or a “joystick.” For example, the stream of data may include commands to activate a series of menus from which sub-menus or individual items may be selected, as are used in commercially available special communications software, such as WORDS-PLUS or Ke:NX. The processor 158 may then control, scroll or select items from computer software programs, operate a printer or other peripheral device (e.g., selecting a font, paragraph, tab or other symbol operator, selecting commands, such as “edit,” “find,” “format,” “insert,” “help,” or controlling CD-ROM or disc drive operations, and/or other Windows and non-Windows functions).
Alternatively, the receiver 156 may be coupled directly to a variety of devices (not shown), such as radio or television controls, lamps, fans, heaters, motors, remote control vehicles, vehicle monitoring or controlling devices, computers, printers, telephones, lifeline units, electronic toys, or augmentative communication systems, to provide a direct interface between the user and the devices.
During use, the detection device 30 is placed on a user's head, i.e., by putting the eyeglasses 20 on as shown in FIG. 1 . The adjustable arm 31 and/or the clamp 27 may be adjusted to optimally orient the emitter 32 and sensor 33 towards the user's eye 300 (shown in FIGS. 6A-6C and 7A-7C). The emitter 32 is activated and a beam of light 340 is directed from the emitter 32 towards the eye 300. The intensity and/or frequency of the emitter 32 and/or the threshold sensitivity of the sensor 33 or other focus may then be adjusted (e.g. manually or automatically using self-adjusting features).
Because of the difference in the reflective characteristics of the surface of the eye 300 itself and the eyelid 302, the intensity of the light reflected off of the eye 300 depends upon whether the eye 300 is open or closed. For example, FIGS. 6A and 6B illustrate an open eye condition, in which a ray of light 340 produced by the emitter 32 strikes the surface of the eye 300 itself and consequently is scattered, as shown by the rays 350. Thus, the resulting light intensity detected by the sensor 33 is relatively low, i.e., the sensor 33 may not receive any substantial return signal.
In FIGS. 7A and 7B , the eye 300 is shown with the eyelid 302 closed as may occur during normal blinks, moments of drowsiness, intentional blinks, or other eyelid movement. Because the light 340 strikes the eyelid 302, it is substantially reflected back to the sensor 33, as shown by the ray 360, resulting in a relatively high light intensity being detected by the sensor 33. Alternatively, as shown in 7C, the beam of light 340 may be broken or cut by the eyelid 302 when the eye 300 is closed.
The sensor 33 consequently produces a light intensity signal that indicates when the eye 300 is open or closed, i.e., corresponding to the time during which reflected light is not detected or detected, respectively, by the sensor 33. Generally, the intensity of the infrared light reflected from the surface of the eyelid is not substantially affected by skin pigmentation. If it is desired to adjust the intensity of light reflected from the eyelid, foil, glitter, reflective moisturizer creams and the like may be applied to increase reflectivity, or black eye liner, absorptive or deflective creams and the like may be applied to reduce reflectivity.
Returning to FIG. 9 , the light intensity detected by the sensor 33 results in an output signal 142 including a series of time-dependent light intensity signals (as shown, for example, in FIG. 12B ). The output signal 142 is received by the CPU 140 coupled to the sensor 33, which compares the length of time of each light intensity signal 142, for example, corresponding to a closed eye condition, with a predetermined threshold. The timer circuitry 146 may provide a threshold time to the CPU 140 for distinguishing normal blinks from intentional and/or other unintentional eyelid movement, which the CPU 140 may then filter out of the output signal 142. The CPU 140 then produces a stream of data 144 which may be used for voluntary and/or involuntary communication.
In one useful application, the detection device 30 may be used to detect impending drowsiness or “micro-sleeps” (i.e., sleep intrusions into wakefulness lasting a few seconds) of a user, with the processing box 130 triggering a warning to alert the user, others in his or her presence, or monitoring equipment of the onset of drowsiness. The threshold of the timer circuitry 146 may be adjusted such that the CPU 140 detects relatively long periods of eye closure, as may occur when a person is falling asleep.
For example, because normal blinks are relatively short, the threshold may be set at a time ranging from close to zero seconds up to several seconds, preferably from about 200 milliseconds to about 300 milliseconds, and most preferably about 250 milliseconds, to distinguish normal blinks from drowsiness-induced eyelid movement. When the CPU 140 detects a drowsiness condition, i.e., detects a high light intensity signal exceeding the predetermined threshold time, it may activate a warning device. The warning device may be included within the processing box 130, such as the speaker 150, or alternatively on the frame, for example, by mounting a warning light (not shown) or an alarm speaker (not shown in FIG. 9 , see FIG. 10C ) on the frame.
Alternatively, the detection device 30 may be used to unobtrusively record or monitor drowsiness-induced eyelid movement, with the CPU 140 producing a stream of data 144 which the transmitter 152 may transmit to the receiving and processing unit 154 (FIG. 9 ). For example, the device 30 may be used in conjunction with a vehicle safety system to monitor a driver's level of awareness or attentiveness. The stream of data 144 may be transmitted to a receiving and processing unit 154 mounted in a vehicle, which may store data on the driver's drowsiness and/or may use the data to make decisions and control the vehicle, e.g., adjust the vehicle's speed or even turn the vehicle's engine off. Thus, the detection device 30 may be used to monitor truck drivers, taxi drivers, ship or airline pilots, train conductors or engineers, radar or airport control tower operators, operators of heavy equipment or factory machinery, scuba divers, students, astronauts, entertainment participants or observers, and the like. The signals may be stored and analyzed in real time for trend changes measured over time to predict drowsiness effects of individuals using the device.
The detection device 30 and system 14 may also be used in a medical diagnostic, therapeutic, research or professional setting to monitor the wakefulness, sleep patterns and/or the effects of drugs, which may affect blink rate, blink velocity, blink duration, or PERCLOS of a patient or vehicle operator. Similar to the method just described, the CPU 140 produces a stream of data 144, which the transmitter may send to a remote receiving and processing unit 154, which may store the stream of data 144 in the memory circuitry 160 for later retrieval and analysis by researchers, medical professionals, or safety personnel (e.g., similar to the way in which flight recorder data may be stored in an aircraft's “black box” recorder). The receiving and processing unit 154 may also display the stream of data 144, for example at a nurse's station, as an additional parameter to continually monitor a patient's physical, mental, or emotional condition. The unit 154 may store and/or produce a signal, e.g., by a series of algorithms, that must be responded to within a predetermined time (e.g., performance vigilance monitoring) to prevent false positives and negatives.
A number of medical conditions may be monitored by the detection device 30 and system 14, such as petit mal epilepsy, in which the eyes flutter at a rate of about three cycles per second, grand mal or psychometer seizures, where the eyes may stare or close repetitively in a jerky manner, myoclonic seizures, in which the lids may open and close in a jerky manner, or tics, or other eye movements, such as encountered by people with Tourette's syndrome. The system may be used to monitor g-lock of pilots caused by g-force effects, hypoxemia of passengers or crew in aircraft due to losses in cabin pressure, nitrogen narcosis or “the bends” in divers, or the effects of gases, chemicals, or biological agents on military personnel or other individuals.
The system may also be used to monitor psychological situations, for example, to detect when a person lies (e.g., by closing their eyes when lying), during hypnosis, to monitor attentiveness, the effects of medications, e.g., L-dopa and other anti-Parkinsonian medications or anti-convulsants, drugs, alcohol, toxins, or the effects of hypoxia or ventilation, and the like. Neurological conditions may also be monitored where the innervation or mechanical function of the eyelid may be affected, such as in Parkinson's disease, muscle diseases, e.g., myotonia, myotonic muscular dystrophy, blepharospasm, photophobia or light sensitivity, encephalopathy, seizures, Bell's palsy, or where the condition may produce eyelid drooping or ptosis, such as third cranial nerve palsy or paresis, brainstem lesions or stroke, tumors, infection, metabolic diseases, trauma, degenerative conditions, e.g., multiple sclerosis, amyotrophic lateral sclerosis, polyneuropathy, myesthenia gravis, botulism, tetanus, tetany, tardive dyskinesia, brainstem encephalitis, and other primary eyelid conditions, such as exophthalmos, thyrotoxicosis or other thyroid conditions.
Similarly, the detector device 30 may be used in biofeedback applications, for example, in biofeedback, hypnosis or psychological therapies of certain conditions (e.g. tic disorders). The detector device 30 may produce a stimulus, e.g. activating a light or speaker, and monitor the user's eyelid movement in anticipation of receiving a response, e.g., a specific sequence of blinks, acknowledging the stimulus within a predetermined time. If the user fails to respond, the processor may store the response, e.g. including response time, and/or may automatically transmit a signal, such as an alarm signal.
In addition, the detection device 30 may be used to monitor individuals in non-medical settings, such as during normal activity in a user's home or elsewhere. For example, individuals with involuntary medical conditions, such as epilepsy or narcolepsy, may be monitored, or other individuals, such as, infants and children, prison inmates, demented patients (e.g., with Alzheimer's disease), law enforcement personnel, military personnel, bank tellers, cashiers, casino workers, students, swing or graveyard shift workers, and the like, may be monitored. Similar application may be applied in a sleep laboratory for monitoring sleep patients to measure parameters, such as onset of sleep, sleep latency, time of eyelid closing or opening, time of awakening during the night, etc., or to animal research where eye blinking may be a factor to be studied. Similarly, the performance and vigilance abilities of the user may be tested and analyzed as a direct function of, or in relationship to, PERCLOS.
When the CPU 140 detects the presence of particular eyelid movement, such as an extensive period of eye closure which may occur, for example, during an epileptic seizure, a syncopal episode, a narcoleptic episode, or when dozing off while driving or working, the CPU 140 may produce an output signal which activates an alarm. Alternatively, the transmitter 152 may send an output signal to shut off equipment being used, to notify medical personnel, such as by automatically activating a telephone to dial emergency services, to signal remote sites, such as police stations, ambulances, vehicle control centers, guardians, and the like.
The system for communication 14 may also find useful application for voluntary communication. A user wearing the detection device 30 may intentionally blink in a predetermined pattern, for example, in Morse code or other blinked code, to communicate an understandable message to people or equipment (e.g., to announce an emergency). The CPU 140 may convert a light intensity signal 142 received from the sensor 33 and corresponding to the blinked code into a stream of data 144, or possibly directly into an understandable message including letters, words and/or commands.
The stream of data 144 may then be displayed on a video 60 display 50 (see FIG. 1 ) coupled to the output port 148, or emitted as synthesized speech on the internal speaker 150. The stream of data 144 may be transmitted by the transmitter 152 via the signal 153 to the receiving and processing unit 154 for displaying messages, or for controlling equipment, such as household devices, connected to the control system 164. In addition to residential settings, the system 14 may be used by individuals in hospitalized or nursing care, for example by intubated, ventilated, restrained, paralyzed or weakened patients, to communicate to attending medical staff and/or to consciously signal a nurse's station. These include all patients who have no physical ability to communicate verbally, but who retain ability to communicate using eye blinking of one or both eyes (e.g., patients with amyotrophic lateral sclerosis, transverse myelitis, locked-in syndrome, cerebravascular strokes, terminal muscular dystrophy and those intubated on ventilation).
The device may be used in any environment or domain, e.g., through water or other substantially transparent fluids. Further, the device 30 may also be used as an emergency notification and/or discrete security tool. A person who may be capable of normal speech may wear the device 30 in the event of circumstances under which normal communication, i.e., speech, is not a viable option. For example, a bank or retail employee who is being robbed or is otherwise present during the commission of a crime may be able to discretely blink out a preprogrammed warning to notify security or to call law enforcement. Alternatively, a person with certain medical conditions may wear the device in the event that they are physically incapacitated, i.e., are unable to move to call for emergency medical care, but are still able to voluntarily move their eyes. In such cases, a pre-recorded message or identifying data (e.g. name of the user, their location, the nature of the emergency, etc.) may be transmitted to a remote location by a specific set of eyeblink codes or preprogrammed message. In this manner, the detection device 30 may be used to monitor patients in an ICU setting, patients on ventilators, prisoners, elderly or disabled persons, heavy equipment operators, truck drivers, motorists, ship and aircraft pilots, train engineers, radar or airport control tower operators, or as a nonverbal or subliminal tool for communication by military guards, police bank tellers, cashiers, taxi-drivers, and the like. The detection device 30 may also be used as a recreational device, for example, as a children's toy similar to a walkie-talkie or to operate a remote control toy vehicle.
In addition, it may be desirable to have the CPU 140 perform an additional threshold comparison to ensure continued use of the detection device 30. For example, additional timer circuitry may be coupled to the CPU 140 such that the CPU 140 may compare the light intensity signals received from the sensor 33 to a second predetermined threshold provided by the timer circuitry. Preferably, the second predetermined threshold corresponds to a time period during which a person would normally blink. If the CPU 140 fails to detect a normal blink within this time period or if the user fails to respond to a predetermined stimulus (e.g. a blinking light or sound), the CPU 140 may produce a signal, activating the speaker 150 or transmitting a warning using the transmitter 152.
This may be useful, if, for example, the detection device 30 is removed by a perpetrator during commission of a crime, falls off because of the onset of a medical episode, as well as to prevent “false alarms,” or to measure the “state of attentiveness” of the user. Alternatively, performance vigilance tasks may be required of the user to determine whether the signal transmitted is a purposeful or “false alarm” signal, and also for measuring attention or drowsiness levels for purposes of biofeedback, and also to measure compliance of the user wearing the device.
Alternatively, the polarity of the output signal 142 may be reversed such that a stream of data is produced only when the eye is opened, for example, when monitoring patients in a sleep lab to measure onset of sleep, sleep latency, time of eyelid closure, etc., or to monitor sleeping prison inmates. For such uses, the CPU 140 may activate an alarm only when an open eye condition is detected, as will be appreciated by those skilled in the art.
Turning to FIG. 8 , another preferred embodiment of the detection device 30 in accordance with the present invention is shown. In this embodiment, the emitter and sensor are a single solid state light emission and detecting biosensor device 132 which are mounted directly onto the eyeglasses 20. The biosensor device 132, which preferably produces and detects infrared light, may be as small as 2 mm ×4 mm and weigh only a few grams, thereby enhancing the convenience, comfort and/or discretion of the detection device 30. Because of the small size, the biosensor device 133 may be mounted directly in the lens 21, as shown in FIG. 8 , on an outside or inside surface of the lens 21, in the bridgework 24 or at another location on the frame 22 that may facilitate detection of eye movement. The biosensor device 132 may measure less than about five millimeters by five millimeters surface area, and may weigh as little as about one ounce, thereby providing a emitter/sensor combination that may be unobtrusive to vision, portable, and may be conveniently incorporated into a light weight eye frame. Because the entire system may be self-contained on the frame, it moves with the user no matter which direction he or she looks and may operate in a variety of environments or domains, day or night, underwater, etc.
Hamamatsu manufactures a variety of infrared emitter and detector devices which may be used for the biosensor device 132, such as Model Nos. L1909, L1915-01, L2791-02, L2792-02, L2959, and 5482-11, or alternatively, a Radio Shack infrared emitter, Model No. 274-142, may be used. Multiple element arrays, e.g., linear optical scanning sensor arrays, appropriate for use with the present invention may be available from Texas Advanced Optoelectronic Solutions, Inc. (TAOS) of Plano, Tex,, such as Model Nos. TSL 201 (64 pixels×1 pixel), TSL 202 (128×1), TSL 208 (512×1), TSL 2301 (102×1). These sensors may be used in combination with lens arrays to facilitate focusing of the detected light, such as the Selfoc lens array for line scanning applications made by NSG America, Inc. of Irvine, Calif.
In addition, multiple biosensor devices 132 may be provided on the eyeglasses 20, for example, a pair of biosensor devices 132 may be provided, as shown in FIG. 8 , for detecting eyelid movement of each eye of the user (not shown). A cable 134 extends from each biosensor device 132 to a processing box 130, similar to the processing box 130 described above. The CPU 140 of the processing box 130 (not shown in FIG. 8 ) may receive and compare the output signal from each biosensor device 132 to further augment distinguishing normal blinks from other eyelid movement.
The pair of biosensor devices 132 may allow use of more sophisticated codes by the user, e.g., blinking each eye individually or together, for communicating more effectively or conveniently, as will be appreciated by those skilled in the art. In one form, a blink of one eye could correspond to a “dot,” and the other eye to a “dash” to facilitate use of Morse code. The output signals from each eye could then be interpreted by the CPU 140 and converted into an understandable message.
In another form, a right eye blink (or series of blinks) may cause an electric wheelchair to move to the right, a left eye blink (or series of blinks) may move to the left, two simultaneous right and left eye blinks may cause the wheelchair to move forward, and/or four simultaneous right and left eye blinks may cause the wheelchair to move backward. Similar combinations or sequences of eye blinks may be used to control the on/off function, or volume or channel control of a television, AM/FM radio, VCR, tape recorder or other electronic or electromechanical device, any augmentative communications or controlling device, or any device operable by simple “on/off” switches (e.g., wireless television remote controls single switch television control units, universal remote controllers, single switch multi-appliance units with AC plug/wall outlet or wall switch modules, computer input adapters, lighted signaling buzzer or vibrating signal boxes, switch modules of all types, video game entertainment controller switch modules and switch-controlled electronic toys).
In additional alternatives, one or more lenses or filters may be provided for controlling the light emitted and/or detected by the biosensor device, an individual emitter and/or detector. For example, the angle of the light emitted may be changed with a prism or other lens, or the light may be columnated or focused through a slit to create a predetermined shaped beam of light directed at the eye or to receive the reflected light by the sensor. An array of lenses may be provided that are adjustable to control the shape, e.g. the width, etc., of the beam of light emitted or to adjust the sensitivity of the sensor. The lenses may be encased along with the emitter in plastic and the like, or provided as a separate attachment, as will be appreciated by those skilled in the art.
Turning now to FIG. 10A , another preferred embodiment of a system for communication 414 is shown, that includes a frame 422 including a biosensor device 432 with associated processor and transmitter circuitry 430 provided directly on the frame 422, for example, to enhance the convenience and discretion of the system for communication 414. The frame 422 may include a bridge piece 424 onto which the biosensor device 432 may be slidably and/or adjustably mounted, and a pair of ear supports 423, 425.
One of the supports 423 may have a larger size compared to the other support 425, for example, to receive the processor and transmitter circuitry 430 embedded or otherwise mounted thereon. A processor 440, similar to the CPU 140 in the processing box 130 previously described, may be provided on the frame 422, and a power source, such as a lithium battery 460, may be inserted or affixed to the support 423. A radio frequency or other transmitter 452 is provided on the support 423, including an antenna 453, which may be embedded or otherwise fastened along the ear support 423, in the temple piece or elsewhere in the frame 422.
The system 414 may also include manual controls (not shown) on the ear support 423 or elsewhere on the frame 422, for example to turn the power off and on, or to adjust the intensity and/or threshold of the biosensor device 432. Thus, a system for communication 414 may be provided that is substantially self-contained on the frame 422, which may or may not include lenses (not shown) similar to eyeglasses. External cables or wires may be eliminated, thereby providing a more convenient and comfortable system for communication.
In another alternative, shown in FIGS. 10B , 10C, and 10D, a linear array 530 of emitters 532 and sensors 533 may be provided, preferably in a vertical arrangement mounted on a nose bridge 524 of an eye frame 522. A CPU 540, battery 460, transmitter antenna 543, and warning indicator 550 may also be provided on the frame 522, preferably in the temple piece 525, similar to the previously described embodiment. An LED 542 or similar stimulus device may also be provided at a predetermined location on the eye frame 522 to allow routine biofeedback responses from the user. In addition, a receiver 544 may be provided for receiving the stream of data created by the CPU 540 and transmitted by the transmitter 543.
As shown particularly in FIG. 10C , each of the sensors 533 and the emitter 532 are coupled to the CPU 540 or other control circuitry for controlling the emitter 532 and for processing the light intensity signals produced by the sensors 532. Thus, the CPU 540 may cycle through the sensors 533 in the array 530 and sequentially process the signal from each of the sensors 533, similar to the processors previously described. More preferably, as shown in FIG. 10D , the emitter 532 includes a lens 534 to focus a beam of light (indicated by individual rays 360a, 360b) onto the eye 300, preferably towards the pupil 301. The sensors 533 are embedded within the nose bridge 524 and a slit 535 is provided for each, the slits 535 having a predetermined size to control the reflected light detected by each sensor 533. Thus, each sensor 535 may detect movement of the eyelid 302 past a particular portion of the eye 300, e.g., to measure PERCLOS, as shown in FIG. 12A . The sensors or emitters may have lenses or columnating devices to focus emitted or reflected light.
The linear array 530 may facilitate measurement of additional parameters related to eyelid movement in addition to mere eye closure. For example, to measure the velocity of the eyelid opening or closing, i.e., the rate of eye closure, the CPU 540 may compare the time delay between the activation of successive sensors 533. In addition, the output signals from the sensors 553 may be processed to measure the percentage of pupil coverage of the eyelid 302, for example, due to partial eye closure, as a function of time, e.g., to monitor when the eye is partially, but not completely, closed, and/or to monitor the percentage of time that the eye is closed (PERCLOS), as shown in FIGS. 12A-12C , e.g., compared to the user's baseline of maximal eye opening.
Turning to FIG. 12D , in a further alternative, a two-dimensional array of sensors, such as a 5×5 array 633 or a 9×11 array, 733 may be provided. Other arrays including any number of elements in the array may be provided, and the invention should not be limited to the exemplary embodiments described herein. The sensors 633, 733 may then be used to measure surface area reflectivity of light from the emitter 632, i.e., the processor (not shown) may process the signals from each sensor in the array 633, 733 to create a stream of data indicating the percentage of surface area of the eye 300 covered by the eyelid 302.
The sensors in the array 633, 733 may be sufficiently sensitive or have sufficient resolution such that they may detect “red reflex” or the equivalent infrared “bright pupil” reflection due to the reflection of light off of the retina through the pupil 301. Thus, the sensors may produce a light intensity signal that includes a substantially zero value, indicating no red reflex or bright pupil, a low output, indicating red reflex or white pupil reflex, and a high output, indicating reflection off of a closed eyelid 302. The red reflex may appear as a bright white light pupil (resulting from infrared light from the emitter(s) reflecting off of the retina when the eyelid is open, or as a dark or “black pupil” if the processor uses subtraction algorithms, as is known in the art. The processor may thereby process the light intensity signals to detect when the pupil 301 is covered by the eyelid 302, i.e., at which point the user cannot see, even though their eye 300 may not be entirely covered by the eyelid 302, generally at a PERCLOS value of about 50-75 percent in primary gaze. Alternatively, as the eyelid, eye, and pupil descend, the sensor(s) may detect a red reflex or bright pupil even through the PERCLOS measurement may be as great as 75 -80 percent or more, e.g., where the eye may still see through a narrow slit-like palpebral fissure opening in downward gaze.
In another alternative, the processor and/or transmitter circuitry (such as the CPU 140 in the processor box 130 of FIG. 2 , or the CPU's 440, 540 of FIGS. 10A and 10B ) may include identification circuitry (not shown), either as a discrete memory chip or other circuit element, or within the CPU itself. The identification circuitry may be preprogrammed with a fixed identification code, or may be programmable, for example, to include selected identification information, such as the identity of the user, the user's location, an identification code for the individual detection device, and the like.
The CPU may selectively add the identification information to the transmitted stream of data 553, or the identification information may be automatically or periodically included in the stream of data 553, thereby allowing the stream of data 553 to be associated with a particular detection device, individual user and/or a specific location. The identification information may be used by the processor, for example, at a remote location, to distinguish between streams of data received from a number of detection devices, which may then be stored, displayed, etc. as previously described. Thus, the detection device may not require users to consciously communicate certain identification or other standard information when the system is used.
As shown in FIG. 11A , the receiver 544 may allow the user to control one or more devices coupled to the receiver 544 through a single switch multi-appliance control unit 550. The control unit 550 includes its own transmitter adapted to transmit on/off or other control signals that may be received by individual control modules 552a-552fe. The user 10 may blink to create a transmitted stream of data 553 that includes commands to turn off and on, or otherwise control, selected appliances using the control unit 550 and control modules 552a-552fe, such as, a radio 554, a television 556, a light 558a., a light 562 controlled by a wall switch 560, a fan 566 plugged into a wall socket 564, and the like.
Alternatively, as shown in FIG. 11B , the receiver 554 may be coupled to other systems, such as a computer 570 and printer 572, a vehicle integration system 574, a lifeline unit 576, a GPS or other satellite transmitter 578, and the like. The transmitted stream of data 553 may be processed alone or along with additional data, such as other vehicle sensor information 573, to further enhance monitoring a user, such as a long-distance truck driver.
Turning to FIG. 13 , yet another embodiment of a system 810 for monitoring eye movement is shown. Generally, the system 810 includes a frame 812 that may include a bridge piece 814 and a pair of ear supports 816. The frame 812 may include a pair of lenses (not shown), such as prescription, shaded, or protective lenses, although they are not necessary for operation of the invention. Alternatively, the system may be provided on other devices that may be worn on a user's head, such as a pilot's oxygen mask, protective eye gear, a patient's ventilator, a scuba or swimming mask, a helmet, a hat, a head band, a head visor, and the like (not shown). The components of the system may be provided at a variety of locations on the device that generally minimize interference with the user's vision and/or normal use of the device.
An array of emitters 820 are provided on the frame 812, preferably in a vertical array 820a and a horizontal array 820b. In a preferred embodiment, the emitters 820 are infrared emitters configured to emit pulses at a predetermined frequency, similar to the embodiments described above. The emitters 820 are arranged on the frame such that they project a reference frame 850 onto the region of the user's eye 300. In a preferred embodiment, the reference frame includes a pair of crossed bands 850a, 850b dividing the region into four quadrants. The intersection of the crossed bands is preferably disposed at a location corresponding substantially to the eye's pupil during primary gaze, i.e., when the user is looking generally straight forward along axis 310 extending directly ahead of the user's eye 300. Alternatively, other reference frames may be provided, generally including a vertical component and a horizontal component.
An array of sensors 822 are also provided on the frame 812 for detecting light from the emitters 820 that is reflected off of the user's eyelid. The sensors 822 preferably generate output signals having an intensity identifying whether the eyelid is closed or open, similar to the embodiments described above. Preferably, the sensors 822 are disposed adjacent to respective emitters 820 for detecting light reflected off of respective portions of the eyelid. Alternatively, sensors 822 may only be provided in a vertical array, e.g., along the bridge piece 814, for monitoring the amount of eyelid closure, similar to the embodiments described above. In a further alternative, the emitters 820 and sensors 822 may be solid state biosensors (not shown) that provide both the emitting and sensing functions in a single device.
Circuitry may be provided for measuring PERCLOS or other parameters using the signals generated by the array of sensors. For example, FIG. 17 shows an exemplary schematic that may be used for processing signals from a five element array, e.g., to obtain PERCLOS measurements or other alertness parameters.
Returning to FIG. 13 , the system 810 also includes a camera 830 provided on the frame 810. Preferably, the camera 830 is mounted on or adjacent the bridge piece 814 offset from the axis 310 such that the camera 830 is oriented towards the region surrounding one of the user's eyes 300 while minimizing interference with the user's vision. The camera 830 preferably includes a bundle of fiberoptic cables 832 that terminate in a lens 834, as shown in FIG. 14 , on a first end mounted adjacent the bridge piece 814 and a second end 837 that is connected to a detector 838, e.g., a CCD or CMOS sensor, such as those used in endoscopes, that may convert an image into a digital video signal. The camera 830 is configured to detect the frequency of light emitted by the emitters 820, e.g., infrared light. The camera 830 may rely on the light projected by the emitters 820, or the fiberoptic cables 832 may include emitters 836 for projecting light, e.g., infrared light, onto the user's eyes and/or face. In addition, the system 810 may include a second camera 840 oriented away from the user's head, e.g., to monitor the user's surroundings.
One of the ear supports 816 may include a panel 818 for mounting a controller or other processor 842, a transmitter 844, an antenna 845, and a battery 846. Preferably, the processor 840 842 is coupled to the emitters 820, the sensors 822, and/or the camera 830 for controlling their operation. The transmitter 844 may be coupled to the processor 842 for receiving the output signals from the sensors 822 and/or the video signals from the camera 830, e.g., to transmit the signals to a remote location, as described below. Alternatively, the transmitter 844 may be coupled directly to output leads from the sensors 822 and the camera 830. The frame 812 may also include manual controls (not shown), e.g., on the ear support 816, for example, to turn the power off and on, or to adjust the intensity and/or threshold of the emitters 820, the sensors 822, and/or the camera 830.
If desired, the system 810 may also include one or more additional sensors on the frame 812. The sensors may be coupled to the processor 842 and/or to the transmitter 844 so that the signals from the sensors may be monitored, recorded, and/or transmitted to a remote location. For example, one or more position sensors 852a, 852b may be provided, e.g., for determining the spatial orientation of the frame 812, and consequently the user's head. For example, actigraphic sensors may be provided to measure tilt or movement of the head, e.g., to monitor whether the user's head is drooping forward or tilting to the side. Acoustic sensors, e.g., a microphone 854 may be provided for detecting environmental noise or sounds produced by the user.
In addition or alternatively, the frame 812 may include one or more sensors for measuring one or more physical characteristics of the user. For example, EEG electrodes 856 may be provided on the ear support 816, above or below the nasion, and/or other region that may contact the patient's skin to measure brain activity, e.g., waking, drowsy, or other sleep-related brain activity. An EKG electrode (not shown) may be provided that is capable of measuring cardiac activity through a skin contact site. A pulse sensor (not shown) may be used to measure cardiovascular pulsations, or an oximetry sensor 858 may be used to measure oxygen saturation levels. A thermistor or other sensor may measure of respiratory air flow, e.g., through the user's nose. A thermister, thermocouple, or other temperature sensor (not shown) may be provided for measuring the user's skin temperature. A sweat detector (not shown) may be provided for measuring moisture on the user's skin.
In addition, the system 810 may include one or more feedback devices on the frame 812. These devices may provide feedback to the user, e.g., to alert and/or wake the user, when a predetermined condition is detected, e.g., a state of drowsiness or lack of consciousness. The feedback devices may be coupled to the processor 842, which may control their activation. For example, a mechanical vibrator device 860 may be provided at a location that may contact the user, e.g., on the ear support 816, that may provide tactile vibrating stimuli through skin contact. An electrode (not shown) may be provided that may produce relatively low power electrical stimuli. A light emitter, such as one or more LED's may provided at desired locations, e.g., above the bridge piece 814. Alternatively, audio devices 862, such as a buzzer or other alarm, may be provided, similar to the previous embodiments. In a further alternative, aroma-emitters may be provided on the frame 810 812, e.g., on or adjacent to the bridge piece 814.
Alternatively, the feedback devices may be provided separate from the frame, but located in a manner capable of providing a feedback response to the user. For example, audio, visual, tactile (e.g., vibrating seat), or olfactory emitters may be provided in the proximity of the user, such as any of the devices described above. In a further alternative, heat or cold generating devices may be provided that are capable of producing thermal stimuli to the user, e.g., a remotely controlled fan or air conditioning unit.
The system 810 may also include components that are remote from the frame 812, similar to the embodiments described above. For example, the system 810 may include a receiver, a processor, and/or a display (not shown) at a remote location from the frame 812, e.g., in the same room, at a nearby monitoring station, or at a more distant location. The receiver may receive signals transmitted by the transmitter 842, including output signals from the sensors 822 or any of the other sensors provided on the frame 812 and/or the video signals from the camera 830.
A processor may be coupled to the receiver for analyzing signals from the components on the frame 812, e.g., to prepare the signals for graphical display. For example, the processor may prepare the video signals from the camera 830 for display on a monitor, thereby allowing personal monitoring of the user. Simultaneously, other parameters may be displayed, either on a single monitor or on separate displays. For example, FIG. 15a FIGS. 15A-15I shows signals indicating the output of various sensors that may be on the frame 812, which may be displayed along a common time axis or otherwise correlated, e.g., to movement of the user's eye and/or level of drowsiness. The processor may superimpose or otherwise simultaneously display the video signal in conjunction with the other sensed parameters to allow a physician or other individual to monitor and personally correlate these parameters to the user's behavior.
In a further alternative, the processor may automatically process the signals to monitor or study the user's behavior. For example, the processor may use the output signals to monitor various parameters related to eye movement, such as eye blink duration (EBD), eye blink frequency, eye blink velocity, eye blink acceleration, interblink duration (IBD), PERCLOS, PEROP (percentage eyelid is open), and the like.
The video signals from the camera 830 may be processed to monitor various eye parameters, such as pupillary size, location, e.g., within the four quadrant defined by the crossed bands 850, eye tracking movement, eye gaze distance, and the like. For example, because the camera 830 is capable of detecting the light emitted by the emitters 822, the camera 830 may detect a reference frame projected onto the region of the user's eye by the emitters. FIG. 16 shows an exemplary video output from a camera included in a system having twenty emitters disposed in a vertical arrangement. The camera may detect twenty discrete regions of light arranged as a vertical band. The camera may also detect a “glint” point, G, and/or a moving bright pupil, P. Thus, the movement of the pupil may be monitored in relation to the glint point, G, and/or in relation to the vertical band 1-20.
Because the emitters 822 are fixed to the frame 812, the reference frame 850 remains substantially stationary. Thus, the processor may determine the location of the pupil in terms of orthogonal coordinates (e.g., x-y or angle-radius) relative to the reference frame 850. Alternatively, if the reference frame is eliminated, the location of the pupil may be determined relative to any stationary “glint” point on the user's eye. For example, the camera 830 itself may project a point of light onto the eye that may be reflected and detected by the camera. This “glint” point remains substantially stationary since the camera 830 is fixed to the frame 812.
In addition, the video signals from a remote camera that may view the user's face from a distance may be used to monitor various facial measures, such as facial expression, yawning frequency, and the like, in addition to or alternatively, the project instead of the projected light reference frame from the emitters. In addition or alternatively, the parameters from other sensors may be processed and correlated, such as head orientation, tilt, body movement, physiological parameters, and the like. Preferably, the processor may correlate these parameters to generate a composite fatigue index (CFI) that is a function of two or more of these parameters. When a predetermined CFI is detected, the system 810 may activate an alarm or other notice to the user and/or to another party at a remote location. Thus, the system 810 may provide a more effective way to monitor the user's fatigue, drowsiness, alertness, mental state, and the like. In a further alternative, the system 810 may be used to generate predetermined outputs, e.g., to activate or deactivate equipment, such as a vehicle being operated by the user when a predetermined condition, e.g., CFI value, is determined by the system 810.
Alternatively, the processor may be provided on the frame 812, e.g. as part of processor 842, for monitoring the parameters for a predetermined event, such as a predetermined CFI value, to occur. Although only a single lens and set of emitters, sensors, and cameras are shown, it will be appreciated that another set may be provided for the other eye of the user of the system 810. In a further alternative, the eye tracking parameters described above may be monitored by a remote camera, e.g., in a fixed position in front of the user, such as the dashboard of a vehicle and the like. The remote camera may be coupled to the processor, either directly or via its own transmitter, as will be appreciated by those skilled in the art.
Thus, a system in accordance with the present invention may monitor or detect one or more parameters, such as those listed below in Table 1.
TABLE 1 |
Potential Biometric Measures |
EYELID MEASURES | EYEGLAZE MEASURES |
Percentage of time (t) and | Eye Tracking Movements (ETM) |
the amount palpebral | including Directional Nystagmus |
fissure is opened | Eye Gaze Distance (EGD) and |
(PEROP-t, -d, -dt), | Direction |
or closed | Eye Movement Distance |
(PERCLOS-t, -d, -dt), | Eye Movement Velocity (EMV) |
lid droop | Eye Movement Acceleration (EMA) |
Eye Blink Duration (EBD) | and Deceleration (EMD) |
Eye Blink Frequency (EBF) | Eye Movement Frequency (EMF) |
Eye Blink Velocity (EBV) | Phoria/eye Drift Measures (PDM) |
Eye Blink Acceleration | HEAD ORIENTATION MEASURES |
(EBAc) | Head Direction or Orientation |
Decceleration (EBDc) | (HDir) |
Interblink duration (IBD) | HEAD MOVEMENT MEASURES |
Eye blink flurries | Head Nodding Frequency (HNF) |
PUPIL MEASURES | Head Tilt (HT) |
Pupillary Appearance or | OTHER NON- |
Disappearance (with eyelid | VIDEO SENSOR METRICS |
movement) | EEG, EKG, pulse, oxygen |
Pupillary Size Measurement | saturation, respiration rate, |
(PSM) | body temp, skin conductance, |
Presence and quality of | actigraphic movements, head |
Pupillary | lilt sensors |
Dilation or Construction | |
(including Hippus) | |
While the invention is susceptible to various modifications, and alternative forms, specific examples thereof have been shown in the drawings and are herein described in detail. It should be understood, however, that the invention is not to be limited to the particular forms or methods disclosed, but to the contrary, the invention is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the appended claims.
Claims (52)
1. A system for monitoring movement of a person's eye, comprising:
a device configured to be worn on a person's head;
an array of emitters on the device for directing light towards an eye of the person when the device is worn, the array of emitters configured for projecting a reference frame towards the eye; and
a camera oriented towards the eye for monitoring movement of the eye relative to the reference frame; and
one or more sensors on the device for detecting light from the array of emitters that is reflected off of the eye or its eyelid, the one or more sensors producing an output signal indicating when the eye is open or closed.
2. The system of claim 1 , wherein the one or more sensors comprise an array of sensors in a predetermined relationship with the array of emitters for detecting light from the array of emitters that is reflected off of respective portions of the eye or its eyelid, each sensor producing an output signal indicating when the respective portion of the eye is covered or not covered by the eyelid.
3. The system of claim 1 , wherein the array of emitters and the one or more sensors are disposed separately and substantially laterally from one another.
4. The system of claim 1 , wherein the array of emitters and the one or more sensors comprise solid state devices capable of operating both as an emitter and as a sensor.
5. The system of claim 1 , wherein the camera is configured for producing a video signal, and wherein the system further comprises a processor for correlating the output signal from the one or more sensors with the video signal from the camera for determining the person's level of drowsiness.
6. The system of claim 5 , further comprising a warning indicator on the device, the warning indicator being activated when the processor determines a predetermined level of drowsiness has occurred.
7. The system of claim 1 , wherein the array of emitters comprises a plurality of emitters disposed in a substantially vertical arrangement on the device.
8. The system of claim 7 , wherein the array of emitters further comprises a plurality of emitters disposed in a substantially horizontal arrangement on the device.
9. The system of claim 1 , wherein the array of emitters is configured for projecting a set of crossed bands towards the eye for dividing a region including the eye into four quadrants.
10. The system of claim 1 , further comprising a transmitter on the device for wireless transmission of video output signals from the camera to a remote location.
11. The system of claim 1 , wherein the array of emitters comprise infrared emitters configured to emit pulses of infrared light.
12. The system of claim 11 , wherein the camera comprises an infrared camera.
13. The system of claim 1 , wherein the camera is mounted on the device.
14. The system of claim 13 , wherein the camera comprises a fiberoptic assembly.
15. The system of claim 13 , wherein the camera comprises at least one of a CCD and CMOS detector.
16. The system of claim 1 , further comprising a sensor on the device for detecting one or more physiological characteristics of the person.
17. The system of claim 16 , wherein the sensor comprises at least one of an EEG electrode, an EKG electrode, an oximetry sensor, a pulse sensor, an airflow sensor, and a temperature sensor.
18. The system of claim 1 , further comprising at least one of an orientation sensor for detecting the spatial orientation of the device and an actigraphic sensor.
19. The system of claim 1 , wherein the device comprises at least one of an eyeglass frame, a hat, a helmet, a visor, and a mask.
20. A system for monitoring movement of a person's eye, comprising:
a frame configured to be worn on a person's head;
an array of emitters on the frame for directing light towards an eye of the person when the frame is worn, the array emitters configured to project a reference frame towards the eye;
an array of sensors on the frame in a predetermined relationship with the array of emitters for detecting light from the array of emitters that is reflected off of respective portions of the eye or its eyelid, each sensor producing an output signal indicating when the respective portion of the eye is covered or not covered by the eyelid;
a camera on the frame for monitoring movement of the eye relative to the reference frame, the camera configured for producing a video signal of a region of the eye and the reference frame; and
a transmitter coupled to the sensor for wireless transmission of the output signal and the video signal to a remote location.
21. The system of claim 20 , further comprising a processor for correlating the output signal and the video signal to determine the person's level of drowsiness.
22. The system of claim 21 , further comprising a display for providing a graphical output of the output signal simultaneous with the video signal.
23. A method for monitoring movement of a person's eye using a detection device including an array of emitters that are directed towards an eye of the person when the detection device is worn, and a camera oriented towards the eye away from the person when the detection device is worn, the method comprising:
emitting light from the array of one or more emitters towards the eye to project a reference frame onto the eye;
monitoring movement of the eye relative to the reference frame the person's surroundings with the camera; and
generating a graphical output of the movement monitored by the camera relative to the reference frame;
wherein the detection device further comprises one or more sensors, and wherein the method further comprises detecting light from the array of one or more emitters reflected off of the eye with the one or more sensors, the one or more sensors producing a light intensity signal indicating when the eye is open or closed.
24. The method of claim 23 , wherein the array of sensors is disposed in a predetermined relationship with the array of one or more emitters for detecting light from the array of one or more emitters that is reflected off of respective portions of the eye or its eyelid, each sensor producing an output signal indicating when the respective portion of the eye is covered or not covered by the eyelid.
25. The method of claim 24 , further comprising correlating the output signal from the one or more sensors with video signals produced by the camera monitoring movement of the eye relative to the reference frame, thereby determining the person's level of alertness.
26. The method of claim 23 , wherein the monitoring step comprises measuring movement of the eye's pupil relative to the reference frame.
27. The method of claim 26 , further comprising graphically displaying the movement of the eye's pupil relative to the reference frame.
28. The method of claim 25 , further comprising providing a warning to the person when the determined level of alertness falls below a predetermined level.
29. A system for monitoring movement of a person's eye, comprising:
a device configured to be worn on a person's head such that the device does not interfere substantially with the person's vision along an axis extending directly ahead of a first eye of the person;
one or more emitters on the device for directing light towards the first eye when the device is worn;
an array of sensors on the device directed towards the first eye when the device is worn, the sensors configured for converting images of the first eye into output signals; and
a processor coupled to the sensors for interpreting the output signals to control one or more devices,
wherein the one or more emitters and the sensors are provided on the device at locations offset from the axis to generally minimize interference with the person's vision and such that the array of sensors are oriented directly towards the region surrounding the first eye.
30. The system of claim 29, further comprising a control system communicating with the processor, the processor configured for directing the control system to control one or more devices based upon the output signals.
31. The system of claim 29, further comprising a computer communicating with the processor, the processor configured for controlling the computer based upon the output signals.
32. The system of claim 31, wherein the processor is configured to interpret the output signals to operate the computer as an eye-activated mouse.
33. The system of claim 29, wherein the one or more emitters comprise a plurality of infrared emitters configured to emit pulses at a predetermined frequency.
34. The system of claim 29, wherein the device comprises a frame comprising a bridge piece extending between a pair of ear supports, the bridge piece including a nose bridge configured to be placed on the person's nose, and wherein the array of sensors are provided on or adjacent the nose bridge lateral from the first eye when the frame is worn to generally minimize interference with the person's vision along the axis.
35. The system of claim 34, wherein the one or more emitters comprise an array of emitters provided on the frame.
36. The system of claim 29, wherein the device comprises a nose bridge configured to be placed on the person's nose, and wherein the array of sensors are provided on or adjacent the nose bridge lateral from the first eye when the device is worn to generally minimize interference with the person's vision along the axis.
37. The system of claim 36, wherein the device comprises a mask.
38. The system of claim 36, wherein the device comprises an eyeglass frame.
39. A method for monitoring movement of a person's eye using a detection device including one or more emitters and an array of sensors that are directed towards a first eye of the person when the detection device is worn, the method comprising:
placing the detection device on a person's head such that the detection device does not interfere substantially with the person's vision along an axis extending directly ahead of the first eye and the array of sensors are offset from the axis and oriented directly towards a region surrounding the first eye;
emitting light from one or more emitters towards the first eye;
detecting light from the one or more emitters reflected off of the first eye with the array of sensors, the array of sensors producing light intensity signals indicating when the first eye is open or closed; and
interpreting the light intensity signals to control one or more devices.
40. The method of claim 39, wherein the array of sensors comprises a two-dimensional array of sensors.
41. The method of claim 39, wherein the light intensity signals are used to control a computer.
42. A system for controlling a computing device, comprising:
a frame configured to be worn on a person's head such that the frame does not interfere substantially with the person's vision along an axis extending directly ahead of a first eye of the person;
a sensor on the frame comprising a lens offset from the axis and oriented directly towards the region surrounding the first eye when the frame is worn, the sensor generating output signals representing video images of the first eye; and
a processor coupled to the sensor for processing the output signals to monitor movement of the first eye relative to a reference frame, the processor communicating with an electronic device remote from the frame and interpreting the output signals to control the electronic device.
43. The system of claim 42, wherein the sensor comprises a detector coupled to the lens by a fiberoptic cable.
44. The system of claim 42, wherein the electronic device comprises a computer, and wherein the processor is configured to interpret the output signals to operate the computer as an eye-activated mouse.
45. The system of claim 42, wherein the frame comprises a bridge piece extending between a pair of ear supports, the bridge piece including a nose bridge configured to be placed on the person's nose, the sensor provided on or adjacent the nose bridge at a location lateral from the first eye when the frame is worn to generally minimize interference with the person's vision along the axis.
46. The system of claim 42, further comprising a plurality of infrared emitters on the frame for directing light towards the first eye of the person when the frame is worn, the emitters configured to emit pulses at a predetermined frequency.
47. The system of claim 46, wherein the emitters are configured for projecting the reference frame onto the first eye of the person wearing the frame.
48. A system for controlling a computing device, comprising:
a frame configured to be worn on a person's head such that the device does not interfere substantially with the person's vision;
a sensor on the frame comprising a lens directed towards the eye of the person when the frame is worn, the sensor generating output signals representing video images of the eye; and
a processor coupled to the sensor for processing the output signals to monitor movement of the eye relative to a reference frame, the processor communicating with an electronic device remote from the frame and interpreting the output signals to control the electronic device,
wherein the processor is configured for monitoring the output signals to detect video images indicating that the person wearing the device has blinked in a predetermined sequence, the processor configured for executing the command on the electronic device based upon the predetermined sequence.
49. A system for monitoring movement of a person's eye, comprising:
an eyeglass frame comprising a nose bridge configured to be placed on the person's nose, and a pair of ear supports configured to be placed over the person's ears such that the frame, when worn on the person's head, does not interfere substantially with the person's vision along an axis extending directly ahead of a first eye of the person;
one or more emitters on the frame for directing light towards the first eye when the frame is worn, the one or more emitters provided on the frame at one or more locations that generally minimize interference with the person's vision along the axis;
a sensor on or adjacent the nose bridge such that the sensor is offset from the axis to generally minimize interference with the person's vision along the axis and oriented directly towards the region surrounding the first eye, the sensor configured for converting images of the first eye into output signals; and
a processor coupled to the sensor for interpreting the output signals to control one or more devices.
50. The system of claim 49, wherein the sensor comprises a lens oriented directly towards the region surrounding the first eye when the frame is worn, and a detector coupled to the lens for converting the images of the first eye into digital video signals comprising the output signals.
51. The system of claim 49, wherein the one or more devices comprise a computer.
52. The system of claim 49, wherein the one or more devices comprise a wheelchair.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US12/199,693 USRE42471E1 (en) | 1996-08-19 | 2008-08-27 | System and method for monitoring eye movement |
Applications Claiming Priority (7)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US08/699,670 US5748113A (en) | 1996-08-19 | 1996-08-19 | Method and apparatus for communication |
US08/978,100 US6246344B1 (en) | 1996-08-19 | 1997-11-25 | Method and apparatus for voluntary communication |
US09/104,258 US6163281A (en) | 1996-08-19 | 1998-06-24 | System and method for communication using eye movement |
US09/740,738 US6542081B2 (en) | 1996-08-19 | 2000-12-18 | System and method for monitoring eye movement |
US11/097,942 USRE39539E1 (en) | 1996-08-19 | 2005-04-01 | System and method for monitoring eye movement |
US11/732,828 USRE41376E1 (en) | 1996-08-19 | 2007-04-03 | System and method for monitoring eye movement |
US12/199,693 USRE42471E1 (en) | 1996-08-19 | 2008-08-27 | System and method for monitoring eye movement |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/740,738 Reissue US6542081B2 (en) | 1996-08-19 | 2000-12-18 | System and method for monitoring eye movement |
Publications (1)
Publication Number | Publication Date |
---|---|
USRE42471E1 true USRE42471E1 (en) | 2011-06-21 |
Family
ID=46205529
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/097,942 Expired - Lifetime USRE39539E1 (en) | 1996-08-19 | 2005-04-01 | System and method for monitoring eye movement |
US11/732,828 Expired - Lifetime USRE41376E1 (en) | 1996-08-19 | 2007-04-03 | System and method for monitoring eye movement |
US12/199,693 Expired - Lifetime USRE42471E1 (en) | 1996-08-19 | 2008-08-27 | System and method for monitoring eye movement |
Family Applications Before (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/097,942 Expired - Lifetime USRE39539E1 (en) | 1996-08-19 | 2005-04-01 | System and method for monitoring eye movement |
US11/732,828 Expired - Lifetime USRE41376E1 (en) | 1996-08-19 | 2007-04-03 | System and method for monitoring eye movement |
Country Status (1)
Country | Link |
---|---|
US (3) | USRE39539E1 (en) |
Cited By (30)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120314045A1 (en) * | 2009-08-26 | 2012-12-13 | Ecole Polytechnique Federale De Lausanne (Epfl) | Wearable systems for audio, visual and gaze monitoring |
US8337404B2 (en) | 2010-10-01 | 2012-12-25 | Flint Hills Scientific, Llc | Detecting, quantifying, and/or classifying seizures using multimodal data |
US8382667B2 (en) | 2010-10-01 | 2013-02-26 | Flint Hills Scientific, Llc | Detecting, quantifying, and/or classifying seizures using multimodal data |
US20130131908A1 (en) * | 2006-03-16 | 2013-05-23 | Gray & Company, Inc. | Navigation and control system for autonomous vehicles |
US8452387B2 (en) | 2010-09-16 | 2013-05-28 | Flint Hills Scientific, Llc | Detecting or validating a detection of a state change from a template of heart rate derivative shape or heart beat wave complex |
US8562536B2 (en) | 2010-04-29 | 2013-10-22 | Flint Hills Scientific, Llc | Algorithm for detecting a seizure from cardiac data |
US8641646B2 (en) | 2010-07-30 | 2014-02-04 | Cyberonics, Inc. | Seizure detection using coordinate data |
US8649871B2 (en) | 2010-04-29 | 2014-02-11 | Cyberonics, Inc. | Validity test adaptive constraint modification for cardiac data used for detection of state changes |
US8684921B2 (en) | 2010-10-01 | 2014-04-01 | Flint Hills Scientific Llc | Detecting, assessing and managing epilepsy using a multi-variate, metric-based classification analysis |
US8725239B2 (en) | 2011-04-25 | 2014-05-13 | Cyberonics, Inc. | Identifying seizures using heart rate decrease |
US20140147019A1 (en) * | 2011-07-11 | 2014-05-29 | Toyota Jidosha Kabushiki Kaisha | Red-eye detection device |
US8831732B2 (en) | 2010-04-29 | 2014-09-09 | Cyberonics, Inc. | Method, apparatus and system for validating and quantifying cardiac beat data quality |
US20150305686A1 (en) * | 2012-11-10 | 2015-10-29 | The Regents Of The University Of California | Systems and methods for evaluation of neuropathologies |
US9265458B2 (en) | 2012-12-04 | 2016-02-23 | Sync-Think, Inc. | Application of smooth pursuit cognitive testing paradigms to clinical drug development |
US9380976B2 (en) | 2013-03-11 | 2016-07-05 | Sync-Think, Inc. | Optical neuroinformatics |
US9402550B2 (en) | 2011-04-29 | 2016-08-02 | Cybertronics, Inc. | Dynamic heart rate threshold for neurological event detection |
US9418617B1 (en) | 2013-03-13 | 2016-08-16 | Google Inc. | Methods and systems for receiving input controls |
US9489817B2 (en) * | 2015-01-29 | 2016-11-08 | Vigo Technologies, Inc. | Infrared sensing of eye and eyelid movements to detect drowsiness |
US9504390B2 (en) | 2011-03-04 | 2016-11-29 | Globalfoundries Inc. | Detecting, assessing and managing a risk of death in epilepsy |
US9681836B2 (en) | 2012-04-23 | 2017-06-20 | Cyberonics, Inc. | Methods, systems and apparatuses for detecting seizure and non-seizure states |
WO2018020334A1 (en) * | 2016-07-28 | 2018-02-01 | Tata Consultancy Services Limited | System and method for aiding communication |
US9883814B1 (en) | 2016-05-05 | 2018-02-06 | Mansour Zarreii | System and method for evaluating neurological conditions |
TWI645366B (en) | 2016-12-13 | 2018-12-21 | 國立勤益科技大學 | Image semantic conversion system and method applied to home care |
US10206591B2 (en) | 2011-10-14 | 2019-02-19 | Flint Hills Scientific, Llc | Seizure detection methods, apparatus, and systems using an autoregression algorithm |
US10220211B2 (en) | 2013-01-22 | 2019-03-05 | Livanova Usa, Inc. | Methods and systems to diagnose depression |
US10448839B2 (en) | 2012-04-23 | 2019-10-22 | Livanova Usa, Inc. | Methods, systems and apparatuses for detecting increased risk of sudden death |
WO2022212029A1 (en) * | 2021-03-31 | 2022-10-06 | Microsoft Technology Licensing, Llc | Head mounted display with obscured light emitting diodes |
US11615688B2 (en) | 2017-12-22 | 2023-03-28 | Resmed Sensor Technologies Limited | Apparatus, system, and method for motion sensing |
US11707197B2 (en) | 2017-12-22 | 2023-07-25 | Resmed Sensor Technologies Limited | Apparatus, system, and method for physiological sensing in vehicles |
US11813081B2 (en) | 2020-06-15 | 2023-11-14 | Beijing Xiaomi Mobile Software Co., Ltd. | Intelligent glasses and glasses box |
Families Citing this family (48)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6483484B1 (en) * | 1998-12-18 | 2002-11-19 | Semiconductor Energy Laboratory Co., Ltd. | Goggle type display system |
US7369951B2 (en) * | 2004-02-27 | 2008-05-06 | Board Of Trustees Of Michigan State University | Digital, self-calibrating proximity switch |
US20070081123A1 (en) * | 2005-10-07 | 2007-04-12 | Lewis Scott W | Digital eyewear |
US11428937B2 (en) | 2005-10-07 | 2022-08-30 | Percept Technologies | Enhanced optical and perceptual digital eyewear |
US9658473B2 (en) | 2005-10-07 | 2017-05-23 | Percept Technologies Inc | Enhanced optical and perceptual digital eyewear |
US8696113B2 (en) | 2005-10-07 | 2014-04-15 | Percept Technologies Inc. | Enhanced optical and perceptual digital eyewear |
US8628478B2 (en) * | 2009-02-25 | 2014-01-14 | Empire Technology Development Llc | Microphone for remote health sensing |
US8160311B1 (en) * | 2008-09-26 | 2012-04-17 | Philip Raymond Schaefer | System and method for detecting facial gestures for control of an electronic device |
US9039631B2 (en) * | 2008-10-09 | 2015-05-26 | Neuro Kinetics | Quantitative, non-invasive, clinical diagnosis of traumatic brain injury using VOG device for neurologic testing |
US8866621B2 (en) * | 2009-02-25 | 2014-10-21 | Empire Technology Development Llc | Sudden infant death prevention clothing |
US8824666B2 (en) * | 2009-03-09 | 2014-09-02 | Empire Technology Development Llc | Noise cancellation for phone conversation |
US8193941B2 (en) | 2009-05-06 | 2012-06-05 | Empire Technology Development Llc | Snoring treatment |
US20100286567A1 (en) * | 2009-05-06 | 2010-11-11 | Andrew Wolfe | Elderly fall detection |
US20100286545A1 (en) * | 2009-05-06 | 2010-11-11 | Andrew Wolfe | Accelerometer based health sensing |
KR20100138725A (en) * | 2009-06-25 | 2010-12-31 | 삼성전자주식회사 | Method and apparatus for processing virtual world |
WO2012061871A1 (en) * | 2010-11-08 | 2012-05-18 | Optalert Australia Pty Ltd | Fitness for work test |
US8698639B2 (en) | 2011-02-18 | 2014-04-15 | Honda Motor Co., Ltd. | System and method for responding to driver behavior |
US9292471B2 (en) | 2011-02-18 | 2016-03-22 | Honda Motor Co., Ltd. | Coordinated vehicle response system and method for driver behavior |
US9033502B2 (en) | 2011-03-18 | 2015-05-19 | Sensomotoric Instruments Gesellschaft Fur Innovative Sensorik Mbh | Optical measuring device and method for capturing at least one parameter of at least one eye wherein an illumination characteristic is adjustable |
EP2499960B1 (en) * | 2011-03-18 | 2015-04-22 | SensoMotoric Instruments Gesellschaft für innovative Sensorik mbH | Method for determining at least one parameter of two eyes by setting data rates and optical measuring device |
EP2790635B1 (en) * | 2011-12-16 | 2016-04-27 | Chordate Medical AB | Ans stimulation |
CN104144668B (en) | 2011-12-16 | 2016-08-24 | 考地特医疗Ab公司 | Equipment and system for vibratory stimulation at least one nasal cavity of experimenter |
US9116545B1 (en) * | 2012-03-21 | 2015-08-25 | Hayes Solos Raffle | Input detection |
US9201512B1 (en) | 2012-04-02 | 2015-12-01 | Google Inc. | Proximity sensing for input detection |
US9128522B2 (en) | 2012-04-02 | 2015-09-08 | Google Inc. | Wink gesture input for a head-mountable device |
US10010270B2 (en) * | 2012-09-17 | 2018-07-03 | Verily Life Sciences Llc | Sensing system |
CN103777351A (en) * | 2012-10-26 | 2014-05-07 | 鸿富锦精密工业(深圳)有限公司 | Multimedia glasses |
WO2014127333A1 (en) * | 2013-02-15 | 2014-08-21 | Emotient | Facial expression training using feedback from automatic facial expression recognition |
US9751534B2 (en) | 2013-03-15 | 2017-09-05 | Honda Motor Co., Ltd. | System and method for responding to driver state |
US10499856B2 (en) | 2013-04-06 | 2019-12-10 | Honda Motor Co., Ltd. | System and method for biological signal processing with highly auto-correlated carrier sequences |
US9936916B2 (en) * | 2013-10-09 | 2018-04-10 | Nedim T. SAHIN | Systems, environment and methods for identification and analysis of recurring transitory physiological states and events using a portable data collection device |
US9454887B1 (en) | 2013-11-19 | 2016-09-27 | Mustafa Q. Matalgah | Doze alert |
US9808157B2 (en) * | 2013-12-30 | 2017-11-07 | Verily Life Sciences Llc | Hands-free interface |
KR102246310B1 (en) * | 2013-12-31 | 2021-04-29 | 아이플루언스, 인크. | Systems and methods for gaze-based media selection and editing |
WO2015116640A1 (en) * | 2014-01-29 | 2015-08-06 | Shazly Tarek A | Eye and head tracking device |
US9955907B2 (en) | 2014-04-23 | 2018-05-01 | Case Western Reserve University | Low frequency non-invasive sensorial stimulation for seizure control |
US9767373B2 (en) * | 2014-09-05 | 2017-09-19 | Ford Global Technologies, Llc | Head-mounted display head pose and activity estimation |
US20160343229A1 (en) * | 2015-05-18 | 2016-11-24 | Frank Colony | Vigilance detection method and apparatus |
US10007845B2 (en) * | 2015-07-06 | 2018-06-26 | Pixart Imaging Inc. | Eye state detecting method and eye state detecting system |
US20170185149A1 (en) * | 2015-12-26 | 2017-06-29 | Intel Corporation | Single sensor brain wave monitor |
JP7385993B2 (en) * | 2016-06-20 | 2023-11-24 | マジック リープ, インコーポレイテッド | Augmented reality display system for evaluation and correction of neurological disorders, including disorders of visual processing and perception |
US10839712B2 (en) * | 2016-09-09 | 2020-11-17 | International Business Machines Corporation | Monitoring learning performance using neurofeedback |
US10925479B2 (en) * | 2016-10-13 | 2021-02-23 | Ronald Michael Kurtz | Networked system of mobile communication platforms for nonpharmacologic constriction of a pupil |
ES2610196A1 (en) * | 2016-12-20 | 2017-04-26 | Universitat D'alacant / Universidad De Alicante | Method and biometric authentication device through the recognition of flashing (Machine-translation by Google Translate, not legally binding) |
US10424232B2 (en) | 2017-12-21 | 2019-09-24 | X Development Llc | Directional light emitters and electronic displays featuring the same |
US11237639B2 (en) | 2018-03-27 | 2022-02-01 | Iowa Adaptive Technologies, Inc. | Method and system for electronic communication by persons with disabilities |
US20230097774A1 (en) * | 2021-09-29 | 2023-03-30 | Pixieray Oy | Eyeglass lens with eye-tracking components |
CN117717340B (en) * | 2024-02-07 | 2024-05-31 | 中汽研汽车检验中心(天津)有限公司 | Driver sleepiness detection method, device, equipment and medium |
Citations (76)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US3689135A (en) * | 1971-01-26 | 1972-09-05 | Laurence R Young | Method for monitoring movement of a subject{40 s eyes |
US3798599A (en) | 1972-02-24 | 1974-03-19 | H Kafafian | Single input controller for a communication system |
US3863243A (en) | 1972-01-19 | 1975-01-28 | Max Skolnick | Sleep inhibiting alarm |
US3966310A (en) | 1974-02-15 | 1976-06-29 | Larson Merlin D | Pupillometer and method of use thereof |
US4102564A (en) | 1975-04-18 | 1978-07-25 | Michael Henry L | Portable device for the accurate measurement of eye movements both in light and obscurity |
US4359724A (en) | 1980-04-28 | 1982-11-16 | Ronald R. Zimmerman | Eyelid movement detector |
US4815839A (en) | 1987-08-03 | 1989-03-28 | Waldorf Ronald A | Infrared/video electronystagmographic apparatus |
US4850691A (en) | 1987-03-18 | 1989-07-25 | University Of Illinois | Method and apparatus for determining pupillary response parameters |
US4852988A (en) * | 1988-09-12 | 1989-08-01 | Applied Science Laboratories | Visor and camera providing a parallax-free field-of-view image for a head-mounted eye movement measurement system |
US4894777A (en) | 1986-07-28 | 1990-01-16 | Canon Kabushiki Kaisha | Operator mental condition detector |
US4953111A (en) | 1987-02-12 | 1990-08-28 | Omron Tateisi Electronics Co. | Doze detector |
US4967186A (en) | 1989-08-18 | 1990-10-30 | Ariold Ludmirsky | Method and apparatus for fatigue detection |
US4988183A (en) | 1988-06-13 | 1991-01-29 | Konan Camera Research Institute, Inc. | Eye movement inspection device |
US5070883A (en) | 1988-12-16 | 1991-12-10 | Konan Camera Research Institute Inc. | Eye movement analyzing device utilizing pupil center-of-gravity data |
US5093567A (en) | 1989-07-14 | 1992-03-03 | Gec-Marconi Limited | Helmet systems with eyepiece and eye position sensing means |
US5183512A (en) | 1991-08-29 | 1993-02-02 | Aquotech, Inc. | Method of cleaning semiconductor wafer and computer disks by purified water treated with electric AC signal |
US5214456A (en) | 1991-10-09 | 1993-05-25 | Computed Anatomy Incorporated | Mapping of corneal topography with display of pupil perimeter |
US5341181A (en) | 1992-11-20 | 1994-08-23 | Godard Roger R | Systems and methods for capturing and presentng visual information |
US5345281A (en) * | 1992-12-17 | 1994-09-06 | John Taboada | Eye tracking system and method |
US5402109A (en) | 1993-04-29 | 1995-03-28 | Mannik; Kallis H. | Sleep prevention device for automobile drivers |
GB2284582A (en) | 1993-11-22 | 1995-06-14 | Toad Innovations Ltd | Vehicle safety device to warn driver of fatigue |
US5447166A (en) | 1991-09-26 | 1995-09-05 | Gevins; Alan S. | Neurocognitive adaptive computer interface method and system based on on-line measurement of the user's mental effort |
EP0679984A1 (en) | 1994-04-22 | 1995-11-02 | Canon Kabushiki Kaisha | Display apparatus |
US5469143A (en) | 1995-01-10 | 1995-11-21 | Cooper; David E. | Sleep awakening device for drivers of motor vehicles |
US5478239A (en) | 1993-12-21 | 1995-12-26 | Maximum Performance, Inc. | Dynamic visual acuity training method and apparatus |
US5481622A (en) | 1994-03-01 | 1996-01-02 | Rensselaer Polytechnic Institute | Eye tracking apparatus and method employing grayscale threshold values |
US5566067A (en) | 1995-03-23 | 1996-10-15 | The President And Fellows Of Harvard College | Eyelid vigilance detector system |
US5570698A (en) | 1995-06-02 | 1996-11-05 | Siemens Corporate Research, Inc. | System for monitoring eyes for detecting sleep behavior |
US5583795A (en) | 1995-03-17 | 1996-12-10 | The United States Of America As Represented By The Secretary Of The Army | Apparatus for measuring eye gaze and fixation duration, and method therefor |
US5682144A (en) | 1995-11-20 | 1997-10-28 | Mannik; Kallis Hans | Eye actuated sleep prevention devices and other eye controlled devices |
US5689241A (en) | 1995-04-24 | 1997-11-18 | Clarke, Sr.; James Russell | Sleep detection and driver alert apparatus |
US5704369A (en) | 1994-07-25 | 1998-01-06 | Beth Israel Hospital Association, Inc. | Non-invasive method for diagnosing Alzeheimer's disease in a patient |
US5726916A (en) | 1996-06-27 | 1998-03-10 | The United States Of America As Represented By The Secretary Of The Army | Method and apparatus for determining ocular gaze point of regard and fixation duration |
US5748113A (en) | 1996-08-19 | 1998-05-05 | Torch; William C. | Method and apparatus for communication |
US5778893A (en) | 1991-04-01 | 1998-07-14 | President And Fellows Of Harvard College | Method of diagnosing and monitoring a treatment for Alzheimer's disease |
US5795306A (en) | 1994-03-10 | 1998-08-18 | Mitsubishi Denki Kabushiki Kaisha | Bodily state detection apparatus |
WO1998049028A1 (en) | 1997-04-25 | 1998-11-05 | Applied Science Group, Inc. | An alertness monitor |
US5861936A (en) | 1996-07-26 | 1999-01-19 | Gillan Holdings Limited | Regulating focus in accordance with relationship of features of a person's eyes |
US5867587A (en) | 1997-05-19 | 1999-02-02 | Northrop Grumman Corporation | Impaired operator detection and warning system employing eyeblink analysis |
US5956125A (en) | 1997-06-19 | 1999-09-21 | Bioprobes, Inc. | System and method for screening for dementia |
US6003991A (en) | 1996-02-17 | 1999-12-21 | Erik Scott Viirre | Eye examination apparatus and method for remote examination of a patient by a health professional |
EP0984347A2 (en) | 1998-08-31 | 2000-03-08 | Sel Semiconductor Energy Laboratory Co., Ltd. | Portable information processing system |
JP2000137792A (en) | 1998-10-30 | 2000-05-16 | Toyota Motor Corp | Eye part detecting device |
US6088470A (en) | 1998-01-27 | 2000-07-11 | Sensar, Inc. | Method and apparatus for removal of bright or dark spots by the fusion of multiple images |
US6087941A (en) | 1998-09-01 | 2000-07-11 | Ferraz; Mark | Warning device for alerting a person falling asleep |
US6090051A (en) | 1999-03-03 | 2000-07-18 | Marshall; Sandra P. | Method and apparatus for eye tracking and monitoring pupil dilation to evaluate cognitive activity |
JP2000201289A (en) | 1999-01-07 | 2000-07-18 | Sony Corp | Image input-output device and image acquiring method |
US6091378A (en) | 1998-06-17 | 2000-07-18 | Eye Control Technologies, Inc. | Video processing methods and apparatus for gaze point tracking |
US6091546A (en) | 1997-10-30 | 2000-07-18 | The Microoptical Corporation | Eyeglass interface system |
US6097295A (en) | 1998-01-28 | 2000-08-01 | Daimlerchrysler Ag | Apparatus for determining the alertness of a driver |
US6116736A (en) | 1999-04-23 | 2000-09-12 | Neuroptics, Inc. | Pupilometer with pupil irregularity detection capability |
US6163281A (en) | 1996-08-19 | 2000-12-19 | Torch; William C. | System and method for communication using eye movement |
US6246344B1 (en) | 1996-08-19 | 2001-06-12 | William C. Torch | Method and apparatus for voluntary communication |
US6246779B1 (en) | 1997-12-12 | 2001-06-12 | Kabushiki Kaisha Toshiba | Gaze position detection apparatus and method |
US6247813B1 (en) | 1999-04-09 | 2001-06-19 | Iritech, Inc. | Iris identification system and method of identifying a person through iris recognition |
US6252977B1 (en) | 1997-12-01 | 2001-06-26 | Sensar, Inc. | Method and apparatus for illuminating and imaging eyes through eyeglasses using multiple sources of illumination |
US20010028309A1 (en) | 1996-08-19 | 2001-10-11 | Torch William C. | System and method for monitoring eye movement |
US6334683B2 (en) | 1997-10-23 | 2002-01-01 | Lasersight Technologies, Inc. | Eye illumination system and method |
US20020024633A1 (en) | 1999-04-09 | 2002-02-28 | Daehoon Kim | Pupil evaluation system |
US6373961B1 (en) | 1996-03-26 | 2002-04-16 | Eye Control Technologies, Inc. | Eye controllable screen pointer |
US6388639B1 (en) | 1996-12-18 | 2002-05-14 | Toyota Jidosha Kabushiki Kaisha | Stereoscopic image display apparatus, method of displaying stereoscopic image, and recording medium |
JP2002309925A (en) | 2001-04-10 | 2002-10-23 | Meidensha Corp | Carburetor temperature regulating method for nox removal device |
US6611618B1 (en) | 1997-11-13 | 2003-08-26 | Schepens Eye Research Institute, Inc. | Wide-band image enhancement |
US20040061680A1 (en) | 2002-07-10 | 2004-04-01 | John Taboada | Method and apparatus for computer control |
US6775060B2 (en) | 2001-07-26 | 2004-08-10 | The Schepens Eye Research Institute, Inc. | Bioptic telescope system embedded into a spectacle lens |
US6820979B1 (en) | 1999-04-23 | 2004-11-23 | Neuroptics, Inc. | Pupilometer with pupil irregularity detection, pupil tracking, and pupil response detection capability, glaucoma screening capability, intracranial pressure detection capability, and ocular aberration measurement capability |
US20050007552A1 (en) | 2003-02-13 | 2005-01-13 | Fergason Patent Properties, Llc | Optical system for monitoring eye movement |
US6864473B2 (en) | 2000-12-07 | 2005-03-08 | The United States Of America As Represented By The United States National Aeronautics And Space Administration | Dynamic optical filtration |
US6997556B2 (en) | 2001-10-01 | 2006-02-14 | Ernst Pfleger | Method for detecting, evaluating, and analyzing look sequences |
US7046215B1 (en) | 1999-03-01 | 2006-05-16 | Bae Systems Plc | Head tracker system |
US7071831B2 (en) | 2001-11-08 | 2006-07-04 | Sleep Diagnostics Pty., Ltd. | Alertness monitor |
WO2006092022A1 (en) | 2005-03-04 | 2006-09-08 | Sleep Diagnostics Pty. Ltd | Measuring alertness |
US7120880B1 (en) | 1999-02-25 | 2006-10-10 | International Business Machines Corporation | Method and system for real-time determination of a subject's interest level to media content |
US7206435B2 (en) | 2002-03-26 | 2007-04-17 | Honda Giken Kogyo Kabushiki Kaisha | Real-time eye detection and tracking under various light conditions |
US7374284B2 (en) | 2003-12-17 | 2008-05-20 | The Schepens Eye Research Institute, Inc. | Peripheral field expansion device |
US7391888B2 (en) | 2003-05-30 | 2008-06-24 | Microsoft Corporation | Head pose assessment methods and systems |
-
2005
- 2005-04-01 US US11/097,942 patent/USRE39539E1/en not_active Expired - Lifetime
-
2007
- 2007-04-03 US US11/732,828 patent/USRE41376E1/en not_active Expired - Lifetime
-
2008
- 2008-08-27 US US12/199,693 patent/USRE42471E1/en not_active Expired - Lifetime
Patent Citations (79)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US3689135A (en) * | 1971-01-26 | 1972-09-05 | Laurence R Young | Method for monitoring movement of a subject{40 s eyes |
US3863243A (en) | 1972-01-19 | 1975-01-28 | Max Skolnick | Sleep inhibiting alarm |
US3798599A (en) | 1972-02-24 | 1974-03-19 | H Kafafian | Single input controller for a communication system |
US3966310A (en) | 1974-02-15 | 1976-06-29 | Larson Merlin D | Pupillometer and method of use thereof |
US4102564A (en) | 1975-04-18 | 1978-07-25 | Michael Henry L | Portable device for the accurate measurement of eye movements both in light and obscurity |
US4359724A (en) | 1980-04-28 | 1982-11-16 | Ronald R. Zimmerman | Eyelid movement detector |
US4894777A (en) | 1986-07-28 | 1990-01-16 | Canon Kabushiki Kaisha | Operator mental condition detector |
US4953111A (en) | 1987-02-12 | 1990-08-28 | Omron Tateisi Electronics Co. | Doze detector |
US4850691A (en) | 1987-03-18 | 1989-07-25 | University Of Illinois | Method and apparatus for determining pupillary response parameters |
US4815839A (en) | 1987-08-03 | 1989-03-28 | Waldorf Ronald A | Infrared/video electronystagmographic apparatus |
US4988183A (en) | 1988-06-13 | 1991-01-29 | Konan Camera Research Institute, Inc. | Eye movement inspection device |
US4852988A (en) * | 1988-09-12 | 1989-08-01 | Applied Science Laboratories | Visor and camera providing a parallax-free field-of-view image for a head-mounted eye movement measurement system |
US5070883A (en) | 1988-12-16 | 1991-12-10 | Konan Camera Research Institute Inc. | Eye movement analyzing device utilizing pupil center-of-gravity data |
US5093567A (en) | 1989-07-14 | 1992-03-03 | Gec-Marconi Limited | Helmet systems with eyepiece and eye position sensing means |
US4967186A (en) | 1989-08-18 | 1990-10-30 | Ariold Ludmirsky | Method and apparatus for fatigue detection |
US5778893A (en) | 1991-04-01 | 1998-07-14 | President And Fellows Of Harvard College | Method of diagnosing and monitoring a treatment for Alzheimer's disease |
US5183512A (en) | 1991-08-29 | 1993-02-02 | Aquotech, Inc. | Method of cleaning semiconductor wafer and computer disks by purified water treated with electric AC signal |
US5447166A (en) | 1991-09-26 | 1995-09-05 | Gevins; Alan S. | Neurocognitive adaptive computer interface method and system based on on-line measurement of the user's mental effort |
US5214456A (en) | 1991-10-09 | 1993-05-25 | Computed Anatomy Incorporated | Mapping of corneal topography with display of pupil perimeter |
US5341181A (en) | 1992-11-20 | 1994-08-23 | Godard Roger R | Systems and methods for capturing and presentng visual information |
US5345281A (en) * | 1992-12-17 | 1994-09-06 | John Taboada | Eye tracking system and method |
US5402109A (en) | 1993-04-29 | 1995-03-28 | Mannik; Kallis H. | Sleep prevention device for automobile drivers |
GB2284582A (en) | 1993-11-22 | 1995-06-14 | Toad Innovations Ltd | Vehicle safety device to warn driver of fatigue |
US5478239A (en) | 1993-12-21 | 1995-12-26 | Maximum Performance, Inc. | Dynamic visual acuity training method and apparatus |
US5481622A (en) | 1994-03-01 | 1996-01-02 | Rensselaer Polytechnic Institute | Eye tracking apparatus and method employing grayscale threshold values |
US5795306A (en) | 1994-03-10 | 1998-08-18 | Mitsubishi Denki Kabushiki Kaisha | Bodily state detection apparatus |
EP0679984A1 (en) | 1994-04-22 | 1995-11-02 | Canon Kabushiki Kaisha | Display apparatus |
US6346929B1 (en) | 1994-04-22 | 2002-02-12 | Canon Kabushiki Kaisha | Display apparatus which detects an observer body part motion in correspondence to a displayed element used to input operation instructions to start a process |
US5704369A (en) | 1994-07-25 | 1998-01-06 | Beth Israel Hospital Association, Inc. | Non-invasive method for diagnosing Alzeheimer's disease in a patient |
US5469143A (en) | 1995-01-10 | 1995-11-21 | Cooper; David E. | Sleep awakening device for drivers of motor vehicles |
US5583795A (en) | 1995-03-17 | 1996-12-10 | The United States Of America As Represented By The Secretary Of The Army | Apparatus for measuring eye gaze and fixation duration, and method therefor |
US5566067A (en) | 1995-03-23 | 1996-10-15 | The President And Fellows Of Harvard College | Eyelid vigilance detector system |
US5689241A (en) | 1995-04-24 | 1997-11-18 | Clarke, Sr.; James Russell | Sleep detection and driver alert apparatus |
US5570698A (en) | 1995-06-02 | 1996-11-05 | Siemens Corporate Research, Inc. | System for monitoring eyes for detecting sleep behavior |
US5682144A (en) | 1995-11-20 | 1997-10-28 | Mannik; Kallis Hans | Eye actuated sleep prevention devices and other eye controlled devices |
US6003991A (en) | 1996-02-17 | 1999-12-21 | Erik Scott Viirre | Eye examination apparatus and method for remote examination of a patient by a health professional |
US6373961B1 (en) | 1996-03-26 | 2002-04-16 | Eye Control Technologies, Inc. | Eye controllable screen pointer |
US5726916A (en) | 1996-06-27 | 1998-03-10 | The United States Of America As Represented By The Secretary Of The Army | Method and apparatus for determining ocular gaze point of regard and fixation duration |
US5861936A (en) | 1996-07-26 | 1999-01-19 | Gillan Holdings Limited | Regulating focus in accordance with relationship of features of a person's eyes |
US5748113A (en) | 1996-08-19 | 1998-05-05 | Torch; William C. | Method and apparatus for communication |
US6163281A (en) | 1996-08-19 | 2000-12-19 | Torch; William C. | System and method for communication using eye movement |
US6246344B1 (en) | 1996-08-19 | 2001-06-12 | William C. Torch | Method and apparatus for voluntary communication |
US20010028309A1 (en) | 1996-08-19 | 2001-10-11 | Torch William C. | System and method for monitoring eye movement |
US6388639B1 (en) | 1996-12-18 | 2002-05-14 | Toyota Jidosha Kabushiki Kaisha | Stereoscopic image display apparatus, method of displaying stereoscopic image, and recording medium |
WO1998049028A1 (en) | 1997-04-25 | 1998-11-05 | Applied Science Group, Inc. | An alertness monitor |
US5867587A (en) | 1997-05-19 | 1999-02-02 | Northrop Grumman Corporation | Impaired operator detection and warning system employing eyeblink analysis |
US5956125A (en) | 1997-06-19 | 1999-09-21 | Bioprobes, Inc. | System and method for screening for dementia |
US6334683B2 (en) | 1997-10-23 | 2002-01-01 | Lasersight Technologies, Inc. | Eye illumination system and method |
US6091546A (en) | 1997-10-30 | 2000-07-18 | The Microoptical Corporation | Eyeglass interface system |
US6611618B1 (en) | 1997-11-13 | 2003-08-26 | Schepens Eye Research Institute, Inc. | Wide-band image enhancement |
US6252977B1 (en) | 1997-12-01 | 2001-06-26 | Sensar, Inc. | Method and apparatus for illuminating and imaging eyes through eyeglasses using multiple sources of illumination |
US6246779B1 (en) | 1997-12-12 | 2001-06-12 | Kabushiki Kaisha Toshiba | Gaze position detection apparatus and method |
US6088470A (en) | 1998-01-27 | 2000-07-11 | Sensar, Inc. | Method and apparatus for removal of bright or dark spots by the fusion of multiple images |
US6097295A (en) | 1998-01-28 | 2000-08-01 | Daimlerchrysler Ag | Apparatus for determining the alertness of a driver |
US6091378A (en) | 1998-06-17 | 2000-07-18 | Eye Control Technologies, Inc. | Video processing methods and apparatus for gaze point tracking |
US6867752B1 (en) | 1998-08-31 | 2005-03-15 | Semiconductor Energy Laboratory Co., Ltd. | Portable information processing system |
EP0984347A2 (en) | 1998-08-31 | 2000-03-08 | Sel Semiconductor Energy Laboratory Co., Ltd. | Portable information processing system |
US6087941A (en) | 1998-09-01 | 2000-07-11 | Ferraz; Mark | Warning device for alerting a person falling asleep |
JP2000137792A (en) | 1998-10-30 | 2000-05-16 | Toyota Motor Corp | Eye part detecting device |
JP2000201289A (en) | 1999-01-07 | 2000-07-18 | Sony Corp | Image input-output device and image acquiring method |
US7120880B1 (en) | 1999-02-25 | 2006-10-10 | International Business Machines Corporation | Method and system for real-time determination of a subject's interest level to media content |
US7046215B1 (en) | 1999-03-01 | 2006-05-16 | Bae Systems Plc | Head tracker system |
US6090051A (en) | 1999-03-03 | 2000-07-18 | Marshall; Sandra P. | Method and apparatus for eye tracking and monitoring pupil dilation to evaluate cognitive activity |
US20020024633A1 (en) | 1999-04-09 | 2002-02-28 | Daehoon Kim | Pupil evaluation system |
US6247813B1 (en) | 1999-04-09 | 2001-06-19 | Iritech, Inc. | Iris identification system and method of identifying a person through iris recognition |
US6116736A (en) | 1999-04-23 | 2000-09-12 | Neuroptics, Inc. | Pupilometer with pupil irregularity detection capability |
US6260968B1 (en) | 1999-04-23 | 2001-07-17 | Neuroptics, Inc. | Pupilometer with pupil irregularity detection capability |
US6820979B1 (en) | 1999-04-23 | 2004-11-23 | Neuroptics, Inc. | Pupilometer with pupil irregularity detection, pupil tracking, and pupil response detection capability, glaucoma screening capability, intracranial pressure detection capability, and ocular aberration measurement capability |
US6864473B2 (en) | 2000-12-07 | 2005-03-08 | The United States Of America As Represented By The United States National Aeronautics And Space Administration | Dynamic optical filtration |
JP2002309925A (en) | 2001-04-10 | 2002-10-23 | Meidensha Corp | Carburetor temperature regulating method for nox removal device |
US6775060B2 (en) | 2001-07-26 | 2004-08-10 | The Schepens Eye Research Institute, Inc. | Bioptic telescope system embedded into a spectacle lens |
US6997556B2 (en) | 2001-10-01 | 2006-02-14 | Ernst Pfleger | Method for detecting, evaluating, and analyzing look sequences |
US7071831B2 (en) | 2001-11-08 | 2006-07-04 | Sleep Diagnostics Pty., Ltd. | Alertness monitor |
US7206435B2 (en) | 2002-03-26 | 2007-04-17 | Honda Giken Kogyo Kabushiki Kaisha | Real-time eye detection and tracking under various light conditions |
US20040061680A1 (en) | 2002-07-10 | 2004-04-01 | John Taboada | Method and apparatus for computer control |
US20050007552A1 (en) | 2003-02-13 | 2005-01-13 | Fergason Patent Properties, Llc | Optical system for monitoring eye movement |
US7391888B2 (en) | 2003-05-30 | 2008-06-24 | Microsoft Corporation | Head pose assessment methods and systems |
US7374284B2 (en) | 2003-12-17 | 2008-05-20 | The Schepens Eye Research Institute, Inc. | Peripheral field expansion device |
WO2006092022A1 (en) | 2005-03-04 | 2006-09-08 | Sleep Diagnostics Pty. Ltd | Measuring alertness |
Non-Patent Citations (4)
Title |
---|
Attorney William A. English, Office Actions and Responses for related U.S. Appl. No. 11/732,828 (EYE-002.1), dated Dec. 6, 2007 to Dec. 7, 2009, 68 pages. |
Hewlett-Packard GmbH: "Helmet-Mounted Display for Data Recall and Direct Consultation During Surgical Operation," Jan. 1996, Research Disclosure, Mason Publications, Hampshire, GB. |
PCT International Search Report for PCT/US2005/011104, Applicant: Dr. William Torch, Forms PCT/ISA/210 & 220, dated Mar. 21, 2006 (9 pages). |
PCT Written Opinion of the International Search Authority for PCT/US2005/011104, Applicant: Dr. William C. Torch, Form PCT/ISA/237, dated Mar. 21, 2006 (11 pages). |
Cited By (47)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130131908A1 (en) * | 2006-03-16 | 2013-05-23 | Gray & Company, Inc. | Navigation and control system for autonomous vehicles |
US20120314045A1 (en) * | 2009-08-26 | 2012-12-13 | Ecole Polytechnique Federale De Lausanne (Epfl) | Wearable systems for audio, visual and gaze monitoring |
US8831732B2 (en) | 2010-04-29 | 2014-09-09 | Cyberonics, Inc. | Method, apparatus and system for validating and quantifying cardiac beat data quality |
US9700256B2 (en) | 2010-04-29 | 2017-07-11 | Cyberonics, Inc. | Algorithm for detecting a seizure from cardiac data |
US8562536B2 (en) | 2010-04-29 | 2013-10-22 | Flint Hills Scientific, Llc | Algorithm for detecting a seizure from cardiac data |
US9241647B2 (en) | 2010-04-29 | 2016-01-26 | Cyberonics, Inc. | Algorithm for detecting a seizure from cardiac data |
US8649871B2 (en) | 2010-04-29 | 2014-02-11 | Cyberonics, Inc. | Validity test adaptive constraint modification for cardiac data used for detection of state changes |
US9220910B2 (en) | 2010-07-30 | 2015-12-29 | Cyberonics, Inc. | Seizure detection using coordinate data |
US8641646B2 (en) | 2010-07-30 | 2014-02-04 | Cyberonics, Inc. | Seizure detection using coordinate data |
US8452387B2 (en) | 2010-09-16 | 2013-05-28 | Flint Hills Scientific, Llc | Detecting or validating a detection of a state change from a template of heart rate derivative shape or heart beat wave complex |
US8948855B2 (en) | 2010-09-16 | 2015-02-03 | Flint Hills Scientific, Llc | Detecting and validating a detection of a state change from a template of heart rate derivative shape or heart beat wave complex |
US8571643B2 (en) | 2010-09-16 | 2013-10-29 | Flint Hills Scientific, Llc | Detecting or validating a detection of a state change from a template of heart rate derivative shape or heart beat wave complex |
US9020582B2 (en) | 2010-09-16 | 2015-04-28 | Flint Hills Scientific, Llc | Detecting or validating a detection of a state change from a template of heart rate derivative shape or heart beat wave complex |
US8382667B2 (en) | 2010-10-01 | 2013-02-26 | Flint Hills Scientific, Llc | Detecting, quantifying, and/or classifying seizures using multimodal data |
US8945006B2 (en) | 2010-10-01 | 2015-02-03 | Flunt Hills Scientific, LLC | Detecting, assessing and managing epilepsy using a multi-variate, metric-based classification analysis |
US8888702B2 (en) | 2010-10-01 | 2014-11-18 | Flint Hills Scientific, Llc | Detecting, quantifying, and/or classifying seizures using multimodal data |
US8337404B2 (en) | 2010-10-01 | 2012-12-25 | Flint Hills Scientific, Llc | Detecting, quantifying, and/or classifying seizures using multimodal data |
US8684921B2 (en) | 2010-10-01 | 2014-04-01 | Flint Hills Scientific Llc | Detecting, assessing and managing epilepsy using a multi-variate, metric-based classification analysis |
US8852100B2 (en) | 2010-10-01 | 2014-10-07 | Flint Hills Scientific, Llc | Detecting, quantifying, and/or classifying seizures using multimodal data |
US9504390B2 (en) | 2011-03-04 | 2016-11-29 | Globalfoundries Inc. | Detecting, assessing and managing a risk of death in epilepsy |
US8725239B2 (en) | 2011-04-25 | 2014-05-13 | Cyberonics, Inc. | Identifying seizures using heart rate decrease |
US9402550B2 (en) | 2011-04-29 | 2016-08-02 | Cybertronics, Inc. | Dynamic heart rate threshold for neurological event detection |
US9177202B2 (en) * | 2011-07-11 | 2015-11-03 | Toyota Jidosha Kabushiki Kaisha | Red-eye detection device |
US20140147019A1 (en) * | 2011-07-11 | 2014-05-29 | Toyota Jidosha Kabushiki Kaisha | Red-eye detection device |
US10206591B2 (en) | 2011-10-14 | 2019-02-19 | Flint Hills Scientific, Llc | Seizure detection methods, apparatus, and systems using an autoregression algorithm |
US10448839B2 (en) | 2012-04-23 | 2019-10-22 | Livanova Usa, Inc. | Methods, systems and apparatuses for detecting increased risk of sudden death |
US9681836B2 (en) | 2012-04-23 | 2017-06-20 | Cyberonics, Inc. | Methods, systems and apparatuses for detecting seizure and non-seizure states |
US11596314B2 (en) | 2012-04-23 | 2023-03-07 | Livanova Usa, Inc. | Methods, systems and apparatuses for detecting increased risk of sudden death |
US20150305686A1 (en) * | 2012-11-10 | 2015-10-29 | The Regents Of The University Of California | Systems and methods for evaluation of neuropathologies |
US10258291B2 (en) * | 2012-11-10 | 2019-04-16 | The Regents Of The University Of California | Systems and methods for evaluation of neuropathologies |
US9265458B2 (en) | 2012-12-04 | 2016-02-23 | Sync-Think, Inc. | Application of smooth pursuit cognitive testing paradigms to clinical drug development |
US11103707B2 (en) | 2013-01-22 | 2021-08-31 | Livanova Usa, Inc. | Methods and systems to diagnose depression |
US10220211B2 (en) | 2013-01-22 | 2019-03-05 | Livanova Usa, Inc. | Methods and systems to diagnose depression |
US9380976B2 (en) | 2013-03-11 | 2016-07-05 | Sync-Think, Inc. | Optical neuroinformatics |
US9418617B1 (en) | 2013-03-13 | 2016-08-16 | Google Inc. | Methods and systems for receiving input controls |
US9489817B2 (en) * | 2015-01-29 | 2016-11-08 | Vigo Technologies, Inc. | Infrared sensing of eye and eyelid movements to detect drowsiness |
US11559243B2 (en) | 2016-05-05 | 2023-01-24 | Mansour Zarreii | System and method for evaluating neurological conditions |
US9883814B1 (en) | 2016-05-05 | 2018-02-06 | Mansour Zarreii | System and method for evaluating neurological conditions |
US11086473B2 (en) | 2016-07-28 | 2021-08-10 | Tata Consultancy Services Limited | System and method for aiding communication |
WO2018020334A1 (en) * | 2016-07-28 | 2018-02-01 | Tata Consultancy Services Limited | System and method for aiding communication |
TWI645366B (en) | 2016-12-13 | 2018-12-21 | 國立勤益科技大學 | Image semantic conversion system and method applied to home care |
US11615688B2 (en) | 2017-12-22 | 2023-03-28 | Resmed Sensor Technologies Limited | Apparatus, system, and method for motion sensing |
US11707197B2 (en) | 2017-12-22 | 2023-07-25 | Resmed Sensor Technologies Limited | Apparatus, system, and method for physiological sensing in vehicles |
US12033485B2 (en) | 2017-12-22 | 2024-07-09 | Resmed Sensor Technologies Limited | Apparatus, system, and method for motion sensing |
US11813081B2 (en) | 2020-06-15 | 2023-11-14 | Beijing Xiaomi Mobile Software Co., Ltd. | Intelligent glasses and glasses box |
WO2022212029A1 (en) * | 2021-03-31 | 2022-10-06 | Microsoft Technology Licensing, Llc | Head mounted display with obscured light emitting diodes |
US11867915B2 (en) | 2021-03-31 | 2024-01-09 | Microsoft Technology Licensing, Llc | Head mounted display with obscured light emitting diodes |
Also Published As
Publication number | Publication date |
---|---|
USRE39539E1 (en) | 2007-04-03 |
USRE41376E1 (en) | 2010-06-15 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
USRE42471E1 (en) | System and method for monitoring eye movement | |
US6542081B2 (en) | System and method for monitoring eye movement | |
AU2005229076B2 (en) | Biosensors, communicators, and controllers monitoring eye movement and methods for using them | |
US6163281A (en) | System and method for communication using eye movement | |
US10039445B1 (en) | Biosensors, communicators, and controllers monitoring eye movement and methods for using them | |
US20110077548A1 (en) | Biosensors, communicators, and controllers monitoring eye movement and methods for using them | |
US8647268B2 (en) | Patient monitoring apparatus | |
US7027621B1 (en) | Method and apparatus for operator condition monitoring and assessment | |
US20100308999A1 (en) | Security and monitoring apparatus | |
CA2980062A1 (en) | Method and apparatus for biological evaluation | |
WO2008127316A1 (en) | Security and monitoring apparatus | |
CN105144199A (en) | Imaging device based occupant monitoring system supporting multiple functions | |
Mabry et al. | Commercial motor vehicle operator fatigue detection technology catalog and review |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: EYE-COM CORPORATION, NEVADA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:TORCH, WILLIAM C., DR.;REEL/FRAME:030964/0128 Effective date: 20130328 Owner name: EYEFLUENCE, INC., NEVADA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:EYE-COM CORPORATION;REEL/FRAME:030964/0386 Effective date: 20130806 |
|
FPAY | Fee payment |
Year of fee payment: 12 |