EP0634031A1 - Apparatus and method for eye tracking interface - Google Patents
Apparatus and method for eye tracking interfaceInfo
- Publication number
- EP0634031A1 EP0634031A1 EP93908689A EP93908689A EP0634031A1 EP 0634031 A1 EP0634031 A1 EP 0634031A1 EP 93908689 A EP93908689 A EP 93908689A EP 93908689 A EP93908689 A EP 93908689A EP 0634031 A1 EP0634031 A1 EP 0634031A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- eye
- signals
- tokens
- processor
- eog
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
- A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
- A61B3/113—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for determining or recording eye movement
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/316—Modalities, i.e. specific diagnostic methods
- A61B5/398—Electrooculography [EOG], e.g. detecting nystagmus; Electroretinography [ERG]
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7203—Signal processing specially adapted for physiological signals or for diagnostic purposes for noise prevention, reduction or removal
- A61B5/7207—Signal processing specially adapted for physiological signals or for diagnostic purposes for noise prevention, reduction or removal of noise induced by motion artifacts
Definitions
- This invention relates to systems for utilizing eye movements for communication and control. More particularly, the invention relates to systems for detecting eye movements, deriving signals from such movements, processing those signals to generate signals suitable for communication and control, and interfacing those signals with equipment designed to perform various functions.
- the latter may include tele-robotics,
- EOG electro-oculography
- ENG electro-nystagmography
- EOG is based on electrical measurement of the
- 35 electrical changes can be detected by electrodes placed near the eyes.
- the detected voltage changes are amplified and used to drive a plotting device, whereby a tracing related to eye movement is obtained.
- these signals are inaccurate and noisy and therefore the EOG method has not been used for implementing eye position tracking.
- One implementation of eye tracking utilizes so-called reflectance methods.
- Reflectance methods are basically divided into two variants or subclasses.
- Limbus tracking involves illuminating the eye with an infra-red light emitting diode and obtaining video images of eye movement.
- the differential reflectance method which use laser-based eye tracking. These methods are very accurate, but also very expensive. Since many applications, e.g. those involving the handicapped, have limited markets, the issue of affordability can be crucial, putting these expensive systems out of the reach of many potential users.
- a further disadvantage of reflectance systems is that their detectors can obstruct the view of the user.
- Still another object of the present invention is to provide a system and method for keeping track of the direction of point of fixation and gaze of a human user and using this information in conjunction with "eye gesture, e.g. , blinks, to derive commands to be carried out by an underlying application.
- a further object of the present invention is to provide a system and method which utilizes eye movement to generate signals enabling interaction in virtual space.
- a still further object of the present invention is to provide a system and method of provide 3D location data from coarsely quantized eye position and velocity measurements.
- a still further object of the present invention is to provide a system and method which enables a user to perform selection tasks by generating command signals through the use of eye movements.
- a further object of the present invention is to provide a system and method for providing commands derived from eye movements which are customized to meet individual application requirements.
- the invention comprises a system and method enabling humans to communicate or dialogue with each other or with external devices using EOG signals generated by eye movement and other eye "gestures".
- One aspect of the invention comprises a portable EOG transducer fitted to the user and a signal processor for extracting command signals from the transducer output for use in controlling task-performing functions.
- the system according to the present invention includes an electro-oculographic (EOG) device which relies upon the fact that the eye has characteristics similar to a battery with the cornea acting as the positive pole and the retina acting as the negative pole.
- EOG electro-oculographic
- the electric signal generated by movements of the eye can be measured in both the horizontal and vertical directions by making contact measurements near the eyes.
- the orientation of the electric associated with the eye changes in conjunction with eye movement. This electrical change can be detected by placing electrodes near the eyes. As each of the eyes move, the voltage detected by the associated electrodes varies in accordance with that motion.
- the analog electrical signal is then analyzed by a processor to derive positional information.
- the processing relies on published psychophysical data on human eye movement patterns.
- the system then correlates the positions of the eyes with spatial coordinates.
- the system In addition to position information, the system also obtains velocity information for developing more versatile and accurate processing.
- the system automatically disengages when it senses non-interaction episodes, e.g., turning to look at someone who enters the room.
- the system then allows re-engagement when the user generates the appropriate commands.
- the user begins operation by first commencing a calibration step.
- the user's eye movement pattern is analyzed as he or she visually tracks simple generated movement patterns provided on a display.
- the system may be calibrated by the user by looking at pre-defined reference points in the environment.
- the calibration parameters make it possible to create a smooth trajectory from the coarse data and provide appropriate visual feedback to the user. Further advantages of the calibration is further explained hereinafter.
- Figure 1 illustrates an example of the signal produced by various kinds of eye movements
- Figures 2A-2C illustrate signals derived by an electro-oculograph.
- FIG. 3 illustrates block diagram of the system according to the invention.
- Figure 4 illustrates one example of the system according to the invention.
- Figure 5 illustrates one configuration for electrodes according to the invention.
- Figure 6 illustrates block diagram of data acquisition stage of an embodiment according to the present invention.
- FIG. 7 illustrates the functions of signal processing according to one embodiment of the present invention.
- Figures 8A and 8B illustrate an example of the steps taken for token generation for different velocities according to one embodiment of the invention.
- FIGS. 9A and 9B illustrate the situation where a display screen is use for calibration an interaction according to one embodiment of the invention. Background and Description of the Embodiments
- vestiJbular vestiJbular
- optokinetic saccadic
- saccadic saccadic
- pursuit The first two have to do with the largely involuntary reflex, serving to stabilize the retinal image in the presence of voluntary or involuntary head motion.
- the saccadic movement is used to "jump" from one object of interest to another. This is the fastest type of eye movement.
- the pursuit movement is used to maintain fixation on a moving object.
- the orientation of the eyes is measured, it is possible to locate the 3D position of a fixated target object by triangulation.
- the accuracy of the location determination depends on the accuracy with which the eye orientation is determined.
- a quantitative estimate of positional accuracy can be based on the observation that there is a change in potential of about 1 micro volt for every degree of change in eye orientation in either direction, [see, e.g., Young, L.R. and Sheena, D. , "Survey of Eye Movement Recording Methods", Behavior Research Methods & Instrumentation, 7 , 5 (1975), 397-429].
- the accuracy and resolution of position calculations responsive to these eye movements are determined by the sophistication of the electronic circuitry (and hence also its cost) utilized to amplify and condition this signal.
- 3D location can be determined with respect to a "head" frame of reference.
- the system would also have to measure head movement and incorporate it in the calculations.
- the EOG technique does not measure head movement directly. However, using the EOG signal it is possible to estimate the distance of the fixated point from the head of the user. For example, this enables the system to determine that the user is fixating at a point outside an object of interest such as a computer display.
- the signal processing steps described herein serve to condition the data so it can be reliably interpreted.
- Some of the noise patterns such as the 60 Hz line frequency are removed using a notch filter.
- Other noise artifacts are mostly transients caused, for example. by the operation of an electrical switch in the vicinity of the electrodes, contraction of the facial or neck muscles, slippage of the electrodes due to sweat, and eye blinking.
- Involuntary eye blinking is considered noise in EOG.
- the signals produced by eye blinks are, in fact, quite regular. This makes it easy to recognize and eliminate them.
- this type of signal is quite distinct from the usual data associated with pursuit or saccadic movements, it can be recognized and categorized as such and exploited for control and communication purposes.
- the EOG technique can recognize and utilize eye "gestures” such as winking, blinking or a combination thereof.
- An example of the signal produced by various kinds of eye movements is shown ' in Figure 1.
- the signal 10 shown in Figure 1 may be derived from signals detected during the movement of the eyes. As shown, slope 12 indicates that a saccadic movement has occurred. Signal 14 illustrates the same signal 10 after it has been filtered while signal 16 illustrates the effect of blinking as indicated by slopes 18.
- Electrodes 20-30 are placed around the eye region. Electrodes 20, 22 and 30 are placed along the horizontal axis of the eye for measuring left and right movement. Electrodes 20 and 22 provide signals in one channel which ultimately drive a tracer 32 for measuring horizontal movement of the LEFT eye. Similarly, electrodes 30 and 22 provide signals in a second channel to actuate a tracer 34 for measuring horizontal movements of the RIGHT eye.
- Electrodes 26 and 28 placed vertically above and below the eye measure vertical movements of the right eye. These electrodes provide signals in a third channel which operates tracer 36 for measuring vertical movement of both eyes.
- a three channel system may be formed for measuring left and right movements of each eye and vertical movements of both eyes.
- the eyes rotate horizontally in opposite directions as distance to the object changes. This enables distance determinations.
- Additional channels may be formed by placing electrodes in other locations affected by the movement of the eyes.
- a fourth channel may be used for measuring the vertical movements of each eye separately. This permits the system to distinguish left and right blinks.
- object 19 moves forward and backward each eye traces its movement generating different voltages in each channel as illustrated in Figure 2.
- Fig. 5 illustrates another possible configuration for electrodes.
- the electrodes 102 are disposed on an eyeglass frame or goggle 100 in appropriate locations and the electrode wires 101 are through a passage in the frame, thence through the ear attachments 105 to cables 107 which connect to the system amplifiers.
- Electrode 103 functions as a ground reference. The use of the goggle is easier than placing electrodes individually on the face.
- Electrodes To measure the EOG signals, two types of electrodes are used. One type uses small 6mm gold-plated disks with electrical contact ensured by use of electrode cream. The other electrodes may be the type used in EKG or polygraph testing. These electrodes come with adhesive and conductive gel around an embedded conductor which makes contact with the skin.
- one electrode provides the reference ground necessary for proper signal measurement.
- This electrode may be either attached to the earlobe with a small gold-plated clip or to the arm with a strap or on the forehead.
- gold-plated electrodes embedded in plastic disks and mounted on a standard audio headphone assembly are used.
- This "Head-Phone” holds the electrodes against the skin on the temples about the eyes with enough pressure to make contact and stay in place.
- This arrangement when used in conjunction with the ear clip ground is easy to apply and allows the user to move about. Means are required however to minimize drift tendencies.
- the system provides automatic zeroing with an adaptive circuit which is described in a following section.
- the gold electrodes are taped to the skin on either side of the eyes with standard first-aid tape. This arrangement works best when not attached close to the corners of the eyes.
- the use of taped-on electrodes is a useful alternative, because they are most often used by professional Ophthalmologists in a clinical setting and produce a lower to moderate amount of drift when used to derive eye movement information.
- Figure 3 illustrates aspects of the system organization. As mentioned above, it is possible to obtain independent measurements from the two eyes. This gives rise to the four channel recording system shown in Figure 3. Electrodes 60, 61, 62, 63 and 64 are positioned to pickup the signals fed to the four channels of the transducer. Electrode 64 is connected to an earlobe representing the reference ground.
- the voltage across the electrodes 62 and 63 is indicative of the vertical orientation of each eye separately.
- electrodes 60 and 61 placed horizontally across the eyes, can measure the horizontal orientation of each eye.
- the voltage registered by the electrodes varies in accordance with the motion. These voltages are fed to the transducer 66 and thence to difference amplifier 68 which serves to amplify the voltages picked up by the electrodes and to supply them, via an optocoupler 70, to a notch filter 72. The latter attenuates 60 H z interference.
- the filtered output of the notch filter is supplied to a 120 Hz low-pass filter 74 which removes high frequency noise.
- the resultant signal is then amplified in a DC amplifier 76, converted to digital form in an A to D converter 78, and inputted to the data bus 80 of a computer 82 for further processing by the computer's software. Preferably, all the additional signal processing occurs in the computer.
- A/D converter 78 may form a part of the computer which also includes processor 94, hard disk storage 84, a complement of RAM memory 86, a monitor 88, and a keyboard 89 for setup. It may also include a floppy disk and communication ports.
- the signals collected for processing can include those that initiate or precede eye movements. These signals are associated with eye movements, and are generated prior to the actual movement of the eyes.
- the behavior of the system is derived from software algorithms that operate on the digitized eye movement signals.
- the system uses the computer monitor 88 and a suitable environment such as an X window/UNIX arrangement, the system interacts with the user to generate, process and interpret the signals received from the electrodes as the user reacts to the display.
- the data extracted from the processed EOG signals are supplied to a graphical user interface embodied in the computer's software.
- the user interface provides command signals to output device 91 for control and communication.
- the data supplied to the interface are utilized in interactive feedback modes.
- FIG. 4 illustrates one example of the system processing according to the present invention.
- the EOG signals for each channel are picked up by the EOG acquisition stage 50 and converted to digital signals.
- a tokenizer 52 is coupled to the acquisition stage 50 and derives serial tokens from the EOG signals.
- a token is a set of signals or numbers that represents eye movements both symbolically and numerically.
- the system may include symbolic tokens corresponding to: left eye blinking; one of the eyes moving left; eyes looking steadily at a point; eyes moving fast to the right; double blinking; one of the eyes moving right; and looking up and down.
- Other movements may also serve as tokens depending upon the needs of a particular application. Examples of numeric values are displacement, speed, and duration.
- a protocol interpreter 54 is coupled to the tokenizer for receiving and interpreting the tokens. It may also receive tokens from other channels.
- a protocol is accessed for interpreting various tokens. It generates various actions according to each token received. For example, as part of the protocol two blinks may mean disengage. Therefore, whenever tokens representing two blinks are generated, the protocol interpreter provides signals via an interface 56 to disengage the operation being performed. As shown in Fig. 4, control and status messages are fed back from each module to the previous module. Except for the acquisition stage 50 which is a hardware item, the above functions may be performed in hardware or by software.
- FIG. 6 illustrates additional system details.
- Each channel illustrated in Fig. 6 includes signals derived from two electrodes for signal pick-up, one electrode disposed at a location near the eye, and the ' other electrode being connected to ground, e.g. , the ear- lobe.
- the signal derived from the pair of electrodes for each channel is coupled to analog amplifier 108 for signal measurement.
- the first and second electrodes of channel 4 are coupled to analog amplifier 108A.
- the first and second electrodes of channels 2, 3 and 4 are similarly connected to analog amplifiers 108B, 108C and 108D, respectively.
- Each analog amplifier 108 includes a first pair of buffers 110 coupled to the pair of electrodes.
- the outputs of the buffers are coupled to a differential amplifier circuit which includes an operational amplifier 118.
- the non-inverting input of operational amplifier 118 is coupled via resistance 114 to the output of buffer 110 and via a resistance 116 to ground.
- the inverting input of the operational amplifier is connected via resistance 114 to the output of its buffer and also to a resistive network 122 and 126, the latter providing offset adjustment.
- the input also receives an offset adjusting voltage via resistance 124.
- the output of the operational amplifier is also fed back to its inverting input via resistor 120.
- the output of the operational amplifier 118 is coupled to an active filter which includes an operational amplifier 131, feedback resistor networks 134 and 132, and capacitor 130.
- resistor 124 is coupled to a variable voltage source, explained below, for controlling the drift that occurs during signal measurement.
- the drift is due to a variety of reasons. For example, after a period of time, when the electrodes are connected to the skin, the effect of perspiration or temperature rise can cause drift which can drive the operational amplifiers into saturation. By varying the voltage across resistor 124 it is possible to maintain the operational amplifier 118 in its linear region and hence offset drift occurrences.
- the signals detected in channels 1-4 by analog amplifiers 108A-108D are fed into an analog to digital (A/D) converter and multiplexer 140.
- A/D analog to digital
- the digital sample corresponding to each channel is then transmitted to the computer 82 over data bus 80. Also connected to the data bus are digital-to- analog (Of A) converters 148, 150, 152 and 154. Each (D/A) converter provides the offset adjustment voltages to each differential amplifier via its resistance 124.
- Of A digital-to- analog
- Computer 82 also provides data for address selection via address bus 90.
- Address bus 90 is connected to channel and address selector 156.
- the output of the channel and address selector is coupled to the enable ports of A/D converter 140 and the D/A converters 148, 150, 152 and 154.
- the channel and address selector receives addressing data via address bus 90 and decodes the addressing data to develop appropriate enable signals which are transmitted to the D/A convertors and the A/D convertor.
- A/D multiplexer 140 is selected, a digital value corresponding to a channel sample voltage is sent to the computer via data bus 80.
- digital data on bus 80 is transmitted to the selected D/A convertor, converted to the equivalent analog signal and sent to the appropriate analog amplifier.
- a clock 146 is coupled to A/D multiplexer 140. It sets the desired sampling rate. For a typical eye- tracking application a sampling rate of 100 samples per second for each channel is sufficient. Hence, signals detected by the analog amplifier stage in each channel are multiplexed to the A/D multiplexer and the corresponding digital samples are derived and fed to computer 82.
- the processor 94 In order to avoid drift occurrences and ensure that the operational amplifiers are operating in their linear regions, the processor 94 (see Fig. 3) measures the detected samples and calculates their average. The processor then determines if there is any drift in the average over a period of time. If so, and in order to adjust for the drift, the processor generates an appropriate offset digital signal which is fed to the- appropriate D/A converter 148, 150, 152 or 153, where it is converted to analog form and sent to the respective channel.
- the automatic drift control according to the present invention allows for accurate calculations of the position signals corresponding to the movement of the eyes.
- each channel has a tokenizer for processing the EOG signal derived from the electrodes. It will be appreciated by those skilled in the art that the tokenizer can be configured both in hardware or software.
- the input to the tokenizer a digital sample corresponding to the derived EOG signal, is fed to a Gaussian filter 180.
- the output of the Gaussian filter is coupled to an Artifact recorder 182.
- the Artifact recorder also receives input from a Trajectory Model 192, enabling the artifact recorder to coarsely measure sharp slopes associated with blinking or winking motions.
- the Artifact Recorder also measures the duration of the blink or wink and deletes the sample(s) corresponding to associated sharp change in signal amplitude.
- the output of the Artifact Recorder is therefore a comparatively smooth signal corresponding to actual motions of the eyes without the effects of blinking or winking.
- the output of the Artifact Recorder is coupled to a Position Estimator 184.
- the Position Estimator is also coupled to the Trajectory Model for estimating the actual position of the eye.
- the output of the Position Estimator is coupled to a Slope Estimator 186.
- the slope estimator is also coupled to the trajectory model for estimating the rate of change of eye displacement as reflected in the slope of the signal.
- a Value Token Generator 190 receives input from the Position Estimator (not shown in Figure 7) , Slope Estimator and from the Symbolic Token Generator.
- the tokens are values that represent eye movements both symbolically and numerically.
- the Tokenizer first classifies the eye movement signal using parameters determined by a calibration test; it defines a token for each classification. Thereafter, during the actual eye movement measurements, the tokenizer assigns the corresponding value or token to the signal derived from eye movement, when that signal falls within the range of classification parameters defining a token and determined at least in part by the calibration test.
- the tokenizer In order to generate appropriate tokens, the tokenizer first filters transient noise from the digital sample of the EOG signal by filter 180.
- the process by which this is done can be termed contextual filtering. ° Such filtering draws upon the literature of signal processing and digital filtering techniques [see, e.g. Ballard, D.H. and Brown, CM., Computer Vision , Prentice- Hall, Englewood Cliffs, NJ, 1982; Cohen, A., Biomedical Signal Processing, CRC Press, Inc., Boca Raton, FL. , 1986; and Juhola, M. , Jantti, V., Pyykko, I., Magnusson, M. , Schalen, L. and Akesson, M.
- the Artifact Recorder receives another input from the Trajectory Mode 192.
- the Trajectory Model continuously predicts the value of the next sample based on a statistical prediction technique. In one embodiment of the present invention, the Trajectory Model predicts the value of incoming data based on a straight line extrapolation. Therefore, if the value of the current data sample is not the same as the one predicted by the Trajectory Model, the average of the sample and the predicted data is taken and substituted for the received data.
- the Trajectory Model may emplody other prediction techniques.
- the Trajectory Model generates a signal indicating the abrupt change.
- the Artifact Recorder realizes that a blink or a wink has occurred and consequently sends a signal to token generators 188 and 190.
- the Symbolic Token Generator 188 generates a value or token representing the blink or the wink;
- Value Token Generator 190 generates a token representing its duration.
- the tokens may be index numbers representing the eye ° gesture.
- winking signals are generated when eyes are involuntary closed periodically; they therefore have a relatively short duration.
- Deliberate blinking signals are generated when eyes are voluntarily closed, for instance in generating a desired action, and therefore have a longer duration.
- the Trajectory Model 192 continuously predicts the expected value of the incoming position data. It sends the estimated value to the Position Estimator based on a straight line extrapolation method. The Position Estimator and the Trajectory Model in conjunction continuously smooth out aberrations in the signal. The Position Estimator then sends the position data to Token Generator 188 and to the Slope Estimator 186. The Slope Estimator is also coupled to the
- Slope tokens can represent "static" condition, i.e. zero velocity; a "slow” state representing small velocity or a pursuit; and a "fast” condition representing saccade or fast velocity.
- Figures 8A and 8B illustrate an example of the steps taken for token generation for different velocities. According to one embodiment of the present invention, three classes of slopes are defined as follows:
- the system begins a calibration procedure.
- the user is instructed to fix the eye at different positions for a specified period of time so that the gaze is directed at different points in the visual field.
- the system collects data corresponding to the eye positions.
- the system then calculates the modes of the position distributions and assigns the mode value S 0 as the first classification.
- the system instructs the user to move his or her eyes slowly in different directions.
- the system collects data corresponding to those eye movements.
- the system then calculates the mode of the slow movement distribution and assigns the mode value Sj as the second classification.
- the system instructs the user to move his or her eyes rapidly in different directions.
- the system collects data corresponding to those eye movements.
- the system then calculates the mode of the rapid movement distribution and assigns the mode value S 2 as the third classification.
- a token may be assigned to each one of the classifications.
- Figure 8B illustrates the steps taken to estimate the mode of collected samples.
- the system begins mode estimation.
- the system collects N samples of data.
- the samples are sorted in an ascending order.
- step 232 the largest of the probability values calculated at step 230 is determined.
- step 234 the argument l/2[S i+w +Sj] corresponding to the maximum probability value is found and assigned as the mode.
- a more rigorous mode estimation process estimates mode for different window sizes W. It then uses a maximum likelihood estimator to pick the mode among the different mode candidates obtained by the different windows S.
- an unknown slope S is classified as I, II, or III according to
- the minimum difference decides membership and an appropriate token is assigned to the estimated velocity.
- calibration is measured as the ratio of actual distance measured in degrees to eye movements measured in number of detectable lefts or rights. Therefore, after every data point is processed a left, right, or no movement is returned. Any data point examined during a left will return a left and so on.
- Figure 9A and 9B illustrate the situation where a display screen 250 of monitor 88 is used for calibration and interaction.
- the calibration is measured as the ratio of actual distance measured in pixels to eye movements measured in number of detectable lefts or rights.
- the user assumes a constant distance from the display.
- the user can set a sensitivity value manually by selecting a "comfortable" value.
- the user may instruct the system to move the cursor a certain amount of pixels for every detectable eye movement.
- the system is calibrated by asking the user to trace with his eyes the movement 252 of cursor 259 which could illustratively span approximately 10,000 pixels, while the number of detectable eye movements are being recorded.
- the user's eye movements are analyzed in the computer 82 (see fig.3), to verify that the pattern of movement and the cursor motion is the same.
- This enables the calibration algorithm to calculate the idiosyncratic signal parameters associated with each particular user, for instance, the amount of voltage fluctuation in the signal corresponding to the left most and right most eye position, the velocity for smooth pursuit motion, and so on.
- This calibration establishes signal parameter ranges that are used in later classification and in symbolic token extraction from the signal as mentioned above.
- the calibration procedures can also assist in minimizing drift.
- eye voltages tend to slowly drift throughout some range.
- the magnitude of drift can be approximated by having the user keep his eyes fixed for an extended period while measuring changes in slope.
- the average slope may be used to adjust for drift.
- Fig. 9b illustrates the next interaction mode which is the command mode.
- the cursor 254 is moved by the system in response to and to track the user's gaze.
- four command “buttons” are displayed: 256 for "start", 258 for "stop", 260 for "slower” and 262 for "fast”.
- the cursor 259 is initially at the center of the display. It can be appreciated that this command generating system could control the speed of a machine by eye movement.
- the START button the cursor follows his or her gaze and moves to that button.
- the START command is activated by a prolonged gaze or by the user winking twice, causing the machine to be started.
- the user can consequently communicate control commands to the computer by guiding the cursor to the appropriate button and making an actuating gesture (e.g., eye blinking or winking).
- the monitor 88 may display various objects.
- the user may provide pre-defined eye gestures to interact with the objects.
- the objects may be characters that can be moved to various locations on the display by providing appropriate eye gestures.
- the problem with using the eyes as a pointing device is that humans also use their eyes for other purposes.
- the user might blink naturally, turn his or her head to look at someone or something else, or quickly dart a glance away and then back to the screen.
- the system is programmed to recognize such irrelevant signals and discard them. Natural blinks cause low voltage blips that do not occur in rapid succession.
- valid blinks for registration of commands must be vigorous and in rapid succession, and thus can be distinguished from natural blinks.
- An alternative technique is to require single eye blinking for transmitting commands. This too can be used to distinguish from natural eye blinks.
- each channel provides a value token corresponding to the position of each eye separately.
- the protocol interpreter may process the value tokens by triangulation techniques to estimate the distance.
- the system When the user looks away from the screen by performing a saccade, the system recognizes the irrelevancy of the motion by comparing it with the classification range of velocities calculated during calibration. Again, this causes the system to disengage.
- the illustrative system communicates location interactively without independent head position measurements. This lack of information is substantially overcome by the calibration step before actual interaction. It is assumed that during an interaction "episode,” the user's head will not move appreciably.
- head movements may be measured by analyzing EOG signals. More accurate head movements measurements may be implemented by adding additional electrodes to detect actual head movements.
- the system includes interface control algorithms to control cursor motion and decision algorithms to drive the overall interface system.
- the user interface for graphical applications employs tools for interactive prototyping of 2D and 3D user interfaces [see, e.g., Giacalone, A. and Kaufman, A. , "Tools for Interactive Prototyping of 2D and 3D User Interfaces", in Augmented Interactive Computer Capabilities, A. Klinger, (ed.). Plenum Press, 1990] and within the already developed framework of the Cube 3D user interface described in Kaufman, A. , "A Three-Di ensional User Interface", NCGA '89 Tenth Annual Conference, Philadelphia, PA, April 1989, 361-367; Kaufman, A. and Yagel, R.
- the above described techniques can record the stereo disparity between the eyes so that depth information can be recovered.
- This enables the system to follow the movement of the 3D point of fixation (i.e., the point that the eye is fixating upon) and measure its 3D velocity.
- This characteristic facilitates use for remote navigation and control applications of the type stated above and permits the operator to exercise navigational control of machinery using gaze directed control in three space.
- the system employs tokens generated from the input channels 1-4 of Figure 3, to measure position and velocity of the point of fixation or gaze point.
- the protocol interpreter 54 of Figure 4 enables interaction using somewhat rough spatial position and velocity input from the eye gaze measurement device.
- the 3D system provides approximate location of eye fixation. After suitable calibration to allow for individual variances, it provides data about the gaze point in 3D as well as the time rate of change (i.e.,. velocity) of this point. This data is somewhat coarse. Therefore, the system embodies rules for interaction between the operator and the machine to overcome the limitations of the information channel. These rules of interaction or protocol, allow reliable and unambiguous communication between human and machine, communication between two or more people at the same time. Furthermore, it is possible to store the eye-movements during an interaction episode and allow the user to follow the same movements upon retrieval to the stored data.
- Another embodiment of the present invention may be applied to wheelchair navigation by a physically challenged human operator.
- it is required to control the direction of motion of the wheelchair and its speed.
- the operator should be free to look around and observe points of interest in the scene.
- certain deliberate eye gesture patterns or signatures
- unconscious or natural eye movement patterns should not be interpreted as wheelchair control commands.
- the eye movement pattern provides a serial information stream. To meet these conditions the protocol recognizes certain predefined tokens in the stream as previously described. Based on the information stream obtained from the 2D system the signal analysis algorithms are designed to detect certain simple patterns or tokens.
- Another application according to one embodiment of the present invention is eye tracking and control while the user is looking through a microscope or a binocular. Since EOG measurements do not obstruct the eye, it is possible to extend such eye tracking movements to microscopic or binocular applications. Accordingly, during calibration for the microscopic use, a calibration plate is first placed for the user's view. The user is then instructed to provide the eye movements or gestures for generating parameter classifications. During the interaction mode, the actual movements are detected and corresponding tokens are generated for the protocol interpreter to provide appropriate actions to the particular interface.
- Calibration for binocular use can similarly be accomplished by providing a marked lens on the binocular for deriving eye movement or gestures and generating parameter classifications.
- Another method for calibrating for binocular use may be employed when a marked lens is not available. In this case, the user is first instructed to calibrate eye movements and gestures with the naked eye. The optical characteristics of the binocular is then provided to the system. The processor then transforms naked eye calibration parameters by incorporating optical characteristics into the field of vision with binoculars.
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Human Computer Interaction (AREA)
- Theoretical Computer Science (AREA)
- Biomedical Technology (AREA)
- Surgery (AREA)
- Ophthalmology & Optometry (AREA)
- Veterinary Medicine (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Animal Behavior & Ethology (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- General Physics & Mathematics (AREA)
- Pathology (AREA)
- Eye Examination Apparatus (AREA)
- Ultra Sonic Daignosis Equipment (AREA)
- Selective Calling Equipment (AREA)
Abstract
Description
Claims
Applications Claiming Priority (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US86109492A | 1992-03-31 | 1992-03-31 | |
US861094 | 1992-03-31 | ||
US6199 | 1993-01-19 | ||
US08/006,199 US5360971A (en) | 1992-03-31 | 1993-01-19 | Apparatus and method for eye tracking interface |
PCT/US1993/003011 WO1993020499A1 (en) | 1992-03-31 | 1993-03-31 | Apparatus and method for eye tracking interface |
Publications (2)
Publication Number | Publication Date |
---|---|
EP0634031A1 true EP0634031A1 (en) | 1995-01-18 |
EP0634031B1 EP0634031B1 (en) | 1999-06-23 |
Family
ID=26675310
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP93908689A Expired - Lifetime EP0634031B1 (en) | 1992-03-31 | 1993-03-31 | Apparatus and method for eye tracking interface |
Country Status (7)
Country | Link |
---|---|
US (1) | US5360971A (en) |
EP (1) | EP0634031B1 (en) |
AT (1) | ATE181605T1 (en) |
AU (1) | AU3941993A (en) |
CA (1) | CA2133416A1 (en) |
DE (1) | DE69325448T2 (en) |
WO (1) | WO1993020499A1 (en) |
Families Citing this family (139)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8352400B2 (en) | 1991-12-23 | 2013-01-08 | Hoffberg Steven M | Adaptive pattern recognition based controller apparatus and method and human-factored interface therefore |
US5270748A (en) * | 1992-01-30 | 1993-12-14 | Mak Technologies, Inc. | High-speed eye tracking device and method |
FR2696574B1 (en) * | 1992-10-06 | 1994-11-18 | Sextant Avionique | Method and device for analyzing a message supplied by means of interaction with a human-machine dialogue system. |
US6000696A (en) * | 1992-10-22 | 1999-12-14 | Semiconductor Energy Laboratory Co., Ltd. | Game machine and game parlor |
US5517021A (en) * | 1993-01-19 | 1996-05-14 | The Research Foundation State University Of New York | Apparatus and method for eye tracking interface |
JPH0749744A (en) * | 1993-08-04 | 1995-02-21 | Pioneer Electron Corp | Head mounting type display input device |
US6061064A (en) | 1993-08-31 | 2000-05-09 | Sun Microsystems, Inc. | System and method for providing and using a computer user interface with a view space having discrete portions |
USRE36450E (en) * | 1994-01-12 | 1999-12-21 | Brain Functions Laboratory, Inc. | Method and apparatus for automatically determining somatic state |
US5601090A (en) * | 1994-07-12 | 1997-02-11 | Brain Functions Laboratory, Inc. | Method and apparatus for automatically determining somatic state |
JP3143558B2 (en) * | 1994-02-02 | 2001-03-07 | キヤノン株式会社 | Image display method and apparatus |
JP3058004B2 (en) * | 1994-03-23 | 2000-07-04 | キヤノン株式会社 | Visual control image display device |
JP2540728B2 (en) * | 1994-03-31 | 1996-10-09 | 株式会社脳機能研究所 | Brain activity automatic determination device |
GB2295457B (en) * | 1994-11-26 | 1999-06-30 | David Kniveton Chadwick | A device for monitoring the myotatic activity of a muscle |
JP3137866B2 (en) * | 1995-02-24 | 2001-02-26 | 株式会社半導体エネルギー研究所 | Gaming equipment |
US5793354A (en) * | 1995-05-10 | 1998-08-11 | Lucent Technologies, Inc. | Method and apparatus for an improved computer pointing device |
US6580448B1 (en) * | 1995-05-15 | 2003-06-17 | Leica Microsystems Ag | Process and device for the parallel capture of visual information |
US5570698A (en) * | 1995-06-02 | 1996-11-05 | Siemens Corporate Research, Inc. | System for monitoring eyes for detecting sleep behavior |
US5682144A (en) * | 1995-11-20 | 1997-10-28 | Mannik; Kallis Hans | Eye actuated sleep prevention devices and other eye controlled devices |
US6078310A (en) * | 1996-06-26 | 2000-06-20 | Sun Microsystems, Inc. | Eyetracked alert messages |
US6025829A (en) * | 1996-10-28 | 2000-02-15 | Welch Allyn, Inc. | Image generator for video display |
US5762612A (en) * | 1997-02-28 | 1998-06-09 | Campbell; Craig | Multimodal stimulation in virtual environments |
US6110362A (en) * | 1997-11-19 | 2000-08-29 | Cohesive Technologies, Inc. | Chemical analysis |
US6931587B1 (en) * | 1998-01-29 | 2005-08-16 | Philip R. Krause | Teleprompter device |
US6204828B1 (en) | 1998-03-31 | 2001-03-20 | International Business Machines Corporation | Integrated gaze/manual cursor positioning system |
DE19835215C2 (en) * | 1998-08-05 | 2000-07-27 | Mannesmann Vdo Ag | Combination instrument |
US6243076B1 (en) * | 1998-09-01 | 2001-06-05 | Synthetic Environments, Inc. | System and method for controlling host system interface with point-of-interest data |
GB9823977D0 (en) * | 1998-11-02 | 1998-12-30 | Scient Generics Ltd | Eye tracking method and apparatus |
US6483484B1 (en) * | 1998-12-18 | 2002-11-19 | Semiconductor Energy Laboratory Co., Ltd. | Goggle type display system |
US7966078B2 (en) | 1999-02-01 | 2011-06-21 | Steven Hoffberg | Network media appliance system and method |
ES2160512B1 (en) * | 1999-09-10 | 2002-05-16 | Univ Alcala Henares | COMMUNICATION SYSTEM MAN-MACHINE BASED ON ELECTROOCULOGRAPHY. . |
US6621918B1 (en) | 1999-11-05 | 2003-09-16 | H Innovation, Inc. | Teleradiology systems for rendering and visualizing remotely-located volume data sets |
US6099124A (en) * | 1999-12-14 | 2000-08-08 | Hidaji; Faramarz | Ophthalmological system and method |
DE10037573B4 (en) * | 2000-08-02 | 2005-05-19 | Robert Bosch Gmbh | Navigation method in a motor vehicle |
GB2372683A (en) * | 2001-02-23 | 2002-08-28 | Ibm | Eye tracking display apparatus |
US20040023191A1 (en) * | 2001-03-02 | 2004-02-05 | Brown Carolyn J. | Adaptive instructional process and system to facilitate oral and written language comprehension |
US7039723B2 (en) | 2001-08-31 | 2006-05-02 | Hinnovation, Inc. | On-line image processing and communication system |
US7959674B2 (en) * | 2002-07-16 | 2011-06-14 | Medtronic, Inc. | Suture locking assembly and method of use |
DE60330980D1 (en) * | 2002-10-15 | 2010-03-04 | Volvo Technology Corp | METHOD FOR EVALUATING THE HEAD AND EYE ACTION OF A PERSON |
ES2253943B1 (en) * | 2003-05-13 | 2007-07-16 | Salvador Miravalls Renart | DEVICE CONTROL DEVICE BY BIOELECTRIC SIGNS PRODUCED BY MOVEMENT OF A MUSCLE. |
US20050143654A1 (en) * | 2003-11-29 | 2005-06-30 | Karel Zuiderveld | Systems and methods for segmented volume rendering using a programmable graphics pipeline |
US8442331B2 (en) | 2004-02-15 | 2013-05-14 | Google Inc. | Capturing text from rendered documents using supplemental information |
US7707039B2 (en) | 2004-02-15 | 2010-04-27 | Exbiblio B.V. | Automatic modification of web pages |
US7812860B2 (en) | 2004-04-01 | 2010-10-12 | Exbiblio B.V. | Handheld device for capturing text from both a document printed on paper and a document displayed on a dynamic display device |
US10635723B2 (en) | 2004-02-15 | 2020-04-28 | Google Llc | Search engines and systems with handheld document data capture devices |
US20050215916A1 (en) * | 2004-03-29 | 2005-09-29 | Fadem Kalford C | Active, multiplexed digital electrodes for EEG, ECG and EMG applications |
US9116890B2 (en) | 2004-04-01 | 2015-08-25 | Google Inc. | Triggering actions in response to optically or acoustically capturing keywords from a rendered document |
US8146156B2 (en) | 2004-04-01 | 2012-03-27 | Google Inc. | Archive of text captures from rendered documents |
US9143638B2 (en) | 2004-04-01 | 2015-09-22 | Google Inc. | Data capture from rendered documents using handheld device |
US7894670B2 (en) | 2004-04-01 | 2011-02-22 | Exbiblio B.V. | Triggering actions in response to optically or acoustically capturing keywords from a rendered document |
US20060081714A1 (en) | 2004-08-23 | 2006-04-20 | King Martin T | Portable scanning device |
US7990556B2 (en) | 2004-12-03 | 2011-08-02 | Google Inc. | Association of a portable scanner with input/output and storage devices |
WO2008028674A2 (en) | 2006-09-08 | 2008-03-13 | Exbiblio B.V. | Optical scanners, such as hand-held optical scanners |
US20060098900A1 (en) | 2004-09-27 | 2006-05-11 | King Martin T | Secure data gathering from rendered documents |
US8489624B2 (en) | 2004-05-17 | 2013-07-16 | Google, Inc. | Processing techniques for text capture from a rendered document |
US8620083B2 (en) | 2004-12-03 | 2013-12-31 | Google Inc. | Method and system for character recognition |
US8874504B2 (en) | 2004-12-03 | 2014-10-28 | Google Inc. | Processing techniques for visual capture data from a rendered document |
EP1602322A1 (en) * | 2004-06-02 | 2005-12-07 | SensoMotoric Instruments GmbH | Method and apparatus for eye tracking latency reduction |
PT1607840E (en) | 2004-06-18 | 2015-05-20 | Tobii Ab | Eye control of computer apparatus |
US20070270678A1 (en) * | 2004-06-18 | 2007-11-22 | Fadem Kalford C | Wireless Electrode for Biopotential Measurement |
US8346620B2 (en) | 2004-07-19 | 2013-01-01 | Google Inc. | Automatic modification of web pages |
US7435227B2 (en) * | 2004-09-13 | 2008-10-14 | Biocognisafe (Bcs) Technologies | Method and apparatus for generating an indication of a level of vigilance of an individual |
KR100594117B1 (en) * | 2004-09-20 | 2006-06-28 | 삼성전자주식회사 | Apparatus and method for inputting key using biosignal in HMD information terminal |
IL165586A0 (en) * | 2004-12-06 | 2006-01-15 | Daphna Palti Wasserman | Multivariate dynamic biometrics system |
US20060149426A1 (en) * | 2005-01-04 | 2006-07-06 | Unkrich Mark A | Detecting an eye of a user and determining location and blinking state of the user |
US7301465B2 (en) | 2005-03-24 | 2007-11-27 | Tengshe Vishwas V | Drowsy driving alarm system |
AU2006280946B2 (en) | 2005-08-19 | 2013-05-02 | Neuronetrix Solutions, Llc | Controller for neuromuscular testing |
US20080058681A1 (en) * | 2006-08-30 | 2008-03-06 | Casali Henry Eloy S | Portable system for monitoring the position of a patient's head during videonystagmography tests (VNG) or electronystagmography (ENG) |
US8370207B2 (en) | 2006-12-30 | 2013-02-05 | Red Dot Square Solutions Limited | Virtual reality system including smart objects |
US9940589B2 (en) * | 2006-12-30 | 2018-04-10 | Red Dot Square Solutions Limited | Virtual reality system including viewer responsiveness to smart objects |
US8341022B2 (en) * | 2006-12-30 | 2012-12-25 | Red Dot Square Solutions Ltd. | Virtual reality system for environment building |
US8073198B2 (en) * | 2007-10-26 | 2011-12-06 | Samsung Electronics Co., Ltd. | System and method for selection of an object of interest during physical browsing by finger framing |
JP5204781B2 (en) * | 2008-05-20 | 2013-06-05 | パナソニック株式会社 | Gaze detection apparatus, imaging apparatus, gaze detection method, program, and integrated circuit |
US20090307601A1 (en) * | 2008-06-05 | 2009-12-10 | Ibm Corporation | Method for reflexive securing of computer displays |
US20110128223A1 (en) * | 2008-08-07 | 2011-06-02 | Koninklijke Phillips Electronics N.V. | Method of and system for determining a head-motion/gaze relationship for a user, and an interactive display system |
KR100995885B1 (en) * | 2008-11-17 | 2010-11-23 | 휴잇테크놀러지스 주식회사 | System and Method of notifying in-vehicle emergency based on eye writing recognition |
US20100182232A1 (en) * | 2009-01-22 | 2010-07-22 | Alcatel-Lucent Usa Inc. | Electronic Data Input System |
US8418055B2 (en) | 2009-02-18 | 2013-04-09 | Google Inc. | Identifying a document by performing spectral analysis on the contents of the document |
EP2406767A4 (en) | 2009-03-12 | 2016-03-16 | Google Inc | Automatically providing content associated with captured information, such as information captured in real-time |
US8447066B2 (en) | 2009-03-12 | 2013-05-21 | Google Inc. | Performing actions based on capturing information from rendered documents, such as documents under copyright |
WO2010149823A1 (en) * | 2009-06-23 | 2010-12-29 | Nokia Corporation | Method and apparatus for processing audio signals |
US9081799B2 (en) | 2009-12-04 | 2015-07-14 | Google Inc. | Using gestalt information to identify locations in printed information |
US9323784B2 (en) | 2009-12-09 | 2016-04-26 | Google Inc. | Image search using text-based elements within the contents of images |
JP4978722B2 (en) * | 2010-08-23 | 2012-07-18 | 株式会社デンソー | Speed control device |
US20120050140A1 (en) * | 2010-08-25 | 2012-03-01 | Border John N | Head-mounted display control |
US9072481B2 (en) * | 2010-09-09 | 2015-07-07 | The Johns Hopkins University | Apparatus and method for assessing vestibulo-ocular function |
US10244988B2 (en) * | 2010-12-16 | 2019-04-02 | Nokia Technologies Oy | Method, apparatus and computer program of using a bio-signal profile |
US9185352B1 (en) | 2010-12-22 | 2015-11-10 | Thomas Jacques | Mobile eye tracking system |
US9013264B2 (en) | 2011-03-12 | 2015-04-21 | Perceptive Devices, Llc | Multipurpose controller for electronic devices, facial expressions management and drowsiness detection |
DE102011103555A1 (en) * | 2011-06-08 | 2012-12-13 | Atlas Elektronik Gmbh | Method for plan-position-indicator-representation of e.g. track symbols on graphical user interface of guidance system in navy, involves representing visual element assigned to information element to operator when signals are passed |
DE102011104524A1 (en) * | 2011-06-15 | 2012-12-20 | Ifakt Gmbh | Method and device for determining and reproducing virtual location-related information for a room area |
KR20140041570A (en) * | 2011-06-24 | 2014-04-04 | 톰슨 라이센싱 | Computer device operable with user's eye movement and method for operating the computer device |
US8885882B1 (en) | 2011-07-14 | 2014-11-11 | The Research Foundation For The State University Of New York | Real time eye tracking for human computer interaction |
US9684374B2 (en) | 2012-01-06 | 2017-06-20 | Google Inc. | Eye reflection image analysis |
US9116545B1 (en) | 2012-03-21 | 2015-08-25 | Hayes Solos Raffle | Input detection |
US9128522B2 (en) | 2012-04-02 | 2015-09-08 | Google Inc. | Wink gesture input for a head-mountable device |
US9171198B1 (en) | 2012-04-02 | 2015-10-27 | Google Inc. | Image capture technique |
US9201512B1 (en) | 2012-04-02 | 2015-12-01 | Google Inc. | Proximity sensing for input detection |
US9072465B2 (en) | 2012-04-03 | 2015-07-07 | Johnson & Johnson Vision Care, Inc. | Blink detection system for electronic ophthalmic lens |
HK1160574A2 (en) * | 2012-04-13 | 2012-07-13 | King Hei Francis Kwong | Secure electronic payment system and process |
US9448635B2 (en) | 2012-04-16 | 2016-09-20 | Qualcomm Incorporated | Rapid gesture re-engagement |
JP5661067B2 (en) * | 2012-05-29 | 2015-01-28 | 株式会社ジェイアイエヌ | Eyewear |
US9007301B1 (en) * | 2012-10-11 | 2015-04-14 | Google Inc. | User interface |
US9746915B1 (en) | 2012-10-22 | 2017-08-29 | Google Inc. | Methods and systems for calibrating a device |
JP5888205B2 (en) * | 2012-11-02 | 2016-03-16 | ソニー株式会社 | Image display device and information input device |
US9612656B2 (en) | 2012-11-27 | 2017-04-04 | Facebook, Inc. | Systems and methods of eye tracking control on mobile device |
US9265458B2 (en) | 2012-12-04 | 2016-02-23 | Sync-Think, Inc. | Application of smooth pursuit cognitive testing paradigms to clinical drug development |
KR20140072524A (en) * | 2012-12-05 | 2014-06-13 | 현대자동차주식회사 | Apparatus for measuring fatigue degree of driver's eye |
US9380976B2 (en) | 2013-03-11 | 2016-07-05 | Sync-Think, Inc. | Optical neuroinformatics |
CA3187490A1 (en) * | 2013-03-15 | 2014-09-18 | Interaxon Inc. | Wearable computing apparatus and method |
US20140358009A1 (en) * | 2013-05-30 | 2014-12-04 | Michael O'Leary | System and Method for Collecting Eye-Movement Data |
WO2014192001A2 (en) * | 2013-05-30 | 2014-12-04 | Umoove Services Ltd. | Smooth pursuit gaze tracking |
EP3019963A4 (en) * | 2013-07-12 | 2017-02-08 | University of Iowa Research Foundation | Augmentative and alternative communication |
WO2015116640A1 (en) * | 2014-01-29 | 2015-08-06 | Shazly Tarek A | Eye and head tracking device |
EP3119343A4 (en) | 2014-03-19 | 2017-12-20 | Intuitive Surgical Operations, Inc. | Medical devices, systems, and methods integrating eye gaze tracking for stereo viewer |
WO2015143067A1 (en) * | 2014-03-19 | 2015-09-24 | Intuitive Surgical Operations, Inc. | Medical devices, systems, and methods using eye gaze tracking |
JP6266417B2 (en) * | 2014-04-14 | 2018-01-24 | 株式会社ジンズ | Information processing apparatus, information processing system, and program |
US10222953B2 (en) * | 2014-04-30 | 2019-03-05 | Disney Enterprises, Inc. | Systems and methods for editing virtual content of a virtual space |
EP3154414B1 (en) | 2014-06-11 | 2021-08-04 | Dignity Health | Systems and methods for non-intrusive deception detection |
WO2016029019A1 (en) * | 2014-08-21 | 2016-02-25 | Dignity Health | Systems and methods for using eye movements to determine traumatic brain injury |
JP5689206B1 (en) * | 2014-12-02 | 2015-03-25 | 株式会社ジェイアイエヌ | Eyewear |
JP2017091327A (en) * | 2015-11-12 | 2017-05-25 | 富士通株式会社 | Pointing support device, pointing support method and pointing support program |
US9469195B1 (en) | 2015-11-17 | 2016-10-18 | International Business Machines Corporation | Adaptive, automatically-reconfigurable, vehicle instrument display |
US9452678B1 (en) | 2015-11-17 | 2016-09-27 | International Business Machines Corporation | Adaptive, automatically-reconfigurable, vehicle instrument display |
US9457665B1 (en) * | 2015-11-17 | 2016-10-04 | International Business Machines Corporation | Adaptive, automatically-reconfigurable, vehicle instrument display |
JP6426589B2 (en) * | 2015-11-27 | 2018-11-21 | 株式会社東芝 | Electro-oculogram detection device, electro-oculogram detection method, eyewear and frame |
CN105867410A (en) * | 2016-04-06 | 2016-08-17 | 东莞北京航空航天大学研究院 | Unmanned aerial vehicle earth station control method and system based on eyeball tracking |
TR201613586A2 (en) * | 2016-09-28 | 2016-11-21 | Kuersat Cagiltay | ELECTRONIC AND MOTORIZED WHEELCHAIR SYSTEM CONTROLLED BY EYE MOVEMENTS |
US10643741B2 (en) | 2016-11-03 | 2020-05-05 | RightEye, LLC | Systems and methods for a web platform hosting multiple assessments of human visual performance |
KR20180055661A (en) | 2016-11-16 | 2018-05-25 | 삼성전자주식회사 | Electronic apparatus and control method thereof |
EP3369373A1 (en) | 2017-03-04 | 2018-09-05 | Tata Consultancy Services Limited | Systems and methods for wavelet based head movement artifact removal from electrooculography (eog) signals |
CN110753567A (en) * | 2017-05-16 | 2020-02-04 | 密歇根大学董事会 | Brain health monitoring system based on eye impedance |
KR102450236B1 (en) | 2017-09-06 | 2022-10-04 | 삼성전자주식회사 | Electronic apparatus, method for controlling thereof and the computer readable recording medium |
JP6774975B2 (en) * | 2018-03-19 | 2020-10-28 | 株式会社東芝 | Eye rotation detectors, electronic devices and systems |
JP6911809B2 (en) * | 2018-05-09 | 2021-07-28 | 日本電信電話株式会社 | Estimating method, estimation program and estimation device |
US11684305B2 (en) | 2018-06-02 | 2023-06-27 | Seyedhesam Sadeghian-Motahar | Electrode array configuration on a flexible substrate for electro-oculogram recording |
US10863812B2 (en) | 2018-07-18 | 2020-12-15 | L'oreal | Makeup compact with eye tracking for guidance of makeup application |
US10860095B2 (en) | 2019-05-02 | 2020-12-08 | Cognixion | Dynamic eye-tracking camera alignment utilizing eye-tracking maps |
US11786694B2 (en) | 2019-05-24 | 2023-10-17 | NeuroLight, Inc. | Device, method, and app for facilitating sleep |
MX2021004961A (en) * | 2021-04-29 | 2022-10-31 | Inst Tecnologico Estudios Superiores Monterrey | Electronic processing device for biometric signals for the handling of actuators. |
Family Cites Families (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4474186A (en) * | 1979-07-17 | 1984-10-02 | Georgetown University | Computerized electro-oculographic (CEOG) system with feedback control of stimuli |
US4561448A (en) * | 1984-02-02 | 1985-12-31 | Colen-Kery, Inc. | Apparatus for sensing and recording biopotential electrical signals |
GB2157000B (en) * | 1984-04-02 | 1988-03-02 | Mo Nii Microchirurg | Automatic processing of electro-oculographic signals |
SU1572492A1 (en) * | 1984-04-02 | 1990-06-23 | Mo Nii Microchirurg | Method of automatic treating electrooculographic signals |
US4651145A (en) * | 1984-05-31 | 1987-03-17 | Medical Research Institute | Communication system for the disabled in which a display target is selected by encephalogram response |
GB2170910A (en) * | 1984-12-19 | 1986-08-13 | Marconi Co Ltd | Eye tracking system |
GB8701288D0 (en) * | 1987-01-21 | 1987-02-25 | Waldern J D | Perception of computer-generated imagery |
EP0301790A3 (en) * | 1987-07-24 | 1990-06-06 | BioControl Systems, Inc. | Biopotential digital controller for music and video applications |
IL93579A0 (en) * | 1990-02-28 | 1990-11-29 | Srd Medical Ltd | Apparatus for rapidly preparing to obtain biosignals |
EP0468340A3 (en) * | 1990-07-24 | 1992-12-16 | Biocontrol Systems, Inc. | Eye directed controller |
-
1993
- 1993-01-19 US US08/006,199 patent/US5360971A/en not_active Expired - Fee Related
- 1993-03-31 WO PCT/US1993/003011 patent/WO1993020499A1/en active IP Right Grant
- 1993-03-31 EP EP93908689A patent/EP0634031B1/en not_active Expired - Lifetime
- 1993-03-31 AT AT93908689T patent/ATE181605T1/en not_active IP Right Cessation
- 1993-03-31 CA CA002133416A patent/CA2133416A1/en not_active Abandoned
- 1993-03-31 AU AU39419/93A patent/AU3941993A/en not_active Abandoned
- 1993-03-31 DE DE69325448T patent/DE69325448T2/en not_active Expired - Fee Related
Non-Patent Citations (1)
Title |
---|
See references of WO9320499A1 * |
Also Published As
Publication number | Publication date |
---|---|
WO1993020499A1 (en) | 1993-10-14 |
DE69325448T2 (en) | 1999-10-28 |
DE69325448D1 (en) | 1999-07-29 |
AU3941993A (en) | 1993-11-08 |
ATE181605T1 (en) | 1999-07-15 |
CA2133416A1 (en) | 1993-10-14 |
US5360971A (en) | 1994-11-01 |
EP0634031B1 (en) | 1999-06-23 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US5517021A (en) | Apparatus and method for eye tracking interface | |
US5360971A (en) | Apparatus and method for eye tracking interface | |
CN111629653B (en) | Brain-computer interface with high-speed eye tracking features | |
JP7496776B2 (en) | Brain-Computer Interface with Adaptation for Fast, Accurate and Intuitive User Interaction - Patent application | |
Majaranta et al. | Eye tracking and eye-based human–computer interaction | |
EP1799105B1 (en) | System and method for mental workload measurement based on rapid eye movement | |
Kaufman et al. | An eye tracking computer user interface | |
US5293187A (en) | Method and apparatus for eye tracking for convergence and strabismus measurement | |
Bulling et al. | What's in the Eyes for Context-Awareness? | |
US7106204B2 (en) | Shared attention detection system and method | |
US20030046254A1 (en) | Apparatus for controlling electrical device using bio-signal and method thereof | |
US20020039111A1 (en) | Automated visual tracking for computer access | |
JP2004527815A (en) | Activity initiation method and system based on sensed electrophysiological data | |
WO1999026126A1 (en) | User interface | |
Sharma et al. | Eye gaze techniques for human computer interaction: A research survey | |
Rantanen et al. | Capacitive measurement of facial activity intensity | |
Sibert et al. | Evaluation and analysis of eye gaze interaction | |
CN115089181A (en) | Fatigue measurement method and system based on eye movement data | |
Ahmed et al. | A survey on human eye-gaze tracking (EGT) system “a comparative study” | |
Peters et al. | Modelling user attention for human-agent interaction | |
CN111933277A (en) | Method, device, equipment and storage medium for detecting 3D vertigo | |
Wilder et al. | Eye tracking in virtual environments | |
US11493994B2 (en) | Input device using bioelectric potential | |
Pazvantov et al. | Development kit for recording and processing of EOG signals for eye tracking | |
Brammi et al. | HCI Based Input Device for Differently Abled |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
17P | Request for examination filed |
Effective date: 19941031 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AT BE CH DE DK ES FR GB GR IE IT LI LU MC NL PT SE |
|
17Q | First examination report despatched |
Effective date: 19971104 |
|
GRAG | Despatch of communication of intention to grant |
Free format text: ORIGINAL CODE: EPIDOS AGRA |
|
GRAG | Despatch of communication of intention to grant |
Free format text: ORIGINAL CODE: EPIDOS AGRA |
|
GRAH | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOS IGRA |
|
RAP1 | Party data changed (applicant data changed or rights of an application transferred) |
Owner name: THE RESEARCH FOUNDATION STATE UNIVERSITY OF NEW YO |
|
GRAH | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOS IGRA |
|
RAP3 | Party data changed (applicant data changed or rights of an application transferred) |
Owner name: THE RESEARCH FOUNDATION OF STATE UNIVERSITY OF NEW |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AT BE CH DE DK ES FR GB GR IE IT LI LU MC NL PT SE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SE Free format text: THE PATENT HAS BEEN ANNULLED BY A DECISION OF A NATIONAL AUTHORITY Effective date: 19990623 Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 19990623 Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT;WARNING: LAPSES OF ITALIAN PATENTS WITH EFFECTIVE DATE BEFORE 2007 MAY HAVE OCCURRED AT ANY TIME BEFORE 2007. THE CORRECT EFFECTIVE DATE MAY BE DIFFERENT FROM THE ONE RECORDED. Effective date: 19990623 Ref country code: GR Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 19990623 Ref country code: ES Free format text: THE PATENT HAS BEEN ANNULLED BY A DECISION OF A NATIONAL AUTHORITY Effective date: 19990623 |
|
REF | Corresponds to: |
Ref document number: 181605 Country of ref document: AT Date of ref document: 19990715 Kind code of ref document: T |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: NV Representative=s name: A. BRAUN, BRAUN, HERITIER, ESCHMANN AG PATENTANWAE |
|
ET | Fr: translation filed | ||
REF | Corresponds to: |
Ref document number: 69325448 Country of ref document: DE Date of ref document: 19990729 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 19990923 Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 19990923 |
|
NLV1 | Nl: lapsed or annulled due to failure to fulfill the requirements of art. 29p and 29m of the patents act | ||
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20000331 Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20000331 Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20000331 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20000331 Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20000331 Ref country code: AT Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20000331 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: FR Payment date: 20000419 Year of fee payment: 8 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: GB Payment date: 20000420 Year of fee payment: 8 Ref country code: DE Payment date: 20000420 Year of fee payment: 8 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
26N | No opposition filed | ||
BERE | Be: lapsed |
Owner name: THE RESEARCH FOUNDATION OF STATE UNIVERSITY OF NE Effective date: 20000331 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MC Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20000930 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: MM4A |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GB Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20010331 |
|
GBPC | Gb: european patent ceased through non-payment of renewal fee |
Effective date: 20010331 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: DE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20020101 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: FR Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20020628 |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: ST |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: FR Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20010331 |