JP7153096B2 - System and method enabling communication by eye feedback - Google Patents

Description

The present invention relates to eye tracking and interfaces and user communication platforms.

Degenerative diseases are characterized by deterioration of the structure and function of body tissues. Degenerative diseases often lead to loss of control over muscles and decreased motor function, impairing a person's ability to exercise. As a result, individuals may experience severe difficulty or a complete inability to communicate verbally and/or by hand movements. Degenerative diseases do not necessarily cause a person's mental and cognitive function to decline at all or at the same rate as motor function declines. Thus, a person with full cognitive activity may lose the ability to physically and linguistically interact with their environment. Such a person's mind can be thought of as essentially trapped or confined within its body.

Some examples of degenerative diseases include Duchenne muscular dystrophy (DMD), Machado Joseph disease (MJD), multiple sclerosis (MS), muscular dystrophy (MD), Parkinson's disease and amyotrophic lateral sclerosis (ALS). is included. Amyotrophic lateral sclerosis (ALS), also known as Lou Gehrig's disease and Charcot's disease, is a disease with neuronal death that causes progressive weakness due to stiff muscles, muscle spasms, and muscle atrophy. Characterized by ALS commonly causes difficulties with speaking, swallowing, and breathing. ALS patients represent a subgroup of fully functioning, cognitively capable people who gradually lose their ability to speak and use their upper extremities.

Apart from people with degenerative diseases, other groups of people can also experience loss of physical abilities while retaining their intellectual abilities (i.e., being unable to communicate by speech while retaining full cognitive abilities). (has difficulty in For example, people with locked-in syndrome (LIS) or neurodegenerative diseases such as Parkinson's or Huntington's disease, victims of strokes and other accidents (e.g. car accidents), have breathing tubes inserted in their throats and therefore speak Patients on a ventilator who cannot move their own limbs (ventilated patients), those who are too weak or unable to move their own limbs, those who are paralyzed, or whose brain is adequate other people who are functioning in the environment but have reached a state where they are unable to interact with their environment due to technical limitations. Such individuals are therefore severely limited in their ability to communicate with others in their environment. Lack of communication can lead to feelings of detachment and isolation, situations in which the patient is stressed and unable to effectively communicate essential needs or obtain vital help (in some cases, life-threatening situations). get).

One possible way is to allow such individuals alternative forms of communication based on their eye-related characteristics. For example, a person's eye direction or focus can be used to provide commands or convey information to their surroundings. Tools exist to support such alternative non-verbal forms of communication, but they are generally relatively expensive, cumbersome, and/or immovable (fixed in position). These tools often require the use of a dedicated computer and are subject to the constant posture of the person using the tool. These tools can track eye position relative to the on-screen display and transform eye movements to mimic the movement of an on-screen mouse cursor. Thus, the patient's focus is translated into the geographic location of the pointer (or cursor) and the patient can use his/her eyes to control the computer, similar to manipulating a computer mouse. For example, the pupil is identified as a point in space, allowing a person to use their eyes to control a computer, similar to how a computer mouse is controlled. Mouse control is not always accurate, and patients with locked-in syndrome are generally allowed to move their head (or other body parts) freely, e.g., due to attached medical equipment (e.g., ventilators). Can not. Such tools that rely on determining the user's exact gaze position typically also require relatively frequent recalibration and require active user participation. The calibration process usually requires precise fixation of the user at multiple positions in front of the screen and is therefore relatively laborious and time consuming. Also, the calibration process can often require the assistance of a third party, which can be costly or inconvenient.

U.S. Pat. No. 8,120,577 to Bouvin et al., entitled "Eye Tracker with Visual Feedback," discloses sending control commands to a computer in response to eye tracker-detected movement sequences of gaze points on a graphic display. Disclose input.

U.S. Patent Application Publication No. 2013/0293488 to Na et al., entitled "Mobile Terminal and Control Method Thereof," describes a camera configured to capture a user's eye gestures and a method based on the captured eye gestures. , a controller configured to display an eye-tracking mode guide on a display and to select a particular function included in the eye-tracking mode guide based on a captured second eye gesture. is doing.

U.S. Pat. No. 8,593,375 to Maltz, entitled "Eye Gaze User," is for a gaze control device designed to accommodate the trade-off between angular accuracy and temporal averaging for gaze direction sensors. A software controlled user interface is disclosed.

U.S. Pat. No. 8,593,375 to Maltz, entitled "Eye Gaze User," is for a gaze control device designed to accommodate the trade-off between angular accuracy and temporal averaging for gaze direction sensors. A software controlled user interface is disclosed.

US Pat. No. 6,456,262 to Bell entitled "Microdisplay with Eye Gaze Detection" discloses a microdisplay in which displayed image elements can be selected by gaze.

U.S. Patent Application Publication No. 2006/0061544 to Min et al., entitled "Apparatus and Method for Inputting Keys Using Biological Signals in Head Mounted Display Information Terminal," describes an EOG (Electronic Eye Group) that detects and receives biological signals as key inputs. Fig. ) Discloses a head-mounted display in which a user inputs keys selected according to the user's biosignals detected using a biosignal detection unit comprising an input unit and an EMG (electromyography) input unit.

An example of a system operated using eye tracking is "EyeWriter™" (notimpossiblelabs.com or http://www.eyewriter.org/). EyeWriter™ was developed for graffiti artists with ALS.

Thus, according to one aspect of the present invention, a system is provided that enables users to communicate using eye-based feedback. The system includes a relative eye orientation sensor, a selection interface and a processor. The selection interface is configured to selectably present a series of communication options to the user. A relative eye orientation sensor is configured to detect light reflected from the user's eyes and provide a correlation signal. A processor is communicatively coupled to the relative eye orientation sensor and selection interface. The processor is configured to determine a relative eye orientation of the user by receiving and processing the correlation signal, the relative eye orientation being at least a relative eye pupil relative to the user's head orientation. including orientation. The processor is further configured to determine a selected communication option based on the determined relative eye orientation and provide instructions for performing the selected communication option. The system may further include at least one light source for illuminating the user's eyes. The light source may be an infrared (IR) light source, and the relative eye orientation sensor may detect IR light reflected from the eye. The selection interface may include visual, audible, and/or tactile interfaces. The light sources and/or relative orientation sensors may be coupled to wearable headgear worn by the user. The selection interface may operate on a mobile computing device. The relative eye orientation sensor may include at least one image sensor configured to capture an image of the user's eye, wherein the processor determines the relative eye orientation through image processing of the captured image. do. The processor may determine the relative eye orientation by detecting the instantaneous relative orientation of the pupils or by detecting transient changes in the pupils. The processor may be communicatively wirelessly coupled to the relative eye orientation sensor and/or the selection interface. Communication options include audible alarms, menu selections, language selections, confirmation messages, sentences, phrases, words, syllables, characters, mobile computing device activation selections, visual interface deactivation selections, instructions to send email/SMS/MMS, and/or Or it may contain instructions for executing a computer application. The visual interface may display at least one communication option through color adjustments, shape adjustments, and/or symbolic images.

Thus, according to another aspect of the invention, a method is provided for enabling a user to communicate using eye-based feedback. The method comprises the steps of selectably presenting a series of communication options to a user having a selection interface and detecting and correlating light reflected from the user's eyes using relative eye orientation sensors. and a procedure for providing a signal. The method further includes determining the relative eye orientation of the user by processing the correlation signal, wherein the relative eye orientation is at least the relative eye orientation of the pupils of the eyes to the orientation of the user's head. including orientation. The method further includes determining a selected communication option based on the determined relative eye orientation, and providing instructions for performing the selected communication option. The method may further comprise illuminating the user's eye with at least one light source. The eye may be illuminated with IR light, and the relative eye orientation sensor is configured to detect the IR light reflected from the eye. Communication options may be presented by the selection interface in a visual, audible, and/or tactile manner. The step of detecting light reflected from the user's eye may include capturing an image of the eye using at least one image sensor, the relative eye orientation being determined by image processing of the captured image. be done. Relative eye orientation may be determined by detecting the instantaneous relative orientation of the pupils or by detecting transient changes in the pupils. Communication options include audible alarms, menu selections, language selections, confirmation messages, sentences, phrases, words, syllables, characters, mobile computing device activation selections, visual interface deactivation selections, email/SMS/MMS sending instructions, and/or It may also include instructions for executing a computer application. The method includes detecting relative changes in the pupil of the eye by comparing successive captured images and tracking a region of interest indicating the position of the pupil by marking the pupil in the captured images. , to set the reference point as the center of the 2D grid, transform the eye movements into coordinates on the grid, and to construct the reference point coordinate system when the pupil is determined to be at the center of the grid. The procedure may further include calibrating by selecting reference points and determining the range of motion of the user.

The invention will be better understood and appreciated from the following detailed description in conjunction with the drawings.

FIG. 1 is a schematic diagram of a system that enables users to communicate using eye-based feedback, constructed and operative in accordance with one embodiment of the present invention. 2 is a diagram of a settings menu and a word menu displayed on the display of FIG. 1; FIG. 3A is an illustration of a letter group menu displayed on the display of FIG. 1; FIG. FIG. 3B is an illustration of the letter group menu of FIG. 3A with the image displayed on the display divided into different sections and selections made. 4 is an illustration of a textual submenu displayed on the display of FIG. 1; FIG. FIG. 5A is an illustration of an image of a user's eye captured by the camera of FIG. 1 with the pupil in the upper quadrant. FIG. 5B is an illustration of an image of a user's eye captured by the camera of FIG. 1 with the pupil in the lower quadrant. FIG. 5C is an illustration of an image of a user's eye captured by the camera of FIG. 1 with the pupil in the left quadrant. FIG. 5D is an illustration of an image of a user's eye captured by the camera of FIG. 1 with the pupil in the right quadrant; FIG. 5E is an image of a user's eye without a pupil captured by the camera of FIG. FIG. 5F is an illustration of an image of a user's eye with the pupil in the center captured by the camera of FIG. FIG. 6 is a flow diagram of a method for enabling users to communicate using eye-based feedback, operating in accordance with one embodiment of the present invention.

The present invention overcomes the shortcomings of the prior art by providing systems and methods that allow users to communicate by eye-based feedback. Specifically, the user can manipulate the relative orientation of his/her eyes by using very basic eye gestures in the form of indicating the general direction in which he/she is looking. Via a visual/audio/tactile interface, he/she can select communication options presented to him/her. The system indicates the orientation of the pupil relative to the eye socket or the orientation of the head by detecting infrared (or other light) reflections from the eye, for example by using an IR light source and an IR camera or other sensor. determine the relative eye orientation of the Accordingly, the present invention provides an alternative form of communication that is sophisticated, accessible, relatively inexpensive, and mobile, thereby enhancing cognitive awareness in patients with locked-in syndrome (LIS), ALS patients, and the like. While maintaining, it helps improve the quality of life of individuals characterized by difficulty or inability to communicate in verbal and/or sign language. More generally, the present invention can be used by any individual, including those who do not exhibit restrictions on communicating in other ways. Additionally, the system allows communication and operation of the device while the user is otherwise engaged.

FIG. 1 is a schematic diagram of a system that enables users to communicate using eye-based feedback, constructed and operative in accordance with one embodiment of the present invention, hereafter generally referenced at 100. See System 100 includes a computing device 114 with primary display 102 , image processor 104 , light source 106 , camera 108 , audio unit 112 , and auxiliary display 124 . Processor 104 is communicatively coupled with primary display 102 , camera 108 , audio unit 112 and computing device 114 . Computing device 114 may be communicatively coupled with processor 104 via a wired or wireless communication link 116 (eg, a two-way Bluetooth connection).

Light source 106 is configured to illuminate at least one eye of user 150 by emitting light 118 . Light source 106 may be configured to emit non-visible light, such as light in the infrared (IR) spectrum. IR light is particularly desirable because it can be used effectively even in dark rooms with poor lighting conditions and because it allows effective illumination of the eye. Alternatively, other light wavelengths (non-IR light) may be used to illuminate the user's 150 eyes. Light source 106 generally emits eye-safe light (eg, in visible or infrared wavelengths). Eye-safe light passes through the various optical elements of system 100 in the desired optical path (i.e., is transmitted through or reflected from the optical elements in the desired direction) without interfering with the visible light path of the displayed image. be able to. As an example, light source 106 may be embodied by four IR LED lights arranged in an array (eg, in a square pattern with each LED at each corner of the square). Alternatively, light source 106 may be embodied by ambient light such as solar radiation or room lighting.

Camera 108 is configured to capture an image of at least one eye of user 150 . The line of sight (LOS) of camera 108 is aligned toward the general direction in which user 150 is facing. The field of view (FOV) of camera 108 is related to the range of possible head movements by user 150 while keeping the eye within the image frame. The system 100 may include multiple cameras 108 with different FOVs that allow imaging of a wider overall FOV than possible with a single camera 108 .

The camera 108 provides an image representation of a real-world scene, including acquisition of any form of electromagnetic radiation in any wavelength range (e.g., light in the visible or invisible spectrum, ultraviolet, infrared, radar waves, microwaves, RF, etc.). can be any kind of device capable of obtaining For example, camera 108 may be a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) camera operating in the visible to near-infrared (NIR) spectrum. Generally, camera 108 operates in a spectral region that matches or at least partially overlaps the operable wavelength range of light source 106 . The main components of camera 108 are filter 111, image sensor 110, lens (not shown), and electronics (not shown). Image sensor 110 may be configured to detect only selected wavelengths of light (ie, a narrowband sensor). Alternatively, filter 111 may be configured to block light other than desired wavelengths from reaching image sensor 110 (eg, with a bandpass filter).

Sensor 110 may be, for example, a photodiode, CCD, or another type of photodetector. Camera 108 generally operates to capture at least one image frame, such as a series of consecutive image frames representing a video image that can be converted into an electronic signal for subsequent processing and/or transmission. By way of example, camera 108 may be embodied by a modified webcam, digital camera, and/or cell phone camera. Accordingly, the term "image" as used herein refers to any form of output from the image sensor described above, including any optical or digital representation of a scene captured in any spectral region, and is a single It includes both image frames and sequences of image frames (ie, "video images").

Although camera 108 is provided as an example of one type of sensor used to determine the relative eye orientation of user 150, system 100 may alternatively substitute camera 108 for a different type of sensor. A sensor may be included. For example, system 100 may alternatively include at least one separate optical sensor positioned individually or in a suitable configuration (such as a sensor array) with respect to user 150 . Accordingly, determination of the relative eye orientation of user 150 is based on analysis of reflected light detected by the separate sensor, rather than image analysis of camera images, as further described below. may be done.

Primary display 102 and auxiliary display 124 are configured to display visual information viewable by user 150 . The display information may include graphic control elements having selectable options, for example in the form of graphic menus or tables containing alphanumeric characters and/or symbols (as further detailed below). Primary display 102 includes, but is not limited to, computer monitors, television monitors, tablet or cell phone displays, plasma or LCD screens, light emitting diode (LED) displays, three-dimensional viewing displays (e.g., holographic displays), and the like. not), and may be any device or mechanism capable of presenting visual information using any suitable format or underlying technology. Primary display 102 may be embodied by a head-mounted display (HMD) embedded within a wearable device worn by user 150, such as a helmet, headband, visor, glasses, goggles, or the like. System 100 may alternatively include audible and/or tactile interfaces that present information in an audible and/or tactile perceptible format in addition to or in place of primary display 102 and auxiliary display 124.

Processor 104 receives instructions and data from various system components. Processor 104 is also configured to perform image processing and analysis on image frames captured by camera 108 and extract information from image features. Processor 104 may be located remotely from other components of system 100 . For example, processor 104 may be part of a server, such as a remote computer or remote computer system or machine accessible over a communication medium or network. Alternatively, processor 104 may be located adjacent to user 150 and/or integrated within other components of system 100 . Processor 104 may be embodied by a low-cost processor with sufficient computing power (eg, Odroid U3 or Raspberry pi2) and may include a text-to-speech engine and Bluetooth functionality.

Audio unit 112 is configured to produce an audible output. The audible output may include various types of sounds of varying tones or intensities, ranging from simple beeps or alarm-type audio notifications to more complex audio patterns. System 100 may generally include multiple audio units 112 that may be configured to produce multiple forms of audible output. Audio unit 112 may be embodied by at least one speaker, such as headphones.

Computing device 114 may be embodied by a mobile computing device such as a smart phone, tablet computer, and/or digital music player, or a stationary computing device such as a desktop computer.

System 100 may include or be used with a support frame (not shown) for supporting the elements of system 100 near user 150 . The support frame may be embodied by spectacles or eye-glasses. The support frame may support light source 106 and camera 108 using strong and flexible movable arms. The flexible movable arm has sufficient elasticity and stiffness to support the camera 108, but is flexible enough to be bent to adjust the imaging range, position and/or viewing angle of the camera 108. It can be made from any material you have. Lightweight materials may be used to reduce weight and pressure on user 150 . For example, the camera 108 may be attached via wires to headgear worn on the user's 150 face and pointed at the user's 150 face such that the camera 108 images at least one eye of the user 150. may This alignment may be automatically or manually readjusted to maintain eye alignment during changes in user's 150 head movements or to accommodate other physical requirements of user 150. may Since a LIS patient may be on a ventilator or other medical equipment, the operability of the support frame should be minimized while minimizing the hindrance or restriction that would occur if the position of system 100 were completely fixed. , such patients may be able to conveniently utilize the system 100 . Also, the support frame may be adjustable and/or configured to meet the requirements of a particular user 150, such as to be usable by people with different types of ventilators. good. An alternative technique for providing stability and support is by mounting elements of system 100 to adjacent counters, vehicles, or wheelchairs.

Components and devices of system 100 may be based on hardware, software, or a combination thereof. It is understood that the functionality associated with each device or component of system 100 may be distributed among multiple devices or components, which may reside in a single location or multiple locations. For example, the functionality associated with processor 104 may be distributed among multiple processing units (such as a dedicated image processor for image processing functions). System 100 may optionally include and/or be associated with additional components not shown in FIG. 1 that enable implementation of the disclosed subject matter. For example, system 100 may include a power supply (not shown) that provides power to the various components, and may also include a memory or storage unit (not shown) for temporary storage of image frames or other types of data. may contain.

For illustrative purposes, operation of the system 100 will now be described from the perspective of a user 150 characterized by difficulty or inability to verbal communication, such as a LIS patient. User 150 sees visual information presented on primary display 102 and/or auxiliary display 124, such as multiple command selections. Camera 108 captures images of the eyes of user 150 illuminated by light source 106 and processor 104 receives and processes the captured images to determine the relative eye orientation of user 150 . The relative eye orientation is the general direction in which the user 150 is looking relative to the direction he/she is facing (e.g., "up," "down," "left," "up," "down," "left," "right", "front"). Thus, the relative eye orientation can be viewed as the orientation of the pupil in the eye socket or the orientation of the pupil relative to the orientation of the user's head. For example, the processor 104 determines the instantaneous relative orientation of the pupil (relative to the eye socket or orientation of the user's 150 head) by comparing the captured image of the eye to at least one previous captured image; or to detect transient changes in pupil orientation. Alternatively, processor 104 processes reflected signals from user's 150 eyes detected by a separate optical sensor, such as a separate IR sensor, rather than images captured by camera 108 . For example, the processor 104 compares each detection of multiple separate sensors placed at different locations around the user's eye (e.g., four sensors evenly distributed around the circumference of the eye) to determine the sensor detection (e.g., which sensors detected the light reflected from the eye, the intensity and/or angle of arrival of the detected reflections), and the relative proximity of the pupil to each sensor, user's 150 relative You may determine the orientation of your eyes.

Display 102 presents multiple options via an option selection table or menu. User 150 indicates an option to select by making an eye gesture such that his/her relative eye orientation correlates with the relative orientation of that option presented on display 102 . Accordingly, processor 104 determines the relative eye orientation of user 150 and determines the option selected by user 150 based on the determined relative eye orientation. For example, if the relative eye orientation is determined to be "right" (i.e., the orientation of the pupil is to the right within the user's 150 eye socket), then the option displayed on the right side of the menu is selected. considered to be User selection can be made without requiring determination of the user's exact gaze direction or line of sight (i.e., the particular point that user 150 is looking at with respect to a particular coordinate system, such as determining a unique coordinate position on display 102). or without determining spatial coordinates), match the very approximate (vague) direction in which the user 150 is looking (e.g., "right", "left", "up", "down", "front") Note that the determination is based only on relative eye orientation.

Processor 104 provides instructions to display 102, audio unit 112, and/or computing device 114 to perform at least one action based on user's 150 determined selection. These actions are, for example, audibly outputting a word, letter, phrase, sentence, or an alarm warning siren; displaying a different menu screen or one or more menu options; displaying text; It may also generate and send text or email messages and/or run certain applications on computing device 114 .

The selection process of system 100 will now be further described with reference to FIGS. The exemplary display screen 200S depicted in FIG. 2 shows two separate menu portions. The upper menu portion 200 is a “setting menu” arranged to allow the user 150 to select or change setting parameters of the system 100 . The lower menu portion 202 is a "word menu" arranged to allow the user 150 to select a particular word or series of words (phrases) from a predetermined set of keywords and phrases. One or more setting options 200A, 200B, 200C, 200D, 200E and one or more word options 202A, 202B, 202C, 202D, 202E are displayed on the display 102, 124 at any given moment. Selecting an option may cause a submenu with additional options to be displayed.

In the illustrated example, option 200A represents "Language Options" which brings up a sub-menu of different language options. Option 200B represents a "standby option" that switches system 100 to a standby (ie, off or low power) mode of operation until reactivated by user 150, such as by performing a particular eye gesture or instruction. Option 200C represents a "help option" that brings up a sub-menu of different options designed to assist user 150 (eg, explain how to use a particular aspect or element of system 100). Option 200D represents an "alert option" which, when selected, causes audio unit 112 to emit an alarm or warning signal. Option 200D is particularly useful when user 150 is in a critical and/or life-threatening situation. An alarm or warning siren may be activated by user 150 making a specific eye gesture or instruction, such as winking four times in quick succession. Option 200E represents a “mobile connectivity option” that establishes communication link 116 between processor 104 and computing device 114 that allows user 150 to run program applications on computing device 114 .

With reference to word menu portion 202, option 202A represents the word "hello", option 202B represents the word "goodbye", option 202C represents the word "thank you", and option 202D represents the word "thank you". Represents the word "help me". For example, selection of option 202D, like setting option 200D, may cause audio unit 112 to emit an alarm signal or warning siren. Option 202E represents the word "application". For example, selection of option 202E, similar to configuration option 200E, causes processor 104 to establish communication link 116 with computing device 114 to allow a user to run a program application on computing device 114. can be ordered to

In one exemplary embodiment, only a single option from the "Settings" menu portion 200 or the "Words" menu portion 202 is displayed on the displays 102, 124 at any one time, allowing the user 150 to focus on the appropriate eye. Performing a movement or gesture allows selecting the currently displayed menu option or continuing to view other available menu options. For example, individual options from each menu portion 200 , 202 may be presented in periodic sequence depending on the relative eye orientation of the user 150 . Specifically, the user 150 can first select the “setting menu” 200 by looking upward (U). Subsequently, by looking to the left (L1), the user 150 may cycle through the "setting options" presented on the displays 102, 124 in a first order, and to the right (R1). Viewing allows the user 150 to cycle through the "setting options" in reverse order. For example, displays 102, 124 initially present only setting options 200A and 202A. By looking up (U) and then in the left (L1) direction, user 150 can instruct system 100 to display setting option 200E (eg, instead of setting option 200A). . Alternatively, by looking up (U) and then looking in the right (R1) direction, user 150 can instruct system 100 to display option 200B (eg, instead of option 200A). . Similarly, by first looking down (D), the user 150 can select the “word menu” 202 . Subsequently, the user 150 may, for example, look to the left (L2) to cause the word options to be repeatedly displayed in a first order, or to the right (R2) to cause the word options to be repeatedly displayed in a second order, and so on. , the "word option" can be repeatedly displayed on the display. For example, by looking down (D) and then in the left (L2) direction, user 150 may instruct system 100 to display word option 202E (eg, instead of option 202A). can. Alternatively, by looking down (D) and then looking in the right (R2) direction, user 150 can instruct system 100 to display option 202B (eg, instead of option 202A). . Note that alternative directional configurations are equally applicable. For example, the user 150 may alternatively cycle through the menu options by directing their gaze upward and downward instead of left and right, and the displays 102, 124 are arranged in an up-down configuration. Instead, different menu portions 200 and 202 may be presented in parallel.

The user 150 closes (winks/blinks) their eyes, or gazes in a particular direction, for example obliquely or at an oblique angle (eg, a 45° angle) to the forward eye orientation, etc. Options can be selected from one of the menus 200, 202 by using predetermined eye-based gestures or indications. User 150 can select the appropriate eye-based gesture to be used during the initialization process of system 100 to ensure that he/she successfully performs the selection process (e.g., one Some users may find it difficult to blink, so may choose an alternative gesture). The selected gesture may be a pre-selected default option to eliminate the need for pre-setup or initialization of system 100 prior to use. For example, the user 150 may look down (D) and then look in the left (L2) and/or right (R2) directions until he/she reaches the desired option (e.g., option 202C). The different options of menu 202 can be cycled through, and then option 202C (the word "thank you") can be selected by looking down (D) and gesturing (eg, winking). can. Selection of a word in word menu 202 instructs audio unit 112 to provide an audible indication of the selected word. For example, selection of option 202C causes people in the vicinity of user 150 to hear the word “thank you” audibly emitted from audio unit 112 .

A temporary visual indication, such as highlighted or bolded text or a colored frame, may indicate that the option has been initially or tentatively selected, but has not yet been validated by user 150, for example. may be applied to the options specified. A confirming eye-related gesture or instruction (such as blinking twice) is then required to authenticate the user selection before executing the associated functionality associated with the selected option. good too. Additionally, shapes or colors may be used to help distinguish between various options within the presented menu. Alternatively, a provisional user selection may be audibly indicated, for example, by directing a partial audio output to user 150 . For example, the initially (tentatively) selected menu option is broadcast to only a single channel of the audio headphones worn by the user 150, providing mono sound reproduction (e.g., right ear/right ear of the headphone). by broadcasting only to the channel). If the user 150 wants the word to be heard in his environment, the selected word will play the appropriate sound (or sounds) associated with the selected option on the audio unit 112 and Optionally authenticated by additional eye-related instructions or gestures (e.g., blinking) that are broadcast on the other audio channel of the headphones worn by the user 150 (e.g., the left ear/left channel of the headphones). be. Between different audio units 112 (e.g., between headphones that can only be heard by the user and speakers that can be heard all around), and between different audio channels of a single audio unit (i.e., mono broadcast), partial By selectively directing the audible display, user 150 is able to perceive and control the audible feedback broadcast by system 100 .

Audio unit 112 may alternatively present available selection options to user 150 in an audible format instead of (or in addition to) visual representation by displays 102, 124. FIG. For example, a list of setting options in setting menu 200 and/or word options in word menu 202 may be provided so that user 150 hears the available options along with audible indications of different eye-related gestures associated with each option. , may be broadcast to user 150 through headphones. For example, user 150 may hear the following audio sequence prompting the user to select: "Left--Hello, Right--Bye, Up--Yes, Down--No, Blink--Repeat Menu." Alternatively, the user 150 can simply listen to a single option, eg "Language". One or more options may be broadcast only to users 150 . The user 150 can navigate between different option menus or select options by performing designated eye-related gestures. For example, user 150 may use a glance to one side (eg, right or left, up or down) to cycle through different word options, and may use a blink or wink gesture to indicate selection. Following user selection, the associated audio output may be publicly broadcast to the surroundings.

As such, the selection interface of system 100 may be a visual interface (visual menus presented on displays 102, 124), an audible interface (audible menus provided by processor 104 and audio unit 112) and/or a tactile interface (not shown). It's okay. Thus, the system 100 can present selection menus equivalent to the settings menu 200 and word menu 202 (FIG. 2) without the displays 102,124. Elimination of displays, as well as stressful devices that can be placed next to the patient's bed, allow recovering patients (e.g., after surgery) to ask nurses or communicate with family members. It is understood that doing so can be particularly beneficial in the setting of a medical institution. Additionally, the elimination of displays 102, 124 helps reduce the number of components of system 100, which can lower overall cost and provide greater mobility and availability.

Referring to exemplary display screen 200S (FIG. 2), displays 102, 124 alternatively display the other menu 200, 202 (or another available menu not currently displayed) instead. Only a single menu portion 200, 202 may be selectively displayed at a given moment, while each menu 200, 202 presents the selectable options that are enabled. For example, settings menu 200 may include a "switch to words menu" option (not shown) and, correspondingly, words menu 202 may include a "switch to settings menu" option (not shown). Allows toggling between each of the two menus 200,202. Such a display configuration helps simplify the selection process by limiting the user's 150 eye movements to only left or right directions (and eliminating the need for upward and downward eye movements). can be useful. The ability to cycle through the displayed menus 200, 202 simply by changing his/her gaze in a general direction (e.g., left or right) is relatively easy and quick to use for the user 150. It will further be appreciated that it allows access to all of the menu options. For example, if a particular menu consists of 10 possible options, any option can be reached within 5 "cycles" of the menu (e.g., word menu 202 contains 5 options). In the example of FIG. 2, option 202D can be reached by starting with option 202A and repeating left two times or right three times). The vocabulary provided in word menu 202 may be limited to a certain number of predetermined words found to be of high importance and high frequency to the target audience or user at issue. The available word options of a word menu 202 may be partially or fully pre-selected and/or edited by user 150 . Each audio output associated with the vocabulary of word menu 202 may be pre-generated (e.g., by pre-recording various audio files corresponding to spoken words and/or phrases) prior to operation of system 100. by), or may be generated in real-time (eg, by converting text to speech or using a phonetic transcription speech synthesizer).

As noted above, certain menu options may indicate different sub-menus that can be displayed, such as word menu 202, which includes menu options that indicate letter group menus. Alternatively, selection of mobile connectivity options 200E, 202E in word menu 202 causes processor 104 to form communication link 116 with computing device 114 running an application that includes a different menu or sub-menu, such as a letter group menu. can be done.

With reference now generally to 300 of an exemplary display screen 300A on a display 102, 124 presenting an example letter option menu including different letter group options 302, 304, 306, 308, 310 for selection. See one FIG. 3A. Letter group menu 300 may be used to compose written text or messages, or to convey the spelling of words that are conveyed or added to word menu 202 . In the illustrated example of FIG. 3A, option 302 represents the letters "ABCDE", option 304 represents the letters "FGHIJ", option 306 represents the letters "KLMNO", and option 308 represents the letters "KLMNO". Represents the letters "PQRST" and option 310 represents the letters "UVWXY". Character group options 302, 304, 306, 308, 310 are presented in a cross-shaped configuration with each character group located in a separate area (i.e., top, bottom, left, right, or center) of display 102, 124. be. Similar to the selection of menu options described above with reference to FIG. 2, user 150 can select the available options of character group menu 300 using only relative eye orientation and/or eye-related gestures. Options can be selected. For example, while viewing letter group menu 300 on display 102, 124, user 150 may select option 302 by gazing upward (U) and option 302 by gazing left (L). 304 can be selected, option 308 can be selected by gazing right (R), or option 310 can be selected by gazing down (D). User 150 may close (wink/blink) one or both eyes, or gaze at an oblique or oblique angle (eg, a 45° angle) relative to the forward relative eye orientation, etc. , the center option 306 can be selected by performing a predetermined eye-related gesture or indication. Options may be conveyed by voice unit 112 announcing the characters described above.

FIG. 3B is an illustration of an exemplary display screen 300B on displays 102, 124 presenting the exemplary letter group menu 300 of FIG. 3A divided into different sections with selection of the letter "G"; refer.

Selection of a character group option (eg, 302 or 304) of the character group menu 300 may result in the display of another submenu (ie, 400) representing individual characters of the selected character group. For example, when the user 150 gazes up to select the letter group option 302 (the letters ABCDE), the display 102, 124 displays the associated letter ABCDE submenu. Hereafter, an exemplary display screen 400A on displays 102, 124 presenting a letter submenu of ABCDE, generally referenced 400, with different individual letter options 402, 404, 406, 408, 410 for selection. Please refer to FIG. 4, which is a diagram of. Character submenu 400 corresponds to selected character group option 302 (FIG. 3A). In the illustrated example of menu 400, option 402 represents the letter 'A', option 404 represents the letter 'B', option 406 represents the letter 'C', and option 408 represents the letter 'C'. option 410 represents the letter "D" and option 410 represents the letter "E". Similar to character group menu 300, individual character options are also presented in a cross-shaped configuration with each character group located in a separate area (i.e., top, bottom, left, right, or center) of display 102, 124. be done. User 150 can select available options of individual character submenu 400 using relative eye orientation and/or eye-related gestures. For example, the user 150 can select option 402 by gazing upwards (U), select option 404 by gazing leftwards (L), and select option 404 by gazing rightwards (R). Option 408 can be selected by gazing, and option 410 can be selected by gazing down (D). User 150 may select center option 406 by performing a predetermined eye-related gesture or instruction, such as closing one or both eyes (winking/blinking).

Selection of individual letters of the submenu 400 may require confirmation by the user 150, a visual representation of the selected letter displayed on the display 102, 124 (e.g., highlighting the selected letter or by presenting a visual edge around it). Alternatively or additionally, a phonetic representation of the selected character may be generated by the phonetic unit 112 . The selected characters may also be added to text messages or text-based instructions being composed by user 150 . For example, looking up and selecting option 402 provides a visual representation of the letter "A" displayed on display 102 and causes user 150 to enter the letter "A" into the text of the message being composed. while instructing the voice unit 112 to produce the sound of the letter "A". The completed word may be added to word menu 202 and saved.

Note that character group menu 300 or individual character menu 400 may alternatively be configured to display any number of options (not necessarily five options). For example, if user 150 has difficulty performing a suitable eye gesture to indicate selection of the "middle" option, only four options may be displayed. In this way, visually presented menus can be more general relative eye orientation (e.g., up, down, left or right) while removing the need for additional eye-related gestures. It can be configured to obtain user selections based solely on the selection. The central area of the menu containing only four options may be left empty or may display the title of the menu (ie, no selectable options). Alternatively, system 100 may provide four diagonal directions (upper right, upper left, lower right, lower left) to indicate selection, for example, for individuals who have difficulty controlling blinking or experience involuntary blinking. ) and an optional straight direction (eg, straight down) may be specified.

Different menu options, in addition to their different relative positions on the displays 102, 124, have unique visual effects or characteristics to provide additional visual distinction for each displayed menu option. may be displayed as For example, each menu option may be displayed with a different color shape or border style so that the different options can be more easily recognized, identified, and distinguished from each other by the user 150. The menus may include a "return option" that allows the user 150 to return to a previously selected option, a previously displayed menu, or a preset determined action. Alternatively, system 100 may be configured to allow user 150 to automatically return to a previous or default menu following final selection of a menu option. As noted above, for selected options, an indication may be provided that the option was first selected. The indication may be, for example, highlighted or bolded text, a color frame, and/or an audible indication emitted to at least one ear of headphones worn by user 150 . System 100 then authenticates or confirms selection of the selected option, thereby providing visual and/or audio indications to inform user 150 that the option has been selected and/or which option has been selected. It may wait for a specified confirmation gesture (such as blinking twice) to do so.

Eye-related gestures for effecting option selection by user 150 may include individual actions or sequences of gestures. Such a sequence may also avoid presenting a submenu while allowing direct selection of available options in a submenu associated with the presented menu. For example, the user 150 performs two consecutive predetermined eye-related gestures while viewing the "character group menu" 300 of FIG. 3A without displaying the "individual character submenu" 400 of FIG. allows you to select individual characters. A first gesture selects a particular character group (e.g., character group 308), and then a second gesture, which may be based on the color or position of individual characters within that character group, selects a particular character group. (eg the letter "S"), thereby avoiding the display of the individual letter submenu 400. For example, individual characters within each displayed character group 302, 304, 306, 308, 310 may be color-coded to correspond to each direction, which may be indicated by color indicators displayed in the associated direction. good. For example, the first letters 'A', 'F', 'K', 'P', and 'U' of each letter group may all be depicted in one color (e.g., orange) and may be drawn in a particular direction. An orange color indicator represented by an arrow pointing (eg, up) may be displayed on the display 102,124. Thus, by looking first to the left (L), user 150 selects option 304 (ie, the entire letter group "FGHIJ") and then up (UP) (ie, toward the orange indicator). , the user 150 selects a particular letter (eg, the letter “F”) from among the letter group 304 . The colors used to display the selected menu options are easily discernible and highly visible to ensure that even users 150 characterized by visual or sensory impairments can perceive them. It may be a specific color that is easy to wear. The graphical elements of the menu items displayed by the displays 102, 124 are not limited to text or alphanumeric characters, such as "spacebar icon", "backspace/delete icon", "back or return to main menu icon", etc. Note that various types of symbols, icons, or images representing various actions or selectable options of may be included. With respect to visual output, user 150 may be provided with an option menu having multiple images, symbols, or pictures (eg, plate of food, glass of water, car, wheelchair, etc.). The selected picture may be displayed to the user's surroundings on an external screen, such as auxiliary display 124, or sent in a message to another device (e.g., to a friend's or nurse's cell phone). good too. Alternatively, selection of a picture can result in an appropriate audio output, for example selection of a picture of a plate of food results in an audible output of the word "food".

The system 100 may include a word prediction application that helps predict and suggest complete words based on the first few letters selected by the user. For example, if the user 150 selects the following letter sequence: "HEL", the system 100 will present the suggested word on the display 102, 124 or by using the voice unit 112 , "help" or "hello". System 100 may even utilize whole word or sentence completions based on the context determined by processor 104 . The user 150 can cycle through different suggested complete words on the display, for example, by looking left or right. Suggested word selection by the user 150 can be made using appropriate eye-related gestures or indications. In this manner, word selection by the user 150 can be accelerated and improved in accuracy (eliminate the need for the user 150 to continue selecting additional letters and avoid subsequent inadvertent erroneous selection by the user). by avoiding character selection). The completed word can be audibly or visually indicated and, if applicable, saved or transferred as an audio and/or text file. The word prediction function may be based on language models and vocabulary libraries and is adaptive, such as learning a user's language habits to improve future suggested word completions associated with that user. may be

Rather than suggesting words by conveying the spelling of individual letters, user 150 may search for the desired word directly on the displayed "word menu." Such menus may include different options representing actual words, or categories of words grouped by topic or subject (eg, "food", "clothes", "color", etc.). Selection of the first displayed word option can cause the display of a sub-menu with logical word options following the first selected word. For example, a word menu may initially present five different categories. When a particular category is selected, a submenu of 5 new words within that category is presented. Each of these new words may lead to additional subcategories, and so on (eg, "clothes"-"shirts"-"dress shirts"). In this way, a fairly large vocabulary of possible words is made available to the user 150 . For example, assuming each word menu screen presents a total of 5 options, user 150 could potentially access over 3000 different words using only 5 eye-related gestures. Yes (ie, if each option on the first four menu screens represents a subcategory of word options).

The user 150 can save preferred frequent words or sentences and allow direct access to those words/sentences in future operational sessions. The system 100 can also adaptively learn the preferences and habits of the user 150 and modify or update the available words or options of different menu screens accordingly. By prioritizing preferred or frequent words or options previously selected by the user, operation of system 100 can be more efficient and more intuitive. System 100 may further include an interface (not shown), for example via the Internet, an input device coupled to the processor, or an application on computing device 114 to allow a particular user (or administrator) to access a particular menu can be customized. Such menu customization allows individual users or specific groups of users to select from words that are more relevant or likely to be accessed frequently by that user/group. This customization also helps accommodate different users with different kinds of special needs.

Reference is now made to Figures 5A-5F. 5A-5F are schematic diagrams of example images of the eyes of user 150 captured by camera 108 of system 100 of FIG. Each image (500A, 500B, 500C, 500D, 500E, 500F) has four quadrants: an upper quadrant 502 corresponding to user 150 looking up; a lower quadrant 502 corresponding to user 150 looking down; A left quadrant 506 corresponding to user 150 looking to the right; and a right quadrant 508 corresponding to user 150 looking to the left. Processor 104 determines the rough or approximate position of the pupil of the eye in the captured image. For example, the processor 104 determines in which region or quadrant the pupil is most likely to be located (i.e., in which the majority of the pupil is located), or whether the pupil is at or near the center of the image. can be determined. If no pupil is found, processor 104 can determine that the eye is blinking or winking.

Based on the general position of the pupil and/or the change in the direction of the pupil (eg, left, right, up or down movement), the processor 104 determines the corresponding position on the display 102, 124 (up, down, left, right or center), and thus the selected/desired option of the menu (200, 300, 400) is displayed. For example, in image 500A (FIG. 5A), pupil P is located in upper quadrant 502 corresponding to option 402 of individual text menu 400 (FIG. 4). In each image 500B, 500C, 500D (FIGS. 5B, 5C, 5D), the pupil P corresponds to each letter option 410 "E", 404 "B" and 408 "B" of the individual letter menu 400, respectively. located in the lower quadrant 504, the left quadrant 506, and the right quadrant 508 (the pupil P located in the left quadrant 506 corresponds to the user looking to the right, the pupil P located in the right quadrant 508 corresponds to the user corresponds to looking to the left (i.e., the captured image mirrors the user's gesture.) Processor 104 determines that in one image, as shown in image 500E (FIG. 5E). , or if (at least the smallest portion of) the pupil P cannot be detected through the minimum sequence of consecutive images, the processor 104 can determine that the user 150 is winking (or blinking), which may correspond to selecting an option (eg, option 406 of menu 400) displayed centrally on the menu screen, or selecting or confirming an option (eg, letter 410 "E") Image 500F (FIG. 5F). , when the processor 104 detects that the pupil P is at or near its center, either in one image or through a minimum sequence of successive images, the processor 104 detects that the user 150 is pointing forward (in any of the directions described above). not), which may correspond to a standby mode in which system 100 waits until user 150 looks in a certain direction or makes an eye gesture.The actual number of divisions may vary. well, it is not limited to quadrants.

Alternatively, instead of dividing image 500 into quadrants, the image is divided into additional or fewer sections during image processing to identify pupil position (and relative eye orientation). good too. For example, the processor 104 may only determine the left and right positions of the pupil by dividing the captured image of the eye into only two sections (e.g., left and right sections separated by a central vertical divider). may Alternatively, processor 104 may determine only the upper and lower positions of the pupil by dividing the captured image of the eye into upper and lower sections (ie, by a central horizontal boundary). Still alternatively, the image of the eye has eight sections (hashtag or pound sign ) may be divided into Dividing the image into multiple sections, when processed to identify the location of the pupils, may provide different oblique or oblique angles to the user's forward gaze (e.g., the forward gaze direction of user 150). The determination of pupil position/relative eye orientation can be improved.

More generally, the processor 104 may divide the eye image into different sections, parts or segments correlated with the particular section or portion of the display 102, 124 in which the particular menu option was presented. For example, because letter group menu 300 and individual letter menu 400 present each menu option in a "cross" shaped configuration, processor 104 can determine the pupil position indicating which menu option is being viewed by user 150. , each processed image may be divided into similar cruciforms. In addition to or instead of the eye pupil P, the processor 104 may track or locate other relevant eye parameters (eg, the cornea) for determining the relative eye orientation of the user 150. may

Processor 104 can also track the position of pupil P through the sequence of captured images. For example, the processor 108 can utilize a reference image (500REF), in which case the position of the pupil P can be determined by comparing a subsequent image (or sequence of subsequent images) to the reference image 500REF. It can be determined whether it has changed compared to its position in , as well as a measure of movement. Such processing may be based on computed image filters to detect and measure differences between the original reference image and subsequent images. In addition to tracking eye pupil position through a sequence of captured images, determining only the orientation of the user's 150 relative eye orientation can also be used to determine camera 108 movement (e.g., camera relative to user's eye between images). motion), or to reduce the need to compensate for background changes. Therefore, no calibration process may be required.

Calibration of system 100 is not required, but several calibration methods may be available, including general or factory calibration, automatic calibration, and manual calibration.

An example of a general calibration process is as follows. The first step is to compare successive images to detect relative changes by performing screen motion detection on the geographic region of the eye (including the pupil region). According to a rule of thumb derived from the shape of the image, it can be assumed that the region of change in the image represents the region of interest. The second step is to mark the pupil circle as a point in space within the captured image to follow a region of interest that indicates the position of the pupil P. The third step is to set the reference point as the center of a two-dimensional (xy) axis grid and transform eye movements to (x,y) coordinates on the grid. With sufficiently high probability, when the calibration process has determined that the pupil P is centered (i.e., close to the center [0,0] of the grid), the reference point (and thus the reference point coordinate system (grid) is built) is selected. The (x,y) coordinates represent the movement of the eye's pupil P in each of the four directions (up, down, left and right). A fourth step may determine the range of motion of the user. The calibration process may adaptively learn the particular range of motion associated with different users. Different users are characterized by different physical characteristics, including different eye and facial features.

Moreover, the position and orientation of camera 108 with respect to each user's face and eyes will likely change during different operating sessions of system 100 . As a result, the calibration process of system 100 adaptively learns the range of movement of pupil P on each user's xy axis (grid) and, if necessary, thresholds (above which Adjust the range set for the pupil movement signal). A non-detection of the pupil P of minimal duration (eg, a few milliseconds) is taken to indicate a wink or blink by the user. The time can be adjusted to prevent unintended selection identification, such as based on the user's natural blinks. It is further noted that the automatic calibration process of system 100 (without prompting the user) eliminates the need for external (third party) assistance for each calibration. Additionally, a system (algorithm) identifies the pupil P and understands the relative eye orientation. Pupil detection capabilities include identifying and comparing positions and ranges of motion, and reliably detecting the presence or absence of pupil P in image 500 . To do this, multiple object detection algorithms and combinations thereof may be used to address the ambiguity issue and relatively large computational requirements. A proprietary calibration algorithm allows the system to automatically calibrate itself. Alternatively, a non-automatic calibration algorithm that does not use any object recognition tools (feature detection) may be used. That is, the algorithm compares images captured during system operation with images captured during manual calibration without identifying the exact position of the eye (pupil P).

System 100 may undergo a general factory calibration prior to an operating session of system 100, which is a single initialization process. Calibration (or initialization) is performed throughout the motion session for each of the potential relative eye orientations (e.g., up, down, left, right and center) that need to be identified during normal use. It may be done automatically, using different designed filters or a given photo. Processor 104 can apply the filter to image 500 to determine the likelihood that image 500 currently being processed corresponds to reference image 500REF. A reference image 500REF corresponding to each basic situation or state. The system may grade the captured images based on the reference image 500REF and select the option with the highest grade.

The system 100 operates using self-calibration or "auto-calibration" (i.e., done automatically during the course of an operating session without the need to prompt the user to initiate and without the user's conscious awareness). may The system 100 can determine when and how often to automatically perform the calibration process according to relevant criteria. Automatic calibration can occur when certain conditions are met, such as when the system can identify the pupil in the image.

System 100 may operate with user or manual calibration by user 150 by providing an audio or visual command, eg, "Look to the right" prior to capturing reference image 500REF. An audio indication, for example resembling a camera shutter sound, may be provided after the image is captured to inform the user 150 that the image has been captured. The reference image 500REF identifies the position of the pupil and the relative boundaries of the user's 150 eyes, or when moving from one position to another (i.e., "looking left" to "looking right"). A similar analysis may be performed to determine the predetermined boundary region that the pupil approaches. This boundary region can further be used to generate a relative eye orientation response by the user 150 when the pupil is observed or measured to cross the boundary, so that the processor 104 responds accordingly. Provide an appropriate response.

To reduce noise and improve image processing quality, the captured image (signal) may be subjected to a noise filter (eg, a Markov signal noise filter). The output for each image 500 is determined as right, left, top, bottom, eyes closed, center, or no discrimination. The final output (in this case one of five different options) may be based on additional criteria such as timing rules. For example, if the pupil is not identified for longer than the threshold duration (such as greater than about 0.2 seconds), the determination is "blink" or closed eyes, and the determination is that the pupil is relocated or identified. until the next image frame (ie, when pupil movement is detected), or until registered as a unique action as described above.

According to the present invention, user commands are directed to vague general eye directions (e.g., "right", "left", "up", "down", "front") rather than precise gaze directions or line of sight. It is determined only from the corresponding his/her relative eye orientation. Such determinations generally compare to the more extensive and time-consuming processing required to identify the user's exact eye gaze by determining the exact coordinates with respect to some particular coordinate system. are relatively basic and involve rapid processing. Thus, the system 100 mimics the way joysticks apply vague rough orientation to provide indications of user selection, as opposed to screen-dependent mouse-tracking mechanisms where pinpoint location is required. can be considered to work in a similar way.

Reference is now made to Figure 6, which is a flow diagram of a method for enabling users to communicate using eye-based feedback. In optional procedure 601, calibration is performed. Calibration of system components may be performed once (eg, prior to initial operation of system 100) or may be performed periodically (during operation of system 100) as needed. Calibration may be performed using only a single eye reflection image captured by camera 108, or by comparing and analyzing a number of such images. Additionally, the calibration may be uniquely tailored to a particular user (i.e., the particular user currently using the system 100), or alternatively may be generic and applicable to many different users. A static calibration may be performed (eg, during an early initialization stage of system 100).

In procedure 602, a series of communication options are selectably presented to a user having a selection interface. 1 and 3, display 102 (and/or display 124) displays a menu 300 with various visual menu options 302, 304, 306, 308 and 310 for selection by the user. Communication options may alternatively be presented using an audio or tactile interface.

At optional step 604, at least one eye of the user is illuminated with a light source. Referring to FIG. 1, light source 106 emits IR light 118 towards user's 150 eyes.

In procedure 606, light reflected from the eye is detected and a correlation signal is provided by a relative eye orientation sensor. Referring to FIG. 1, image sensor 110 receives IR light 118 reflected from the eye of user 150 and relays a correlation signal to processor 104 . The eye orientation sensor may alternatively be embodied by a non-imaging sensor such as one or more separate sensors or sensor arrays. Referring to FIG. 5A, image sensor 110 produces image 500A of the area around the eye of user 150, with pupil P located in the upper quadrant region of image 500A.

In procedure 608, the correlation signal is processed to determine the relative eye orientation, including the orientation of the pupils relative to the orientation of the user's head. 1 and 5A, processor 104 calculates a relative image of user 150 based on images captured by image sensor 110 (or by using other eye reflection data captured by non-image sensors). determine the orientation of the eyes. The determined relative eye orientation represents the orientation of the pupils relative to the orientation of the head of the user 150 (the orientation of the head represents the general direction in which the user 150 is facing). For example, referring to FIG. 5A, processor 104 determines from image 500A that pupil P is located in upper quadrant region 502 and thus the relative eye orientation of user 150 is in the "up" position. The processor 104 may determine the relative eye orientation of the user 150 by detecting the instantaneous relative orientation of the pupil P or by detecting transient changes in the pupil P.

In procedure 610, a selected communication option is determined based on the determined relative eye orientation of the user. 1, 5A and 3A, processor 104 selects options presented on display 102 (and/or by audio unit 112) in response to the determined relative eye orientation of user 150. do. For example, if the processor 104 determines from the image 500A that the relative eye orientation of the user 150 is in the "up" position, the processor 104 selects the communication option 302 ("ABCDE") presented on the menu 300 as select.

In procedure 612, instructions are provided to implement the selected communication option. Referring to FIG. 1, processor 104 instructs audio unit 112 to provide an audible indication associated with the selected communication option. For example, audio unit 112 broadcasts a word selected by user 150, such as the word “hello” associated with word option 202A of word menu 202. FIG. Additionally, processor 104 may direct display 102 to present a visual representation associated with the selected communication option. For example, display 102 highlights a selected menu option, such as "Hello" word option 202A. Alternatively, processor 104 may instruct computing device 114 to execute the selected application or perform an action associated with the selected communication option. For example, computing device 114 sends a message or email following, for example, selection of "email application" option 202E of word menu 202. FIG.

As described above, the processor 104 translates eye movements, via a wireless link (e.g., Bluetooth), into a field that is in the area and that the user 150 uses to communicate with his/her environment. The actions are broadcast to a computing device 114 (eg, a smart phone) capable of running different applications that can run different applications. The computer device 114 can be used to send text messages, run different applications, and/or control a computer (using programs designed for control using selection tables or menus). After the user selects to run an application on computing device 114 , processor 104 may continue to send and receive data to/from computing device 114 over communication link 116 .

According to one embodiment of the present invention, system 100 can be used to control external devices or apparatus such as power wheelchairs or computer controlled appliances. For example, a user may indicate a selected direction (e.g., forward, backward, left, right) using his/her relative eye orientation (e.g., up, down, left, right) to A power wheelchair can be guided or navigated. The user may provide directions by looking in the direction of a particular menu option of the “powered wheelchair navigation” menu presented on display 102 , 124 . The system 100 determines the indicated selection (eg, rotate wheelchair left) and causes the power wheelchair to perform the selected option. System 100 may be communicatively coupled to such external devices via wireless communication links (eg, communication link 116).

eye-based communication in which the system 100 determines only the relative eye orientation of the user 150 and does not require the user 150 to have full use of vision or a satisfactory level of vision; It is further noted that user 150 may alternatively be an individual who is visually impaired or lacks complete visual ability, as it provides .

Accordingly, the present invention provides a simple and intuitive operation, relatively affordable device that helps improve the quality of life of individuals with special needs by providing a means to effectively communicate with the environment. Provide affordable and affordable solutions. The system is mobile, portable, and suitable for use in many situations and positions (lying, sitting, head down, etc.). Furthermore, the system of the present invention does not require third party assistance for calibration.

It is understood that the systems and methods of the present invention are not limited to individuals characterized by locked-in syndrome (LIS), but are equally equally adaptable for use by other types of people, including those with visual impairments. deaf. The systems and methods of the present invention may be incorporated into applications on computing devices such as smart phones, tablet computers, or digital music players that at least include sensors (such as embedded cameras) for detecting light reflections from the eyes. . Thus, the present invention composes and sends messages, or telephonically communicates with third parties, based solely on commands provided by a user using eye gestures correlated with his/her relative eye orientation. A variety of communication actions can be facilitated and initiated using a computing device, such as starting a conversation.

Although specific embodiments of the disclosed subject matter have been described to enable those skilled in the art to practice the invention, the foregoing description is intended to be exemplary only. It shall not be used to limit the scope of the disclosed subject matter, which should be determined by reference to the claims that follow.

Claims (13)

In a communication system,
An audible selection interface configured to selectably present a series of communication options to a user, each audible communication option being selected upon determination of a different eye gesture associated with each audible communication option. an audible selection interface , comprising:
at least one camera configured to capture a sequence of sequential images of the user's eye;
a processor communicatively coupled, optionally wirelessly, with said at least one camera and said audible selection interface, said processor comprising:
(i) receiving and processing the captured image to determine a relative eye orientation, which is the orientation of the pupil of the eye within the orbit, said determination of the relative eye orientation comprising:
a. marking the pupil in the captured image and tracking a region of interest indicating detection and location of the pupil;
b. setting a reference point as the center of a 2D grid and constructing a coordinate system of the reference points relative to the center of the grid;
c. transforming the position of the pupil into coordinates on the grid;
and
(ii) determining an eye gesture based on the relative orientation of the eyes by determining in which of at least two sections of the 2D grid the region of interest is located;
(iii) a processor configured to determine a user-selected communication option based on the determined eye gesture, and (iv) provide instructions to perform the user-selected communication option;
A system characterized by comprising: 3. The system of claim 1, further comprising at least one light source configured to illuminate the eye of the user, the light source optionally comprising a light emitting diode (LED). system. 3. The system of claim 2, wherein the light source comprises an infrared (IR) light source and the at least one camera is configured to detect IR light reflected from the eye. . 2. The system of claim 1, wherein the eye gesture comprises:
upward with respect to the eye socket;
downward with respect to the eye socket;
transversely to the eye socket;
obliquely with respect to the eye socket;
an oblique orientation with respect to the eye socket;
a frontal orientation that matches the orientation of the eye socket;
winking the eye;
blinking said eye; and
A system characterized by being selected from a list consisting of: 3. The system of claim 2, wherein at least one of the light source and the at least one camera are coupled to wearable headgear worn by the user, or the selection interface operates on a mobile computing device. characterized system. 2. The system of claim 1, wherein at least one of said communication options includes:
an audible alarm;
menu selection and
language selection and
confirmation message and
sentence and
phrase and
word and
syllables and
letters and
a mobile computing device boot selection;
a visual interface deactivation selection;
an email/SMS/MMS transmission instruction;
computer application execution instructions;
A system characterized by being selected from a list consisting of: A method of communicating, the method comprising:
A procedure for audibly and selectably presenting a series of communication options to a user via a selection interface, each audible communication option having a different eye gesture determination associated with each audible communication option. a procedure configured to be selected when
capturing a sequence of sequential images of the user's eye with at least one camera;
A procedure for processing said eye image to determine relative eye orientation, comprising: determining a relative orientation that is an orientation of a pupil in said eye socket of said user; teeth,
a. marking the pupil in the captured image and tracking a region of interest indicating detection and location of the pupil;
b. setting a reference point as the center of a 2D grid and constructing a coordinate system of the reference points relative to the center of the grid;
c. transforming the position of the pupil into coordinates on the grid;
a step comprising
determining an eye gesture of the user based on the relative orientation of the eyes by determining in which of at least two sections of the 2D grid the region of interest is located;
determining a user-selected communication option based on the determined eye gesture;
providing instructions to implement the user-selected communication option;
A method characterized by 8. The method of claim 7, further comprising illuminating said eyes of said user with at least one light source, optionally IR light being used to illuminate said eyes, said at least one camera is configured to detect IR light reflected from said eye, and optionally said light source and said at least one camera are coupled to wearable headgear worn by said user. Method. 8. The method of claim 7, wherein the eye gesture is
upward with respect to the eye socket;
downward with respect to the eye socket;
transversely to the eye socket;
obliquely with respect to the eye socket;
an oblique orientation with respect to the eye socket;
a frontal orientation that matches the orientation of the eye socket;
winking the eye;
blinking said eye; and
, wherein the method is selected from a list consisting of 8. The method of claim 7, wherein the user communication option comprises:
an audible alarm;
menu selection and
language selection and
confirmation message and
sentence and
phrase and
word and
syllables and
letters and
a mobile computing device boot selection;
a visual interface deactivation selection;
an e-mail transmission command;
an SMS/MMS transmission command;
computer application execution instructions;
, wherein the method is selected from a list consisting of In a communication system,
An audible selection interface configured to selectably present a series of communication options to a user, each audible communication option being selected upon determination of a different eye gesture associated with each audible communication option. an audible selection interface configured in a
at least one camera configured to capture an image of the eye;
a processor communicatively, optionally wirelessly, coupled to said at least one camera and said selection interface;
with
The processor receives and processes the eye image data,
(i) dividing the image into at least two sections;
(ii) determining the approximate location of the pupil of the eye in one of the at least two sections of the image of the eye;
(iii) determining an eye gesture based on approximate pupil position in one of said at least two sections of said eye image;
(iv) determining a user-selected communication option based on the determined eye gesture;
(v) providing instructions for implementing said user-selected communication options;
A system characterized by: 12. The system for communicating according to claim 11, wherein said selection interface is an audible interface. A communicating system according to claim 11 or 12, wherein
Determining the approximate position of the pupil of the eye comprises:
(i) marking the pupil in the captured image and tracking a region of interest indicating detection and location of the pupil;
(ii) setting a reference point as the center of a 2D grid to construct a coordinate system of the reference points relative to the center of said grid;
(iii) transforming the position of the pupil into coordinates on the grid;
A system characterized by comprising:

Images