New! View global litigation for patent families

US20100076334A1 - Alzheimer's cognitive enabler - Google Patents

Alzheimer's cognitive enabler Download PDF

Info

Publication number
US20100076334A1
US20100076334A1 US12233817 US23381708A US20100076334A1 US 20100076334 A1 US20100076334 A1 US 20100076334A1 US 12233817 US12233817 US 12233817 US 23381708 A US23381708 A US 23381708A US 20100076334 A1 US20100076334 A1 US 20100076334A1
Authority
US
Grant status
Application
Patent type
Prior art keywords
patient
device
eeg
ace
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US12233817
Inventor
Martine Rothblatt
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Unither Neurosciences Inc
Original Assignee
Unither Neurosciences Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06KRECOGNITION OF DATA; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
    • G06K9/00Methods or arrangements for reading or recognising printed or written characters or for recognising patterns, e.g. fingerprints
    • G06K9/00624Recognising scenes, i.e. recognition of a whole field of perception; recognising scene-specific objects
    • G06K9/00664Recognising scenes such as could be captured by a camera operated by a pedestrian or robot, including objects at substantially different ranges from the camera
    • G06K9/00671Recognising scenes such as could be captured by a camera operated by a pedestrian or robot, including objects at substantially different ranges from the camera for providing information about objects in the scene to a user, e.g. as in augmented reality applications
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Detecting, measuring or recording for diagnostic purposes; Identification of persons
    • A61B5/04Detecting, measuring or recording bioelectric signals of the body or parts thereof
    • A61B5/0476Electroencephalography
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Detecting, measuring or recording for diagnostic purposes; Identification of persons
    • A61B5/04Detecting, measuring or recording bioelectric signals of the body or parts thereof
    • A61B5/0476Electroencephalography
    • A61B5/0484Electroencephalography using evoked response
    • A61B5/04842Electroencephalography using evoked response visually
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Detecting, measuring or recording for diagnostic purposes; Identification of persons
    • A61B5/04Detecting, measuring or recording bioelectric signals of the body or parts thereof
    • A61B5/0476Electroencephalography
    • A61B5/0484Electroencephalography using evoked response
    • A61B5/04845Electroencephalography using evoked response acoustically or auditory
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Detecting, measuring or recording for diagnostic purposes; Identification of persons
    • A61B5/40Detecting, measuring or recording for evaluating the nervous system
    • A61B5/4076Diagnosing or monitoring particular conditions of the nervous system
    • A61B5/4088Diagnosing of monitoring cognitive diseases, e.g. Alzheimer, prion diseases or dementia
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRICAL DIGITAL DATA PROCESSING
    • G06F1/00Details of data-processing equipment not covered by groups G06F3/00 - G06F13/00, e.g. cooling, packaging or power supply specially adapted for computer application
    • G06F1/16Constructional details or arrangements
    • G06F1/1613Constructional details or arrangements for portable computers
    • G06F1/163Wearable computers, e.g. on a belt
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06KRECOGNITION OF DATA; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
    • G06K9/00Methods or arrangements for reading or recognising printed or written characters or for recognising patterns, e.g. fingerprints
    • G06K9/00624Recognising scenes, i.e. recognition of a whole field of perception; recognising scene-specific objects
    • G06K9/00711Recognising video content, e.g. extracting audiovisual features from movies, extracting representative key-frames, discriminating news vs. sport content

Abstract

A computing device that enables user interaction when the user is in a mentally debilitating state includes a memory configured to store electroencephalogram (EEG) waveform signals. The EEG waveform signals are generated by associated recognition data. Response data is generated based on the associated recognition data and stored in memory. An input component is configured to receive an EEG waveform signal from a user. A processor is configured to compare the received EEG waveform signal with the stored EEG waveform signals to determine if the received EEG waveform signal matches a stored EEG waveform signal. An output component is configured to automatically output the response data generated by the associated recognition data based on the determination from the processor.

Description

    BACKGROUND OF THE INVENTION
  • [0001]
    1. Field of the Invention
  • [0002]
    Embodiments of the present invention relate to apparatus, systems and methods for improving the quality of life of an individual suffering from a debilitating disease as well as members of interest affected by the sufferer's diminished capacity.
  • [0003]
    2. Description of the Related Art
  • [0004]
    Debilitating diseases which affect a person's mental capacity are especially difficult to endure, both for the individual suffering from the disease as well as family members, friends and caregivers who must care for the individual. One form of a mentally debilitating disease is dementia. Dementia is a brain disorder that seriously affects a person's ability to carry out daily activities. The most common form of dementia among older people is Alzheimer's disease, which initially involves the parts of the brain that control thought, memory, and language. In its early stages, memory loss, shown as a difficulty to remember recently learned facts, is the most common symptom, although it is often initially misdiagnosed as age-related memory-loss or stress. Later symptoms include confusion, anger, mood swings, language breakdown, long-term memory loss, and the general withdrawal of the sufferer as his or her senses decline. Each individual experiences the symptoms of Alzheimer's disease in unique ways. The duration of the disease is between 5 and 20 years.
  • [0005]
    Currently, there is no cure for Alzheimer's disease. Also, there are no effective ways for an Alzheimer's disease sufferer to effectively interact with family and friends at the various stages of the disease.
  • [0006]
    Therefore, a need exists for an apparatus, system and method that allows an Alzheimer's disease suffer to effectively interact with family, friends, caregivers, etc. at the various stages of the disease.
  • SUMMARY OF THE DISCLOSURE
  • [0007]
    Accordingly, an Alzheimer's cognitive enabler is disclosed that allows an Alzheimer's disease suffer to effectively interact with family, friends, caregivers, etc. at the various stages of the disease.
  • [0008]
    According to one embodiment of the present invention, a computing device that enables user interaction when the user is in a mentally debilitating state includes a memory configured to store electroencephalogram (EEG) waveform signals. The EEG waveform signals are generated by associated recognition data. Response data is generated based on the associated recognition data and stored in memory. An input component is configured to receive an EEG waveform signal from a user. A processor is configured to compare the received EEG waveform signal with the stored EEG waveform signals to determine if the received EEG waveform signal matches a stored EEG waveform signal. An output component is configured to automatically output the response data generated by the associated recognition data based on the determination from the processor.
  • [0009]
    According to another embodiment of the present invention, a computer system that enables user interaction when the user is in a mentally debilitating state includes a computing device and a server computer. The computing device includes a memory, an input component, a processor and an output component. The memory is configured to store EEG waveform signals. The EEG waveform signals are generated by associated recognition data. Response data is generated based on the associated recognition data and stored in memory. The input component is configured to receive an EEG waveform signal from a user. The processor is configured to compare the received EEG waveform signal with the stored EEG waveform signals to determine if the received EEG waveform signal matches a stored EEG waveform signal and the output component is configured to automatically output the response data generated by the associated recognition data based on the determination from the processor. The server computer is configured to update EEG waveform signals, associated recognition data and response data stored in the memory of the computing device.
  • [0010]
    According to a further embodiment of the present invention, a method for enabling user interaction when the user is in a mentally debilitating state includes storing electroencephalogram (EEG) waveform signals, whereby the EEG waveform signals are generated by associated recognition data. The method also includes the steps of storing response data generated based on the associated recognition data and receiving an EEG waveform signal from a user. The method further includes the steps of comparing the received EEG waveform signal with the stored EEG waveform signals and determining that the received EEG waveform signal matches a stored EEG waveform signal. The method also includes the step of automatically outputting the response data generated by the associated recognition data based on the determining step.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • [0011]
    These and other features, aspects and advantages of the present invention will become better understood with regard to the following description, appended claims and accompanying drawings where:
  • [0012]
    FIG. 1 illustrates a user with an embodiment of the Alzheimer's Cognitive Enabler (ACE) device of the present invention;
  • [0013]
    FIG. 2 illustrates an operating environment in which the ACE device, system and method of the present invention may operate; and
  • [0014]
    FIG. 3 is a flowchart depicting steps performed within the ACE device in accordance with one embodiment of the present invention.
  • DETAILED DESCRIPTION
  • [0015]
    Unless otherwise specified, “a” or “an” means “one or more.”
  • [0016]
    Recent advancements in non-invasive electroencephalogram (EEG) waveform monitoring for the gaming industry have enabled a significant array of EEG waveform signals to be associated with ideational content. In addition, recent advancements in image and voice recognition technologies have enabled these technologies to be used easily with personal computers. These technologies, however, have been limited to being practiced in their on particular industry and have not been linked together to assist individuals suffering from Alzheimer's disease. In general, an Alzheimer's disease patient is mentally capable of more than they can articulate or express. According to an embodiment of the present invention, an Alzheimer's cognitive enabler (ACE) allows the Alzheimer's disease patient to draw upon his or her less-affected cognitive waveforms to trigger recognition and conversation functions that are blocked by beta amyloid plaque in the brain, but can be made available on the ACE. In other words, the ACE may be equated to or conceptualized as a “cognitive wheelchair” providing mental assistance as compared to physical assistance provided by a conventional wheelchair.
  • [0017]
    Consider, for example, ACE device 100 being carried by user 110 (i.e., an Alzheimer's patient) in FIG. 1. In this exemplary embodiment illustrated in FIG. 1, the ACE device 100 has access to input devices such as an EEG device 150, video camera 121, microphone 124, and input device 122 (e.g., a keyboard, a hand-writing recognition device, etc.). The ACE device 100 also has access to a visual display device 132 and a speaker output device 134. In some embodiments of the present invention, the input and output devices may be integrated together with the ACE device 100 into a single device, while in alternative embodiments the ACE device 100 may be a separate device that has the ability to accept input from and provide output to (e.g., via physical or wireless connections) any of a variety of devices which may be accessible (e.g., carried by the user or located nearby in the environment). The ACE device 100 further includes a processor 160 for processing information stored in its memory 165 as discussed in greater detail below.
  • [0018]
    The invention is further illustrated by, though in no way limited to, the following particular embodiments and examples. In the example illustrated in FIG. 1, a variety of situations are illustrated for which user 110 may wish to store information that can be later used to assist the user 110 to interact with persons 170. According to one embodiment of the present invention, the ACE device 100 is prescribed for user 110 at the user's first sign of cognitive decline. During this still relatively healthy stage, the Alzheimer's patient, family member(s) and/or caregivers would load the ACE device 100 with the patient's salient images, voices data, memories, etc. and associated EEG waveforms. Via artificial intelligence technology, however, it is not critical that every bit of data has an associated EEG waveform. For example, there may be an EEG waveform associated with the image of a patient's son. However, there need not be an EEG waveform associated with an image of the son together with the patient. The artificial intelligence program running the ACE device 100 uses relational database technology to bring together related images after an index image is EEG triggered. More generally, the ACE device 100 is configured to conduct a normal conversation with a person based on stored data and advanced artificial intelligence-based chatbot software upon an EEG waveform associated with just recognition of the person. In general, any EEG waveform will trigger some conversational capability out of the ACE device 100, with such conversational capability being as close to the motivations that gave rise to the EEG waveform as the stored associations permit. While wearing EEG device 150, the Alzheimer's patient is able to train the ACE device 100 to react as he or she would in given situations when interacting with others. Thereafter, once the disease has progressed, the ACE device 100 can be used to interact with others on behalf of the Alzheimer's patient using images and sound recognition routines, conversational software and patient-specific settings previously stored and continuously updated as discussed in greater detail below.
  • [0019]
    For example, upon encountering person 170 for the first time, the user 110 may wish to store a variety of information about the person for later recollection. In particular, the user 110 can use the video camera 121 and microphone 124 to capture video and audio recordings of the encounter. The user 110 can also use the video camera 121 to capture other current information about the environment which may assist in later recollection about the encounter, such as a video image of the location in which the encounter occurred. In addition, the user 110 can use the microphone 124 to record dictated information about the person 170, such as the person's name, address, e-mail address, phone number, etc. The ACE device 100 can then store this dictated information as an audio recording, or can instead perform voice recognition on the dictation in order to produce a textual version of the information. Alternately, the user 110 can provide some, or all of the information about the person to the ACE device 100 directly as text via text input device 121. In other embodiments, the user 110 can provide information to the ACE device 100 via any other available input means, such as transmitted information from a portable device (not shown) that the person might be carrying (e.g., another ACE device). After receiving and processing the various information about the encounter with the person 170, the ACE device 100 associates the various information with the received EEG waveform signal from the EEG device 150.
  • [0020]
    According to one embodiment of the present invention, the ACE device 100 is configured such that an Alzheimer's patient who sees person 170 again, but fails to recognize him, could automatically trigger an appropriate greeting for person 170 based on the EEG waveform generated when the Alzheimer's patient's see that person again but is unable to recognize him.
  • [0021]
    The ACE device 100 may be in the form of a general-purpose body-mounted wearable computer worn by user 110. Many wearable computers travel with the user, such as being strapped or attached to a user's body or clothing or being mounted in a holster. The wearable computer has a variety of user-worn user input devices including the microphone 124, a hand-held flat panel display with character recognition capabilities, and various other user input devices 222. Similarly, the computer has a variety of user-worn output devices that include the hand-held flat panel display, an earpiece speaker, an eyeglass-mounted display, etc. In addition to the various user-worn user input devices, the computer can also receive information from various user sensor input devices and from environment sensor input devices, including the video camera 121. The ACE device 100 can receive and process the various input information and can present information to the user 110 on the various output devices. Thus, as the user 110 moves about in various environments, the ACE device 100 receives various input information from the input devices that can be stored.
  • [0022]
    The general-purpose body-mounted wearable computer may include a touch screen feature (which may approximate the size of a human head, but could be any size, for example) that enables the ACE device 100 to respond to tactile as well as audio and video inputs. For example, if the Alzheimer's patient was in a late stage of the disease, the patient's omnipresent, naturally moving image on the touch screen could be kissed, and the programming features of the ACE device 100 would recognize that input and respond in the Alzheimer patient's voice, with a reply such as “THANK YOU FOR THE KISS GRANDSON.”
  • [0023]
    Those skilled in the art will appreciate that specialized versions of the body-mounted computer can be created for a variety of purposes. Those skilled in the art will also appreciate that a variety of such physiological conditions can be monitored, and that other specialized versions of the device can similarly be implemented.
  • System Environment
  • [0024]
    Referring to FIG. 2, an ACE system and method of the present invention operates primarily in a computer network environment. In this embodiment, the ACE system 20 includes a program system administration site 10 which includes server computer 12 and a system database 14 and a number of workstations 18 that communicate with the server computer 12. The workstations 18 may be used by patients/caregivers 18 a, family members 18 b or friends 18 c or any other system participant 18 d. According to an embodiment of the present invention, the workstation 18 may be a stand alone personal computer (PC) based system with wireless internet connectivity, and Smart Phones with Wi-Fi connectivity to an off-site server, the same as the ACE device 100. The off-site server may provide software upgrades, act as a repository for certain database functions and provide diagnostic functions. For example, the wireless-linked feature of the ACE device 100 and the ACE system 20 helps to ensure that no data is lost in the event of a failure, theft, breakage or loss of the ACE device 100. At user-settable periods all data from the ACE device 100 is automatically uploaded to server computer 12 and then to system database 14 for safe storage. This stored data can be readily downloaded into a replacement ACE device. The personal computer typically will require a video monitor, sound and video cards. Alternatively, the workstation 18 may be any device capable of interacting with a computer network, including such devices as personal digital assistance (PDA) and cellular phones.
  • [0025]
    Additional system requirements include a PC compatible video camera, PC compatible directional microphone, speakers, and a Blue Tooth commercial EEG headset. In the caregiver/institutional setting, a tablet PC, and voice activated microphones may be required in addition to the above. Embodiments of the present invention also may require onsite training and a patient support facility 24 in some cases or the patient may require assistance in utilizing the invention, in both instances the ACE system 20 is amenable to additional users assisting the patient in providing the input or in training the patient in utilizing the supplemental system input devices. Resident on the sever computer 12 of the ACE system 20 is an Alzheimer's disease recognition program 16. Also, with the wireless-linked feature of the ACE device 100 and ACE system 20, responsiveness of the ACE device 100 to be continually improved via new software downloaded from server computer 12. At the server computer 12, software will analyze the ACE patterns of interaction the are uploaded each day to tune the ACE device's 100 program to better match the Alzheimer's patient's healthy pattern of conversation. According to one embodiment of the present invention, server 12 has installed therein, newer versions of conversational software that are more sophisticated in replicating normal conversational patterns. Such software is configured to review saved conversations between the patient and family to identify errors and adjust conversational parameters. This is analogous to how “auto-tune” software works with digital music, in that a person's voice can be analyzed for being off-tune and corrected.
  • System Training and Data Base Elements
  • [0026]
    According to one preferred embodiment of the present invention, the ACE device 100 and ACE system 20 are trained via patient interaction in each of the following instances: recording the patient's voice, obtaining varying minutes of digital video of the patient's facial expressions (speaking, laughing, and joy, sadness) to obtain a viseme database. As used throughout this specification, a viseme is defined as a unit of speech in the visual domain; how ones face forms words and expressions. The ACE device 100 and ACE system 20 build a lexicon of patient's visemes from video taping the patient. In the case where the patient cannot participate, the lexicon is built synthetically. In the synthetic process, the patient's face is photographed in high definition, for example, and donor visemes are digitally overlaid onto the photographed face of the patient and the result is akin to live animation.
  • [0027]
    According to one embodiment of the present invention, videotaping can be performed by trained staff, a caregiver, etc. The patient may also construct, or be assisted in constructing a database of family members and friends (Family/Friends) with whom the patient would wish to interact. The database may also include the contact information for caregivers and health care providers. According to the operation of the present invention, voice samples of these individuals may be collected from Wi-Fi capable cell phone interactions, as well as prearranged voice recording conducted by training staff. Afterwards, the patient may construct or be aided in constructing, or have constructed by the training staff, a database of responses for their phone (cellular and landline) and PC systems, as well as establishing the pre-sets for when the system responds for them.
  • [0028]
    According to an exemplary embodiment of the present invention, baseline voice samples of the patient may be used for Alzheimer analysis. The patient may be instructed by a staff member in the training and use of the EEG device 150. As stated previously, the EEG device 150 serves as an auxiliary data input tool and data collection device for the ACE device 100 and ACE system 20 with the EEG signals serving in place of the PC's mouse, acting as a drawing and pointing/selection agent on the screen, and one capable of dragging and dropping screen based objects. According to an alternative embodiment of the present invention, if the EEG signals are confused, or unrecognized, the software is configured to default to running as if the appropriate EEG signal was triggered. For example, if “daughter” has a specific EEG, signal but it is not recognized, it will be possible for “daughter” to specify that it is the daughter visiting the patient, and then the ACE device 100 will converse as if the EEG signal for “daughter” had been triggered. Familiarity with the EEG device 150 will also assist the patient in accessing the device and system as the patient's health declines. Baseline EEG samples obtained from patient use may be uploaded to the server 12 and the data may be then analyzed. The patient may be trained to use the voice recognition software as an auxiliary data entry tool. This will allow the patient to more readily enter his or her autobiographical memories, and to use the PC more fully in conjunction with the EEG device 150. Voice samples from the usage may be utilized for voice synthesis as well as diagnostic analysis.
  • Profiles
  • [0029]
    According to one preferred implementation of the present invention, the data collected through phone conversations and interviews with the patient and the patient's Family/Friends may be utilized to assist in constructing affirmative conversations for the ACE device 100 and ACE system 20 to deploy. In the instance that the patient is sufficiently incapacitated and unable to participate in the interview process, in a preferred implementation of the present invention, staff psychologists may conduct interviews with Family/Friends, review personal material and construct a patient autobiographical inventory that will serve as the basis for constructing a conversational database and the autobiographical database stored either in the ACE device 100 or stored remotely on the ACE system 20.
  • [0030]
    According to the operation of the ACE device 100 and the ACE system 20, if a user 110 is unable to recognize a person initiating a conversation, the EEG waveform signal generated from the EEC device 150 is used to retrieve the correct responses. According to an alternative operation of the ACE device 100 and ACE system 20, voice and face recognition software is used to identify the initiator of a conversation and then the ACE device 100 and ACE system 20 supplies a correct response set based on the recognition of the initiator of the conversation. In the preferred implementation of the present invention, the response is profiled based upon the historical psychodynamics between that individual and the patient. The ACE device 100 and ACE system include a profiling methodology which is a psychological framework employed to maximize the therapeutic investment of energy into the suspense of disbelief experienced by the patient's Friend/Family in interacting with a patient avatar. According to the present invention, a patient avatar is defined as an audio construct of the patient, one that will have the patient's face, facial expression and voice. In the instance that the patient was an early participant of the system and partook in the system training, the patient avatar will appear on the monitor speaking in the patient's own voice, and to a very large degree, with the patients' own facial expressions. If the patient was unable to participate, the voice will be gender appropriate and pitch and tone balanced. According to an exemplary embodiment of the present invention, the facial expression may be human and warm and the face will be that of the patient.
  • [0031]
    According to an alternative embodiment of the present invention, the ACE device 100 and ACE system 20 may include an emotion software engine that is based upon actual digitally sampled emotions from an Alzheimer's patient. According to this implementation of the present invention, the ACE device 100 and ACE system 20 are configured to portray realistic emotions of the Alzheimer's patient, triggered by EEG waveform signals and/or program-analyzed semantic cues in an conversational software engine.
  • [0032]
    According to one embodiment of the present invention, the conversational database generates an appropriate conversational segment in response to a unique speaker from the information stored in the system database 14 or in the memory of the ACE device 100 and then further processed. An example conversation is provided below:
  • [0033]
    SARAH: HI NANA, HOW ARE YOU?
  • [0034]
    NANA: I'M DOING SO-SO TODAY PRECIOUS, HOW IS YOUR LITTLE JO-JO?
  • [0035]
    Sarah, the grand-daughter, is recognized by the ACE device 100 or the ACE system 20, either by the associated EEG waveform, voice recognition software or face recognition software, the conversation generation then looks up Sarah, recognize her grandmother's pet name for her and that she has a pet named Jo-jo.
  • System Inputs
  • [0036]
    As part of the ACE device 100 and ACE system 20, inputs according to an embodiment of the present invention, including videotaping of a patient in scripted conversation conducted by training staff. According to one preferred implementation of the present invention, this may be held over multiple sessions of 15 minute duration totaling 60 minutes in aggregate, for example. The sessions are designed to capture the patient in a variety of emotional responses and facial patterns. The trainer/technician performing the video taping in addition to following the script may be trained in how to provoke the various emotive responses desired in order to capture a variety of facial expression of the patient. The resulting video is then processed into the viseme database for the patient avatar as discussed above. For example, high quality head shots of the patient's designated Family/Friends may be taken by the training staff and input into the ACE device 100 and ACE system 20 for the facial recognition component. The training staff may also sample Family/Friend vocal recordings for input into the voice recognition component. In the event that the patient is of diminished capacity and cannot withstand the rigors of the interview process, the trainer may take high definition images of the patient's face. Visemes may then be constructed synthetically from the images. The process is one of overlaying the patients face over a “digital donor”, one carries out the process of mapping say, trainer Mary's visemes, removing Mary's characteristic facial features leaving behind a dynamic template, and then mapping the patient's facial features onto the template. This may be a one time process, completed on site by the training staff.
  • System Output
  • [0037]
    As part of the ACE device 100 and ACE system 20 outputs according to an embodiment of the present invention, text, audio-visual, or audio responses through the patient's phone system (land as well as cellular) and PC/PDA/Smartphone either at the patient's request or at a predetermined setting in response to the diagnostic data gathered from the patient may be output. Video may be processed and synchronized with the patient's visemes to produce a patient avatar. According to one preferred implementation of the present invention, the processing may be conducted by training staff onsite. The ACE device 100 and ACE system 20, when prompted by the initiator, may then cue the conversational database load either in the system database 14 or locally in memory of the ACE device 100 for the initiator and prepare to integrate the speech synthesis engine with the patient avatar in response to the initiators spoken word.
  • System Diagnostics
  • [0038]
    According to one preferred embodiment of the present invention, voice data obtained from voice control of the PC and from cellular conversations may be analyzed for early onset of the Alzheimer's using currently established diagnostic benchmarks developed for voice recognition technology and the indicia of Alzheimer's disease. This data set may be compared against baseline voice data collected at the patient's initial use of the system. EEG waveform data collected from the patient may be analyzed using currently established protocols for the prediction of or indicia of Alzheimer's disease. Through these two processes a reasonable degree of statistical likelihood exists in being able to understand the degree of incapacity of the patient at a given time and whether to invoke the patient's choices in terms of responding to phone calls or PC communications, or in calling for caregivers, health providers or family members.
  • System and Program Model
  • [0039]
    According to a further preferred embodiment of the present invention, ACE system 20 is a system platform that is designed to assist a patient diagnosed in the pre-onset stages of Alzheimer's disease through the later stages of the disease. In the pre-onset stage, the preferred model envisions a patient's interacting directly with the ACE device 100 and ACE system 20, providing inputs both for system training, patient diagnosis and patient memory storage. As the patient becomes infirmed, or feels unable to complete or attend to specific communication based tasks, the device and system provide responses for the patient. If the patient becomes institutionalized or requires a caregiver and reaches a stage of diminished conversational capacity, the ACE device 100 and ACE system 20 are configured to recognize a vocal prompt and or the face of a member from the Family/Friend indexed in the database 14 or local memory, which will initialize the device and system and will commence conversation with the indexed individual.
  • [0040]
    In the pre-onset stage of Alzheimer's disease, the patient interacts with the device and system through a number of dimensions. Phone conversations provide profile data for the Family/Friend database-namely, the individuals within the database, voice samples, frequency and duration of interaction; cellular phone data is transmitted via Wi-Fi to the server 12 for storage and analysis. PC based activities may be gradually controlled by supplemental devices, viz., the patient's oral commands (voice recognition) and through EEG waveform based commands. Data collected from the device and system may be transmitted to the server 12 for diagnostic analysis and for use in the patient avatar. If the patient feels fatigued, or if the EEG waveform based diagnostics advise the patient or the patient's caregiver that the patient is entering a deficit, the device and system may be engaged by default or by the patient's choice to intervene on any or all communication channels available to the patient. If the perceived deficit reaches a predetermined threshold, the device and system is configured to alert caregivers, healthcare providers, and family members and friends as desired.
  • [0041]
    In the intermediate stage of Alzheimer's disease, the patient has some capacity for conversation, or may easily feel fatigue. According to a preferred implementation of the present invention, the patient may wear and may interact with the ACE device and ACE system through the EEG device 150 and may guide the conversation with Family/Friends, selecting conversational elements from the conversational database or elements from his or her autobiographical database via a tablet PC. In this implementation of the present invention, the patient may also turn the patient avatar on or off. The pre-sets established will determine whether the patient is in a state to make the determination.
  • [0042]
    When a patient is in a state of decline such that self sustaining conversation is either a burden to the patient or a mental impracticality, the ACE device and system is configured to intervene between the patient and family member, or the patient and the patient's friend. The patient is present, juxtaposed with a video monitor of either the ACE device or ACE system. In one preferred implementation of the present invention, one speaker at a time engages the patient, with each speaker wearing a voice activated microphone. Upon speaking to the patient, the device and system is activated via voice and facial recognition using the software provided therein; the patient avatar displays recognition, and enters into a conversation with the initiator. The patient avatar is configured to converse approximately as one would expect an elderly family member: short answers, of durations 20 seconds or less, and for a total conversation length of no more than five minutes per family participant, for example.
  • [0043]
    According to an embodiment of the present invention, the conversations with the same party normally will not repeat the same content during the next conservation if (i) the original database is sufficiently robust and (ii) the conversation doesn't duplicate the content, for example. Even in the second instance, for example, there is no assurance that the content of the patient avatar's content would be the same. The content of the conversation may be uploaded to a server, such as server 12 for diagnostic analysis. Specific events referenced by the patient avatar may be accessed by the Family/Friend by accessing the autobiographical database events referenced for that day.
  • Telemetry
  • [0044]
    According to an embodiment of the present invention, the ACE device and system are also configured to monitor patient pain and mood levels via EEG waveform signals output as well as a standard set of outputs (e.g., heartbeat, pulse rate, blood pressure, etc.). All data collected is available to the patient's health care providers and approved subscribers historically and in real-time via Wi-Fi (PC/Smartphone/PDA) or a remote server.
  • [0045]
    FIG. 3 is a flowchart depicting steps performed within the ACE device in accordance with one embodiment of the present invention. The process begins from a start state S400 and proceeds to process step S401, wherein EEG waveform signals are stored in memory. The EEG waveform signals are generated by associated recognition data. At process step S402, response data generated based on the associated recognition data is also stored in memory. After the information has been stored, at process step S403, an EEG waveform signal from a user is received. After receiving the EEG waveform signal, the process proceeds to process step S404 where the received EEG waveform signal is compared with the stored EEG waveform signals. After the comparison step, the process proceeds to decision step S405 where it is determined whether or not the received EEG waveform signal matches a stored EEG waveform signal. If the received EEG waveform signal matches one of the stored EEG waveform signals, the process proceeds to process step S406 where the response data generated by the associated recognition data is automatically outputted, otherwise, the process returns to process step S403.
  • [0046]
    Alzheimer's disease is characterized by a slow, gradual decline in cognition which cause great sorrow and diminished quality of life to the Alzheimer's disease sufferer as well as his or her family. The ACE device, system and method can significantly ameliorate the diminished quality of life.
  • [0047]
    Based on this description of exemplary embodiments, other embodiments will be readily apparent to one of skill in the art. Thus, these exemplary embodiments should be considered as limiting the scope, which is defined according to the following claims.

Claims (22)

  1. 1. A computing device that enables user interaction when the user is in a mentally debilitating state, comprising:
    a memory configured to store electroencephalogram (EEG) waveform signals, the EEG waveform signals being generated by associated recognition data,
    wherein response data is generated based on the associated recognition data and stored in memory;
    an input component configured to receive an EEG waveform signal from a user;
    a processor configured to compare the received EEG waveform signal with the stored EEG waveform signals to determine that the received EEG waveform signal matches a stored EEG waveform signal; and
    an output component configured to automatically output the response data generated by the associated recognition data based on the determination from the processor.
  2. 2. The computing device according to claim 1, wherein the associated recognition data include images data.
  3. 3. The computing device according to claim 1, wherein the associated recognition data includes voice data.
  4. 4. The computing device according to claim 1, wherein response data includes voice data.
  5. 5. A computing system that enables user interaction when the user is in a mentally debilitating state, comprising:
    a computing device, comprising:
    a memory configured to store electroencephalogram (EEG) waveform signals, the EEG waveform signals being generated by associated recognition data,
    wherein response data is generated based on the associated recognition data and stored in memory;
    an input component configured to receive an EEG waveform signal from a user;
    a processor configured to compare the received EEG waveform signal with the stored EEG waveform signals to determine that the received EEG waveform signal matches a stored EEG waveform signal; and
    an output component configured to automatically output the response data generated by the associated recognition data based on the determination from the processor; and
    a server computer configured to update EEG waveform signals, associated recognition data and response data stored in the memory of the computing device.
  6. 6. The computing system according to claim 5, further comprising a system database.
  7. 7. The computing system according to claim 5, wherein the computing device is a wearable computer.
  8. 8. The computing system according to claim 5, wherein the updating is performed via wireless transmission.
  9. 9. The computing system according to claim 5, wherein the computing device includes a cellular phone.
  10. 10. The computing system according to claim 5, wherein the computing device includes a personal computer.
  11. 11. The computing system according to claim 5, wherein the associated recognition data include images data.
  12. 12. The computing system according to claim 5, wherein the associated recognition data includes voice data.
  13. 13. The computing system according to claim 5, wherein response data includes voice data.
  14. 14. The computing system according to claim 6, wherein the system database includes a viseme database.
  15. 15. The computing system according to claim 6, wherein the system database includes a conversational database.
  16. 16. The computing system according to claim 6, wherein the system database includes an autographical database.
  17. 17. A method for enabling user interaction when the user is in a mentally debilitating state, comprising:
    storing electroencephalogram (EEG) waveform signals, the EEG waveform signals being generated by associated recognition data, storing response data generated based on the associated recognition data;
    receiving an EEG waveform signal from a user;
    comparing the received EEG waveform signal with the stored EEG waveform signals;
    determining that the received EEG waveform signal matches a stored EEG waveform signal; and
    automatically outputting the response data generated by the associated recognition data based on the determining step.
  18. 18. The method according to claim 17, further comprising updating the stored EEG waveform signals, associated recognition data and response data.
  19. 19. The method according to claim 18, wherein the updating is performed via wireless transmission.
  20. 20. The method according to claim 17, further comprising creating a conversational database for the response data.
  21. 21. The method according to claim 17, further comprising creating an autographical database.
  22. 22. The method according to claim 17, wherein the user suffers from Alzheimer's disease.
US12233817 2008-09-19 2008-09-19 Alzheimer's cognitive enabler Abandoned US20100076334A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US12233817 US20100076334A1 (en) 2008-09-19 2008-09-19 Alzheimer's cognitive enabler

Applications Claiming Priority (10)

Application Number Priority Date Filing Date Title
US12233817 US20100076334A1 (en) 2008-09-19 2008-09-19 Alzheimer's cognitive enabler
ES09815029T ES2528029T3 (en) 2008-09-19 2009-09-14 cognitive enabler for Alzheimer's disease
EP20090815029 EP2344031B1 (en) 2008-09-19 2009-09-14 Alzheimer's cognitive enabler
CN 200980135706 CN102149319B (en) 2008-09-19 2009-09-14 Alzheimer's cognitive enabler
CA 2737183 CA2737183A1 (en) 2008-09-19 2009-09-14 Alzheimer's cognitive enabler
PCT/US2009/056837 WO2010033461A3 (en) 2008-09-19 2009-09-14 Alzheimer's cognitive enabler
JP2011527893A JP5643207B2 (en) 2008-09-19 2009-09-14 Computing device that enables a user interaction user when in the state of being mentally debilitating, computing systems and methods
KR20117008452A KR101520524B1 (en) 2008-09-19 2009-09-14 Alzheimer's Cognitive Enabler
US13966686 US20140180159A1 (en) 2008-09-19 2013-08-14 Alzheimer's cognitive enabler
US14724982 US20150262016A1 (en) 2008-09-19 2015-05-29 Computing device for enhancing communications

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US13966686 Continuation US20140180159A1 (en) 2008-09-19 2013-08-14 Alzheimer's cognitive enabler

Publications (1)

Publication Number Publication Date
US20100076334A1 true true US20100076334A1 (en) 2010-03-25

Family

ID=42038374

Family Applications (3)

Application Number Title Priority Date Filing Date
US12233817 Abandoned US20100076334A1 (en) 2008-09-19 2008-09-19 Alzheimer's cognitive enabler
US13966686 Abandoned US20140180159A1 (en) 2008-09-19 2013-08-14 Alzheimer's cognitive enabler
US14724982 Pending US20150262016A1 (en) 2008-09-19 2015-05-29 Computing device for enhancing communications

Family Applications After (2)

Application Number Title Priority Date Filing Date
US13966686 Abandoned US20140180159A1 (en) 2008-09-19 2013-08-14 Alzheimer's cognitive enabler
US14724982 Pending US20150262016A1 (en) 2008-09-19 2015-05-29 Computing device for enhancing communications

Country Status (7)

Country Link
US (3) US20100076334A1 (en)
EP (1) EP2344031B1 (en)
JP (1) JP5643207B2 (en)
CN (1) CN102149319B (en)
CA (1) CA2737183A1 (en)
ES (1) ES2528029T3 (en)
WO (1) WO2010033461A3 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110157221A1 (en) * 2009-12-29 2011-06-30 Ptucha Raymond W Camera and display system interactivity

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103561652B (en) * 2011-06-01 2017-02-15 皇家飞利浦有限公司 A method and system for assisting a patient
US9864431B2 (en) 2016-05-11 2018-01-09 Microsoft Technology Licensing, Llc Changing an application state using neurological data

Citations (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6011991A (en) * 1998-12-07 2000-01-04 Technology Patents, Llc Communication system and method including brain wave analysis and/or use of brain activity
US6309361B1 (en) * 1998-05-04 2001-10-30 Kirtley E. Thornton Method for improving memory by identifying and using QEEG parameters correlated to specific cognitive functioning
US6434419B1 (en) * 2000-06-26 2002-08-13 Sam Technology, Inc. Neurocognitive ability EEG measurement method and system
US6463321B2 (en) * 1999-08-10 2002-10-08 Thuris Corporation Method and computer program product for assessing neurological conditions and treatments using evoked response potentials
US6466232B1 (en) * 1998-12-18 2002-10-15 Tangis Corporation Method and system for controlling presentation of information to a user based on the user's condition
US20020188217A1 (en) * 2001-06-07 2002-12-12 Lawrence Farwell Method and apparatus for brain fingerprinting, measurement, assessment and analysis of brain function
US6513046B1 (en) * 1999-12-15 2003-01-28 Tangis Corporation Storing and recalling information to augment human memories
US6527715B2 (en) * 1998-10-30 2003-03-04 The United States Of America As Represented By The Secretary Of The Army System and method for predicting human cognitive performance using data from an actigraph
US6553252B2 (en) * 1998-10-30 2003-04-22 The United States Of America As Represented By The Secretary Of The Army Method and system for predicting human cognitive performance
US6622036B1 (en) * 2000-02-09 2003-09-16 Cns Response Method for classifying and treating physiologic brain imbalances using quantitative EEG
US6842877B2 (en) * 1998-12-18 2005-01-11 Tangis Corporation Contextual responses based on automated learning techniques
US20050228785A1 (en) * 2004-04-02 2005-10-13 Eastman Kodak Company Method of diagnosing and managing memory impairment using images
US20060008785A1 (en) * 2004-07-09 2006-01-12 William Clark Method of and system for interacting with cognitively impaired individuals
US20060009978A1 (en) * 2004-07-02 2006-01-12 The Regents Of The University Of Colorado Methods and systems for synthesis of accurate visible speech via transformation of motion capture data
US6996261B2 (en) * 2001-01-30 2006-02-07 Decharms R Christopher Methods for physiological monitoring, training, exercise and regulation
US7080322B2 (en) * 1998-12-18 2006-07-18 Tangis Corporation Thematic response to a computer user's context, such as by a wearable personal computer
US20070299360A1 (en) * 2006-06-21 2007-12-27 Lexicor Medical Technology, Llc Systems and Methods for Analyzing and Assessing Dementia and Dementia-Type Disorders
US20080138783A1 (en) * 2006-12-06 2008-06-12 Microsoft Corporation Memory training via visual journal

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8564544B2 (en) * 2006-09-06 2013-10-22 Apple Inc. Touch screen device, method, and graphical user interface for customizing display of content category icons
US7761414B2 (en) * 2007-01-07 2010-07-20 Apple Inc. Asynchronous data synchronization amongst devices

Patent Citations (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6309361B1 (en) * 1998-05-04 2001-10-30 Kirtley E. Thornton Method for improving memory by identifying and using QEEG parameters correlated to specific cognitive functioning
US6527715B2 (en) * 1998-10-30 2003-03-04 The United States Of America As Represented By The Secretary Of The Army System and method for predicting human cognitive performance using data from an actigraph
US6743167B2 (en) * 1998-10-30 2004-06-01 The United States Of America As Represented By The Secretary Of The Army Method and system for predicting human cognitive performance using data from an actigraph
US6553252B2 (en) * 1998-10-30 2003-04-22 The United States Of America As Represented By The Secretary Of The Army Method and system for predicting human cognitive performance
US6011991A (en) * 1998-12-07 2000-01-04 Technology Patents, Llc Communication system and method including brain wave analysis and/or use of brain activity
US7080322B2 (en) * 1998-12-18 2006-07-18 Tangis Corporation Thematic response to a computer user's context, such as by a wearable personal computer
US6842877B2 (en) * 1998-12-18 2005-01-11 Tangis Corporation Contextual responses based on automated learning techniques
US6466232B1 (en) * 1998-12-18 2002-10-15 Tangis Corporation Method and system for controlling presentation of information to a user based on the user's condition
US7137069B2 (en) * 1998-12-18 2006-11-14 Tangis Corporation Thematic response to a computer user's context, such as by a wearable personal computer
US6463321B2 (en) * 1999-08-10 2002-10-08 Thuris Corporation Method and computer program product for assessing neurological conditions and treatments using evoked response potentials
US6549915B2 (en) * 1999-12-15 2003-04-15 Tangis Corporation Storing and recalling information to augment human memories
US6513046B1 (en) * 1999-12-15 2003-01-28 Tangis Corporation Storing and recalling information to augment human memories
US7155456B2 (en) * 1999-12-15 2006-12-26 Tangis Corporation Storing and recalling information to augment human memories
US6622036B1 (en) * 2000-02-09 2003-09-16 Cns Response Method for classifying and treating physiologic brain imbalances using quantitative EEG
US6434419B1 (en) * 2000-06-26 2002-08-13 Sam Technology, Inc. Neurocognitive ability EEG measurement method and system
US6996261B2 (en) * 2001-01-30 2006-02-07 Decharms R Christopher Methods for physiological monitoring, training, exercise and regulation
US20020188217A1 (en) * 2001-06-07 2002-12-12 Lawrence Farwell Method and apparatus for brain fingerprinting, measurement, assessment and analysis of brain function
US20050228785A1 (en) * 2004-04-02 2005-10-13 Eastman Kodak Company Method of diagnosing and managing memory impairment using images
US20060009978A1 (en) * 2004-07-02 2006-01-12 The Regents Of The University Of Colorado Methods and systems for synthesis of accurate visible speech via transformation of motion capture data
US20060008785A1 (en) * 2004-07-09 2006-01-12 William Clark Method of and system for interacting with cognitively impaired individuals
US20070299360A1 (en) * 2006-06-21 2007-12-27 Lexicor Medical Technology, Llc Systems and Methods for Analyzing and Assessing Dementia and Dementia-Type Disorders
US20080138783A1 (en) * 2006-12-06 2008-06-12 Microsoft Corporation Memory training via visual journal

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110157221A1 (en) * 2009-12-29 2011-06-30 Ptucha Raymond W Camera and display system interactivity
US9319640B2 (en) * 2009-12-29 2016-04-19 Kodak Alaris Inc. Camera and display system interactivity

Also Published As

Publication number Publication date Type
WO2010033461A3 (en) 2010-06-10 application
US20140180159A1 (en) 2014-06-26 application
EP2344031B1 (en) 2014-10-22 grant
KR20110070981A (en) 2011-06-27 application
CA2737183A1 (en) 2010-03-25 application
CN102149319B (en) 2013-07-17 grant
JP5643207B2 (en) 2014-12-17 grant
US20150262016A1 (en) 2015-09-17 application
ES2528029T3 (en) 2015-02-03 grant
CN102149319A (en) 2011-08-10 application
EP2344031A2 (en) 2011-07-20 application
JP2012502741A (en) 2012-02-02 application
EP2344031A4 (en) 2012-11-21 application
WO2010033461A2 (en) 2010-03-25 application

Similar Documents

Publication Publication Date Title
Gunes Automatic, dimensional and continuous emotion recognition
Kapoor et al. Automatic prediction of frustration
Nasoz et al. Emotion recognition from physiological signals using wireless sensors for presence technologies
Oberman et al. Face to face: Blocking facial mimicry can selectively impair recognition of emotional expressions
US20140347265A1 (en) Wearable computing apparatus and method
US20050163302A1 (en) Customer service system and method using physiological data
US20080208015A1 (en) System, apparatus and method for real-time health feedback on a mobile device based on physiological, contextual and self-monitored indicators of mental and physical health states
US20070106127A1 (en) Automated patient monitoring and counseling system
US6878111B2 (en) System for measuring subjective well being
US20040210159A1 (en) Determining a psychological state of a subject
Weber-Fox et al. Phonologic processing in adults who stutter: Electrophysiological and behavioral evidence
Ward Stuttering and cluttering: frameworks for understanding and treatment
US20090119154A1 (en) Determining a demographic characteristic based on computational user-health testing of a user interaction with advertiser-specified content
US20060029912A1 (en) Aural rehabilitation system and a method of using the same
US20090132275A1 (en) Determining a demographic characteristic of a user based on computational user-health testing
US20090069641A1 (en) Method for analyzing stress based on multi-measured bio-signals
Croot et al. Single word production in nonfluent progressive aphasia
US20090118593A1 (en) Determining a demographic characteristic based on computational user-health testing of a user interaction with advertiser-specified content
Grünerbl et al. Smartphone-based recognition of states and state changes in bipolar disorder patients
US20150099946A1 (en) Systems, environment and methods for evaluation and management of autism spectrum disorder using a wearable data collection device
Worrall et al. The validity of functional assessments of communication and the Activity/Participation components of the ICIDH-2: do they reflect what really happens in real-life?
US20130281798A1 (en) Cognitive biometric systems to monitor emotions and stress
US20090112621A1 (en) Computational user-health testing responsive to a user interaction with advertiser-configured content
McMillan et al. Cascading influences on the production of speech: Evidence from articulation
US20090005654A1 (en) Computational user-health testing

Legal Events

Date Code Title Description
AS Assignment

Owner name: UNITHER NEUROSCIENCES, INC.,MARYLAND

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:ROTHBLATT, MARTINE;REEL/FRAME:021806/0832

Effective date: 20080919