CA3220064A1 - Methods and devices for real-time word and speech decoding from neural activity - Google Patents

Methods and devices for real-time word and speech decoding from neural activity Download PDF

Info

Publication number
CA3220064A1
CA3220064A1 CA3220064A CA3220064A CA3220064A1 CA 3220064 A1 CA3220064 A1 CA 3220064A1 CA 3220064 A CA3220064 A CA 3220064A CA 3220064 A CA3220064 A CA 3220064A CA 3220064 A1 CA3220064 A1 CA 3220064A1
Authority
CA
Canada
Prior art keywords
attempted
speech
subject
word
electrical signal
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CA3220064A
Other languages
English (en)
French (fr)
Inventor
Edward Chang
David Moses
Jessie LIU
Sean METZGER
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
University of California San Diego UCSD
Original Assignee
Individual
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Individual filed Critical Individual
Publication of CA3220064A1 publication Critical patent/CA3220064A1/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/015—Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7235—Details of waveform analysis
    • A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/25—Bioelectric electrodes therefor
    • A61B5/279—Bioelectric electrodes therefor specially adapted for particular uses
    • A61B5/291—Bioelectric electrodes therefor specially adapted for particular uses for electroencephalography [EEG]
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/25—Bioelectric electrodes therefor
    • A61B5/279—Bioelectric electrodes therefor specially adapted for particular uses
    • A61B5/291—Bioelectric electrodes therefor specially adapted for particular uses for electroencephalography [EEG]
    • A61B5/293—Invasive
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/316—Modalities, i.e. specific diagnostic methods
    • A61B5/369—Electroencephalography [EEG]
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/316—Modalities, i.e. specific diagnostic methods
    • A61B5/369—Electroencephalography [EEG]
    • A61B5/37—Intracranial electroencephalography [IC-EEG], e.g. electrocorticography [ECoG]
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/316—Modalities, i.e. specific diagnostic methods
    • A61B5/369—Electroencephalography [EEG]
    • A61B5/372—Analysis of electroencephalograms
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/40—Detecting, measuring or recording for evaluating the nervous system
    • A61B5/4058—Detecting, measuring or recording for evaluating the nervous system for evaluating the central nervous system
    • A61B5/4064—Evaluating the brain
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/48—Other medical applications
    • A61B5/4803—Speech analysis specially adapted for diagnostic purposes
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7235—Details of waveform analysis
    • A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
    • A61B5/7267—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7271—Specific aspects of physiological measurement analysis
    • A61B5/7278—Artificial waveform generation or derivation, e.g. synthesizing signals from measured signals
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/74—Details of notification to user or communication with user or patient; User input means
    • A61B5/7405—Details of notification to user or communication with user or patient; User input means using sound
    • A61B5/741—Details of notification to user or communication with user or patient; User input means using sound using synthesised speech
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/74—Details of notification to user or communication with user or patient; User input means
    • A61B5/742—Details of notification to user or communication with user or patient; User input means using visual displays
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L13/00—Speech synthesis; Text to speech systems
    • G10L13/08—Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/24—Speech recognition using non-acoustical features

Landscapes

  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Biomedical Technology (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Surgery (AREA)
  • Molecular Biology (AREA)
  • Animal Behavior & Ethology (AREA)
  • Medical Informatics (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Biophysics (AREA)
  • Pathology (AREA)
  • Veterinary Medicine (AREA)
  • Psychiatry (AREA)
  • Artificial Intelligence (AREA)
  • Psychology (AREA)
  • Physiology (AREA)
  • Neurosurgery (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Mathematical Physics (AREA)
  • Evolutionary Computation (AREA)
  • Fuzzy Systems (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Neurology (AREA)
  • Human Computer Interaction (AREA)
  • Theoretical Computer Science (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Computational Linguistics (AREA)
  • General Engineering & Computer Science (AREA)
  • Dermatology (AREA)
  • General Physics & Mathematics (AREA)
  • Machine Translation (AREA)
  • Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)
  • Electrically Operated Instructional Devices (AREA)
CA3220064A 2021-05-26 2022-05-26 Methods and devices for real-time word and speech decoding from neural activity Pending CA3220064A1 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US202163193351P 2021-05-26 2021-05-26
US63/193,351 2021-05-26
PCT/US2022/031101 WO2022251472A1 (en) 2021-05-26 2022-05-26 Methods and devices for real-time word and speech decoding from neural activity

Publications (1)

Publication Number Publication Date
CA3220064A1 true CA3220064A1 (en) 2022-12-01

Family

ID=84229189

Family Applications (1)

Application Number Title Priority Date Filing Date
CA3220064A Pending CA3220064A1 (en) 2021-05-26 2022-05-26 Methods and devices for real-time word and speech decoding from neural activity

Country Status (8)

Country Link
US (1) US20240366157A1 (enExample)
EP (1) EP4329615A4 (enExample)
JP (1) JP2024521768A (enExample)
KR (1) KR20240024095A (enExample)
CN (1) CN117693315A (enExample)
AU (1) AU2022282378A1 (enExample)
CA (1) CA3220064A1 (enExample)
WO (1) WO2022251472A1 (enExample)

Families Citing this family (27)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11790169B2 (en) * 2021-04-02 2023-10-17 Salesforce, Inc. Methods and systems of answering frequently asked questions (FAQs)
EP4569511A4 (en) * 2022-08-09 2026-04-15 Univ Leland Stanford Junior SYSTEMS AND METHODS FOR DECODING SPEECH FROM NEURAL ACTIVITY
US12530535B2 (en) * 2022-10-31 2026-01-20 Zoom Communications, Inc. Intelligent prediction of next step sentences from a communication session
CN116206150A (zh) * 2023-01-09 2023-06-02 阿里巴巴（中国）有限公司 任务处理方法、地物分类方法及任务模型训练方法
US20240264670A1 (en) * 2023-02-03 2024-08-08 Georgia Tech Research Corporation Systems and Methods for Determining the Coupling Response of a Non-Linear Variant System
CN116225222A (zh) * 2023-02-26 2023-06-06 北京航空航天大学 基于轻量级梯度提升决策树的脑机交互意图识别方法及系统
US20240398317A1 (en) * 2023-06-05 2024-12-05 Northwestern University Method and system to decode speech production from non-frontal, non-post-central brain cortices
EP4704703A1 (en) * 2023-06-06 2026-03-11 The Regents of University of California Methods and systems for translation of neural activity into embodied digital-avatar animation
US12469507B2 (en) 2023-06-14 2025-11-11 Microsoft Technology Licensing, Llc Predictive context-based decoder correction
US20240419895A1 (en) * 2023-06-14 2024-12-19 Microsoft Technology Licensing, Llc Context-based decoder correction
WO2025076530A1 (en) * 2023-10-06 2025-04-10 Precision Neuroscience Corporation Systems and methods for visualizing brain activity in real time at high spatial and temporal resolution
CN117058514B (zh) * 2023-10-12 2024-04-02 之江实验室 基于图神经网络的多模态脑影像数据融合解码方法和装置
WO2025080841A1 (en) 2023-10-12 2025-04-17 The Regents Of The University Of California Methods for inside-out deployment of microelectrode arrays and devices and systems for same
CN117130490B (zh) * 2023-10-26 2024-01-26 天津大学 一种脑机接口控制系统及其控制方法和实现方法
CN117131426B (zh) * 2023-10-26 2024-01-19 一网互通(北京)科技有限公司 基于预训练的品牌识别方法、装置及电子设备
CN117238277B (zh) * 2023-11-09 2024-01-19 北京水滴科技集团有限公司 意图识别方法、装置、存储介质及计算机设备
CN117851769B (zh) * 2023-11-30 2024-06-21 浙江大学 一种面向侵入式脑机接口的汉字书写解码方法
US20250218434A1 (en) * 2023-12-29 2025-07-03 Cx360, Inc. Automated prompt finder
CN117708546B (zh) * 2024-02-05 2024-05-10 北京智冉医疗科技有限公司 基于侵入式脑机接口的高通量神经信号的解码方法及装置
US12588859B2 (en) * 2024-03-21 2026-03-31 The Education University Of Hong Kong System and method for interacting with human brain activities using EEG-fNIRS neurofeedback
CN118095447B (zh) * 2024-04-12 2024-06-25 清华大学 大语言模型分布式推理方法及装置、介质
CN118095295B (zh) * 2024-04-28 2024-07-09 昆明理工大学 渐进式预训练和提示增强低资源语言的跨语言摘要方法
CN118766414B (zh) * 2024-04-30 2025-05-09 中国科学院心理研究所 一种基于跨模态脑功能连接图谱的书写能力脑指纹构建方法及系统
WO2025235580A1 (en) * 2024-05-08 2025-11-13 The Regents Of The University Of California Systems and methods for decoding biosignals of a person indicative of speech
CN119402342B (zh) * 2024-09-24 2026-04-17 中国南方电网有限责任公司 电力通信网的故障定位方法、系统及电子设备
CN119691418B (zh) * 2024-11-26 2025-11-14 神州数码(中国)有限公司 模型评估方法、装置、电子设备和计算机可读存储介质
CN121096350A (zh) * 2025-09-03 2025-12-09 哈尔滨工业大学 一种基于脑电信号的语音解码方法、系统、设备及介质

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014102722A1 (en) * 2012-12-26 2014-07-03 Sia Technology Ltd. Device, system, and method of controlling electronic devices via thought
US10130809B2 (en) * 2014-06-13 2018-11-20 Nervana, LLC Transcutaneous electrostimulator and methods for electric stimulation
US12008987B2 (en) * 2018-04-30 2024-06-11 The Board Of Trustees Of The Leland Stanford Junior University Systems and methods for decoding intended speech from neuronal activity
WO2021021714A1 (en) * 2019-07-29 2021-02-04 The Regents Of The University Of California Method of contextual speech decoding from the brain

Also Published As

Publication number Publication date
KR20240024095A (ko) 2024-02-23
WO2022251472A1 (en) 2022-12-01
CN117693315A (zh) 2024-03-12
AU2022282378A1 (en) 2023-12-14
JP2024521768A (ja) 2024-06-04
US20240366157A1 (en) 2024-11-07
WO2022251472A9 (en) 2023-11-09
EP4329615A1 (en) 2024-03-06
EP4329615A4 (en) 2025-01-01

Similar Documents

Publication Publication Date Title
US20240366157A1 (en) Methods And Devices For Real-Time Word And Speech Decoding From Neural Activity
Metzger et al. Generalizable spelling using a speech neuroprosthesis in an individual with severe limb and vocal paralysis
Metzger et al. A high-performance neuroprosthesis for speech decoding and avatar control
Makin et al. Machine translation of cortical activity to text with an encoder–decoder framework
Moses et al. Real-time decoding of question-and-answer speech dialogue using human cortical activity
Willett et al. A high-performance speech neuroprosthesis
Moses et al. Neuroprosthesis for decoding speech in a paralyzed person with anarthria
US20250252958A1 (en) Method of Contextual Speech Decoding From the Brain
US20260060621A1 (en) Detection of disease conditions and comorbidities
Livezey et al. Deep learning as a tool for neural data analysis: speech classification and cross-frequency coupling in human sensorimotor cortex
Bouchard et al. Functional organization of human sensorimotor cortex for speech articulation
Mora-Cortes et al. Language model applications to spelling with brain-computer interfaces
Kunz et al. Inner speech in motor cortex and implications for speech neuroprostheses
Qian et al. A survey of automatic speech recognition for dysarthric speech
EP4704703A1 (en) Methods and systems for translation of neural activity into embodied digital-avatar animation
Gwilliams et al. Neural dynamics of phoneme sequencing in real speech jointly encode order and invariant content
Wu et al. Adaptive LDA classifier enhances real-time control of an EEG brain–computer interface for decoding imagined syllables
Stavisky Restoring speech using brain–computer interfaces
Zhang et al. A brain-to-text framework for decoding natural tonal sentences
Feng et al. Acoustic inspired brain-to-sentence decoder for logosyllabic language
Qiu et al. A review of brain–computer interface-based language decoding: From signal interpretation to intelligent communication
Tan et al. Effective phoneme decoding with hyperbolic neural networks for high-performance speech BCIs
Alonso-Vázquez et al. EEG-based classification of spoken words using machine learning approaches
Yang et al. Recognizing tonal and nontonal mandarin sentences for EEG-based brain–computer interface
Salomons et al. Frame-Based Phone Classification Using EMG Signals

Legal Events

Date Code Title Description
MFA Maintenance fee for application paid

Free format text: FEE DESCRIPTION TEXT: MF (APPLICATION, 3RD ANNIV.) - STANDARD

Year of fee payment: 3

U00 Fee paid

Free format text: ST27 STATUS EVENT CODE: A-1-1-U10-U00-U101 (AS PROVIDED BY THE NATIONAL OFFICE); EVENT TEXT: MAINTENANCE REQUEST RECEIVED

Effective date: 20250516

U11 Full renewal or maintenance fee paid

Free format text: ST27 STATUS EVENT CODE: A-1-1-U10-U11-U102 (AS PROVIDED BY THE NATIONAL OFFICE); EVENT TEXT: MAINTENANCE FEE PAYMENT PAID IN FULL

Effective date: 20250516