US10496168B2 - Calibration techniques for handstate representation modeling using neuromuscular signals - Google Patents

Calibration techniques for handstate representation modeling using neuromuscular signals Download PDF

Info

Publication number
US10496168B2
US10496168B2 US16/257,979 US201916257979A US10496168B2 US 10496168 B2 US10496168 B2 US 10496168B2 US 201916257979 A US201916257979 A US 201916257979A US 10496168 B2 US10496168 B2 US 10496168B2
Authority
US
United States
Prior art keywords
user
gesture
statistical model
neuromuscular
calibration
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
US16/257,979
Other languages
English (en)
Other versions
US20190227627A1 (en
Inventor
Patrick Kaifosh
Tudor Giurgica-Tiron
Adam Berenzweig
Steven Kober
Adam Al-natsheh
Alexandre Barachant
Zhuo Wang
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Meta Platforms Technologies LLC
Original Assignee
Ctrl Labs Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to US16/257,979 priority Critical patent/US10496168B2/en
Application filed by Ctrl Labs Corp filed Critical Ctrl Labs Corp
Publication of US20190227627A1 publication Critical patent/US20190227627A1/en
Assigned to COGNESCENT CORPORATION reassignment COGNESCENT CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: GIURGICA-TIRON, Tudor
Assigned to CTRL-LABS CORPORATION reassignment CTRL-LABS CORPORATION CHANGE OF NAME (SEE DOCUMENT FOR DETAILS). Assignors: COGNESCENT CORPORATION
Assigned to CTRL-LABS CORPORATION reassignment CTRL-LABS CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: AL-NATSHEH, Adam, Barachant, Alexandre, BERENZWEIG, ADAM, KAIFOSH, PATRICK, KOBER, STEVEN, WANG, ZHUO
Priority to US16/671,066 priority patent/US11163361B2/en
Publication of US10496168B2 publication Critical patent/US10496168B2/en
Application granted granted Critical
Assigned to FACEBOOK TECHNOLOGIES, INC. reassignment FACEBOOK TECHNOLOGIES, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: CTRL-LABS CORPORATION
Assigned to FACEBOOK TECHNOLOGIES, LLC reassignment FACEBOOK TECHNOLOGIES, LLC CORRECTIVE ASSIGNMENT TO CORRECT THE RECEIVING PARTY DATA PREVIOUSLY RECORDED AT REEL: 051649 FRAME: 0001. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Assignors: CTRL-LABS CORPORATION
Priority to US17/741,263 priority patent/US20220269346A1/en
Assigned to META PLATFORMS TECHNOLOGIES, LLC reassignment META PLATFORMS TECHNOLOGIES, LLC CHANGE OF NAME (SEE DOCUMENT FOR DETAILS). Assignors: FACEBOOK TECHNOLOGIES, LLC
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/015—Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
    • A61B5/1121—Determining geometric values, e.g. centre of rotation or angular range of movement
    • A61B5/1122—Determining geometric values, e.g. centre of rotation or angular range of movement of movement trajectories
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
    • A61B5/1126—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb using a particular sensing technique
    • A61B5/1128—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb using a particular sensing technique using image analysis
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/316—Modalities, i.e. specific diagnostic methods
    • A61B5/389—Electromyography [EMG]
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/68—Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient
    • A61B5/6801—Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient specially adapted to be attached to or worn on the body surface
    • A61B5/6813—Specially adapted to be attached to a specific body part
    • A61B5/6824—Arm or wrist
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/014—Hand-worn input/output arrangements, e.g. data gloves
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0346—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B2560/00—Constructional details of operational features of apparatus; Accessories for medical measuring apparatus
    • A61B2560/02—Operational features
    • A61B2560/0223—Operational features of calibration, e.g. protocols for calibrating sensors
    • A61B2560/0238—Means for recording calibration data
    • A61B5/04004—
    • A61B5/04012—
    • A61B5/0492—
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
    • A61B5/1107—Measuring contraction of parts of the body, e.g. organ, muscle
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
    • A61B5/1112—Global tracking of patients, e.g. by using GPS
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
    • A61B5/1113—Local tracking of patients, e.g. in a hospital or private home
    • A61B5/1114—Tracking parts of the body
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/25—Bioelectric electrodes therefor
    • A61B5/279—Bioelectric electrodes therefor specially adapted for particular uses
    • A61B5/296—Bioelectric electrodes therefor specially adapted for particular uses for electromyography [EMG]
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/30—Input circuits therefor
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/316—Modalities, i.e. specific diagnostic methods

Definitions

  • the application In some computer applications that generate musculoskeletal representations of the human body, it is desirable for the application to know or infer the spatial positioning, orientation and movement of a user's body to provide a realistic representation of body movement. For example, in a virtual reality (VR) environment, tracking the spatial position of the user's hand enables the application to represent hand motion in the VR environment, which allows the user to interact with (e.g., by grasping or manipulating) virtual objects within the VR environment.
  • VR virtual reality
  • Some existing techniques for tracking movements of a user's body using wearable sensors include using information obtained from multiple Inertial Measurement Units (IMUs) affixed to different parts of the user's body, and using external imaging devices (e.g., fixed-position cameras) to reconstruct the position and orientation of parts of the user's body.
  • IMUs Inertial Measurement Units
  • external imaging devices e.g., fixed-position cameras
  • Some embodiments are directed to predicting information about the positioning and movements of portions of a user's body (e.g., arm and/or hand) represented as a multi-segment articulated rigid body system with joints connecting the multiple segments of the rigid body system.
  • Signals recorded by wearable neuromuscular sensors placed at locations on the user's body are provided as input to a statistical model trained to predict estimates of the position (e.g., absolute position, relative position, orientation) and forces associated with a plurality of rigid segments in a computer-based representation (e.g., a musculoskeletal representation associated with a hand) when a user performs one or more movements.
  • a handstate of the musculoskeletal representation.
  • a trained statistical model interprets neuromuscular signals recorded by the wearable neuromuscular sensors into position and force estimates (handstate information) that are used to update the computer-based representation.
  • the computer-based representation is updated in real time and a visual representation (e.g., of a hand within a virtual reality environment) is optionally rendered based on the current handstate estimates.
  • inventions are directed to a computerized system configured to calibrate performance of one or more statistical models used to generate a musculoskeletal representation.
  • the system comprises a user interface configured to instruct a user to perform at least one gesture while wearing a wearable device having a plurality of neuromuscular sensors arranged thereon, and at least one computer processor.
  • the at least one computer processor is programmed to control presentation of instructions via the user interface to instruct the user to perform the at least one gesture; and update at least one parameter of the one or more statistical models based, at least in part on a plurality of neuromuscular signals recorded by the plurality of neuromuscular sensors during performance of the at least one gesture by the user.
  • inventions are directed to a computerized system configured to calibrate performance of one or more statistical models used to generate a musculoskeletal representation.
  • the system comprises at least one computer processor programmed to identify, based on an output of the one or more statistical models, at least one gesture characteristic that the one or more statistical model is poor at estimating, initiate a calibration routine to update at least one parameter of the one or more statistical models to improve the performance of the one or more statistical models to accurately predict the at least one identified gesture characteristic, receive a plurality of neuromuscular signals recorded by a plurality of neuromuscular sensors during performance of at least one gesture that includes the at least one gesture characteristic, and update the at least one parameter of the one or more statistical models based, at least in part on a plurality of neuromuscular signals recorded by the neuromuscular sensors during performance of the at least one gesture by the user.
  • the at least one computer processor is further programmed to update at least one parameter of the one or more statistical models based, at least in part, on predetermined estimates of one or more of position information, force information, and neuromuscular information for the at least one gesture.
  • the at least one computer processor is further programmed to: control at least one external device to capture at least one measurement of position and/or force information during performance of the at least one gesture; and update the at least one parameter of the one or more statistical models based, at least in part, on the at least one measurement of position and/or force information.
  • the at least one external device comprises an imaging device configured to capture at least one image, and the at least one measurement of position is determined based, at least in part, on the at least one image.
  • the at least one external device comprises an inertial measurement unit.
  • the at least one external device comprises a plurality of external devices, each of which is configured to capture at least one measurement of position and/or force information during performance of the at least one gesture
  • the at least one computer processor is further programmed to: combine the at least one measurement of position and/or force information captured by each of the plurality of external devices; and update the at least one parameter of the one or more statistical models based, at least in part, on the combined measurements of position and/or force.
  • updating the at least one parameter of the one or more statistical models comprises training the one or more statistical models based, at least in part, on the plurality of neuromuscular signals recorded by the neuromuscular sensors during performance of the at least one gesture by the user.
  • the one or more statistical models include a trained statistical model, and updating the at least one parameter of the one or more statistical models comprises adjusting one or more parameters provided as input to the trained statistical model without retraining the trained statistical model.
  • the at least one computer processor is further programmed to provide feedback to the user, and the feedback indicates to the user whether the performance of the at least one gesture has been performed as instructed.
  • providing feedback to the user comprises providing visual feedback.
  • instructing the user to perform at least one gesture comprises instructing the user to perform a predetermined set of gestures.
  • controlling presentation of the instructions via the user interface comprises instructing the user to perform the gestures of the predetermined set in succession.
  • the at least one computer processor is further programmed to select a new gesture to be performed by the user.
  • selecting a new gesture to be performed by the user comprises randomly selecting the new gesture from a set of gestures.
  • selecting a new gesture to be performed by the user comprises selecting a gesture previously performed by the user.
  • the at least one computer processor is further programmed to: identify, based on an output of the one or more statistical models, at least one gesture characteristic that the one or more statistical model is poor at estimating; and select the new gesture to be performed by the user based on the identified at least one gesture characteristic.
  • the at least one gesture characteristic comprises a plurality of gesture characteristics
  • selecting the new gesture to be performed by the user based on the identified at least one gesture characteristic comprises selecting a new gesture that includes the plurality of gesture characteristics
  • each of a first gesture and a second gesture in a set of gestures includes at least two of the plurality of gesture characteristics
  • selecting the new gesture to be performed by the user comprises selecting the first gesture as the new gesture when the first gesture includes more of the plurality of gesture characteristics than the second gesture.
  • the at least one computer processor is further programmed to: control presentation of instructions via the user interface to instruct the user to perform the new gesture; and update at least one parameter of the one or more statistical models based, at least in part on the plurality of neuromuscular signals recorded by the neuromuscular sensors during performance of the new gesture by the user.
  • the instructions provided via the user interface includes at least one user-interpretable instruction.
  • the at least one user-interpretable instruction comprises a percentage of force exerted by one or more portions of a hand of the user.
  • the at least one user-interpretable instruction comprises a percentage of maximum force exerted by pinching at least two fingers of the hand.
  • the at least one computer processor is further programmed to provide feedback to the user, wherein the feedback indicates to the user a response of the one or more statistical models to performance of the at least one gesture by the user based on the user-interpretable instruction.
  • the at least one computer processor is further programmed to render a visual representation associated with the hand based on an output of the one or more statistical models.
  • the output of the one or more statistical models comprises position and force estimates
  • rendering the visual representation associated with the hand comprises rendering at least one force metric based on the force estimates on the visual representation associated with the hand.
  • the at least one computer processor is further programmed to provide feedback to the user describing information about the accuracy and/or completeness of the at least one gesture.
  • the visual representation is rendered during the updating of the at least one parameter of the one or more statistical models.
  • the at least one computer processor is further programmed to: continuously track a calibration of a performance of the one or more statistical models during usage of the one or more statistical models following training; and initiate a calibration routine when the calibration of the performance deviates from an acceptable calibration threshold by a particular amount.
  • initiating a calibration routine comprises: controlling presentation of instructions via the user interface to instruct the user to perform one or more additional gestures that includes at least one gesture characteristic for which performance of the one or more statistical models is poor, and updating at least one parameter of at least one of the one or more statistical models based, at least in part, on the plurality of neuromuscular signals recorded by the neuromuscular sensors during performance of the one or more additional gestures by the user.
  • the musculoskeletal representation is a musculoskeletal representation associated with a hand
  • the one or more statistical model are configured to output position and force estimates used to update the musculoskeletal representation associated with the hand.
  • updating at least one parameter of the one or more statistical models comprises updating a plurality of model parameters
  • the at least one computer processor is further programmed to: store a user profile that includes user-specific calibration parameters, wherein the user-specific calibration parameters includes a subset of the plurality of model parameters.
  • the user-specific calibration parameters include at least one joint stiffness parameter.
  • Other embodiments are directed to a computerized system configured to calibrate performance of one or more statistical models used to generate a musculoskeletal representation, the system comprising: at least one computer processor programmed to: identify, based on an output of the one or more statistical models, at least one gesture characteristic that the one or more statistical model is poor at estimating; initiate a calibration routine to update at least one parameter of the one or more statistical models to improve the performance of the one or more statistical models to accurately predict the at least one identified gesture characteristic; receive a plurality of neuromuscular signals recorded by a plurality of neuromuscular sensors during performance of at least one gesture that includes the at least one gesture characteristic; and update the at least one parameter of the one or more statistical models based, at least in part, on a plurality of neuromuscular signals recorded by the neuromuscular sensors during performance of the at least one gesture by the user.
  • Other embodiments are directed to a method of calibrating performance of one or more statistical models used to generate a musculoskeletal representation.
  • the method comprises controlling presentation of instructions via a user interface to instruct the user to perform the at least one gesture, and updating at least one parameter of the one or more statistical models based, at least in part, on a plurality of neuromuscular signals recorded by a plurality of neuromuscular sensors during performance of the at least one gesture by the user.
  • inventions are directed to a computer-readable medium encoded with a plurality of instructions that, when executed by at least one processor, perform a method.
  • the method comprises controlling presentation of instructions via a user interface to instruct the user to perform the at least one gesture, and updating at least one parameter of the one or more statistical models based, at least in part, on a plurality of neuromuscular signals recorded by a plurality of neuromuscular sensors during performance of the at least one gesture by the user.
  • updating the statistical model configuration comprises selecting a different statistical model to generate the updated musculoskeletal representation than the statistical model used to generate to the musculoskeletal representation.
  • Other embodiments are directed to a method of calibrating performance of one or more statistical models used to generate a musculoskeletal representation, the method comprising: instructing, via a user interface, a user to perform at least one gesture while wearing a wearable device having a plurality of neuromuscular sensors arranged thereon; controlling presentation of instructions via the user interface to instruct the user to perform the at least one gesture; updating at least one parameter of the one or more statistical models based, at least in part, on a plurality of neuromuscular signals recorded by the plurality of neuromuscular sensors during performance of the at least one gesture by the user; identifying, based on an output of the one or more statistical models, a plurality of gesture characteristics that the one or more statistical model is poor at estimating; selecting a new gesture for the user to perform that includes the identified plurality of gesture characteristics; controlling presentation of instructions via the user interface to instruct the user to perform the new gesture; and updating at least one parameter of the one or more statistical models based, at least in part on the plurality of neuromuscular signals recorded by the
  • FIG. 1 is a schematic diagram of a computer-based system for generating a musculoskeletal representation based on neuromuscular sensor data in accordance with some embodiments of the technology described herein;
  • FIG. 2 is a flowchart of an illustrative process for calibrating performance of one or more statistical models in accordance of some embodiments of the technology described herein;
  • FIG. 3 is a flowchart of an illustrative process for calibrating performance of a statistical model in accordance with some embodiments of the technology described herein;
  • FIG. 4 is a flowchart of an illustrative process for initiating a calibration routine to update parameters of a trained statistical model during runtime in accordance with some embodiments of the technology described herein;
  • FIG. 5 is a flowchart of an illustrative process for generating a statistical model for predicting musculoskeletal position information using signals recorded from sensors, in accordance with some embodiments of the technology described herein;
  • FIG. 6A illustrates a wearable system with sixteen EMG sensors arranged circumferentially around an elastic band configured to be worn around a user's lower arm or wrist, in accordance with some embodiments of the technology described herein;
  • FIG. 6B is a cross-sectional view through one of the sixteen EMG sensors illustrated in FIG. 6A ;
  • FIGS. 7A and 7B schematically illustrate components of a computer-based system on which some embodiments are implemented.
  • FIG. 7A illustrates a wearable portion of the computer-based system
  • FIG. 7B illustrates a dongle portion connected to a computer, wherein the dongle portion is configured to communicate with the wearable portion.
  • Robustness, consistency, and reliability are important features of a system for estimating the position, movement, and force exerted by a part of the human body, in part, because such features support realistic highly-immersive experiences in virtual environments and support the use of gesture-based machine control frameworks.
  • Some conventional systems use neuromuscular signals to estimate the position and movement of a portion of a user's body represented as a multi-segment articulated rigid body system (e.g., a user's arm, hand, leg, etc.) in an autonomous manner.
  • System calibration is typically performed in such systems using a sequence of prescribed movements or by changing the gain of signal amplification circuitry, though these systems generally have limited capabilities with respect to tuning, personalization, and retraining.
  • Some conventional camera-based systems for tracking a user's hand perform system calibration by calibrating imaging sensors or by configuring the system to distinguish individual users.
  • Camera-based systems have some limitations relative to systems that use neuromuscular sensors to infer the position or movement of a part of the body, particularly with regard to occlusion and limited spatial functionality for fixed-camera based systems.
  • Neuromuscular signals contain information about motor-unit activity levels which determine the movements of joints and exerted forces.
  • the degree to which estimates of joint angles and exerted forces generalize across users is limited by, variability between people with regard to characteristics such as joint stiffness, joint segment moments and inertia, and the relationship between motor unit activity and joint torque.
  • the models used to make the predictions may perform sub-optimally.
  • some embodiments are directed to techniques for tuning model parameters based on neuromuscular data collected during a calibration procedure to account for user-specific physiological and anatomical differences.
  • All or portions of the human musculoskeletal system can be logically represented in a model implemented in a multi-segment articulated rigid body system.
  • a model may be partitioned into multiple data structures including segments, logical interfaces, and other suitable data structures which, in ensemble, emulate biological functions and properties of the human musculoskeletal system.
  • segments can represent rigid parts of the musculoskeletal system and logical interfaces can represent joints of the musculoskeletal system.
  • Each of the data structures (e.g., segments and logical interfaces) can include attributes (variable and constant) denoting properties of the biological part that such data structure represents.
  • the multi-segment articulated rigid body system may be configured to emulate biological functions and properties of the human musculoskeletal system.
  • All or portions of the human musculoskeletal system can be modeled as a multi-segment articulated rigid body system, with joint interfaces between different segments and joint angles defining the spatial relationships between connected segments in the model. Constraints on the movement at the joints are governed by the type of joint connecting the segments and the biological structures (e.g., muscles, tendons, ligaments, and other biological structures) that restrict the range of movement at the joint.
  • the shoulder joint connecting the upper arm to the torso and the hip joint connecting the upper leg to the torso are ball and socket joints that permit extension and flexion movements as well as rotational movements.
  • the elbow joint connecting the upper arm and the forearm and the knee joint connecting the upper leg and the lower leg allow for a more limited range of motion.
  • the multi-segment articulated rigid body system can be used to model portions of the human musculoskeletal system.
  • Some segments of the human musculoskeletal system e.g., the forearm segment
  • may include multiple rigid structures e.g., the ulna and radius bones of the forearm
  • a model of an articulated rigid body system for use with some embodiments of the technology described herein may include segments that represent a combination of body parts that are not strictly rigid bodies.
  • rigid bodies are objects that exhibit various attributes of motion (e.g., position, orientation, angular velocity, acceleration). Knowing the motion attributes of one segment of the rigid body enables the motion attributes for other segments of the rigid body to be determined based on constraints in how the segments are connected. For example, a hand can be modeled as a multi-segment articulated body with the joints in the wrist and each finger forming the interfaces between the multiple segments in the model. Movements of the segments in the rigid body model can be emulated as an articulated rigid body system in which position (e.g., actual position, relative position, or orientation) information of a segment relative to other segments in the model are predicted using a trained statistical model, as described in more detail below.
  • position e.g., actual position, relative position, or orientation
  • the multi-segment articulated rigid body system emulates portions of the human body, for example, a hand or a combination of a hand with one or more portions of an arm.
  • Information used to describe position relationships between segments and force relationships for individual segments or combinations of segments in the musculoskeletal representation is referred to herein as the handstate of the musculoskeletal representation.
  • the techniques described herein are also applicable to musculoskeletal representations of portions of the body other than the hand including, but not limited to, an arm, a leg, a foot, a torso, a neck, or any combination of the foregoing.
  • the multi-segment articulated rigid body system is configured to predict force information associated with one or more segments of the musculoskeletal representation.
  • the multi-segment articulated rigid body system can compute linear forces or rotational (torque) forces exerted by one or more segments.
  • linear forces include, but are not limited to, the force of a finger or hand pressing on a solid object such as a table, and a force exerted when two segments (e.g., two fingers) are pinched together.
  • rotational forces include, but are not limited to, rotational forces created when segments in the wrist or fingers are twisted or flexed.
  • a handstate can further include force information determined as a portion of a current handstate estimate that includes one or more of pinching force information, grasping force information, or information about co-contraction forces between muscles represented by the musculoskeletal representation.
  • FIG. 1 illustrates a system 100 in accordance with some embodiments.
  • the system includes a plurality of sensors 102 configured to record signals resulting from the movement of portions of a human body.
  • Sensors 102 may include autonomous sensors.
  • autonomous sensors refers to sensors configured to measure the movement of body segments without requiring the use of external devices.
  • sensors 102 may also include one or more auxiliary (e.g., non-autonomous) sensors in combination with autonomous sensors.
  • auxiliary sensors include, but are not limited to, wearable (e.g. body-mounted) cameras, global positioning systems, or laser scanning systems.
  • Autonomous sensors may include a plurality of neuromuscular sensors configured to record signals arising from neuromuscular activity in skeletal muscle of a human body.
  • the term “neuromuscular activity” as used herein refers to neural activation of spinal motor neurons that innervate a muscle, muscle activation, muscle contraction, or any combination of the neural activation, muscle activation, and muscle contraction.
  • Neuromuscular sensors may include one or more electromyography (EMG) sensors, one or more mechanomyography (MMG) sensors, one or more sonomyography (SMG) sensors, a combination of two or more types of EMG sensors, MMG sensors, and SMG sensors, and/or one or more sensors of any suitable type that are configured to detect neuromuscular signals.
  • EMG electromyography
  • MMG mechanomyography
  • SMG sonomyography
  • the plurality of neuromuscular sensors may be used to sense muscular activity related to a movement of the part of the body controlled by muscles from which the neuromuscular sensors are arranged to sense the muscle activity. Spatial information (e.g., position and/or orientation information) and force information describing the movement may be predicted based on the sensed neuromuscular signals as the user moves over time.
  • Spatial information e.g., position and/or orientation information
  • force information describing the movement may be predicted based on the sensed neuromuscular signals as the user moves over time.
  • Autonomous sensors may include one or more Inertial Measurement Units (IMUs), which measure a combination of physical aspects of motion, using, for example, an accelerometer, a gyroscope, a magnetometer, or any combination of one or more accelerometers, gyroscopes and magnetometers.
  • IMUs may be used to sense information about the movement of the part of the body on which the IMU is attached and information derived from the sensed data (e.g., position and/or orientation information) may be tracked as the user moves over time.
  • one or more IMUs may be used to track movements of portions of a user's body proximal to the user's torso relative to the sensor (e.g., arms, legs) as the user moves over time.
  • the IMU(s) and neuromuscular sensors may be arranged to detect movement of different parts of the human body.
  • the IMU(s) may be arranged to detect movements of one or more body segments proximal to the torso (e.g., an upper arm), whereas the neuromuscular sensors may be arranged to detect movements of one or more body segments distal to the torso (e.g., a forearm or wrist).
  • autonomous sensors may be arranged in any suitable way, and embodiments of the technology described herein are not limited based on the particular sensor arrangement.
  • At least one IMU and a plurality of neuromuscular sensors may be co-located on a body segment to track movements of body segment using different types of measurements.
  • an IMU sensor and a plurality of EMG sensors are arranged on a wearable device configured to be worn around the lower arm or wrist of a user.
  • the IMU sensor may be configured to track movement information (e.g., positioning and/or orientation over time) associated with one or more arm segments, to determine, for example whether the user has raised or lowered their arm
  • the EMG sensors may be configured to determine movement information associated with wrist or hand segments to determine, for example, whether the user has an open or closed hand configuration.
  • Each of the autonomous sensors includes one or more sensing components configured to sense information about a user.
  • the sensing components may include one or more accelerometers, gyroscopes, magnetometers, or any combination thereof to measure characteristics of body motion, examples of which include, but are not limited to, acceleration, angular velocity, and sensed magnetic field around the body.
  • the sensing components may include, but are not limited to, electrodes configured to detect electric potentials on the surface of the body (e.g., for EMG sensors) vibration sensors configured to measure skin surface vibrations (e.g., for MMG sensors), and acoustic sensing components configured to measure ultrasound signals (e.g., for SMG sensors) arising from muscle activity.
  • the output of one or more of the sensing components may be processed using hardware signal processing circuitry (e.g., to perform amplification, filtering, and/or rectification).
  • at least some signal processing of the output of the sensing components may be performed in software.
  • signal processing of autonomous signals recorded by the autonomous sensors may be performed in hardware, software, or by any suitable combination of hardware and software, as aspects of the technology described herein are not limited in this respect.
  • the recorded sensor data may be processed to compute additional derived measurements that are then provided as input to a statistical model, as described in more detail below.
  • recorded signals from an IMU sensor may be processed to derive an orientation signal that specifies the orientation of a rigid body segment over time.
  • Autonomous sensors may implement signal processing using components integrated with the sensing components, or at least a portion of the signal processing may be performed by one or more components in communication with, but not directly integrated with the sensing components of the autonomous sensors.
  • At least some of the plurality of autonomous sensors are arranged as a portion of a wearable device configured to be worn on or around part of a user's body.
  • a wearable device configured to be worn on or around part of a user's body.
  • an IMU sensor and a plurality of neuromuscular sensors are arranged circumferentially around an adjustable and/or elastic band such as a wristband or armband configured to be worn around a user's wrist or arm.
  • at least some of the autonomous sensors may be arranged on a wearable patch configured to be affixed to a portion of the user's body.
  • multiple wearable devices, each having one or more IMUs and/or neuromuscular sensors included thereon may be used to predict musculoskeletal position information for movements that involve multiple parts of the body.
  • sensors 102 only includes a plurality of neuromuscular sensors (e.g., EMG sensors). In other embodiments, sensors 102 includes a plurality of neuromuscular sensors and at least one “auxiliary” sensor configured to continuously record a plurality of auxiliary signals.
  • auxiliary sensors include, but are not limited to, other autonomous sensors such as IMU sensors, and auxiliary sensors such as an imaging device (e.g., a camera), a radiation-based sensor for use with a radiation-generation device (e.g., a laser-scanning device), or other types of sensors such as a heart-rate monitor.
  • System 100 also includes one or more computer processors programmed to communicate with sensors 102 .
  • signals recorded by one or more of the sensors may be provided to the processor(s), which may be programmed to execute one or more machine learning algorithms that process signals output by the sensors 102 to train one or more statistical models 104 , and the trained (or retrained) statistical model(s) 104 may be stored for later use in generating a musculoskeletal representation 106 , as described in more detail below.
  • the statistical model may be implemented using one or more neural networks.
  • Such neural network(s) may be implemented as one or more recurrent neural networks.
  • the recurrent neural network may be a fully recurrent neural network, a recursive neural network, a variational autoencoder, a Hopfield neural network, an associative memory neural network, an Elman neural network, a Jordan neural network, an echo state neural network, a second order recurrent neural network, and/or any other suitable type of recurrent neural network.
  • neural networks that are not recurrent neural networks may be used.
  • deep neural networks, convolutional neural networks, and/or feedforward neural networks may be used.
  • the neural network(s) used in accordance with some embodiments may be implemented based on a variety of topologies and/or architectures including deep neural networks with fully connected (dense) layers, Long Short-Term Memory (LSTM) layers, convolutional layers, Temporal Convolutional Layers (TCL), or other suitable type of deep neural network topology and/or architecture.
  • the neural network(s) may have different types of output layers including output layers with logistic sigmoid activation functions, hyperbolic tangent activation functions, linear units, rectified linear units, or other suitable type of nonlinear unit.
  • the neural network(s) may be configured to represent the probability distribution over n different classes via, for example, a softmax function or include an output layer that provides a parameterized distribution e.g., mean and variance of a Gaussian distribution.
  • a softmax function or include an output layer that provides a parameterized distribution e.g., mean and variance of a Gaussian distribution.
  • Other non-limiting examples of statistical models that may be used in accordance with some embodiments to predict handstate information based on recorded signals from sensors 102 are discussed in more detail below in connection with FIG. 5 .
  • System 100 also optionally includes a display controller configured to display a visual representation 108 (e.g., of a hand).
  • a display controller configured to display a visual representation 108 (e.g., of a hand).
  • one or more computer processors may implement one or more trained statistical models configured to predict handstate information based, at least in part, on signals recorded by sensors 102 .
  • the predicted handstate information is used to update the musculoskeletal representation 106 , which is then optionally used to render a visual representation 108 based on the updated musculoskeletal representation incorporating the current handstate information.
  • Real-time reconstruction of the current handstate and subsequent rendering of the visual representation reflecting the current handstate information in the musculoskeletal model may provide visual feedback to the user about the effectiveness of the trained statistical model to accurately represent an intended handstate.
  • system 100 include components configured to render a visual representation.
  • handstate estimates output from the trained statistical model and a corresponding updated musculoskeletal representation are used to determine a state of a user's hand (e.g., in a virtual reality environment) even though a visual representation based on the updated musculoskeletal representation is not rendered.
  • a computer application configured to simulate a virtual reality environment may be instructed to display a visual representation of the user's hand. Positioning, movement, and/or forces applied by portions of the hand within the virtual reality environment may be displayed based on the output of the trained statistical model(s).
  • the visual representation may be dynamically updated based on current reconstructed handstate information as continuous signals are recorded by the sensors 102 and processed by the trained statistical model(s) 104 to provide an updated computer-generated representation of the user's position, movement, and/or force that is updated in real-time.
  • some embodiments are directed to using a statistical model for predicting musculoskeletal information based on signals recorded from wearable autonomous sensors.
  • the statistical model may be used to predict the musculoskeletal position information without having to place sensors on each segment of the rigid body that is to be represented in the computer-generated musculoskeletal representation.
  • the types of joints between segments in a multi-segment articulated rigid body model constrain movement of the rigid body.
  • different individuals tend to move in characteristic ways when performing a task that can be captured in statistical patterns of individual user behavior. At least some of these constraints on human body movement may be explicitly incorporated into statistical models used for prediction in accordance with some embodiments. Additionally or alternatively, the constraints may be learned by the statistical model though training based on recorded sensor data.
  • Constraints imposed in the construction of the statistical model are those set by anatomy and the physics of a user's body, while constraints derived from statistical patterns are those set by human behavior for one or more users from which sensor measurements are measured.
  • the constraints may comprise part of the statistical model itself being represented by information (e.g., connection weights between nodes) in the model.
  • some embodiments are directed to using a statistical model for predicting handstate information to enable the generation and/or real-time update of a computer-based musculoskeletal representation.
  • the statistical model may be used to predict the handstate information based on IMU signals, neuromuscular signals (e.g., EMG, MMG, and SMG signals), external device signals (e.g., camera or laser-scanning signals), or a combination of IMU signals, neuromuscular signals, and external device signals detected as a user performs one or more movements.
  • statistical models that predict estimates (e.g., handstate estimates) for updating a musculoskeletal representation include some constraints imposed in the construction of the statistical model set by anatomy and physics of a user's body and other constraints derived from statistical patterns set by human behavior for one or more users from which sensor measurements are measured.
  • the constraints derived from statistical patterns set by human behavior may be initially learned by a statistical model by training the model using training data collected from a plurality of users to produce a user-independent model.
  • Such a user-independent model may perform well in estimating position and/or force information (e.g., handstate estimates) based on recorded neuromuscular signals for some (e.g., simple) gestures, and may perform poorly in estimating such information for other (e.g., complex) gestures.
  • position and/or force information e.g., handstate estimates
  • one or more parameters of the user-independent model may be updated based on user-specific data. Updating parameter(s) of a statistical model for a particular user and/or for usage during a particular recording session is referred to herein as “calibration” of the model performance.
  • FIG. 2 illustrates a process 200 for calibrating performance of a statistical model that may be used in accordance with system 100 , described in connection with FIG. 1 .
  • the performance of one or more statistical models is calibrated by presenting the user with explicit instructions to perform one or more gestures.
  • the system may include a user interface configured to present instructions to the user to perform one or more gestures while wearing a wearable device on which a plurality of neuromuscular sensors are arranged and configured to record a plurality of neuromuscular signals.
  • the user interface may be a visual interface, an audio interface, or any other suitable type of user interface for presenting the instructions to the user for performing one or more gestures.
  • the instructions may be presented using text, static visual representations (e.g., images), dynamic visual representations (e.g., videos, animations), or any combination of the foregoing.
  • gestures refers to a static or dynamic configuration of one or more body parts including the position of the one or more body parts and forces associated with the configuration.
  • gestures include discrete gestures, such as pressing the palm of a hand down on a solid surface or grasping a ball, continuous gestures, such as a waving a finger back and forth or throwing a ball, or a combination of discrete and continuous gestures such as grasping and throwing a ball.
  • Gestures may be defined by an application configured to prompt a user to perform the gestures or, alternatively, gestures may be arbitrarily defined by a user.
  • process 200 proceeds to act 204 , where a plurality of neuromuscular signals are recorded by a plurality of neuromuscular sensors arranged on the body of a user during performance of the gesture(s) in accordance with the presented instructions.
  • the plurality of neuromuscular sensors may be arranged on one or wearable devices configured to be worn by a user during performance of one or more gestures.
  • Process 200 then proceeds to act 206 , where the statistical model configuration used to generate a musculoskeletal representation is updated based on the recorded plurality of neuromuscular signals.
  • the statistical model configuration may be updated by selecting a different statistical model used to generate the musculoskeletal model.
  • At least one parameter of a statistical model may be updated based on the recorded plurality of neuromuscular signals.
  • the one or more parameters of the statistical model may be updated in any suitable way.
  • one or more parameters of the statistical model may be updated by training the statistical model based, at least in part, on the recorded neuromuscular signals.
  • the one or more parameters of the statistical model may be updated without training or re-training the model.
  • the statistical model may receive as input the one or more parameters and updating the one or more parameters may comprise providing different values of the one or more parameters as input to the model.
  • the statistical model may be characterized by one or more parameters that are not provided as input to the model, but are nonetheless configurable, such that the parameter(s) may be updated based on neuromuscular signals recorded during performance of a gesture.
  • the configurable parameters may include a scaling factor by which recorded neuromuscular signals are scaled prior to being provided as input to the model.
  • FIG. 3 illustrates a process 300 for updating one or more parameters of a statistical model in accordance with some embodiments.
  • a plurality of neuromuscular signals are recorded during performance of a gesture.
  • the neuromuscular signals may be recorded in response to prompting the user to perform one or more gestures as part of a training or retraining protocol to train the model. For example, a user may be instructed to hold their fingers out, orient their wrist parallel to their arm (e.g., not flexed or extended), and move their wrist with an ulnar deviation (e.g., towards their pinky finger).
  • the plurality of neuromuscular signals may be recorded during runtime usage of the system after the model has been trained, and the performance of the model to accurately predict estimates may be tracked and used to determine whether to perform “online” calibration of the model, as discussed in more detail below.
  • ground truth data representing the expected estimates output from the model in response to particular inputs are determined in act 306 .
  • the ground truth data may be predetermined and defined based on the particular gesture that the user is instructed to perform. For example, when the estimates output from the model are handstate estimates that include a set of joint angles and forces for a musculoskeletal representation associated with a hand, a gesture may be defined by a particular set of joint angles and forces. In such a situation, the ground truth data may be determined as the particular set of joint angles and forces that define the gesture. Additionally or alternatively, the ground truth data may be determined based on neuromuscular information for the gesture. For example, the gesture may be characterized by a generic neuromuscular signal shape which may be used, at least in part, to determine the ground truth data.
  • the ground truth data may be determined by one or more measurements captured by one or more external devices, examples of which include imaging devices (e.g., cameras) and inertial measurement units (IMUs).
  • imaging devices e.g., cameras
  • IMUs inertial measurement units
  • the external device is an imaging device configured to capture images
  • the imaging device may be controlled to capture at least one image as the user performs the gesture as the neuromuscular signals are recorded, and the one or more measurements may be determined based on the captured image(s).
  • multiple external devices are used to determine the ground truth data.
  • an IMU may be used to determine some position/orientation such as large joint angles in the arm and wrist and an imaging device may be used to capture other position/orientation information such as smaller joint angles in the fingers.
  • Other combinations of external devices may alternatively be used to determine position and/or force information as ground truth data, and embodiments are not limited in this respect.
  • the measurements from the multiple devices may be combined in any suitable way, and the statistical model may be updated based on the combined measurements.
  • Process 300 then proceeds to act 308 , where at least one parameter of the statistical model is updated based on the plurality of neuromuscular signals recorded in act 304 and the ground truth data determined in act 306 .
  • updating parameter(s) of a statistical model are described above in connection with act 206 of process 200 .
  • Process 300 then proceeds to act 310 , where feedback is optionally presented to the user to provide the user with information about whether the user's performance of the gesture was performed correctly as instructed (when instructions are provided).
  • the feedback may be provided audibly, visually, using haptic feedback, or using any combination of the foregoing.
  • the feedback may be provided as a static or dynamic visual representation of the gesture performed by the user to enable the user to compare how the model interpreted the user's performance of the gesture.
  • the feedback may illustrate a comparison between how the model estimates the user's performance of the gesture and how the gesture was intended to be performed based on the instructions provided.
  • the feedback may then be interpreted by the user to determine whether the gesture was performed correctly and/or whether the model predicted the handstate estimates correctly based on the performed gesture.
  • acts 304 - 310 may be repeated until the system and/or the user is satisfied that the model output can be used to accurately represent the performed gesture.
  • Process 300 then proceeds to act 312 , where it is determined whether the process should be repeated with a new gesture. When it is determined to repeat the process with a new gesture, process 300 proceeds to act 314 where a new gesture is selected. Otherwise, the process ends.
  • Determining whether acts 304 - 310 of process 300 should be repeated with a new gesture may be determined in any suitable way.
  • a passive calibration procedure the user may be instructed to perform a predetermined set of gestures and determining whether to perform a new gesture may be determined based on whether all of the gestures in the set have been performed.
  • This calibration procedure is referred to as “passive” herein as the gestures performed are not actively determined (e.g., based on the performance of the model), but are predetermined based on the gestures in the set.
  • Passive calibration may be used, for example, when a user first uses the system to provide user-specific initial training of the statistical model or when the user uses the system to control a new application that associates particular gestures with control commands in the application.
  • Passive calibration may be used to ensure that the statistical model is trained on a base set of uniform gestures relevant to a particular application.
  • the gestures in the set may be selected in act 314 randomly from the set for performance by the user during calibration or the user may be instructed to perform at least some of the gestures in the set in a particular order.
  • the particular order may specify that the model be trained on simple gestures prior to training the model on more complex gestures.
  • some embodiments employ an active calibration procedure, where a new gesture to be performed is not determined based on a predetermined set of gestures, but instead selection of a new gesture is informed based on performance of the model.
  • an active calibration procedure new gestures to perform are actively suggested to the user based on an online (e.g., Bayesian) estimation of the model performance. For example, it may be determined that the model performs well for estimating certain gesture characteristics but performs poorly for estimating other gesture characteristics.
  • each gesture may be defined by a plurality of gesture characteristics (e.g., joint angles and forces), and the gesture that is selected during an active calibration procedure may be selected based on the gesture characteristic(s) that the model estimates poorly and gestures that include the gesture characteristic(s).
  • a new gesture may be selected randomly from a set of gestures having gesture characteristics which the model estimates poorly.
  • a new gesture to be performed by the user may be selected as a gesture previously performed by the user, but on which performance of the model is poorly calibrated.
  • a new gesture is selected based, at least in part, on whether the particular gesture is defined by multiple gesture characteristics that the model estimates poorly.
  • a gesture that includes a maximum number of gesture characteristics for which the model performs poorly may be selected.
  • the user may be provided with explicit instructions explaining the gesture to be performed by, for example, displaying on a user interface a static or dynamic visual representation of the gesture to be performed.
  • the instructions provided to the user may include at least one user-interpretable instruction that may be subjectively interpreted by the user to perform the gesture.
  • the user actively participates in the calibration by determining, at least in part, how to perform the gesture.
  • the user may be provided with a series of prompts that instructs the user to pinch two fingers at different percentages (e.g., 25%, 50%, 75%, 100%) of force exerted between the fingers relative to the maximum exertion.
  • the user defines how a percentage of maximal force maps to the neuromuscular signals recorded during performance of the gesture.
  • Other examples of user-interpretable instruction include, but are not limited to, moving a slider along a bar or line, interacting (e.g., clenching) with an object, and pressing against a surface.
  • feedback may be provided to a user during a passive or active calibration procedure when user-interpretable instructions are included.
  • the feedback may include rendering a visual representation associated with a hand.
  • the visual representation may include force metrics to provide feedback on exerted forces to the user. Force metrics may be provided to the user in any suitable way, examples of which include, but are not limited to, using different colors, bar plots, scatterplots, time series plots, etc. In some embodiments, metrics in addition to or as an alternative to force metrics may also be provided as feedback to a user.
  • additional metrics may include the extent to which the gesture is completed or the degree to which the gesture is completed (e.g., 50% completed), metrics describing whether the completed or partially-completed gesture is accurate, or any other suitable metrics that provide feedback to the user as the user is performing the gesture may be provided.
  • the calibration procedures described herein may be performed at different times. For example, training or retraining the model may occur prior to use of the system by a new user and/or for a new application. Such calibration individualizes the statistical model(s) based on user-specific physiological and/or anatomical characteristics. Additionally or alternatively, the calibration procedures may be performed as the user is using the system.
  • FIG. 4 illustrates a process 400 for performing calibration during normal use of the system.
  • the performance of the model is tracked to predict the performance of the model in accurately predicting gestures and/or gesture characteristics during runtime.
  • process 400 proceeds to act 404 , where a calibration routine is initiated.
  • the determination to initiate a calibration routine may be made in any suitable way. For example, one or more quality measures may be determined for model estimates corresponding to gestures or gesture characteristics and it may be determined to initiate a calibration routine when the quality measure(s) fall below a threshold for the gestures/gesture characteristics.
  • a calibration routine may be initiated if the probabilities among several possible gestures exhibits relatively high entropy.
  • a well-calibrated statistical model may calculate the likelihood of one gesture to be high (e.g., greater than 0.8, greater than 0.9, close to 1) and the posterior probabilities for other gestures to be small.
  • the model frequently identifies gestures with low likelihood (generally indicating the presence or estimation of many ‘long tail’ gestures)
  • a calibration routine may be initiated or a user notification may be presented to the user to determine whether the user was in fact initiating uncommon hand gestures.
  • a calibration routine may be used to account for a user's anatomy.
  • a detector circuit may be configured to detect whether a user has the palmaris longus muscle in their arm.
  • One way to determine if a user has a palmaris longus muscle is to instruct the user to pinch the fingertips of their pinky and thumb while flexing their wrist.
  • an appropriate calibration routine may be initiated that is specifically configured to calibrate for individuals with a palmar longus muscle.
  • a calibration routine may be initiated to account for the effect of a user's anatomy on the conductive profile of the user's body (e.g., their arm).
  • Adipose tissue affects the filtering of neuromuscular activity recorded by electromyography at the skin surface, so a calibration routine optimized for improving a statistical model based on the location and amount of adipose tissue may be initiated.
  • a calibration routine optimized for improving a statistical model based on the location and amount of adipose tissue may be initiated.
  • the location and amount of adipose tissue in a user may be measured or estimated in various ways, including via ultrasonography.
  • the effect of a particular muscle activation on the movement of a part of the body depends on individual anatomy, including the size of the arm or hand and properties of the user's joints (e.g., joint stiffness, joint segment moments and inertia, and the relationship between motor unit activity and joint torque), and variations in individual anatomy that affect how joints move based on a given muscle activation may cause degraded model performance and cause a calibration routine to be initiated.
  • individual anatomy including the size of the arm or hand and properties of the user's joints (e.g., joint stiffness, joint segment moments and inertia, and the relationship between motor unit activity and joint torque), and variations in individual anatomy that affect how joints move based on a given muscle activation may cause degraded model performance and cause a calibration routine to be initiated.
  • Initiation of a calibration routine may be made apparent to the user by, for example, the system entering a “calibration mode” where the user is prompted with instructions to perform one or more gestures to calibrate performance of the model, as discussed above. Halting usage of the system and entering a calibration mode may enable the model performance to improve quickly by having the user focus on performing gestures with characteristics that the model estimates poorly.
  • initiation of a calibration routine may occur during routine usage of the system without the user becoming aware that calibration is being performed.
  • the system may identify gestures and/or gesture characteristics for which model calibration is desired, and when gestures/gesture characteristics are performed by the user during ordinary usage of the system, calibration may be performed using neuromuscular signals recorded during performance of such gestures.
  • the user's use of the system is not interrupted, and the model adapts through normal use of the system.
  • process 400 proceeds to act 406 , where a plurality of neuromuscular signals are recorded during performance of gestures that include the identified gesture characteristic(s) on which the model performs poorly.
  • Process 400 then proceeds to act 408 , where at least one parameter of the model is updated based on the recorded plurality of neuromuscular signals.
  • updating the parameter(s) of a statistical model may be implemented by training/retraining the model, modifying inputs to the model or changing other configurable parameters of the model.
  • some simple updates to the parameter(s) of the model may occur during runtime usage of the model, whereas more complex updates (e.g., retraining the model) may occur offline when the system is not being used by the user.
  • the model updates happen is not a limitation of the technology described herein.
  • model parameters examples of which include, but are not limited to, a user's muscular anatomical features, a location of single muscles, muscle gain, and joint stiffness, are not expected to change substantially from session to session.
  • Such model parameters may be stored in a user profile for the user, and the user profile may be accessed each time the user uses the system in a new session.
  • a user's user profile is updated with each new calibration session, for example, to track a user's long-term changes in physiology and/or anatomy.
  • Other model parameters may be both user dependent and session dependent. For example, the placement of the neuromuscular sensors on the user's body may change from session to session or may even change during a session resulting in varying model estimates.
  • the placement of neuromuscular sensors in an armband may be rotated circumferentially around the arm; the armband may be placed on the arm oriented to be facing either proximally or distally; the armband may be placed on a different arm relative to an earlier calibration session; or the armband may be shifted along the proximal-distal axis of the arm (e.g., closer to the hand or closer to the shoulder).
  • a user may experience muscle fatigue during a session, causing the pattern of motor unit activation to change, and a calibration session may be useful to account for the changing pattern of motor unit activation. Model parameters relating to session-dependent events may need to be updated each session and as such, may not be stored in the user's profile.
  • session-dependent events may be characteristic of a user (e.g., propensity of a muscle to experience fatigue) and as such, may be stored in the user's profile.
  • a determination is made during or at the end of a session regarding which model parameter updates made during the session should persist and be stored in the user's profile and which model parameter updates made during the session are session dependent and should be discarded following the session. The determination may be made based on the type of update, examples of which are described above.
  • FIG. 5 describes a process 500 for generating (sometimes termed “training” herein) a statistical model using signals recorded from sensors 102 .
  • Process 500 may be executed by any suitable computing device(s), as aspects of the technology described herein are not limited in this respect.
  • process 500 may be executed by one or more computer processors described with reference to FIGS. 7A and 7B .
  • one or more acts of process 500 may be executed using one or more servers (e.g., servers included as a part of a cloud computing environment).
  • at least a portion of act 510 relating to training of a statistical model (e.g., a neural network) may be performed using a cloud computing environment.
  • Process 500 begins at act 502 , where a plurality of sensor signals are obtained for one or multiple users performing one or more movements (e.g., typing on a keyboard).
  • the plurality of sensor signals may be recorded as part of process 500 .
  • the plurality of sensor signals may have been recorded prior to the performance of process 500 and are accessed (rather than recorded) at act 502 .
  • the plurality of sensor signals may include sensor signals recorded for a single user performing a single movement or multiple movements.
  • the user may be instructed to perform a sequence of movements for a particular task (e.g., opening a door) and sensor signals corresponding to the user's movements may be recorded as the user performs the task he/she was instructed to perform.
  • the sensor signals may be recorded by any suitable number of sensors located in any suitable location(s) to detect the user's movements that are relevant to the task performed.
  • the sensor signals may be recorded by multiple neuromuscular sensors circumferentially (or otherwise) arranged around the user's lower right arm to detect muscle activity in the lower right arm that give rise to the right hand movements and one or more IMU sensors arranged to predict the joint angle of the user's arm relative to the user's torso.
  • sensor signals may be recorded by multiple neuromuscular sensors circumferentially (or otherwise) arranged around the user's leg to detect muscle activity in the leg that give rise to the movements of the foot and one or more IMU sensors arranged to predict the joint angle of the user's leg relative to the user's torso.
  • the sensor signals obtained in act 502 correspond to signals from one type of sensor (e.g., one or more IMU sensors or one or more neuromuscular sensors) and a statistical model may be trained based on the sensor signals recorded using the particular type of sensor, resulting in a sensor-type specific trained statistical model.
  • the obtained sensor signals may comprise a plurality of EMG sensor signals arranged around the lower arm or wrist of a user and the statistical model may be trained to predict musculoskeletal position information for movements of the wrist and/or hand during performance of a task such as grasping and twisting an object such as a doorknob.
  • a separate statistical model may be trained for each of the types of sensors and the outputs of the sensor-type specific models may be combined to generate a musculoskeletal representation of the user's body.
  • the sensor signals obtained in act 502 from two or more different types of sensors may be provided to a single statistical model that is trained based on the signals recorded from the different types of sensors.
  • an IMU sensor and a plurality of EMG sensors are arranged on a wearable device configured to be worn around the forearm of a user, and signals recorded by the IMU and EMG sensors are collectively provided as inputs to a statistical model, as discussed in more detail below.
  • the sensor signals obtained in act 502 are recorded at multiple time points as a user performs one or multiple movements.
  • the recorded signal for each sensor may include data obtained at each of multiple time points.
  • the recorded sensor signals for the user may comprise a time series of K n-dimensional vectors ⁇ x k
  • a user may be instructed to perform a task multiple times and the sensor signals and position information may be recorded for each of multiple repetitions of the task by the user.
  • the plurality of sensor signals may include signals recorded for multiple users, each of the multiple users performing the same task one or more times. Each of the multiple users may be instructed to perform the task and sensor signals and position information corresponding to that user's movements may be recorded as the user performs (once or repeatedly) the task he/she was instructed to perform.
  • Collecting sensor signals and position information from a single user performing the same task repeatedly and/or from multiple users performing the same task one or multiple times facilitates the collection of sufficient training data to generate a statistical model that can accurately predict musculoskeletal position information associated with performance of the task.
  • a user-independent statistical model may be generated based on training data corresponding to the recorded signals from multiple users, and as the system is used by a user, the statistical model is trained based on recorded sensor data such that the statistical model learns the user-dependent characteristics to refine the prediction capabilities of the system for the particular user.
  • the plurality of sensor signals may include signals recorded for a user (or each of multiple users) performing each of multiple tasks one or multiple times. For example, a user may be instructed to perform each of multiple tasks (e.g., grasping an object, pushing an object, and pulling open a door) and signals corresponding to the user's movements may be recorded as the user performs each of the multiple tasks he/she was instructed to perform. Collecting such data may facilitate developing a statistical model for predicting musculoskeletal position information associated with multiple different actions that may be taken by the user. For example, training data that incorporates musculoskeletal position information for multiple actions may facilitate generating a statistical model for predicting which of multiple possible movements a user may be performing.
  • the sensor data obtained at act 502 may be obtained by recording sensor signals as each of one or multiple users performs each of one or more tasks one or more multiple times.
  • position information describing the spatial position of different body segments during performance of the task(s) may be obtained in act 504 .
  • the position information is obtained using one or more external devices or systems that track the position of different points on the body during performance of a task. For example, a motion capture system, a laser scanner, a device to measure mutual magnetic induction, or some other system configured to capture position information may be used.
  • a plurality of position sensors may be placed on segments of the fingers of the right hand and a motion capture system may be used to determine the spatial location of each of the position sensors as the user performs a task such as grasping an object.
  • the sensor data obtained at act 502 may be recorded simultaneously with recording of the position information obtained in act 504 .
  • position information indicating the position of each finger segment over time as the grasping motion is performed is obtained.
  • process 500 proceeds to act 506 , where the sensor signals obtained in act 502 and/or the position information obtained in act 504 are optionally processed.
  • the sensor signals or the position information signals may be processed using amplification, filtering, rectification, or other types of signal processing.
  • process 500 proceeds to act 508 , where musculoskeletal position characteristics are determined based on the position information (as collected in act 504 or as processed in act 506 ).
  • a set of derived musculoskeletal position characteristic values are determined based on the recorded position information, and the derived values are used as training data for training the statistical model.
  • the position information may be used to determine joint angles that define angles between each connected pair of rigid segments at each of multiple time points during performance of a task.
  • the position information obtained in act 504 may be represented by a vector of n joint angles at each of a plurality of time points, where n is the number of joints or connections between segments in the articulated rigid body model.
  • process 500 proceeds to act 510 , where the time series information obtained at acts 502 and 508 is combined to create training data used for training a statistical model at act 510 .
  • the obtained data may be combined in any suitable way.
  • each of the sensor signals obtained at act 502 may be associated with a task or movement within a task corresponding to the musculoskeletal position characteristics (e.g., joint angles) determined based on the positional information recorded in act 504 as the user performed the task or movement.
  • the sensor signals may be associated with musculoskeletal position characteristics (e.g., joint angles) and the statistical model may be trained to predict that the musculoskeletal representation will be characterized by particular musculoskeletal position characteristics between different body segments when particular sensor signals are recorded during performance of a particular task.
  • musculoskeletal position characteristics e.g., joint angles
  • the sensor data for the different types of sensors may be recorded using the same or different sampling rates.
  • the sensor data may be resampled (e.g., up-sampled or down-sampled) such that all sensor data provided as input to the statistical model corresponds to time series data at the same time resolution. Resampling at least some of the sensor data may be performed in any suitable way including, but not limited to using interpolation for upsampling and using decimation for downsampling.
  • some embodiments employ a statistical model configured to accept multiple inputs asynchronously.
  • the statistical model may be configured to model the distribution of the “missing” values in the input data having a lower sampling rate.
  • the timing of training of the statistical model occur asynchronously as input from multiple sensor data measurements becomes available as training data.
  • process 500 proceeds to act 512 , where a statistical model for predicting musculoskeletal position information is trained using the training data generated at act 510 .
  • the statistical model being trained may take as input a sequence of data sets each of the data sets in the sequence comprising an n-dimensional vector of sensor data.
  • the statistical model may provide output that indicates, for each of one or more tasks or movements that may be performed by a user, the likelihood that the musculoskeletal representation of the user's body will be characterized by a set of musculoskeletal position characteristics (e.g., a set of joint angles between segments in an articulated multi-segment body model).
  • the statistical model may take as input a sequence of vectors ⁇ x k
  • a derived value provided as input to the statistical model may comprise features extracted from the data from all or a subset of the sensors at and/or prior to time t j (e.g., a covariance matrix, a power spectrum, a combination thereof, or any other suitable derived representation).
  • the statistical model may provide output indicating, a probability that a musculoskeletal representation of the user's body will be characterized by a set of musculoskeletal position characteristics.
  • the statistical model may be trained to predict a set of joint angles for segments in the fingers in the hand over time as a user grasps an object.
  • the trained statistical model may output, a set of predicted joint angles for joints in the hand corresponding to the sensor input.
  • the statistical model may be a neural network and, for example, may be a recurrent neural network.
  • the recurrent neural network may be a long short-term memory (LSTM) neural network. It should be appreciated, however, that the recurrent neural network is not limited to being an LSTM neural network and may have any other suitable architecture.
  • the recurrent neural network may be a fully recurrent neural network, a recursive neural network, a variational autoencoder, a Hopfield neural network, an associative memory neural network, an Elman neural network, a Jordan neural network, an echo state neural network, a second order recurrent neural network, and/or any other suitable type of recurrent neural network.
  • neural networks that are not recurrent neural networks may be used. For example, deep neural networks, convolutional neural networks, and/or feedforward neural networks, may be used.
  • the output layer of the neural network may provide a set of output values corresponding to a respective set of possible musculoskeletal position characteristics (e.g., joint angles).
  • the neural network may operate as a non-linear regression model configured to predict musculoskeletal position characteristics from raw or pre-processed sensor measurements. It should be appreciated that, in some embodiments, any other suitable non-linear regression model may be used instead of a neural network, as aspects of the technology described herein are not limited in this respect.
  • the neural network can be implemented based on a variety of topologies and/or architectures including deep neural networks with fully connected (dense) layers, Long Short-Term Memory (LSTM) layers, convolutional layers, Temporal Convolutional Layers (TCL), or other suitable type of deep neural network topology and/or architecture.
  • the neural network can have different types of output layers including output layers with logistic sigmoid activation functions, hyperbolic tangent activation functions, linear units, rectified linear units, or other suitable type of nonlinear unit.
  • the neural network can be configured to represent the probability distribution over n different classes via, for example, a softmax function or include an output layer that provides a parameterized distribution e.g., mean and variance of a Gaussian distribution.
  • the statistical model may comprise a hidden Markov model, a Markov switching model with the switching allowing for toggling among different dynamic systems, dynamic Bayesian networks, and/or any other suitable graphical model having a temporal component. Any such statistical model may be trained at act 512 using the sensor data obtained at act 502 .
  • the statistical model may take as input, features derived from the sensor data obtained at act 502 .
  • the statistical model may be trained at act 512 using features extracted from the sensor data obtained at act 502 .
  • the statistical model may be a support vector machine, a Gaussian mixture model, a regression based classifier, a decision tree classifier, a Bayesian classifier, and/or any other suitable classifier, as aspects of the technology described herein are not limited in this respect.
  • Input features to be provided as training data to the statistical model may be derived from the sensor data obtained at act 502 in any suitable way.
  • the sensor data may be analyzed as time series data using wavelet analysis techniques (e.g., continuous wavelet transform, discrete-time wavelet transform, etc.), Fourier-analytic techniques (e.g., short-time Fourier transform, Fourier transform, etc.), and/or any other suitable type of time-frequency analysis technique.
  • wavelet analysis techniques e.g., continuous wavelet transform, discrete-time wavelet transform, etc.
  • Fourier-analytic techniques e.g., short-time Fourier transform, Fourier transform, etc.
  • the sensor data may be transformed using a wavelet transform and the resulting wavelet coefficients may be provided as inputs to the statistical model.
  • values for parameters of the statistical model may be estimated from the training data generated at act 510 .
  • parameters of the neural network e.g., weights
  • parameters of the statistical model may be estimated from the training data.
  • parameters of the statistical model may be estimated using gradient descent, stochastic gradient descent, and/or any other suitable iterative optimization technique.
  • the statistical model may be trained using stochastic gradient descent and backpropagation through time.
  • the training may employ a cross-entropy loss function and/or any other suitable loss function, as aspects of the technology described herein are not limited in this respect.
  • process 500 proceeds to act 514 , where the trained statistical model is stored (e.g., in datastore—not shown).
  • the trained statistical model may be stored using any suitable format, as aspects of the technology described herein are not limited in this respect. In this way, the statistical model generated during execution of process 500 may be used at a later time, for example, to predict musculoskeletal position information (e.g., joint angles) for a given set of input sensor data, as described below.
  • sensor signals are recorded from a plurality of sensors (e.g., arranged on or near the surface of a user's body) that record activity associated with movements of the body during performance of a task.
  • the recorded signals may be optionally processed and provided as input to a statistical model trained using one or more techniques described above in connection with FIG. 5 .
  • the continuously recorded signals (raw or processed) may be continuously or periodically provided as input to the trained statistical model for prediction of musculoskeletal position information (e.g., joint angles) for the given set of input sensor data.
  • the trained statistical model is a user-independent model trained based on autonomous sensor and position information measurements from a plurality of users.
  • the trained model is a user-dependent model trained on data recorded from the individual user from which the data associated with the sensor signals is also acquired.
  • the predicted musculoskeletal position information is output from the trained statistical model.
  • the predicted musculoskeletal position information may comprise a set of musculoskeletal position information values (e.g., a set of joint angles) for a multi-segment articulated rigid body model representing at least a portion of the user's body.
  • the musculoskeletal position information may comprise a set of probabilities that the user is performing one or more movements from a set of possible movements.
  • a computer-based musculoskeletal representation of the user's body is generated based, at least in part, on the musculoskeletal position information output from the trained statistical model.
  • the computer-based musculoskeletal representation may be generated in any suitable way.
  • a computer-based musculoskeletal model of the human body may include multiple rigid body segments, each of which corresponds to one or more skeletal structures in the body.
  • the upper arm may be represented by a first rigid body segment
  • the lower arm may be represented by a second rigid body segment
  • the palm of the hand may be represented by a third rigid body segment
  • each of the fingers on the hand may be represented by at least one rigid body segment (e.g., at least fourth-eighth rigid body segments).
  • a set of joint angles between connected rigid body segments in the musculoskeletal model may define the orientation of each of the connected rigid body segments relative to each other and a reference frame, such as the torso of the body.
  • the computer-based musculoskeletal representation of the user's body may be updated based on the updated set of joint angles determined based on the output of the statistical model. In this way the computer-based musculoskeletal representation is dynamically updated in real-time as sensor data is continuously recorded.
  • the computer-based musculoskeletal representation may be represented and stored in any suitable way, as embodiments of the technology described herein are not limited with regard to the particular manner in which the representation is stored. Additionally, although referred to herein as a “musculoskeletal” representation, to reflect that muscle activity may be associated with the representation in some embodiments, as discussed in more detail below, it should be appreciated that some musculoskeletal representations used in accordance with some embodiments may correspond to skeletal structures, muscular structures or a combination of skeletal structures and muscular structures in the body.
  • direct measurement of neuromuscular activity and/or muscle activity underlying the user's movements may be combined with the generated musculoskeletal representation.
  • Measurements from a plurality of sensors placed at locations on a user's body may be used to create a unified representation of muscle recruitment by superimposing the measurements onto a dynamically-posed skeleton.
  • muscle activity sensed by neuromuscular sensors and/or information derived from the muscle activity e.g., force information
  • FIG. 6A illustrates a wearable system with sixteen neuromuscular sensors 610 (e.g., EMG sensors) arranged circumferentially around an elastic band 620 configured to be worn around a user's lower arm or wrist. As shown, EMG sensors 610 are arranged circumferentially around elastic band 620 . It should be appreciated that any suitable number of neuromuscular sensors may be used. The number and arrangement of neuromuscular sensors may depend on the particular application for which the wearable device is used. For example, a wearable armband or wristband can be used to generate control information for controlling an augmented reality system, a robot, controlling a vehicle, scrolling through text, controlling a virtual avatar, or any other suitable control task.
  • EMG sensors 610 e.g., EMG sensors
  • FIG. 6A illustrates a wearable system with sixteen neuromuscular sensors 610 (e.g., EMG sensors) arranged circumferentially around an elastic band 620 configured to be worn around a user's lower arm or wrist.
  • EMG sensors 610
  • sensors 610 includes a set of neuromuscular sensors (e.g., EMG sensors).
  • sensors 610 can include a set of neuromuscular sensors and at least one “auxiliary” sensor configured to continuously record auxiliary signals.
  • auxiliary sensors include, but are not limited to, other sensors such as IMU sensors, microphones, imaging sensors (e.g., a camera), radiation based sensors for use with a radiation-generation device (e.g., a laser-scanning device), or other types of sensors such as a heart-rate monitor.
  • the sensors 610 may be coupled together using flexible electronics 630 incorporated into the wearable device.
  • FIG. 6B illustrates a cross-sectional view through one of the sensors 610 of the wearable device shown in FIG. 6A .
  • the output of one or more of the sensing components can be optionally processed using hardware signal processing circuitry (e.g., to perform amplification, filtering, and/or rectification).
  • hardware signal processing circuitry e.g., to perform amplification, filtering, and/or rectification
  • at least some signal processing of the output of the sensing components can be performed in software.
  • signal processing of signals sampled by the sensors can be performed in hardware, software, or by any suitable combination of hardware and software, as aspects of the technology described herein are not limited in this respect.
  • a non-limiting example of a signal processing chain used to process recorded data from sensors 610 is discussed in more detail below in connection with FIGS. 7A and 7B
  • FIGS. 7A and 7B illustrate a schematic diagram with internal components of a wearable system with sixteen EMG sensors, in accordance with some embodiments of the technology described herein.
  • the wearable system includes a wearable portion 710 ( FIG. 7A ) and a dongle portion 720 ( FIG. 7B ) in communication with the wearable portion 710 (e.g., via Bluetooth or another suitable short range wireless communication technology).
  • the wearable portion 710 includes the sensors 610 , examples of which are described in connection with FIGS. 6A and 6B .
  • the output of the sensors 610 is provided to analog front end 730 configured to perform analog processing (e.g., noise reduction, filtering, etc.) on the recorded signals.
  • analog processing e.g., noise reduction, filtering, etc.
  • the processed analog signals are then provided to analog-to-digital converter 732 , which converts the analog signals to digital signals that can be processed by one or more computer processors.
  • An example of a computer processor that may be used in accordance with some embodiments is microcontroller (MCU) 734 illustrated in FIG. 7A .
  • MCU 734 may also include inputs from other sensors (e.g., IMU sensor 740 ), and power and battery module 742 .
  • the output of the processing performed by MCU may be provided to antenna 750 for transmission to dongle portion 720 shown in FIG. 7B .
  • Dongle portion 720 includes antenna 752 configured to communicate with antenna 750 included as part of wearable portion 710 . Communication between antenna 750 and 752 may occur using any suitable wireless technology and protocol, non-limiting examples of which include radiofrequency signaling and Bluetooth. As shown, the signals received by antenna 752 of dongle portion 720 may be provided to a host computer for further processing, display, and/or for effecting control of a particular physical or virtual object or objects.
  • the embodiments can be implemented in any of numerous ways.
  • the embodiments may be implemented using hardware, software or a combination thereof.
  • the software code can be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers.
  • any component or collection of components that perform the functions described above can be generically considered as one or more controllers that control the above-discussed functions.
  • the one or more controllers can be implemented in numerous ways, such as with dedicated hardware or with one or more processors programmed using microcode or software to perform the functions recited above.
  • one implementation of the embodiments of the present invention comprises at least one non-transitory computer-readable storage medium (e.g., a computer memory, a portable memory, a compact disk, etc.) encoded with a computer program (i.e., a plurality of instructions), which, when executed on a processor, performs the above-discussed functions of the embodiments of the present invention.
  • the computer-readable storage medium can be transportable such that the program stored thereon can be loaded onto any computer resource to implement the aspects of the present invention discussed herein.
  • the reference to a computer program which, when executed, performs the above-discussed functions is not limited to an application program running on a host computer. Rather, the term computer program is used herein in a generic sense to reference any type of computer code (e.g., software or microcode) that can be employed to program a processor to implement the above-discussed aspects of the present invention.
  • embodiments of the invention may be implemented as one or more methods, of which an example has been provided.
  • the acts performed as part of the method(s) may be ordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts simultaneously, even though shown as sequential acts in illustrative embodiments.

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Biomedical Technology (AREA)
  • General Health & Medical Sciences (AREA)
  • Human Computer Interaction (AREA)
  • General Physics & Mathematics (AREA)
  • Surgery (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • Pathology (AREA)
  • Animal Behavior & Ethology (AREA)
  • Biophysics (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Neurosurgery (AREA)
  • Neurology (AREA)
  • Dermatology (AREA)
  • Physiology (AREA)
  • Dentistry (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Geometry (AREA)
  • User Interface Of Digital Computer (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
US16/257,979 2016-07-25 2019-01-25 Calibration techniques for handstate representation modeling using neuromuscular signals Active US10496168B2 (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US16/257,979 US10496168B2 (en) 2018-01-25 2019-01-25 Calibration techniques for handstate representation modeling using neuromuscular signals
US16/671,066 US11163361B2 (en) 2018-01-25 2019-10-31 Calibration techniques for handstate representation modeling using neuromuscular signals
US17/741,263 US20220269346A1 (en) 2016-07-25 2022-05-10 Methods and apparatuses for low latency body state prediction based on neuromuscular data

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201862621838P 2018-01-25 2018-01-25
US16/257,979 US10496168B2 (en) 2018-01-25 2019-01-25 Calibration techniques for handstate representation modeling using neuromuscular signals

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US16/657,029 Continuation-In-Part US11361522B2 (en) 2016-07-25 2019-10-18 User-controlled tuning of handstate representation model parameters

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/671,066 Continuation US11163361B2 (en) 2016-07-25 2019-10-31 Calibration techniques for handstate representation modeling using neuromuscular signals

Publications (2)

Publication Number Publication Date
US20190227627A1 US20190227627A1 (en) 2019-07-25
US10496168B2 true US10496168B2 (en) 2019-12-03

Family

ID=67298623

Family Applications (2)

Application Number Title Priority Date Filing Date
US16/257,979 Active US10496168B2 (en) 2016-07-25 2019-01-25 Calibration techniques for handstate representation modeling using neuromuscular signals
US16/671,066 Active US11163361B2 (en) 2016-07-25 2019-10-31 Calibration techniques for handstate representation modeling using neuromuscular signals

Family Applications After (1)

Application Number Title Priority Date Filing Date
US16/671,066 Active US11163361B2 (en) 2016-07-25 2019-10-31 Calibration techniques for handstate representation modeling using neuromuscular signals

Country Status (4)

Country Link
US (2) US10496168B2 (zh)
EP (1) EP3742961A4 (zh)
CN (1) CN111902077B (zh)
WO (1) WO2019147996A1 (zh)

Cited By (30)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10656711B2 (en) 2016-07-25 2020-05-19 Facebook Technologies, Llc Methods and apparatus for inferring user intent based on neuromuscular signals
US10687759B2 (en) 2018-05-29 2020-06-23 Facebook Technologies, Llc Shielding techniques for noise reduction in surface electromyography signal measurement and related systems and methods
US10842407B2 (en) 2018-08-31 2020-11-24 Facebook Technologies, Llc Camera-guided interpretation of neuromuscular signals
US10905383B2 (en) 2019-02-28 2021-02-02 Facebook Technologies, Llc Methods and apparatus for unsupervised one-shot machine learning for classification of human gestures and estimation of applied forces
US10921764B2 (en) 2018-09-26 2021-02-16 Facebook Technologies, Llc Neuromuscular control of physical objects in an environment
US10937414B2 (en) 2018-05-08 2021-03-02 Facebook Technologies, Llc Systems and methods for text input using neuromuscular information
US10970936B2 (en) 2018-10-05 2021-04-06 Facebook Technologies, Llc Use of neuromuscular signals to provide enhanced interactions with physical objects in an augmented reality environment
US10970374B2 (en) 2018-06-14 2021-04-06 Facebook Technologies, Llc User identification and authentication with neuromuscular signatures
US10990174B2 (en) 2016-07-25 2021-04-27 Facebook Technologies, Llc Methods and apparatus for predicting musculo-skeletal position information using wearable autonomous sensors
US11036302B1 (en) 2018-05-08 2021-06-15 Facebook Technologies, Llc Wearable devices and methods for improved speech recognition
US11045137B2 (en) 2018-07-19 2021-06-29 Facebook Technologies, Llc Methods and apparatus for improved signal robustness for a wearable neuromuscular recording device
US11069148B2 (en) 2018-01-25 2021-07-20 Facebook Technologies, Llc Visualization of reconstructed handstate information
US11079846B2 (en) 2013-11-12 2021-08-03 Facebook Technologies, Llc Systems, articles, and methods for capacitive electromyography sensors
US11163361B2 (en) 2018-01-25 2021-11-02 Facebook Technologies, Llc Calibration techniques for handstate representation modeling using neuromuscular signals
US11179066B2 (en) 2018-08-13 2021-11-23 Facebook Technologies, Llc Real-time spike detection and identification
US11216069B2 (en) 2018-05-08 2022-01-04 Facebook Technologies, Llc Systems and methods for improved speech recognition using neuromuscular information
US11337652B2 (en) 2016-07-25 2022-05-24 Facebook Technologies, Llc System and method for measuring the movements of articulated rigid bodies
US11481030B2 (en) 2019-03-29 2022-10-25 Meta Platforms Technologies, Llc Methods and apparatus for gesture detection and classification
US11481031B1 (en) 2019-04-30 2022-10-25 Meta Platforms Technologies, Llc Devices, systems, and methods for controlling computing devices via neuromuscular signals of users
US11493993B2 (en) 2019-09-04 2022-11-08 Meta Platforms Technologies, Llc Systems, methods, and interfaces for performing inputs based on neuromuscular control
US11567573B2 (en) 2018-09-20 2023-01-31 Meta Platforms Technologies, Llc Neuromuscular text entry, writing and drawing in augmented reality systems
US11635736B2 (en) 2017-10-19 2023-04-25 Meta Platforms Technologies, Llc Systems and methods for identifying biological structures associated with neuromuscular source signals
US11644799B2 (en) 2013-10-04 2023-05-09 Meta Platforms Technologies, Llc Systems, articles and methods for wearable electronic devices employing contact sensors
US11666264B1 (en) 2013-11-27 2023-06-06 Meta Platforms Technologies, Llc Systems, articles, and methods for electromyography sensors
US11797087B2 (en) 2018-11-27 2023-10-24 Meta Platforms Technologies, Llc Methods and apparatus for autocalibration of a wearable electrode sensor system
US11868531B1 (en) 2021-04-08 2024-01-09 Meta Platforms Technologies, Llc Wearable device providing for thumb-to-finger-based input gestures detected based on neuromuscular signals, and systems and methods of use thereof
US11907423B2 (en) 2019-11-25 2024-02-20 Meta Platforms Technologies, Llc Systems and methods for contextualized interactions with an environment
US11921471B2 (en) 2013-08-16 2024-03-05 Meta Platforms Technologies, Llc Systems, articles, and methods for wearable devices having secondary power sources in links of a band for providing secondary power in addition to a primary power source
US11961494B1 (en) 2019-03-29 2024-04-16 Meta Platforms Technologies, Llc Electromagnetic interference reduction in extended reality environments
US11974857B2 (en) 2019-10-08 2024-05-07 Unlimited Tomorrow, Inc. Biometric sensor array

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9880632B2 (en) 2014-06-19 2018-01-30 Thalmic Labs Inc. Systems, devices, and methods for gesture identification
CN110312471B (zh) 2016-07-25 2022-04-29 脸谱科技有限责任公司 从神经肌肉活动测量中导出控制信号的自适应系统
US11331045B1 (en) 2018-01-25 2022-05-17 Facebook Technologies, Llc Systems and methods for mitigating neuromuscular signal artifacts
EP3743790A4 (en) 2018-01-25 2021-03-17 Facebook Technologies, Inc. RECONSTRUCTION OF HAND STATE ON THE BASIS OF MULTIPLE ENTRIES
EP3801743A4 (en) 2018-05-25 2021-08-04 Facebook Technologies, LLC PROCESSES AND APPARATUS FOR OBTAINING A SUB-MUSCULAR COMMAND
WO2020172261A1 (en) * 2019-02-19 2020-08-27 Coapt Llc Electromyographic control systems and methods for the coaching of exoprosthetic users
US20210100482A1 (en) * 2019-10-04 2021-04-08 Tactual Labs Co. Capactive based mechanomyography
CN112132192A (zh) * 2020-09-07 2020-12-25 北京海益同展信息科技有限公司 一种模型训练方法、装置、电子设备及存储介质
CN112347951B (zh) * 2020-11-11 2023-07-11 哈尔滨工业大学（深圳） 一种手势识别方法、装置、存储介质及数据手套
CN114515146B (zh) * 2020-11-17 2024-03-22 北京机械设备研究所 基于电学测量的智能手势识别方法及系统
CN113221708B (zh) * 2021-04-30 2023-11-10 北京科技大学 面部运动单元检测模型训练方法及装置
US20220391697A1 (en) * 2021-06-04 2022-12-08 Apple Inc. Machine-learning based gesture recognition with framework for adding user-customized gestures
CN114093488B (zh) * 2022-01-20 2022-04-22 武汉泰乐奇信息科技有限公司 一种基于骨骼识别的医生技能水平评判方法和装置

Citations (171)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4896120A (en) 1986-03-24 1990-01-23 Zvi Kamil Instrumentation amplifier arrangement
JPH05277080A (ja) 1992-04-01 1993-10-26 Dia Medical Syst Kk 生体電気現象検出装置
US5625577A (en) 1990-12-25 1997-04-29 Shukyohojin, Kongo Zen Sohonzan Shorinji Computer-implemented motion analysis method using dynamics
US6005548A (en) 1996-08-14 1999-12-21 Latypov; Nurakhmed Nurislamovich Method for tracking and displaying user's spatial position and orientation, a method for representing virtual reality for a user, and systems of embodiment of such methods
US6009210A (en) 1997-03-05 1999-12-28 Digital Equipment Corporation Hands-free interface to a virtual reality environment using head tracking
US6244873B1 (en) 1998-10-16 2001-06-12 At&T Corp. Wireless myoelectric control apparatus and methods
US6411843B1 (en) 1999-05-28 2002-06-25 Respironics, Inc. Method and apparatus for producing a model EMG signal from a measured EMG signal
US20030144829A1 (en) 2002-01-25 2003-07-31 Geatz Michael W. System and method for sensing and evaluating physiological parameters and modeling an adaptable predictive analysis for symptoms management
US20030171921A1 (en) 2002-03-04 2003-09-11 Ntt Docomo, Inc. Speech recognition system, speech recognition method, speech synthesis system, speech synthesis method, and program product
US6658287B1 (en) 1998-08-24 2003-12-02 Georgia Tech Research Corporation Method and apparatus for predicting the onset of seizures based on features derived from signals indicative of brain activity
US6720984B1 (en) 2000-06-13 2004-04-13 The United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration Characterization of bioelectric potentials
US20040092839A1 (en) 2002-10-24 2004-05-13 Shin Sang-Hoon Apparatus and method for measuring local skin impedance using multiple array elecrtodes
US6774885B1 (en) 1999-01-20 2004-08-10 Motek B.V. System for dynamic registration, evaluation, and correction of functional human behavior
JP2005095561A (ja) 2003-08-21 2005-04-14 Yoshiyuki Yamaumi 装着式動作補助装置、装着式動作補助装置の制御方法および制御用プログラム
US6942621B2 (en) 2002-07-11 2005-09-13 Ge Medical Systems Information Technologies, Inc. Method and apparatus for detecting weak physiological signals
US7089148B1 (en) 2000-10-30 2006-08-08 The United States Of America As Represented By The Secretary Of The Navy Method and apparatus for motion tracking of an articulated rigid body
CN1838933A (zh) 2003-08-21 2006-09-27 山海嘉之 穿着式动作辅助装置、穿着式动作辅助装置的控制方法和控制用程序
US20070172797A1 (en) 2006-01-12 2007-07-26 Kabushiki Kaisha Toyota Chuo Kenkyusho Method of constructing computer-based musculoskeletal model by redefining directions of pivot axes of joints in the same model
US20070256494A1 (en) 2004-06-16 2007-11-08 Yoshihiko Nakamura Muscular Strength Acquiring Method and Device Based on Musculoskeletal Model
US20070285399A1 (en) 2006-06-12 2007-12-13 Microsoft Corporation Extended eraser functions
US20080052643A1 (en) 2006-08-25 2008-02-28 Kabushiki Kaisha Toshiba Interface apparatus and interface method
US7351975B2 (en) 2005-03-29 2008-04-01 Duke University Sensor system for identifying and tracking movements of multiple sources
US20080214360A1 (en) 2006-03-03 2008-09-04 Garmin Ltd. Method and apparatus for estimating a motion parameter
US20080221487A1 (en) 2007-03-07 2008-09-11 Motek Bv Method for real time interactive visualization of muscle forces and joint torques in the human body
US20090082692A1 (en) 2007-09-25 2009-03-26 Hale Kelly S System And Method For The Real-Time Evaluation Of Time-Locked Physiological Measures
WO2009042313A1 (en) 2007-09-26 2009-04-02 Medtronic, Inc. Chopper-stabilized analog-to-digital converter
US20090112080A1 (en) 2006-04-21 2009-04-30 Quantum Applied Science & Research, Inc. System for Measuring Electric Signals
US7574253B2 (en) 2003-09-26 2009-08-11 Northwestern University Signal processing using non-linear regression with a sinusoidal model
US7580742B2 (en) 2006-02-07 2009-08-25 Microsoft Corporation Using electroencephalograph signals for task classification and activity recognition
US20090327171A1 (en) 2008-06-26 2009-12-31 Microsoft Corporation Recognizing gestures from forearm emg signals
US20090326406A1 (en) * 2008-06-26 2009-12-31 Microsoft Corporation Wearable electromyography-based controllers for human-computer interface
US20100030532A1 (en) 2008-06-12 2010-02-04 Jasbir Arora System and methods for digital human model prediction and simulation
US20100063794A1 (en) 2003-08-28 2010-03-11 Hernandez-Rebollar Jose L Method and apparatus for translating hand gestures
US20100106044A1 (en) 2008-10-27 2010-04-29 Michael Linderman EMG measured during controlled hand movement for biometric analysis, medical diagnosis and related analysis
US7787946B2 (en) 2003-08-18 2010-08-31 Cardiac Pacemakers, Inc. Patient monitoring, diagnosis, and/or therapy systems and methods
US7805386B2 (en) 2006-05-16 2010-09-28 Greer Douglas S Method of generating an encoded output signal using a manifold association processor having a plurality of pairs of processing elements trained to store a plurality of reciprocal signal pairs
US20100292617A1 (en) 2009-05-15 2010-11-18 Kin Fong Lei method and system for quantifying an intention of movement of a user
US20100293115A1 (en) 2006-08-21 2010-11-18 Kaveh Seyed Momen Method, system and apparatus for real-time classification of muscle signals from self -selected intentional movements
US20100315266A1 (en) 2009-06-15 2010-12-16 Microsoft Corporation Predictive interfaces with usability constraints
US7901368B2 (en) 2005-01-06 2011-03-08 Braingate Co., Llc Neurally controlled patient ambulation system
US20110077484A1 (en) 2009-09-30 2011-03-31 Nellcor Puritan Bennett Ireland Systems And Methods For Identifying Non-Corrupted Signal Segments For Use In Determining Physiological Parameters
US20110092826A1 (en) 2009-10-20 2011-04-21 Tatung Company System and method for measuring ECG and breath signals by using two polar electrodes
US20120066163A1 (en) 2010-09-13 2012-03-15 Nottingham Trent University Time to event data analysis method and system
US8190249B1 (en) 2005-08-01 2012-05-29 Infinite Biomedical Technologies, Llc Multi-parametric quantitative analysis of bioelectrical signals
US20120188158A1 (en) 2008-06-26 2012-07-26 Microsoft Corporation Wearable electromyography-based human-computer interface
US20120265480A1 (en) 2010-01-20 2012-10-18 Omron Healthcare Co., Ltd. Body movement detection device
US20120283526A1 (en) 2009-12-16 2012-11-08 Ictalcare A/S Method and a System for the Prediction of Epileptic Seizures
US8311623B2 (en) 2006-04-15 2012-11-13 The Board Of Trustees Of The Leland Stanford Junior University Systems and methods for estimating surface electromyography
US8351651B2 (en) 2010-04-26 2013-01-08 Microsoft Corporation Hand-location post-process refinement in a tracking system
US20130077820A1 (en) 2011-09-26 2013-03-28 Microsoft Corporation Machine learning gesture detection
US8421634B2 (en) 2009-12-04 2013-04-16 Microsoft Corporation Sensing mechanical energy to appropriate the body for data input
US8435191B2 (en) 2007-07-18 2013-05-07 Siemens Aktiengesellschaft Support structure for a sensor strip and sensor strip for mounting on said support structure
US20130141375A1 (en) 2011-12-06 2013-06-06 Lester F. Ludwig Gesteme (gesture primitive) recognition for advanced touch user interfaces
US8484022B1 (en) 2012-07-27 2013-07-09 Google Inc. Adaptive auto-encoders
US20130207889A1 (en) 2011-07-13 2013-08-15 Lumo Bodytech, Inc. System and Method of Biomechanical Posture Detection and Feedback Including Sensor Normalization
US20130217998A1 (en) 2009-02-02 2013-08-22 Jointvue, Llc Motion Tracking System with Inertial-Based Sensing Units
US20130317382A1 (en) 2012-05-25 2013-11-28 Emotiv Lifesciences Inc. System and Method for Providing and Aggregating Biosignals and Action Data
US20130317648A1 (en) 2012-05-25 2013-11-28 California Institute Of Technology Biosleeve human-machine interface
US20140052150A1 (en) 2010-08-02 2014-02-20 The Johns Hopkins University Method for presenting force sensor information using cooperative robot control and audio feedback
US20140092009A1 (en) 2006-07-14 2014-04-03 Ailive Inc. Methods and Systems for Dynamic Calibration of Movable Game Controllers
US20140098018A1 (en) 2012-10-04 2014-04-10 Microsoft Corporation Wearable sensor for tracking articulated body-parts
US8718980B2 (en) 2009-09-11 2014-05-06 Qualcomm Incorporated Method and apparatus for artifacts mitigation with multiple wireless sensors
US8744543B2 (en) 2005-09-29 2014-06-03 Covidien Lp System and method for removing artifacts from waveforms
US8754862B2 (en) 2010-07-11 2014-06-17 Lester F. Ludwig Sequential classification recognition of gesture primitives and window-based parameter smoothing for high dimensional touchpad (HDTP) user interfaces
US20140196131A1 (en) 2013-01-07 2014-07-10 Salutron, Inc. User authentication based on a wrist vein pattern
US20140198034A1 (en) 2013-01-14 2014-07-17 Thalmic Labs Inc. Muscle interface device and method for interacting with content displayed on wearable head mounted displays
US20140223462A1 (en) 2012-12-04 2014-08-07 Christopher Allen Aimone System and method for enhancing content using brain-state data
US20140245200A1 (en) 2013-02-25 2014-08-28 Leap Motion, Inc. Display control with gesture-selectable control paradigms
US20140240103A1 (en) 2013-02-22 2014-08-28 Thalmic Labs Inc. Methods and devices for combining muscle activity sensor signals and inertial sensor signals for gesture-based control
US20140249397A1 (en) 2013-03-01 2014-09-04 Thalmic Labs Inc. Differential non-contact biopotential sensor
US20140278441A1 (en) 2013-03-15 2014-09-18 Qualcomm Incorporated Systems and methods for switching processing modes using gestures
US20140304665A1 (en) 2013-04-05 2014-10-09 Leap Motion, Inc. Customized gesture interpretation
US8880163B2 (en) 2010-07-13 2014-11-04 Commissariat A L'energie Atomique Et Aux Energies Alternatives Method and a system for classifying neural signals, and a method of selecting electrodes for direct neural control
US20140334083A1 (en) 2013-05-13 2014-11-13 Thalmic Labs Inc. Systems, articles and methods for wearable electronic devices that accommodate different user forms
US8890875B2 (en) 2007-05-11 2014-11-18 Commissariat A L'energie Atomique Processing method for capturing movement of an articulated structure
US20140344731A1 (en) 2013-05-17 2014-11-20 Leap Motion, Inc. Dynamic interactive objects
WO2014194257A1 (en) 2013-05-31 2014-12-04 President And Fellows Of Harvard College Soft exosuit for assistance with human motion
US20140355825A1 (en) 2013-06-03 2014-12-04 Samsung Electronics Co., Ltd. Method and apparatus for estimating pose
WO2014197443A1 (en) 2013-06-03 2014-12-11 Kacyvenski Isaiah Motion sensor and analysis
US20140365163A1 (en) 2011-10-21 2014-12-11 Commissariat A L'energie Atomique Et Aux Ene Alt Method of detecting activity with a motion sensor, and corresponding device and computer program
US20140376773A1 (en) 2013-06-21 2014-12-25 Leap Motion, Inc. Tunable operational parameters in motion-capture and touchless interface operation
US20150006120A1 (en) 2013-06-26 2015-01-01 Dassault Systémes Simulia Corp. Musculo-Skeletal Modeling Using Finite Element Analysis, Process Integration, and Design Optimization
US20150010203A1 (en) 2011-06-20 2015-01-08 Veldandi Muninder Methods, apparatuses and computer program products for performing accurate pose estimation of objects
US20150025355A1 (en) 2013-07-22 2015-01-22 Thalmic Labs Inc. Systems, articles and methods for strain mitigation in wearable electronic devices
US20150029092A1 (en) 2013-07-23 2015-01-29 Leap Motion, Inc. Systems and methods of interpreting complex gestures
US20150035827A1 (en) 2012-03-29 2015-02-05 Sony Corporation Information processing device, information processing method, and information processing system
US20150045699A1 (en) 2012-04-03 2015-02-12 Camegie Mellon University Musculoskeletal activity recognition system and method
US20150051470A1 (en) 2013-08-16 2015-02-19 Thalmic Labs Inc. Systems, articles and methods for signal routing in wearable electronic devices
WO2015027089A1 (en) 2013-08-23 2015-02-26 Thalmic Labs Inc. Systems, articles, and methods for human-electronics interfaces
US20150070274A1 (en) 2013-06-17 2015-03-12 3Divi Company Methods and systems for determining 6dof location and orientation of head-mounted display and associated user movements
US20150070270A1 (en) 2013-09-06 2015-03-12 Thalmic Labs Inc. Systems, articles, and methods for electromyography-based human-electronics interfaces
US20150084860A1 (en) 2013-09-23 2015-03-26 Thalmic Labs Inc. Systems, articles, and methods for gesture identification in wearable electromyography devices
US20150109202A1 (en) 2013-10-22 2015-04-23 Thalmic Labs Inc. Systems, articles, and methods for gesture identification in wearable electromyography devices
US20150124566A1 (en) 2013-10-04 2015-05-07 Thalmic Labs Inc. Systems, articles and methods for wearable electronic devices employing contact sensors
US20150128094A1 (en) 2013-11-05 2015-05-07 At&T Intellectual Property I, L.P. Gesture-Based Controls Via Bone Conduction
US20150141784A1 (en) 2013-11-12 2015-05-21 Thalmic Labs Inc. Systems, articles, and methods for capacitive electromyography sensors
WO2015073713A1 (en) 2013-11-13 2015-05-21 Hrl Laboratories, Llc System for controlling brain machine interfaces and neural prosthetic systems
US20150148641A1 (en) 2013-11-27 2015-05-28 Thalmic Labs Inc. Systems, articles, and methods for electromyography sensors
US20150157944A1 (en) 2013-12-06 2015-06-11 Glenn I. Gottlieb Software Application for Generating a Virtual Simulation for a Sport-Related Activity
US20150169074A1 (en) 2013-12-12 2015-06-18 Thalmic Labs Inc. Systems, articles, and methods for gesture identification in wearable electromyography devices
US20150193949A1 (en) 2014-01-06 2015-07-09 Oculus Vr, Llc Calibration of multiple rigid bodies in a virtual reality system
WO2015123445A1 (en) 2014-02-14 2015-08-20 Thalmic Labs Inc. Systems, articles, and methods for elastic electrical cables and wearable electronic devices employing same
US20150261318A1 (en) 2014-03-12 2015-09-17 Michael Scavezze Gesture parameter tuning
US20150261306A1 (en) 2014-03-17 2015-09-17 Thalmic Labs Inc. Systems, devices, and methods for selecting between multiple wireless connections
US20150277575A1 (en) 2014-03-27 2015-10-01 Thalmic Labs Inc. Systems, devices, and methods for wearable electronic devices such as state machines
US20150302168A1 (en) 2014-04-17 2015-10-22 The Boeing Company Method and System for Tuning a Musculoskeletal Model
US20150309563A1 (en) * 2013-09-17 2015-10-29 Medibotics Llc Motion Recognition Clothing [TM] with Flexible Electromagnetic, Light, or Sonic Energy Pathways
US20150309582A1 (en) 2014-02-28 2015-10-29 Vikas Gupta Gesture operated wrist mounted camera system
US20150313496A1 (en) * 2012-06-14 2015-11-05 Medibotics Llc Mobile Wearable Electromagnetic Brain Activity Monitor
US20150325202A1 (en) 2014-05-07 2015-11-12 Thalmic Labs Inc. Systems, devices, and methods for wearable computers with heads-up displays
US20150346701A1 (en) 2014-05-27 2015-12-03 Leap Motion, Inc. Systems and methods of gestural interaction in a pervasive computing environment
US9218574B2 (en) 2013-05-29 2015-12-22 Purepredictive, Inc. User interface for machine learning
US20150370326A1 (en) 2014-06-23 2015-12-24 Thalmic Labs Inc. Systems, articles, and methods for wearable human-electronics interface devices
US20150370333A1 (en) 2014-06-19 2015-12-24 Thalmic Labs Inc. Systems, devices, and methods for gesture identification
WO2015199747A1 (en) 2014-06-23 2015-12-30 Thalmic Labs Inc. Systems, articles, and methods for wearable human-electronics interface devices
US9235934B2 (en) 2004-01-30 2016-01-12 Electronic Scripting Products, Inc. Computer interface employing a wearable article with an absolute pose detection component
US20160011668A1 (en) 2014-07-11 2016-01-14 Microsoft Corporation 3d gesture recognition
US9240069B1 (en) 2015-06-30 2016-01-19 Ariadne's Thread (Usa), Inc. Low-latency virtual reality display system
WO2016041088A1 (en) 2014-09-19 2016-03-24 Sulon Technologies Inc. System and method for tracking wearable peripherals in augmented reality and virtual reality applications
US20160187992A1 (en) 2014-04-03 2016-06-30 Honda Motor Co., Ltd. Smart tutorial for gesture control system
US20160235323A1 (en) 2013-09-25 2016-08-18 Mindmaze Sa Physiological parameter measurement and feedback system
US20160262687A1 (en) 2013-11-04 2016-09-15 Imperial Innovations Limited Biomechanical activity monitoring
US20160274758A1 (en) 2015-03-20 2016-09-22 Thalmic Labs Inc. Systems, devices, and methods for mitigating false positives in human-electronics interfaces
US9459697B2 (en) 2013-01-15 2016-10-04 Leap Motion, Inc. Dynamic, free-space user interactions for machine control
US20160292497A1 (en) 2015-04-06 2016-10-06 The Texas A&M University System Fusion of inertial and depth sensors for movement measurements and recognition
US20160313798A1 (en) 2015-04-22 2016-10-27 Medibotics Llc Nerd of the Rings -- Devices for Measuring Finger Motion and Recognizing Hand Gestures
US20160313899A1 (en) 2015-04-24 2016-10-27 Thalmic Labs Inc. Systems, methods, and computer program products for interacting with electronically displayed presentation materials
US20160313801A1 (en) 2015-01-02 2016-10-27 Wearable Devices Ltd. Method and apparatus for a gesture controlled interface for wearable devices
US20160350973A1 (en) 2015-05-28 2016-12-01 Microsoft Technology Licensing, Llc Shared tactile interaction and user safety in shared space multi-person immersive virtual reality
US20170031502A1 (en) 2014-09-26 2017-02-02 Sensel Inc. Systems and methods for manipulating a virtual environment
US20170061817A1 (en) 2015-08-28 2017-03-02 Icuemotion, Llc System for movement skill analysis and skill augmentation and cueing
US9597015B2 (en) 2008-02-12 2017-03-21 Portland State University Joint angle tracking with inertial sensors
US20170080346A1 (en) 2014-05-01 2017-03-23 Mohamad Abbas Methods and systems relating to personalized evolving avatars
US20170090604A1 (en) 2015-09-30 2017-03-30 Stmicroelectronics, Inc. Encapsulated pressure sensor
US9613262B2 (en) 2014-01-15 2017-04-04 Leap Motion, Inc. Object detection and tracking for providing a virtual device experience
US9612661B2 (en) 2015-01-02 2017-04-04 Wearable Devices Ltd. Closed loop feedback interface for wearable devices
WO2017062544A1 (en) 2015-10-06 2017-04-13 University Of Pittsburgh-Of The Commonwealth System Of Higher Education Method, device and system for sensing neuromuscular, physiological, biomechanical, and musculoskeletal activity
US20170123487A1 (en) 2015-10-30 2017-05-04 Ostendo Technologies, Inc. System and methods for on-body gestural interfaces and projection displays
US20170119472A1 (en) 2015-10-30 2017-05-04 Orthosensor Inc Spine measurement system and method therefor
US20170124816A1 (en) 2015-10-28 2017-05-04 Disney Enterprises, Inc. Automatic object detection and state estimation via electronic emissions sensing
US9659403B1 (en) 2014-01-06 2017-05-23 Leap Motion, Inc. Initializing orientation in space for predictive information for free space gesture control and communication
WO2017092225A1 (zh) 2015-12-04 2017-06-08 深圳大学 基于emg的可穿戴式文本输入系统及方法
US9687168B2 (en) 2009-07-30 2017-06-27 University Of Cape Town Non-invasive deep muscle electromyography
US9696795B2 (en) 2015-02-13 2017-07-04 Leap Motion, Inc. Systems and methods of creating a realistic grab experience in virtual reality/augmented reality environments
US20170188980A1 (en) 2016-01-06 2017-07-06 Empire Technology Development Llc Wearable sensor based body modeling
WO2017120669A1 (en) 2016-01-12 2017-07-20 Bigmotion Technologies Inc. Systems and methods for human body motion capture
US9741169B1 (en) 2014-05-20 2017-08-22 Leap Motion, Inc. Wearable augmented reality devices with object detection and tracking
US20170259167A1 (en) 2016-03-14 2017-09-14 Nathan Sterling Cook Brainwave virtual reality apparatus and method
US9766709B2 (en) 2013-03-15 2017-09-19 Leap Motion, Inc. Dynamic user interactions for display control
WO2017172185A1 (en) 2016-03-31 2017-10-05 Intel Corporation Sensor signal processing to determine finger and/or hand position
US9785247B1 (en) 2014-05-14 2017-10-10 Leap Motion, Inc. Systems and methods of tracking moving hands and recognizing gestural interactions
US9788789B2 (en) 2013-08-30 2017-10-17 Thalmic Labs Inc. Systems, articles, and methods for stretchable printed circuit boards
KR101790147B1 (ko) 2016-06-22 2017-10-25 재단법인 실감교류인체감응솔루션연구단 가상 객체 조작 시스템 및 방법
US20170308118A1 (en) 2014-11-12 2017-10-26 Kyocera Corporation Wearable device
US9864431B2 (en) 2016-05-11 2018-01-09 Microsoft Technology Licensing, Llc Changing an application state using neurological data
US20180024635A1 (en) 2016-07-25 2018-01-25 Patrick Kaifosh Methods and apparatus for predicting musculo-skeletal position information using wearable autonomous sensors
US20180020951A1 (en) 2016-07-25 2018-01-25 Patrick Kaifosh Adaptive system for deriving control signals from measurements of neuromuscular activity
US20180020978A1 (en) 2016-07-25 2018-01-25 Patrick Kaifosh System and method for measuring the movements of articulated rigid bodies
US20180024634A1 (en) 2016-07-25 2018-01-25 Patrick Kaifosh Methods and apparatus for inferring user intent based on neuromuscular signals
US20180154140A1 (en) 2015-06-02 2018-06-07 Battelle Memorial Institute Neural sleeve for neuromuscular stimulation, sensing and recording
US20180153430A1 (en) * 2016-12-02 2018-06-07 Pison Technology, Inc. Detecting and Using Body Tissue Electrical Signals
US20180153444A1 (en) 2016-12-05 2018-06-07 Intel Corporation Body movement tracking
US20180307314A1 (en) 2012-06-14 2018-10-25 Medibotics Llc Circumferential Array of Electromyographic (EMG) Sensors
US20180333575A1 (en) 2015-06-02 2018-11-22 Battelle Memorial Institute Non-invasive motor impairment rehabilitation system
US20180360379A1 (en) 2015-06-26 2018-12-20 Carnegie Mellon University System for Wearable, Low-Cost Electrical Impedance Tomography for Non-Invasive Gesture Recognition
US20190025919A1 (en) 2017-01-19 2019-01-24 Mindmaze Holding Sa System, method and apparatus for detecting facial expression in an augmented reality system
US20190038166A1 (en) 2018-01-03 2019-02-07 Intel Corporation Detecting fatigue based on electroencephalogram (eeg) data
US10203751B2 (en) 2016-05-11 2019-02-12 Microsoft Technology Licensing, Llc Continuous motion controls operable using neurological data
US20190076716A1 (en) 2017-09-12 2019-03-14 Intel Corporation Activity training system
US20190121305A1 (en) 2017-10-19 2019-04-25 Ctrl-Labs Corporation Systems and methods for identifying biological structures associated with neuromuscular source signals
US20190150777A1 (en) 2017-11-17 2019-05-23 Ctrl-Labs Corporation Dual-supply analog circuitry for sensing surface emg signals

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150323998A1 (en) 2014-05-06 2015-11-12 Qualcomm Incorporated Enhanced user interface for a wearable electronic device
CN204515353U (zh) * 2015-03-31 2015-07-29 深圳市长桑技术有限公司 一种智能手表
US10716513B2 (en) * 2015-04-17 2020-07-21 Heartflow, Inc. Systems and methods for cardiovascular blood flow and musculoskeletal modeling for predicting device failure or clinical events
US10496168B2 (en) 2018-01-25 2019-12-03 Ctrl-Labs Corporation Calibration techniques for handstate representation modeling using neuromuscular signals
CN107397649B (zh) * 2017-08-10 2020-08-14 燕山大学 一种基于径向基神经网络上肢外骨骼运动意图识别方法

Patent Citations (228)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4896120A (en) 1986-03-24 1990-01-23 Zvi Kamil Instrumentation amplifier arrangement
US5625577A (en) 1990-12-25 1997-04-29 Shukyohojin, Kongo Zen Sohonzan Shorinji Computer-implemented motion analysis method using dynamics
JPH05277080A (ja) 1992-04-01 1993-10-26 Dia Medical Syst Kk 生体電気現象検出装置
US6005548A (en) 1996-08-14 1999-12-21 Latypov; Nurakhmed Nurislamovich Method for tracking and displaying user's spatial position and orientation, a method for representing virtual reality for a user, and systems of embodiment of such methods
US6009210A (en) 1997-03-05 1999-12-28 Digital Equipment Corporation Hands-free interface to a virtual reality environment using head tracking
US6658287B1 (en) 1998-08-24 2003-12-02 Georgia Tech Research Corporation Method and apparatus for predicting the onset of seizures based on features derived from signals indicative of brain activity
US6244873B1 (en) 1998-10-16 2001-06-12 At&T Corp. Wireless myoelectric control apparatus and methods
US6774885B1 (en) 1999-01-20 2004-08-10 Motek B.V. System for dynamic registration, evaluation, and correction of functional human behavior
US6411843B1 (en) 1999-05-28 2002-06-25 Respironics, Inc. Method and apparatus for producing a model EMG signal from a measured EMG signal
US6720984B1 (en) 2000-06-13 2004-04-13 The United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration Characterization of bioelectric potentials
US7089148B1 (en) 2000-10-30 2006-08-08 The United States Of America As Represented By The Secretary Of The Navy Method and apparatus for motion tracking of an articulated rigid body
US20030144829A1 (en) 2002-01-25 2003-07-31 Geatz Michael W. System and method for sensing and evaluating physiological parameters and modeling an adaptable predictive analysis for symptoms management
US20030171921A1 (en) 2002-03-04 2003-09-11 Ntt Docomo, Inc. Speech recognition system, speech recognition method, speech synthesis system, speech synthesis method, and program product
US6942621B2 (en) 2002-07-11 2005-09-13 Ge Medical Systems Information Technologies, Inc. Method and apparatus for detecting weak physiological signals
US20040092839A1 (en) 2002-10-24 2004-05-13 Shin Sang-Hoon Apparatus and method for measuring local skin impedance using multiple array elecrtodes
US7787946B2 (en) 2003-08-18 2010-08-31 Cardiac Pacemakers, Inc. Patient monitoring, diagnosis, and/or therapy systems and methods
JP2005095561A (ja) 2003-08-21 2005-04-14 Yoshiyuki Yamaumi 装着式動作補助装置、装着式動作補助装置の制御方法および制御用プログラム
CN1838933A (zh) 2003-08-21 2006-09-27 山海嘉之 穿着式动作辅助装置、穿着式动作辅助装置的控制方法和控制用程序
US20100280628A1 (en) 2003-08-21 2010-11-04 University Of Tsukuba Wearable action-assist device, and method and program for controlling wearable action-assist device
US20100063794A1 (en) 2003-08-28 2010-03-11 Hernandez-Rebollar Jose L Method and apparatus for translating hand gestures
US7574253B2 (en) 2003-09-26 2009-08-11 Northwestern University Signal processing using non-linear regression with a sinusoidal model
US9235934B2 (en) 2004-01-30 2016-01-12 Electronic Scripting Products, Inc. Computer interface employing a wearable article with an absolute pose detection component
US20070256494A1 (en) 2004-06-16 2007-11-08 Yoshihiko Nakamura Muscular Strength Acquiring Method and Device Based on Musculoskeletal Model
US7901368B2 (en) 2005-01-06 2011-03-08 Braingate Co., Llc Neurally controlled patient ambulation system
US7351975B2 (en) 2005-03-29 2008-04-01 Duke University Sensor system for identifying and tracking movements of multiple sources
US8190249B1 (en) 2005-08-01 2012-05-29 Infinite Biomedical Technologies, Llc Multi-parametric quantitative analysis of bioelectrical signals
US8744543B2 (en) 2005-09-29 2014-06-03 Covidien Lp System and method for removing artifacts from waveforms
US20070172797A1 (en) 2006-01-12 2007-07-26 Kabushiki Kaisha Toyota Chuo Kenkyusho Method of constructing computer-based musculoskeletal model by redefining directions of pivot axes of joints in the same model
US7580742B2 (en) 2006-02-07 2009-08-25 Microsoft Corporation Using electroencephalograph signals for task classification and activity recognition
US20080214360A1 (en) 2006-03-03 2008-09-04 Garmin Ltd. Method and apparatus for estimating a motion parameter
US8311623B2 (en) 2006-04-15 2012-11-13 The Board Of Trustees Of The Leland Stanford Junior University Systems and methods for estimating surface electromyography
US20090112080A1 (en) 2006-04-21 2009-04-30 Quantum Applied Science & Research, Inc. System for Measuring Electric Signals
US7805386B2 (en) 2006-05-16 2010-09-28 Greer Douglas S Method of generating an encoded output signal using a manifold association processor having a plurality of pairs of processing elements trained to store a plurality of reciprocal signal pairs
US20070285399A1 (en) 2006-06-12 2007-12-13 Microsoft Corporation Extended eraser functions
US20140092009A1 (en) 2006-07-14 2014-04-03 Ailive Inc. Methods and Systems for Dynamic Calibration of Movable Game Controllers
US8437844B2 (en) 2006-08-21 2013-05-07 Holland Bloorview Kids Rehabilitation Hospital Method, system and apparatus for real-time classification of muscle signals from self-selected intentional movements
US20100293115A1 (en) 2006-08-21 2010-11-18 Kaveh Seyed Momen Method, system and apparatus for real-time classification of muscle signals from self -selected intentional movements
US20080052643A1 (en) 2006-08-25 2008-02-28 Kabushiki Kaisha Toshiba Interface apparatus and interface method
US20090082701A1 (en) 2007-03-07 2009-03-26 Motek Bv Method for real time interactive visualization of muscle forces and joint torques in the human body
JP2010520561A (ja) 2007-03-07 2010-06-10 モーテック・ビー．ブイ． 人体における筋力と関節トルクとをリアルタイムでインタラクティブに視覚化する方法
WO2008109248A2 (en) 2007-03-07 2008-09-12 Motek Bv Method for real time interactive visualization of muscle forces and joint torques in the human body
US20080221487A1 (en) 2007-03-07 2008-09-11 Motek Bv Method for real time interactive visualization of muscle forces and joint torques in the human body
US8890875B2 (en) 2007-05-11 2014-11-18 Commissariat A L'energie Atomique Processing method for capturing movement of an articulated structure
US8435191B2 (en) 2007-07-18 2013-05-07 Siemens Aktiengesellschaft Support structure for a sensor strip and sensor strip for mounting on said support structure
US20090082692A1 (en) 2007-09-25 2009-03-26 Hale Kelly S System And Method For The Real-Time Evaluation Of Time-Locked Physiological Measures
EP2198521B1 (en) 2007-09-26 2012-06-06 Medtronic, Inc. Chopper-stabilized analog-to-digital converter
WO2009042313A1 (en) 2007-09-26 2009-04-02 Medtronic, Inc. Chopper-stabilized analog-to-digital converter
US9597015B2 (en) 2008-02-12 2017-03-21 Portland State University Joint angle tracking with inertial sensors
US20100030532A1 (en) 2008-06-12 2010-02-04 Jasbir Arora System and methods for digital human model prediction and simulation
US8892479B2 (en) 2008-06-26 2014-11-18 Microsoft Corporation Recognizing finger gestures from forearm EMG signals
US20120188158A1 (en) 2008-06-26 2012-07-26 Microsoft Corporation Wearable electromyography-based human-computer interface
US9037530B2 (en) 2008-06-26 2015-05-19 Microsoft Technology Licensing, Llc Wearable electromyography-based human-computer interface
US8170656B2 (en) 2008-06-26 2012-05-01 Microsoft Corporation Wearable electromyography-based controllers for human-computer interface
US20090326406A1 (en) * 2008-06-26 2009-12-31 Microsoft Corporation Wearable electromyography-based controllers for human-computer interface
US20090327171A1 (en) 2008-06-26 2009-12-31 Microsoft Corporation Recognizing gestures from forearm emg signals
US8447704B2 (en) 2008-06-26 2013-05-21 Microsoft Corporation Recognizing gestures from forearm EMG signals
US20130232095A1 (en) 2008-06-26 2013-09-05 Microsoft Corporation Recognizing finger gestures from forearm emg signals
US20100106044A1 (en) 2008-10-27 2010-04-29 Michael Linderman EMG measured during controlled hand movement for biometric analysis, medical diagnosis and related analysis
US20130217998A1 (en) 2009-02-02 2013-08-22 Jointvue, Llc Motion Tracking System with Inertial-Based Sensing Units
US20100292617A1 (en) 2009-05-15 2010-11-18 Kin Fong Lei method and system for quantifying an intention of movement of a user
US20100315266A1 (en) 2009-06-15 2010-12-16 Microsoft Corporation Predictive interfaces with usability constraints
US9687168B2 (en) 2009-07-30 2017-06-27 University Of Cape Town Non-invasive deep muscle electromyography
US8718980B2 (en) 2009-09-11 2014-05-06 Qualcomm Incorporated Method and apparatus for artifacts mitigation with multiple wireless sensors
US20110077484A1 (en) 2009-09-30 2011-03-31 Nellcor Puritan Bennett Ireland Systems And Methods For Identifying Non-Corrupted Signal Segments For Use In Determining Physiological Parameters
US20110092826A1 (en) 2009-10-20 2011-04-21 Tatung Company System and method for measuring ECG and breath signals by using two polar electrodes
US8421634B2 (en) 2009-12-04 2013-04-16 Microsoft Corporation Sensing mechanical energy to appropriate the body for data input
US20120283526A1 (en) 2009-12-16 2012-11-08 Ictalcare A/S Method and a System for the Prediction of Epileptic Seizures
US20120265480A1 (en) 2010-01-20 2012-10-18 Omron Healthcare Co., Ltd. Body movement detection device
US8351651B2 (en) 2010-04-26 2013-01-08 Microsoft Corporation Hand-location post-process refinement in a tracking system
US8754862B2 (en) 2010-07-11 2014-06-17 Lester F. Ludwig Sequential classification recognition of gesture primitives and window-based parameter smoothing for high dimensional touchpad (HDTP) user interfaces
US8880163B2 (en) 2010-07-13 2014-11-04 Commissariat A L'energie Atomique Et Aux Energies Alternatives Method and a system for classifying neural signals, and a method of selecting electrodes for direct neural control
US20140052150A1 (en) 2010-08-02 2014-02-20 The Johns Hopkins University Method for presenting force sensor information using cooperative robot control and audio feedback
US20120066163A1 (en) 2010-09-13 2012-03-15 Nottingham Trent University Time to event data analysis method and system
US20150010203A1 (en) 2011-06-20 2015-01-08 Veldandi Muninder Methods, apparatuses and computer program products for performing accurate pose estimation of objects
US20130207889A1 (en) 2011-07-13 2013-08-15 Lumo Bodytech, Inc. System and Method of Biomechanical Posture Detection and Feedback Including Sensor Normalization
US20130077820A1 (en) 2011-09-26 2013-03-28 Microsoft Corporation Machine learning gesture detection
US20140365163A1 (en) 2011-10-21 2014-12-11 Commissariat A L'energie Atomique Et Aux Ene Alt Method of detecting activity with a motion sensor, and corresponding device and computer program
US20130141375A1 (en) 2011-12-06 2013-06-06 Lester F. Ludwig Gesteme (gesture primitive) recognition for advanced touch user interfaces
US20150035827A1 (en) 2012-03-29 2015-02-05 Sony Corporation Information processing device, information processing method, and information processing system
US20150045699A1 (en) 2012-04-03 2015-02-12 Camegie Mellon University Musculoskeletal activity recognition system and method
US9278453B2 (en) 2012-05-25 2016-03-08 California Institute Of Technology Biosleeve human-machine interface
US9867548B2 (en) 2012-05-25 2018-01-16 Emotiv, Inc. System and method for providing and aggregating biosignals and action data
US20130317648A1 (en) 2012-05-25 2013-11-28 California Institute Of Technology Biosleeve human-machine interface
US20130317382A1 (en) 2012-05-25 2013-11-28 Emotiv Lifesciences Inc. System and Method for Providing and Aggregating Biosignals and Action Data
US20180307314A1 (en) 2012-06-14 2018-10-25 Medibotics Llc Circumferential Array of Electromyographic (EMG) Sensors
US20150313496A1 (en) * 2012-06-14 2015-11-05 Medibotics Llc Mobile Wearable Electromagnetic Brain Activity Monitor
US8484022B1 (en) 2012-07-27 2013-07-09 Google Inc. Adaptive auto-encoders
US20140098018A1 (en) 2012-10-04 2014-04-10 Microsoft Corporation Wearable sensor for tracking articulated body-parts
US20140223462A1 (en) 2012-12-04 2014-08-07 Christopher Allen Aimone System and method for enhancing content using brain-state data
US20140196131A1 (en) 2013-01-07 2014-07-10 Salutron, Inc. User authentication based on a wrist vein pattern
US20140198035A1 (en) 2013-01-14 2014-07-17 Thalmic Labs Inc. Wearable muscle interface systems, devices and methods that interact with content displayed on an electronic display
US20140198034A1 (en) 2013-01-14 2014-07-17 Thalmic Labs Inc. Muscle interface device and method for interacting with content displayed on wearable head mounted displays
US9459697B2 (en) 2013-01-15 2016-10-04 Leap Motion, Inc. Dynamic, free-space user interactions for machine control
KR20150123254A (ko) 2013-02-22 2015-11-03 탈믹 랩스 인크 제스처-기반 제어를 위해 근활성도 센서 신호와 관성 센서 신호를 결합하는 방법 및 기기
CN105190578A (zh) 2013-02-22 2015-12-23 赛尔米克实验室公司 用于基于手势控制的组合肌肉活动传感器信号和惯性传感器信号的方法和设备
EP2959394A1 (en) 2013-02-22 2015-12-30 Thalmic Labs Inc. Methods and devices that combine muscle activity sensor signals and inertial sensor signals for gesture-based control
US9299248B2 (en) 2013-02-22 2016-03-29 Thalmic Labs Inc. Method and apparatus for analyzing capacitive EMG and IMU sensor signals for gesture control
JP2016507851A (ja) 2013-02-22 2016-03-10 サルミック ラブス インコーポレイテッド ジェスチャに基づいて制御するための筋活動センサ信号と慣性センサ信号とを結合する方法および装置
WO2014130871A1 (en) 2013-02-22 2014-08-28 Thalmic Labs Inc. Methods and devices that combine muscle activity sensor signals and inertial sensor signals for gesture-based control
US20140240223A1 (en) 2013-02-22 2014-08-28 Thalmic Labs Inc. Method and apparatus for analyzing capacitive emg and imu sensor signals for gesture control
CA2902045A1 (en) 2013-02-22 2014-08-28 Thalmic Labs Inc. Methods and devices that combine muscle activity sensor signals and inertial sensor signals for gesture-based control
US20140240103A1 (en) 2013-02-22 2014-08-28 Thalmic Labs Inc. Methods and devices for combining muscle activity sensor signals and inertial sensor signals for gesture-based control
US20140245200A1 (en) 2013-02-25 2014-08-28 Leap Motion, Inc. Display control with gesture-selectable control paradigms
US20140249397A1 (en) 2013-03-01 2014-09-04 Thalmic Labs Inc. Differential non-contact biopotential sensor
US20140278441A1 (en) 2013-03-15 2014-09-18 Qualcomm Incorporated Systems and methods for switching processing modes using gestures
US9766709B2 (en) 2013-03-15 2017-09-19 Leap Motion, Inc. Dynamic user interactions for display control
US20140304665A1 (en) 2013-04-05 2014-10-09 Leap Motion, Inc. Customized gesture interpretation
US20140334083A1 (en) 2013-05-13 2014-11-13 Thalmic Labs Inc. Systems, articles and methods for wearable electronic devices that accommodate different user forms
WO2014186370A1 (en) 2013-05-13 2014-11-20 Thalmic Labs Inc. Systems, articles and methods for wearable electronic devices that accommodate different user forms
US10152082B2 (en) 2013-05-13 2018-12-11 North Inc. Systems, articles and methods for wearable electronic devices that accommodate different user forms
US20140344731A1 (en) 2013-05-17 2014-11-20 Leap Motion, Inc. Dynamic interactive objects
US9218574B2 (en) 2013-05-29 2015-12-22 Purepredictive, Inc. User interface for machine learning
WO2014194257A1 (en) 2013-05-31 2014-12-04 President And Fellows Of Harvard College Soft exosuit for assistance with human motion
WO2014197443A1 (en) 2013-06-03 2014-12-11 Kacyvenski Isaiah Motion sensor and analysis
US20140355825A1 (en) 2013-06-03 2014-12-04 Samsung Electronics Co., Ltd. Method and apparatus for estimating pose
US20150070274A1 (en) 2013-06-17 2015-03-12 3Divi Company Methods and systems for determining 6dof location and orientation of head-mounted display and associated user movements
US20140376773A1 (en) 2013-06-21 2014-12-25 Leap Motion, Inc. Tunable operational parameters in motion-capture and touchless interface operation
US20150006120A1 (en) 2013-06-26 2015-01-01 Dassault Systémes Simulia Corp. Musculo-Skeletal Modeling Using Finite Element Analysis, Process Integration, and Design Optimization
US9408316B2 (en) 2013-07-22 2016-08-02 Thalmic Labs Inc. Systems, articles and methods for strain mitigation in wearable electronic devices
US20150025355A1 (en) 2013-07-22 2015-01-22 Thalmic Labs Inc. Systems, articles and methods for strain mitigation in wearable electronic devices
US20150029092A1 (en) 2013-07-23 2015-01-29 Leap Motion, Inc. Systems and methods of interpreting complex gestures
US20150051470A1 (en) 2013-08-16 2015-02-19 Thalmic Labs Inc. Systems, articles and methods for signal routing in wearable electronic devices
WO2015027089A1 (en) 2013-08-23 2015-02-26 Thalmic Labs Inc. Systems, articles, and methods for human-electronics interfaces
CA2921954A1 (en) 2013-08-23 2015-02-26 Thalmic Labs Inc. Systems, articles, and methods for human-electronics interfaces
US20150057770A1 (en) 2013-08-23 2015-02-26 Thaimic Labs Inc. Systems, articles, and methods for human-electronics interfaces
US9788789B2 (en) 2013-08-30 2017-10-17 Thalmic Labs Inc. Systems, articles, and methods for stretchable printed circuit boards
US9372535B2 (en) 2013-09-06 2016-06-21 Thalmic Labs Inc. Systems, articles, and methods for electromyography-based human-electronics interfaces
US20150070270A1 (en) 2013-09-06 2015-03-12 Thalmic Labs Inc. Systems, articles, and methods for electromyography-based human-electronics interfaces
US20150309563A1 (en) * 2013-09-17 2015-10-29 Medibotics Llc Motion Recognition Clothing [TM] with Flexible Electromagnetic, Light, or Sonic Energy Pathways
US9483123B2 (en) 2013-09-23 2016-11-01 Thalmic Labs Inc. Systems, articles, and methods for gesture identification in wearable electromyography devices
US20150084860A1 (en) 2013-09-23 2015-03-26 Thalmic Labs Inc. Systems, articles, and methods for gesture identification in wearable electromyography devices
US20160235323A1 (en) 2013-09-25 2016-08-18 Mindmaze Sa Physiological parameter measurement and feedback system
US20150124566A1 (en) 2013-10-04 2015-05-07 Thalmic Labs Inc. Systems, articles and methods for wearable electronic devices employing contact sensors
US20180150033A1 (en) 2013-10-04 2018-05-31 Thalmic Labs Inc. Systems, articles and methods for wearable electronic devices employing contact sensors
US20150109202A1 (en) 2013-10-22 2015-04-23 Thalmic Labs Inc. Systems, articles, and methods for gesture identification in wearable electromyography devices
US9389694B2 (en) 2013-10-22 2016-07-12 Thalmic Labs Inc. Systems, articles, and methods for gesture identification in wearable electromyography devices
US20160262687A1 (en) 2013-11-04 2016-09-15 Imperial Innovations Limited Biomechanical activity monitoring
US20150128094A1 (en) 2013-11-05 2015-05-07 At&T Intellectual Property I, L.P. Gesture-Based Controls Via Bone Conduction
US20180067553A1 (en) 2013-11-12 2018-03-08 Thalmic Labs Inc. Systems, articles, and methods for capacitive electromyography sensors
US10042422B2 (en) 2013-11-12 2018-08-07 Thalmic Labs Inc. Systems, articles, and methods for capacitive electromyography sensors
US10101809B2 (en) 2013-11-12 2018-10-16 Thalmic Labs Inc. Systems, articles, and methods for capacitive electromyography sensors
US20150141784A1 (en) 2013-11-12 2015-05-21 Thalmic Labs Inc. Systems, articles, and methods for capacitive electromyography sensors
WO2015073713A1 (en) 2013-11-13 2015-05-21 Hrl Laboratories, Llc System for controlling brain machine interfaces and neural prosthetic systems
US20180064363A1 (en) 2013-11-27 2018-03-08 Thalmic Labs Inc. Systems, articles, and methods for electromyography sensors
US10188309B2 (en) 2013-11-27 2019-01-29 North Inc. Systems, articles, and methods for electromyography sensors
US20150148641A1 (en) 2013-11-27 2015-05-28 Thalmic Labs Inc. Systems, articles, and methods for electromyography sensors
US20180344195A1 (en) 2013-11-27 2018-12-06 Thalmic Labs Inc. Systems, articles, and methods for electromyography sensors
WO2015081113A1 (en) 2013-11-27 2015-06-04 Cezar Morun Systems, articles, and methods for electromyography sensors
US20150157944A1 (en) 2013-12-06 2015-06-11 Glenn I. Gottlieb Software Application for Generating a Virtual Simulation for a Sport-Related Activity
US9367139B2 (en) 2013-12-12 2016-06-14 Thalmic Labs Inc. Systems, articles, and methods for gesture identification in wearable electromyography devices
US20150169074A1 (en) 2013-12-12 2015-06-18 Thalmic Labs Inc. Systems, articles, and methods for gesture identification in wearable electromyography devices
US20150193949A1 (en) 2014-01-06 2015-07-09 Oculus Vr, Llc Calibration of multiple rigid bodies in a virtual reality system
US9659403B1 (en) 2014-01-06 2017-05-23 Leap Motion, Inc. Initializing orientation in space for predictive information for free space gesture control and communication
US9613262B2 (en) 2014-01-15 2017-04-04 Leap Motion, Inc. Object detection and tracking for providing a virtual device experience
US20150234426A1 (en) 2014-02-14 2015-08-20 Thalmic Labs Inc. Systems, articles, and methods for elastic electrical cables and wearable electronic devices employing same
US9600030B2 (en) 2014-02-14 2017-03-21 Thalmic Labs Inc. Systems, articles, and methods for elastic electrical cables and wearable electronic devices employing same
KR20160121552A (ko) 2014-02-14 2016-10-19 탈믹 랩스 인크 탄성 전기 케이블 및 이를 사용하는 웨어러블 전자 장치를 위한 시스템, 물품, 및 방법
JP2017509386A (ja) 2014-02-14 2017-04-06 サルミック ラブス インコーポレイテッド 弾性電気ケーブルのためのシステム、製品、及び方法、並びに、それを用いるウェアラブル電子デバイス
WO2015123445A1 (en) 2014-02-14 2015-08-20 Thalmic Labs Inc. Systems, articles, and methods for elastic electrical cables and wearable electronic devices employing same
CA2939644A1 (en) 2014-02-14 2015-08-20 Thalmic Labs Inc. Systems, articles, and methods for elastic electrical cables and wearable electronic devices employing same
EP3104737A1 (en) 2014-02-14 2016-12-21 Thalmic Labs Inc. Systems, articles, and methods for elastic electrical cables and wearable electronic devices employing same
CN106102504A (zh) 2014-02-14 2016-11-09 赛尔米克实验室公司 用于弹性电力缆线的系统、制品和方法以及采用弹性电力缆线的可佩戴电子装置
US20150309582A1 (en) 2014-02-28 2015-10-29 Vikas Gupta Gesture operated wrist mounted camera system
US20150261318A1 (en) 2014-03-12 2015-09-17 Michael Scavezze Gesture parameter tuning
US20150261306A1 (en) 2014-03-17 2015-09-17 Thalmic Labs Inc. Systems, devices, and methods for selecting between multiple wireless connections
US10199008B2 (en) 2014-03-27 2019-02-05 North Inc. Systems, devices, and methods for wearable electronic devices as state machines
US20150277575A1 (en) 2014-03-27 2015-10-01 Thalmic Labs Inc. Systems, devices, and methods for wearable electronic devices such as state machines
US20160187992A1 (en) 2014-04-03 2016-06-30 Honda Motor Co., Ltd. Smart tutorial for gesture control system
US20150302168A1 (en) 2014-04-17 2015-10-22 The Boeing Company Method and System for Tuning a Musculoskeletal Model
US20170080346A1 (en) 2014-05-01 2017-03-23 Mohamad Abbas Methods and systems relating to personalized evolving avatars
US20150325202A1 (en) 2014-05-07 2015-11-12 Thalmic Labs Inc. Systems, devices, and methods for wearable computers with heads-up displays
US9785247B1 (en) 2014-05-14 2017-10-10 Leap Motion, Inc. Systems and methods of tracking moving hands and recognizing gestural interactions
US9741169B1 (en) 2014-05-20 2017-08-22 Leap Motion, Inc. Wearable augmented reality devices with object detection and tracking
US20150346701A1 (en) 2014-05-27 2015-12-03 Leap Motion, Inc. Systems and methods of gestural interaction in a pervasive computing environment
US20180120948A1 (en) 2014-06-19 2018-05-03 Thalmic Labs Inc. Systems, devices, and methods for gesture identification
US20150370333A1 (en) 2014-06-19 2015-12-24 Thalmic Labs Inc. Systems, devices, and methods for gesture identification
US9880632B2 (en) 2014-06-19 2018-01-30 Thalmic Labs Inc. Systems, devices, and methods for gesture identification
US10216274B2 (en) 2014-06-23 2019-02-26 North Inc. Systems, articles, and methods for wearable human-electronics interface devices
US20150370326A1 (en) 2014-06-23 2015-12-24 Thalmic Labs Inc. Systems, articles, and methods for wearable human-electronics interface devices
WO2015199747A1 (en) 2014-06-23 2015-12-30 Thalmic Labs Inc. Systems, articles, and methods for wearable human-electronics interface devices
US20160011668A1 (en) 2014-07-11 2016-01-14 Microsoft Corporation 3d gesture recognition
WO2016041088A1 (en) 2014-09-19 2016-03-24 Sulon Technologies Inc. System and method for tracking wearable peripherals in augmented reality and virtual reality applications
US20170031502A1 (en) 2014-09-26 2017-02-02 Sensel Inc. Systems and methods for manipulating a virtual environment
US20170308118A1 (en) 2014-11-12 2017-10-26 Kyocera Corporation Wearable device
US9720515B2 (en) 2015-01-02 2017-08-01 Wearable Devices Ltd. Method and apparatus for a gesture controlled interface for wearable devices
US9612661B2 (en) 2015-01-02 2017-04-04 Wearable Devices Ltd. Closed loop feedback interface for wearable devices
US20160313801A1 (en) 2015-01-02 2016-10-27 Wearable Devices Ltd. Method and apparatus for a gesture controlled interface for wearable devices
US9696795B2 (en) 2015-02-13 2017-07-04 Leap Motion, Inc. Systems and methods of creating a realistic grab experience in virtual reality/augmented reality environments
US20180088765A1 (en) 2015-03-20 2018-03-29 Thalmic Labs Inc. Systems, devices, and methods for mitigating false positives in human-electronics interfaces
US20180101289A1 (en) 2015-03-20 2018-04-12 Thalmic Labs Inc. Systems, devices, and methods for mitigating false positives in human-electronics interfaces
US20160274758A1 (en) 2015-03-20 2016-09-22 Thalmic Labs Inc. Systems, devices, and methods for mitigating false positives in human-electronics interfaces
US20180095630A1 (en) 2015-03-20 2018-04-05 Thalmic Labs Inc. Systems, devices, and methods for mitigating false positives in human-electronics interfaces
US20160292497A1 (en) 2015-04-06 2016-10-06 The Texas A&M University System Fusion of inertial and depth sensors for movement measurements and recognition
US9891718B2 (en) 2015-04-22 2018-02-13 Medibotics Llc Devices for measuring finger motion and recognizing hand gestures
US20160313798A1 (en) 2015-04-22 2016-10-27 Medibotics Llc Nerd of the Rings -- Devices for Measuring Finger Motion and Recognizing Hand Gestures
US20160313899A1 (en) 2015-04-24 2016-10-27 Thalmic Labs Inc. Systems, methods, and computer program products for interacting with electronically displayed presentation materials
US20160350973A1 (en) 2015-05-28 2016-12-01 Microsoft Technology Licensing, Llc Shared tactile interaction and user safety in shared space multi-person immersive virtual reality
US20180154140A1 (en) 2015-06-02 2018-06-07 Battelle Memorial Institute Neural sleeve for neuromuscular stimulation, sensing and recording
US20180333575A1 (en) 2015-06-02 2018-11-22 Battelle Memorial Institute Non-invasive motor impairment rehabilitation system
US20180360379A1 (en) 2015-06-26 2018-12-20 Carnegie Mellon University System for Wearable, Low-Cost Electrical Impedance Tomography for Non-Invasive Gesture Recognition
US9240069B1 (en) 2015-06-30 2016-01-19 Ariadne's Thread (Usa), Inc. Low-latency virtual reality display system
US20170061817A1 (en) 2015-08-28 2017-03-02 Icuemotion, Llc System for movement skill analysis and skill augmentation and cueing
US20170090604A1 (en) 2015-09-30 2017-03-30 Stmicroelectronics, Inc. Encapsulated pressure sensor
WO2017062544A1 (en) 2015-10-06 2017-04-13 University Of Pittsburgh-Of The Commonwealth System Of Higher Education Method, device and system for sensing neuromuscular, physiological, biomechanical, and musculoskeletal activity
US20170124816A1 (en) 2015-10-28 2017-05-04 Disney Enterprises, Inc. Automatic object detection and state estimation via electronic emissions sensing
US20170123487A1 (en) 2015-10-30 2017-05-04 Ostendo Technologies, Inc. System and methods for on-body gestural interfaces and projection displays
US20170119472A1 (en) 2015-10-30 2017-05-04 Orthosensor Inc Spine measurement system and method therefor
WO2017092225A1 (zh) 2015-12-04 2017-06-08 深圳大学 基于emg的可穿戴式文本输入系统及方法
US20170188980A1 (en) 2016-01-06 2017-07-06 Empire Technology Development Llc Wearable sensor based body modeling
WO2017120669A1 (en) 2016-01-12 2017-07-20 Bigmotion Technologies Inc. Systems and methods for human body motion capture
US20170259167A1 (en) 2016-03-14 2017-09-14 Nathan Sterling Cook Brainwave virtual reality apparatus and method
WO2017172185A1 (en) 2016-03-31 2017-10-05 Intel Corporation Sensor signal processing to determine finger and/or hand position
US10203751B2 (en) 2016-05-11 2019-02-12 Microsoft Technology Licensing, Llc Continuous motion controls operable using neurological data
US9864431B2 (en) 2016-05-11 2018-01-09 Microsoft Technology Licensing, Llc Changing an application state using neurological data
KR101790147B1 (ko) 2016-06-22 2017-10-25 재단법인 실감교류인체감응솔루션연구단 가상 객체 조작 시스템 및 방법
US20180020951A1 (en) 2016-07-25 2018-01-25 Patrick Kaifosh Adaptive system for deriving control signals from measurements of neuromuscular activity
US20180024635A1 (en) 2016-07-25 2018-01-25 Patrick Kaifosh Methods and apparatus for predicting musculo-skeletal position information using wearable autonomous sensors
US20180020978A1 (en) 2016-07-25 2018-01-25 Patrick Kaifosh System and method for measuring the movements of articulated rigid bodies
US20180024634A1 (en) 2016-07-25 2018-01-25 Patrick Kaifosh Methods and apparatus for inferring user intent based on neuromuscular signals
US10070799B2 (en) 2016-12-02 2018-09-11 Pison Technology, Inc. Detecting and using body tissue electrical signals
US20180153430A1 (en) * 2016-12-02 2018-06-07 Pison Technology, Inc. Detecting and Using Body Tissue Electrical Signals
US20180153444A1 (en) 2016-12-05 2018-06-07 Intel Corporation Body movement tracking
US20190025919A1 (en) 2017-01-19 2019-01-24 Mindmaze Holding Sa System, method and apparatus for detecting facial expression in an augmented reality system
US20190076716A1 (en) 2017-09-12 2019-03-14 Intel Corporation Activity training system
US20190121305A1 (en) 2017-10-19 2019-04-25 Ctrl-Labs Corporation Systems and methods for identifying biological structures associated with neuromuscular source signals
US20190121306A1 (en) 2017-10-19 2019-04-25 Ctrl-Labs Corporation Systems and methods for identifying biological structures associated with neuromuscular source signals
US20190150777A1 (en) 2017-11-17 2019-05-23 Ctrl-Labs Corporation Dual-supply analog circuitry for sensing surface emg signals
US20190038166A1 (en) 2018-01-03 2019-02-07 Intel Corporation Detecting fatigue based on electroencephalogram (eeg) data

Non-Patent Citations (64)

* Cited by examiner, † Cited by third party
Title
Arkenbout et al., Robust Hand Motion Tracking Through Data Fusion of 5Dt Data Glove and Nimble VR Kinect Camera Measurements. Sensors. 2015;15:31644-71.
Benko et al., Enhancing Input on and Above the Interactive Surface with Muscle Sensing. The ACM International Conference on Interactive Tabletops and Surfaces. ITS '09. 2009:93-100.
Boyali et al., Spectral Collaborative Representation based Classification for hand gestures recognition on electromyography signals. Biomedical Signal Processing and Control. 2016;24:11-18.
Cheng et al., A Novel Phonology- and Radical-Coded Chinese Sign Language Recognition Framework Using Accelerometer and Surface Electromyography Sensors. Sensors. 2015;15:23303-24.
Csapo et al., Evaluation of Human-Myo Gesture Control Capabilities in Continuous Search and Select Operations. 7th IEEE International Conference on Cognitive Infocommunications. 2016;000415-20.
Davoodi et al., Development of a Physics-Based Target Shooting Game to Train Amputee Users of Multijoint Upper Limb Prostheses. Presence. Massachusetts Institute of Technology. 2012;21(1):85-95.
Delis et al., Development of a Myoelectric Controller Based on Knee Angle Estimation. Biodevices 2009. International Conference on Biomedical Electronics and Devices. Jan. 17, 2009. 7 pages.
Diener et al., Direct conversion from facial myoelectric signals to speech using Deep Neural Networks. 2015 International Joint Conference on Neural Networks (IJCNN). Oct. 1, 2015. 7 pages.
Ding et al., HMM with improved feature extraction-based feature parameters for identity recognition of gesture command operators by using a sensed Kinect-data stream. Neurocomputing. 2017;262:108-19.
Farina et al., Man/machine interface based on the discharge timings of spinal motor neurons after targeted muscle reinnervation. Nature. Biomedical Engineering. 2017;1:1-12.
Favorskaya et al., Localization and Recognition of Dynamic Hand Gestures Based on Hierarchy of Manifold Classifiers. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences. 2015;XL-5/W6:1-8.
Gallina et al., Surface EMG Biofeedback. Surface Electromyography: Physiology, Engineering, and Applications. 2016:485-500.
Hauschild et al., A Virtual Reality Environment for Designing and Fitting Neural Prosthetic Limbs. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2007;15(1):9-15.
International Preliminary Report on Patentability for International Application No. PCT/US2017/043686 dated Feb. 7, 2019.
International Preliminary Report on Patentability for International Application No. PCT/US2017/043693 dated Feb. 7, 2019.
International Preliminary Report on Patentability for International Application No. PCT/US2017/043791 dated Feb. 7, 2019.
International Preliminary Report on Patentability for International Application No. PCT/US2017/043792 dated Feb. 7, 2019.
International Search Report and Written Opinion for International Application No. PCT/US19/20065 dated May 16, 2019.
International Search Report and Written Opinion for International Application No. PCT/US2017/043686 dated Oct. 6, 2017.
International Search Report and Written Opinion for International Application No. PCT/US2017/043693 dated Oct. 6, 2017.
International Search Report and Written Opinion for International Application No. PCT/US2017/043791 dated Oct. 5, 2017.
International Search Report and Written Opinion for International Application No. PCT/US2017/043792 dated Oct. 5, 2017.
International Search Report and Written Opinion for International Application No. PCT/US2018/056768 dated Jan. 15, 2019.
International Search Report and Written Opinion for International Application No. PCT/US2018/061409 dated Mar. 12, 2019.
International Search Report and Written Opinion for International Application No. PCT/US2018/063215 dated Mar. 21, 2019.
International Search Report and Written Opinion for International Application No. PCT/US2019/015134 dated May 15, 2019.
International Search Report and Written Opinion for International Application No. PCT/US2019/015167 dated May 21, 2019.
International Search Report and Written Opinion for International Application No. PCT/US2019/015174 dated May 21, 2019.
International Search Report and Written Opinion for International Application No. PCT/US2019/015180 dated May 28, 2019.
International Search Report and Written Opinion for International Application No. PCT/US2019/015183 dated May 3, 2019.
International Search Report and Written Opinion for International Application No. PCT/US2019/015238 dated May 16, 2019.
International Search Report and Written Opinion for International Application No. PCT/US2019/015244 dated May 16, 2019.
Jiang, Purdue University Graduate School Thesis/Dissertation Acceptance. Graduate School Form 30. Updated Jan. 15, 2015. 24 pages.
Kawaguchi et al., Estimation of Finger Joint Angles Based on Electromechanical Sensing of Wrist Shape. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2017;25(9):1409-18.
Kim et al., Real-Time Human Pose Estimation and Gesture Recognition from Depth Images Using Superpixels and SVM Classifier. Sensors. 2015;15:12410-27.
Koerner, Design and Characterization of the Exo-Skin Haptic Device: A Novel Tendon Actuated Textile Hand Exoskeleton. 2017. 5 pages.
Lee et al., Motion and Force Estimation System of Human Fingers. Journal of Institute of Control, Robotics and Systems. 2011;17(10):1014-1020.
Li et al., Motor Function Evaluation of Hemiplegic Upper-Extremities Using Data Fusion from Wearable Inertial and Surface EMG Sensors. Sensors. MDPI. 2017;17(582):1-17.
Lopes et al., Hand/arm gesture segmentation by motion using IMU and EMG sensing. ScienceDirect. Elsevier. Procedia Manufacturing. 2017;11:107-13.
Martin et al., A Novel Approach of Prosthetic Arm Control using Computer Vision, Biosignals, and Motion Capture. IEEE. 2014. 5 pages.
Mcintee, A Task Model of Free-Space Movement-Based Geastures. Dissertation. Graduate Faculty of North Carolina State University. Computer Science. 2016. 129 pages.
Mendes et al., Sensor Fusion and Smart Sensor in Sports and Biomedical Applications. Sensors. 2016;16(1569):1-31.
Naik et al., Source Separation and Identification issues in bio signals: A solution using Blind source seperation. Intech. 2009. 23 pages.
Naik et al., Subtle Hand gesture identification for HCI using Temporal Decorrelation Source Separation BSS of surface EMG. Digital Image Computing Techniques and Applications. IEEE Computer Society. 2007;30-7.
Negro et al., Multi-channel intramuscular and surface EMG decomposition by convolutive blind source separation. Journal of Neural Engineering. 2016;13:1-17.
PCT/US19/20065, dated May 16, 2019, International Search Report and Written Opinion.
PCT/US2019/015134, dated May 15, 2019, International Search Report and Written Opinion.
PCT/US2019/015167, dated May 21, 2019, International Search Report and Written Opinion.
PCT/US2019/015174, dated May 21, 2019, International Search Report and Written Opinion.
PCT/US2019/015180, dated May 28, 2019, International Search Report and Written Opinion.
PCT/US2019/015183, dated May 3, 2019, International Search Report and Written Opinion.
PCT/US2019/015238, dated May 16, 2019, International Search Report and Written Opinion.
PCT/US2019/015244, dated May 16, 2019, International Search Report and Written Opinion.
Saponas et al., Demonstrating the Feasibility of Using Forearm Electromyography for Muscle-Computer Interfaces. CHI 2008 Proceedings. Physiological Sensing for Input. 2008:515-24.
Saponas et al., Enabling Always-Available Input with Muscle-Computer Interfaces. UIST '09. 2009:167-76.
Saponas et al., Making Muscle-Computer Interfaces More Practical. CHI 2010: Brauns and Brawn. 2010:851-4.
Sartori et al., Neural Data-Driven Musculoskeletal Modeling for Personalized Neurorehabilitation Technologies. IEEE Transactions on Biomedical Engineering. 2016;63(5):879-93.
Sauras-Perez et al., A Voice and Pointing Gesture Interaction System for Supporting Human Spontaneous Decisions in Autonomous Cars. Clemson University. All Dissertations. 2017. 174 pages.
Shen et al., I am a Smartwatch and I can Track my User's Arm. University of Illinois at Urbana-Champaign. MobiSys' 16.
Son et al., Evaluating the utility of two gestural discomfort evaluation methods. PLOS One. 2017. 21 pages.
Strbac et al., Microsoft Kinect-Based Artificial Perception System for Control of Functional Electrical Stimulation Assisted Grasping. Hindawi Publishing Corporation. BioMed Research International. 2014. 13 pages.
Torres, Myo Gesture Control Armband. PCMag. Https://www.pcmag.com/article2/0,2817,2485462,00.asp 2015. 9 pages.
Wodzinski et al., Sequential Classification of Palm Gestures Based on A* Algorithm and MLP Neural Network for Quadrocopter Control. Metrol. Meas. Syst., 2017;24(2):265-76.
Xue et al., Multiple Sensors Based Hand Motion Recognition Using Adaptive Directed Acyclic Graph. Applied Sciences. MDPI. 2017;7(358):1-14.

Cited By (33)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11921471B2 (en) 2013-08-16 2024-03-05 Meta Platforms Technologies, Llc Systems, articles, and methods for wearable devices having secondary power sources in links of a band for providing secondary power in addition to a primary power source
US11644799B2 (en) 2013-10-04 2023-05-09 Meta Platforms Technologies, Llc Systems, articles and methods for wearable electronic devices employing contact sensors
US11079846B2 (en) 2013-11-12 2021-08-03 Facebook Technologies, Llc Systems, articles, and methods for capacitive electromyography sensors
US11666264B1 (en) 2013-11-27 2023-06-06 Meta Platforms Technologies, Llc Systems, articles, and methods for electromyography sensors
US10990174B2 (en) 2016-07-25 2021-04-27 Facebook Technologies, Llc Methods and apparatus for predicting musculo-skeletal position information using wearable autonomous sensors
US11337652B2 (en) 2016-07-25 2022-05-24 Facebook Technologies, Llc System and method for measuring the movements of articulated rigid bodies
US10656711B2 (en) 2016-07-25 2020-05-19 Facebook Technologies, Llc Methods and apparatus for inferring user intent based on neuromuscular signals
US11635736B2 (en) 2017-10-19 2023-04-25 Meta Platforms Technologies, Llc Systems and methods for identifying biological structures associated with neuromuscular source signals
US11163361B2 (en) 2018-01-25 2021-11-02 Facebook Technologies, Llc Calibration techniques for handstate representation modeling using neuromuscular signals
US11069148B2 (en) 2018-01-25 2021-07-20 Facebook Technologies, Llc Visualization of reconstructed handstate information
US10937414B2 (en) 2018-05-08 2021-03-02 Facebook Technologies, Llc Systems and methods for text input using neuromuscular information
US11036302B1 (en) 2018-05-08 2021-06-15 Facebook Technologies, Llc Wearable devices and methods for improved speech recognition
US11216069B2 (en) 2018-05-08 2022-01-04 Facebook Technologies, Llc Systems and methods for improved speech recognition using neuromuscular information
US10687759B2 (en) 2018-05-29 2020-06-23 Facebook Technologies, Llc Shielding techniques for noise reduction in surface electromyography signal measurement and related systems and methods
US11129569B1 (en) 2018-05-29 2021-09-28 Facebook Technologies, Llc Shielding techniques for noise reduction in surface electromyography signal measurement and related systems and methods
US10970374B2 (en) 2018-06-14 2021-04-06 Facebook Technologies, Llc User identification and authentication with neuromuscular signatures
US11045137B2 (en) 2018-07-19 2021-06-29 Facebook Technologies, Llc Methods and apparatus for improved signal robustness for a wearable neuromuscular recording device
US11179066B2 (en) 2018-08-13 2021-11-23 Facebook Technologies, Llc Real-time spike detection and identification
US10842407B2 (en) 2018-08-31 2020-11-24 Facebook Technologies, Llc Camera-guided interpretation of neuromuscular signals
US10905350B2 (en) 2018-08-31 2021-02-02 Facebook Technologies, Llc Camera-guided interpretation of neuromuscular signals
US11567573B2 (en) 2018-09-20 2023-01-31 Meta Platforms Technologies, Llc Neuromuscular text entry, writing and drawing in augmented reality systems
US10921764B2 (en) 2018-09-26 2021-02-16 Facebook Technologies, Llc Neuromuscular control of physical objects in an environment
US10970936B2 (en) 2018-10-05 2021-04-06 Facebook Technologies, Llc Use of neuromuscular signals to provide enhanced interactions with physical objects in an augmented reality environment
US11797087B2 (en) 2018-11-27 2023-10-24 Meta Platforms Technologies, Llc Methods and apparatus for autocalibration of a wearable electrode sensor system
US11941176B1 (en) 2018-11-27 2024-03-26 Meta Platforms Technologies, Llc Methods and apparatus for autocalibration of a wearable electrode sensor system
US10905383B2 (en) 2019-02-28 2021-02-02 Facebook Technologies, Llc Methods and apparatus for unsupervised one-shot machine learning for classification of human gestures and estimation of applied forces
US11481030B2 (en) 2019-03-29 2022-10-25 Meta Platforms Technologies, Llc Methods and apparatus for gesture detection and classification
US11961494B1 (en) 2019-03-29 2024-04-16 Meta Platforms Technologies, Llc Electromagnetic interference reduction in extended reality environments
US11481031B1 (en) 2019-04-30 2022-10-25 Meta Platforms Technologies, Llc Devices, systems, and methods for controlling computing devices via neuromuscular signals of users
US11493993B2 (en) 2019-09-04 2022-11-08 Meta Platforms Technologies, Llc Systems, methods, and interfaces for performing inputs based on neuromuscular control
US11974857B2 (en) 2019-10-08 2024-05-07 Unlimited Tomorrow, Inc. Biometric sensor array
US11907423B2 (en) 2019-11-25 2024-02-20 Meta Platforms Technologies, Llc Systems and methods for contextualized interactions with an environment
US11868531B1 (en) 2021-04-08 2024-01-09 Meta Platforms Technologies, Llc Wearable device providing for thumb-to-finger-based input gestures detected based on neuromuscular signals, and systems and methods of use thereof

Also Published As

Publication number Publication date
EP3742961A4 (en) 2021-03-31
WO2019147996A1 (en) 2019-08-01
US11163361B2 (en) 2021-11-02
CN111902077B (zh) 2023-08-04
EP3742961A1 (en) 2020-12-02
CN111902077A (zh) 2020-11-06
US20190227627A1 (en) 2019-07-25
US20200125172A1 (en) 2020-04-23

Similar Documents

Publication Publication Date Title
US11163361B2 (en) Calibration techniques for handstate representation modeling using neuromuscular signals
US10489986B2 (en) User-controlled tuning of handstate representation model parameters
US10817795B2 (en) Handstate reconstruction based on multiple inputs
US11127143B2 (en) Real-time processing of handstate representation model estimates
US11069148B2 (en) Visualization of reconstructed handstate information
US10842407B2 (en) Camera-guided interpretation of neuromuscular signals
US20210405750A1 (en) Methods and apparatus for predicting musculo-skeletal position information using wearable autonomous sensors
US20190223748A1 (en) Methods and apparatus for mitigating neuromuscular signal artifacts

Legal Events

Date Code Title Description
FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY

FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO SMALL (ORIGINAL EVENT CODE: SMAL); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Free format text: ENTITY STATUS SET TO SMALL (ORIGINAL EVENT CODE: SMAL); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

AS Assignment

Owner name: COGNESCENT CORPORATION, NEW YORK

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:GIURGICA-TIRON, TUDOR;REEL/FRAME:050740/0745

Effective date: 20161219

Owner name: CTRL-LABS CORPORATION, NEW YORK

Free format text: CHANGE OF NAME;ASSIGNOR:COGNESCENT CORPORATION;REEL/FRAME:050740/0870

Effective date: 20171003

AS Assignment

Owner name: CTRL-LABS CORPORATION, NEW YORK

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:KAIFOSH, PATRICK;BERENZWEIG, ADAM;KOBER, STEVEN;AND OTHERS;SIGNING DATES FROM 20190717 TO 20190923;REEL/FRAME:050832/0794

STCF Information on status: patent grant

Free format text: PATENTED CASE

AS Assignment

Owner name: FACEBOOK TECHNOLOGIES, INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:CTRL-LABS CORPORATION;REEL/FRAME:051649/0001

Effective date: 20200116

AS Assignment

Owner name: FACEBOOK TECHNOLOGIES, LLC, CALIFORNIA

Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE RECEIVING PARTY DATA PREVIOUSLY RECORDED AT REEL: 051649 FRAME: 0001. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNOR:CTRL-LABS CORPORATION;REEL/FRAME:051867/0136

Effective date: 20200116

FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

AS Assignment

Owner name: META PLATFORMS TECHNOLOGIES, LLC, CALIFORNIA

Free format text: CHANGE OF NAME;ASSIGNOR:FACEBOOK TECHNOLOGIES, LLC;REEL/FRAME:060199/0876

Effective date: 20220318

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 4