EP3068349A1 - System for controlling brain machine interfaces and neural prosthetic systems - Google Patents

System for controlling brain machine interfaces and neural prosthetic systems

Info

Publication number
EP3068349A1
EP3068349A1 EP14861977.8A EP14861977A EP3068349A1 EP 3068349 A1 EP3068349 A1 EP 3068349A1 EP 14861977 A EP14861977 A EP 14861977A EP 3068349 A1 EP3068349 A1 EP 3068349A1
Authority
EP
European Patent Office
Prior art keywords
motion
commands
prosthetic device
joint
motion commands
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP14861977.8A
Other languages
German (de)
French (fr)
Other versions
EP3068349A4 (en
Inventor
Vincent DE SAPIO
Narayan Srinivasa
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
HRL Laboratories LLC
Original Assignee
HRL Laboratories LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by HRL Laboratories LLC filed Critical HRL Laboratories LLC
Publication of EP3068349A1 publication Critical patent/EP3068349A1/en
Publication of EP3068349A4 publication Critical patent/EP3068349A4/en
Pending legal-status Critical Current

Links

Classifications

    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61FFILTERS IMPLANTABLE INTO BLOOD VESSELS; PROSTHESES; DEVICES PROVIDING PATENCY TO, OR PREVENTING COLLAPSING OF, TUBULAR STRUCTURES OF THE BODY, e.g. STENTS; ORTHOPAEDIC, NURSING OR CONTRACEPTIVE DEVICES; FOMENTATION; TREATMENT OR PROTECTION OF EYES OR EARS; BANDAGES, DRESSINGS OR ABSORBENT PADS; FIRST-AID KITS
    • A61F2/00Filters implantable into blood vessels; Prostheses, i.e. artificial substitutes or replacements for parts of the body; Appliances for connecting them with the body; Devices providing patency to, or preventing collapsing of, tubular structures of the body, e.g. stents
    • A61F2/50Prostheses not implantable in the body
    • A61F2/68Operating or control means
    • A61F2/70Operating or control means electrical
    • A61F2/72Bioelectric control, e.g. myoelectric
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61FFILTERS IMPLANTABLE INTO BLOOD VESSELS; PROSTHESES; DEVICES PROVIDING PATENCY TO, OR PREVENTING COLLAPSING OF, TUBULAR STRUCTURES OF THE BODY, e.g. STENTS; ORTHOPAEDIC, NURSING OR CONTRACEPTIVE DEVICES; FOMENTATION; TREATMENT OR PROTECTION OF EYES OR EARS; BANDAGES, DRESSINGS OR ABSORBENT PADS; FIRST-AID KITS
    • A61F2/00Filters implantable into blood vessels; Prostheses, i.e. artificial substitutes or replacements for parts of the body; Appliances for connecting them with the body; Devices providing patency to, or preventing collapsing of, tubular structures of the body, e.g. stents
    • A61F2/50Prostheses not implantable in the body
    • A61F2/68Operating or control means
    • A61F2/70Operating or control means electrical
    • A61F2002/701Operating or control means electrical operated by electrically controlled means, e.g. solenoids or torque motors
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61FFILTERS IMPLANTABLE INTO BLOOD VESSELS; PROSTHESES; DEVICES PROVIDING PATENCY TO, OR PREVENTING COLLAPSING OF, TUBULAR STRUCTURES OF THE BODY, e.g. STENTS; ORTHOPAEDIC, NURSING OR CONTRACEPTIVE DEVICES; FOMENTATION; TREATMENT OR PROTECTION OF EYES OR EARS; BANDAGES, DRESSINGS OR ABSORBENT PADS; FIRST-AID KITS
    • A61F2/00Filters implantable into blood vessels; Prostheses, i.e. artificial substitutes or replacements for parts of the body; Appliances for connecting them with the body; Devices providing patency to, or preventing collapsing of, tubular structures of the body, e.g. stents
    • A61F2/50Prostheses not implantable in the body
    • A61F2/76Means for assembling, fitting or testing prostheses, e.g. for measuring or balancing, e.g. alignment means
    • A61F2002/7615Measuring means
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/049Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs

Definitions

  • the present invention relates to a robotic control system and, more particularly, to a system for controlling robotic prosthetic devices given motor intent inferred from neuroimatiinti data.
  • BMI Brain machine interfaces
  • neural prosthetics offer great hope for restoring function to people with spinal cord injuries and amputees, as well augmenting and enhancing the abilities of people with full motor function
  • prosthetic device control there have been fewer advances, with most approaches rooted in conventional robotic control.
  • BMI and neural prosthetics research which addresses the decoding of cortical signals for the downsiream execution of motion commands by an externa! device (see, for example, the List of Incorporated Literature References, Literature Reference Nos. I, 9, .1 1 5 1.2, and 15).
  • Most of this work focuses on the neural decoding with less rigor applied to the prosthesis control hi cases where a robotic arm prosthesis is control led, the motion trajectory is typically converted into joint commands and executed using joint space control (see, for example.
  • Task/posture decomposition has particular advantages over joint space control in B Is. It is based on an abstraction analogous to the manner in which cortical signals are encoded (e.g., eye-centered, hand-centered Cartesian coordinates). Further, it is generali abie to whole body prosthetic control involving multiple prosthetic limbs and .high degree-of-iteedom kinematics. Joint space prosthetic control is limited in this capacity due to its reliance on inverse kinematics solutions. Additionally, task/posture decomposition allows for the specification of postural behaviors based on minimizing important objective functions (e.g. power consumption, virtual muscle effort, etc.) consistent with the execution of cortical motion commands.
  • important objective functions e.g. power consumption, virtual muscle effort, etc.
  • the system includes one or more processors and a memory, the memory having executable instructions encoded thereon, such that upon execution of the instructions, the one or more processors perform several operations, such as recei ving neuroimaging data of a user from a neuroimaging device.
  • the neuroimaging data is then decoded to infer spatial motion intent of the user, where the spatial motion intent includes desired motion commands of the torque controlled prosthetic device represented in a coordinate system.
  • the system executes, with a prosthesis controller, the motion commands as torque commands to cause the torque controlled prosthetic device to move according to the spatial motion intent of the user.
  • the system includes at least one torque controlled prosthetic device operabiy connected with the one or more processors.
  • the system performs an operation of receiving, in the controller, sensory information regarding a current state of the prosthetic device,
  • commands are executed using a task decomposition and posture decomposition, wherein the task decomposition is a task space control and the posture decomposition is formulated as a cost potential which represents a cost, function.
  • the motion commands are executed as torque commands that generate a desired, task space control while minimizing the cost potential.
  • the motio commands are executing using spiking neural network.
  • the prosthesis controller is a neuromorphic
  • prosthesis controller and further comprises: a neuromorphic spike encoder to represent the motion command as a set of neural spikes; a neuromorphic motor mapper to map the neural spikes representing Cartesian
  • the system performs operations of receiving a model of the prosthetic device and a musculoskeletal mode!
  • the musculoskeletal model includes musculoskeletal dynamics that include steadv state tendon forces; eeneratina, with a sensorimotor controller, simulated neural excitations given the motion commands to drive a set of muscle activations in a musculoskeletal simulation; and generating, with the prosthesis controller,, simulated actuator joint torques given the motion commands to drive a simulated prosthetic device.
  • the present invention also includes a computer program
  • the computer program product includes computer-readable instructions stored on a non-transitory computer-readable medium that are executable b a computer having one or more processors, such that upon execution of the instructions, the one or more processors perform the operations listed herein.
  • the computer implemented method includes an act of causing a computer to execute such instructions and perform the resulting operations.
  • FIG. t is a block diagram depicting the components of system
  • FIG. 2 is an illustration of a computer program product embodying an aspect of the present invention
  • FIG. 3 is a process flow-chart illustrating how high-level motor intent from the brain is decocted and a task-level .motion command is
  • FIG. 4A is a simplified model of a human arm actuated by 14 muscles
  • FIG. 4B s a table illustrating corresponding maximum isometric
  • FIG. 5 A is an il lustration of a redundant muscle-actuated model of the human ann used as a physiological template for controlling a prosthetic arm;
  • FIG. 5B is a graph illustrating time histories of joint motion, while performing a simulation run of the model as illustrated i FIG. 5 A;
  • FIG. 5C is a graph illustrating time histories of hand motion while performing a simulation run of the mode l as illustrated in FIG. 5A;
  • FIG. 5D is a graph illustrating time histories of muscle effort while performing a simulation run of the model as illustrated in FIG. 5 A;
  • FIG. 6 A is a flow chart illustrating an artificial spiking neural network where motor map redundancy is resolved by training the network only on unique solutions of the inverse mapping given by ⁇ 3 ⁇ 4
  • FIG. 6B is a flow chart illustrating an artificial spiking neural network where motor map redundancy is resolved by explicitly inhibiting the learning of solutions where the cost is high;
  • FIG. 7 is flow chart illustrating how high-level motor intent from the brai is decoded and a task-level motion command is transmitted, to the prosthetic device controller;
  • FIG. 8 is a flow chart illustrating how a decoded cortical output is sent to a simulated prosthesis controller and a simulated sensorimotor controller; [00041 ]
  • FIG. 9 is a flow chart illustrating the neural and musculoskeletal
  • FIG. 10 is a functional block diagram depicting a system-level
  • FIG. 9 showing an abstracted representation of neural and musculoskeletal physiology
  • FIG. 1 1 is an illustration depicting an active state museulotendon
  • FIG. 12 is an illustration depicting a neuromuscular
  • FIG. 13 is an illustration of a task-level sensorimotor controller
  • FIG. 14 is a functional block diagram depicting an abstracted
  • FIG. 15 is an illustration depicting a neuromuscular
  • musculoskeletal system feed-forward path
  • musculoskeletai system dynamics showing the musculoskeletai system dynamics augmented with a set ofholonomic constraints and Lagrange multipliers
  • FIG , 16 is an illustration of a constrained task-level sensorimotor
  • F G. 17 is an illustration of a, constrained task-level sensorimotor
  • controller which generates motion consistent, with the input motion commands in the presence of olonomic system constraints.
  • the present invention relates to a robotic control system and, more particularly, to a system for controlling robotic prosthetic devices given motor intent inferred from neuioimaging data.
  • the following descriptio is presented to enable one of ordinary skill in the art to make and use the invention and to incorporate it in the context of particular applications.
  • Various modifications, as well as a variety of uses in. different applications will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to a wide range of aspects.
  • the present invention is not intended to be limited to the aspects presented, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
  • Buneo (1 A., Jarvis, M. R., Batista, A. P., & Andersen, R. A., (2002), Direct visuomotor transformati ns for reaching. Nature, 416(6881 ), 632-636.
  • Kiiatib, O,, (1 95), inertia! properties in robotic manipulation An object-level framework, The International Journal of Robotics Research,. 14(1 ), 1 -36.
  • the present invention has three "principal" aspects.
  • the first is a
  • the system for controlling robotic prosthetic devices gi en motor intent inferred from neuroimaging data.
  • the system is typically in the form of a computer system operating software or in the form of a "hard-coded" instruction set.
  • This system may be incorporated into a wide variety of devices that provide different functionalities.
  • the system is incorporated into a robot having actuators and appendages or other motion operable components and any other components as may be required to provide the functionality as described herein.
  • the second principal aspect is a method, typically in the form of software, operated using a data processing system (computer).
  • the third principal aspect is a computer program product.
  • the computer program product generally represents computer-readable instructions stored on a non-transitory computer- readable medium such as an optical storage device, e.g., a compact disc (CD) or digital versatile disc (DVD), or a magnetic storage device such as a floppy disk or magnetic tape.
  • a non-transitory computer- readable medium such as an optical storage device, e.g., a compact disc (CD) or digital versatile disc (DVD), or a magnetic storage device such as a floppy disk or magnetic tape.
  • Computer-readable media include hard disks, read-only memory (ROM), and flash-type memories. These aspects will be described in more detail below.
  • FIG. I A block diagram depicting an example of a system (i.e., computer system 100) of the present invention is provided in FIG. I .
  • the computer system 100 is configured - to perform calculations, processes, operations, and or functions associated with a program or algorithm.
  • certain processes and steps discussed herein are realized as a series of instructions (e.g., software program) that reside within computer readable memory units and are executed by one or more processors of the computer system 100. When executed, the instructions cause the computer system 100 to perform specific actions and exhibit specific behavior, such as described herein.
  • the computer system 1 0 may include an address/data bus 102 that is configured to communicate information. Additionally, one or more data processing units, such as a processor 104 (or processors), are coupled with the address/data bus 10:2. The processor 104 is configured to process information and instructions. In an aspect, the processor 104 is a
  • processor 104 may be a different type of processor such as a parallel processor, or a field programmable gate array.
  • the computer system 100 is configured to utilize one or more data storage units.
  • the computer system 100 may include a volatile memory unit 106 (e.g., random access memory (“RAM”), static RAM, dynamic RAM, etc.) coupled with the address data bus 102, wherein a volatile memory unit 106 is configured t store information and instructions for the processor 104,
  • the computer system 1 0 further may include a non- volatile memory unit 108 (e.g., read-only memory (“ROM”),
  • PROM programmable ROM
  • ROM erasable programmable ROM
  • the computer system 100 may execute instructions retrieved from an online data storage unit such as in "Cloud” computing, in an aspect, the computer system 100 also may include one or more interfaces, such as an interface 110, coupled with the address/data bus 102, The one or more interfaces are configured to enable the computer system 100 to interface with other electronic devices and computer systems.
  • the commu ication interfaces implemented by the one or more interfaces may include wireline (e.g., serial cables, modems, network adaptors, etc.) and/or wireless (e.g.. wireless modems, wireless network adaptors, etc.) communication technology.
  • the computer system 100 may include an input device .1 12 coupled with the address/data bus 102, wherein the input device 1 12 is configured to communicate information and command selections to the processor 100.
  • the input device 1 12 is an alphanumeric input device, such as a keyboard, that may include alphanumeric and/or function keys.
  • the input device 1.12 may be an input device other than an alphanumeric input device
  • the computer system 100 may include a cursor control device 1 14 coupled with the address/data bus 102, wherein the cursor control device 1 14 is configured to communicate user input information and/or command selections to the processor 100.
  • the cursor control device .1 14 is implemented using a device such as a mouse, a track-ball, a track-pad, an optical tracking device, or a touch screen.
  • the cursor control device 1 14 is directed and/or activated via input from the input device 112, such as in response to the use of special keys and key sequence commands associated with the input device 1 12.
  • the cursor control device 1. 1 is configured to be directed or guided by voice commands.
  • the computer system 100 further may include one or more optional computer usable data storage devices, such as a storage device 116, coupled with the address/data bus 102.
  • the storage device 1 16 is configured to store information and/or computer executable instructions, in one aspect, the storage device 1 16 is a storage device such as a magnetic or optical disk drive (e.g., hard disk drive ⁇ "HDD"), floppy diskette, compact disk read only memory (“CD-ROM”), digital versatile disk (“DVD”)).
  • a display device 1 18 is coupled with the address/data bus 102, wherein the display device 118 is configured to display video and/or graphics.
  • the display device 118 may include a cathode ray rube ("CRT"), liquid crystal display (“LCD”), field emission display (“FED”), plasma display, or any othe display device suitable for displaying video and/or graphic images and alphanumeric characters recognizable to a user.
  • CTR cathode ray rube
  • LCD liquid crystal display
  • FED field emission display
  • plasma display or any othe display device suitable for displaying video and/or graphic images and alphanumeric characters recognizable to a user.
  • the computer system 100 presented herein is an example computing environment in accordance with an aspect.
  • the non-limiting example of the computer system 100 is not strictly limited to being a computer system.
  • an aspect provides thai the computer system 100 represents a t pe of data processing analysis that may be used in accordance with various aspects described herein.
  • other computing systems may also be implemented, indeed, the spirit and scope of the present technology is not limited to any single data processing environment.
  • one or more operations of various aspects of the present technology are control led or implemented using computer-executable instructions, such as program modules, being executed by a computer.
  • program modules include routines, programs, objects, components and/or data structures that are configured to perform particular tasks or implement particular abstract data types.
  • an aspect provides that one or more aspects of the present technology are implemented by utilising one or more distributed computing environments, such as where tasks are performed by remote processing devices that are linked through a communications network, or such as where various program modules are located in both local and remote computer-storage medi including memory-storage devices.
  • the computer program product is depicted as floppy disk 200 or an optical disk 202 such as a CD or DVD.
  • the computer program product generally represents computer-readable instructions stored o any compatible non-transitory computer-readable medium.
  • the term "instructions” as used with respect to this invention generally indicates a set of operations to be performed on a computer, and may represent pieces of a whole program or individual, separable, software modules.
  • Non-limitin examples of "instruction” include computer program code (source or object code) and "hard -coded” electronics (i.e. computer operations coded into a computer chip).
  • the "instruction" is stored on any non-transitory computer-readable medium, such as in the memory of a computer or on a floppy disk, a CD-ROM, and a flash drive. In either event, the instructions are encoded on a non- transitory computer-readable medium.
  • Brain machine interfaces (B I) and neural prosthetics offer great hope for restoring function to people with spinal cord injuries and amputees, as wel l augmenting and enhancing the abilities of people with full motor function.
  • There ha ve been a number of advances in neural imaging as well as in decoding motor intent from neuroiinagiiig data.
  • prosthetic device control there have been fewer advances, with most approaches rooted in conventional robotic control.
  • the system described herein provides a system-level architecture for controlling robotic prosthetic devices given motor intent inferred from neuroimaging data.
  • the system also provides stimulation architecture to facilitate virtual testing of the specific controller and prosthetic device design, with, respect to the human subject, prior to any clinical
  • One controller is based on a bio-inspired task/posture decomposition of motion intended for impiementation on conventional digital computing hardware.
  • the other controller is based on neuroraorphic mapping of task motion to joint motion with a bio-inspired resolution of kinematic redundancy., whic can he implemented on oeuromorphic computing hardware.
  • a unique aspect is that both approaches utilize bio-inspired abstractions for controlling prosthetic devices in a natural human-like manner rather than utilizing conventional inverse kinematic approaches.
  • BMi cortical commands
  • human augmentation technologies can be implemented for enhancing the capabilities of the manufacturing floor worker. These technologies include assistive exoskeletons and other devices to increase worker productivity and reduce injury.
  • the invention offers significant capabilities for the development of next generation BMls, such as user performance enhancement, assistive robotic devices, and neural prosthetic technologies.
  • This desired motion command is referred to herein as 3 ⁇ 4 a multidimensional vector of task coord inates (e.g. Cartesian coordinates associated motion of a. single hand or coordinated motion of both hands).
  • the goal is to control the prosthetic device based on this motor intent.
  • inverse kinematics is conventionally used to map a desired Cartesian goal into a set of joint angles, ⁇ sf 1 (3 ⁇ 4 ⁇
  • the prosthetic device is controlled by servoing individual joints or by using computed torque control.
  • a human limb e.g., the human arm
  • a task space representation of the dynamics is given by, f ⁇ A($)x- p(q.,q) - p(q), where x is a description of task coordinates that are to be controlled, f are the control forces that act at the task, and .f * f X and .p ⁇ q ⁇ are the task, space mass .matrix, centrifugal and Coriolis force vector, and gravity force vector, respectively.
  • Task space was described by hatib in Literature Reference No. .10.
  • control of the prosthetic device can be decomposed into a task-level component and a complementary postural component (due to the kinematic redundancy). Based on the choice of a particular generalized inverse of the Jaeobian these tw components of motion control are guaranteed to be dynamically consistent with each other and, as such, control can be synthesized to execute the cortical motion command while simultaneously achieving som postural objective for the prosthetic device.
  • FIG. 3 depicts the system-level architecture associated with this approach. High-level motor intent from the brain 302 is decoded (with a cortical decoder 304) and a task-level motion command 306 is transmitted to the prosthetic device controller 308. High-level motor intent is captured using a neuroimaging device or any other suitable device operable for capturing neuroimaging data 300, non-limiting examples of which include neuroimaging equipment such as
  • the cortical decoder 304 uses a decoding algorithm to infer spatial, motion intent 306 from the nemoiraadna data 300.
  • this motor intent is in the form of desired motion commands represented in different Cartesian coordinate systems (eye-centered, hand-centered, etc.) using visumoior transfonmtions.
  • the controller 308 executes the motion command using a task/posture decomposition. Relevant physiological criteri (i.e., postural criterion) or device power consumption criteria can be used to drive the postural component of motion 310, consistent with the conical command. Sensory information 312 (e.g., joint angles of the current state of the prosthetic- device 1 ⁇ from the prosthetic device 314 is fed back to the controller 308, resulting in closed loop control The architecture also accommodates sensory information 312 from the prosthetic device 314 being sent to the brain 302 to facilitate afferent proprioception.
  • Relevant physiological criteri i.e., postural criterion
  • Device power consumption criteria can be used to drive the postural component of motion 310, consistent with the conical command.
  • Sensory information 312 e.g., joint angles of the current state of the prosthetic- device 1 ⁇ from the prosthetic device 314 is fed back to the controller 308, resulting in closed loop control
  • the architecture also accommodates sensory information 3
  • a potential. can be specified as representing some cost function.
  • An example of suitable cost functions was described by De Sapio in Literature ' Reference No. 6.
  • the task-level motion command can be controlled while simultaneously controlling motion in the complementary posture space to minimize This can be accomplished by applying gradient descent in the null/posture space. That is.
  • the control torque which generates the desired task space control while minimizing the instantaneous cost potential in the posture space, is then.
  • a physiological cost criterion like muscle effort, can he selectively represented.
  • De Sapio, et a!. (see Literature Reference No. 6), proposed the following muscle effort criterion related to minimizing muscle activation, where g ⁇ q) is the gravity torque, M ⁇ q) is the matrix of muscle moment arms, and IT / 0? ⁇ is the diagonal matrix mapping muscle acti vation, «, to muscle force.
  • the elements of f for a given muscle can be modeled by, where ⁇ $) is the muscle length, and f a and 1 ⁇ 2 represent, the maximum isometric muscle force and the optimal muscle fiber length respectively.
  • FIG. 4A illustrates a simplified biomechanical model of the human arm actuated by 14 muscles
  • IG. 4B is a table illustrating the maximum isometric forces. / s , and optimal fiber lengths, l Mf> for the biomechanical model of
  • FIG. 4A The control law is defined as,
  • control torque is defined as, where a dissipative term, and a gain, are added on the gradient descent term in the posture space portion of the control torque.
  • FIG. 5A illustrates a redundant muscle-actuated model of the human arm used as a physiological template for controlling a prosthetic arm
  • PIGs. SB, 5C and 5D are graphs illustrating time histories of joint motion, hand motion, and muscle effort, respecti vely, for a simulation run using the model of FIG. 5A.
  • the model of FIG. 5 A depicts initial and final configurations (joint angles), q ⁇ ) and - (i f ), associated with gradient descent movement to a target, x f .
  • the motion corresponds to gradient descent of the rnuscie effort, subject to the task requirement.
  • the posture control seeks to reduce the muscle effort but is also constrained by the task requirement.
  • the controller achieves the final target objective while the posture control simultaneously seeks to reduce the muscle effort (consisteni with the task requirement). No compensation for the dynamics (except for gravity) was included in the control. Thus, there is no feedback linearization of the inertia! terms present in the control . Normally, perfect feedback linearization would produce straight line motion to the goal, in the absence of feedback, linearization non-straight line motion results.
  • the biomechanical model of FIG. 4A is used to generate a physiological criterion that is encoded in the prosthesis controller.
  • the ac tuation system of the prosthetic arm need not be analogous to its biological muscle-driver* counterpart Rather, the biological muscie model defines a set of virtual muscles that direc t the postural control of the prosthetic device in a bio-inspired manner, eve if conventional robot actuators (e.g., DC motors, etc. ) are used in the prosthetic device to control joint movements.
  • conventional robot actuators e.g., DC motors, etc.
  • the control architecture depicted in FIG, 3 is intended to be
  • kinematic redundancy in the prosthetic device was resol ved using a. task/posture decomposition, in the case of motor map learning by an arti ficial spiking neural network , kinematic redundancy also needs to be addressed since a differential movement in x would result in an infinite space of differential movements in q.
  • FIG. 6.A and 6B depict artificial spiking neural networks with two different approaches for resolving kinematic redundancy. Both networks learn the inverse kinematic mapping using trajectory babbling 600, 61 1 of the prosthetic arm. Kinematic redundancy impacts the trainin such that for a given configuration (set. of joint angles), q 603, and Cartesian displacement. Ax 601, there will be multiple va lues of configuration (joint) space displacement, A*
  • this can be resolved by using a specific generalized inverse of / (i.e.,
  • the spiking neural network of FIG. 6A trains on the output of the trajectory babbling in Cartesian space 600.
  • the signals 603 and Ax 601 serve as the context for the training and the signal &q 604 serves as the signal to be learned with respect, to that context.
  • Signal 603 represents the current joint angle configuration that is being sensed at any given time.
  • the spikes representing these signals act as inputs to layer 1 605 of spiki g neurons.
  • the outputs of layer 1 605 act as spike inputs to layer 2 606.
  • An additional layer 3 60? of neurons is employed on the context side (q 603 and Ax 601 input side).
  • each neuron codes for a single combined representation of both the configuration, q 603, and Cartesian displacement.
  • Ax see Literature Reference No, 14
  • the synaptic weights of the spiking neurons in that layer are reinforced consistent with learning the training signals 609.
  • Spike timing-dependent plasticity (see Literature Reference Nos. 2 and 14) is utilized to modulate the synaptic weights in the layers. After sufficient training on motor babbling data the spike timing-dependent plasticity can be turned, off as the neurons in layer 4 60? will have ieamed the differential, kinematic mapping. Note that x(t+ 1 ) and q(t+ 1 ) denote a new value for a next time step.
  • the generalized inverse, * 602 can be chose based on the minimization of some quadratic Form. For example, to find a solution thai minimizes I f I 3 , the following solution can be used: &q ⁇ 4 ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ) ⁇ 1 ⁇ .
  • FIG. 6B the redundancy problem can also be resolved by adding a layer of neurons 612 that explicitly inhibits the learning of solutions where the cost (physiological or otherwise) is high.
  • the structure of Fig. 6B is similar to Fig, 6A, However, rather than using a generalized inverse to compute a unique value of &q to team for a given configuration, ⁇ ?, and Cartesian displacement, ⁇ ⁇ , forward kinematics 610 is used to compute the unique forward mapping from the configuration (joint) space motor babbling 61 1 to Cartesian space.
  • a layer of neurons 612 is incorporated to inhibit the learning 613 f q signals with high cost values. In this way training will be directed at learning Aq values associated with minimal cost values.
  • FIG. 7 that map Cartesian displacement, &x 601, to configuration (joint) space displacement, &q.
  • a feedback loop 700 is specified in FIG. 7 to control the prosthetic arm.
  • the overall system consists of output from a conical decoder, which specifies intentional mo vement 306 of the arm in Cartesian space. This signal is encoded as spikes 706 b the aeuromorphic spike encoder 702. These spikes, as well, as joint angle measurements from the prosthetic arm 714 encoded as spikes 720, act as input to the neuromorphic motor mapper 708.
  • a spiking output representing the learned mapping to the
  • FIG. 7 depicts the system-level architecture associated with the approach depicted in FIG. 6B.
  • the prosthetic device controller i.e., neuromorphic prosthesis controller
  • the prosthetic device controller 308 includes a neuromorphic spike encoder 702 (to represent the motion command 306 as a set of spikes 706), a neuromorphic motor mapper 708 (as illustrated in FIG. 6A and/or 6B) to map the neural spikes representing Cartesian displacements to neural spikes representing configuration (joint) space displacements, a spike decoder 710 (to decode the neural spikes representing configuration (joint) space displacements) and generate joint space commands, and a join t servo 712 to execute the joint space
  • a neuromorphic spike encoder 702 to represent the motion command 306 as a set of spikes 706
  • a neuromorphic motor mapper 708 as illustrated in FIG. 6A and/or 6B
  • a spike decoder 710 to decode the neural spikes representing configuration (jo
  • the sensed joint angles 71 are fed hack to the joint servo 712 and to the motor mapper 70S (after spike encoding 720).
  • Relevant physiological criteria or device power consumption criteria can incorporated into the neuromorphic motor mapper (as illustrated in FIG, 6A and/or 6B) to drive the postural component of motion, consistent with the cortical command.
  • the system can also he modified to allow human-in-the- ⁇ virtual testing of bio-integration issues associated with the prosthetic hardware.
  • oeuroioia ing equipment 802 (e.g., EEG, tMRI, etc.) attached to the subject is sent to a simulated prosthesis controller 804 (whereas the prosthesis controller 308 in FIG, 3 is controlling an actual piece of hardware) and a simulated sensorimotor controller 806.
  • the simulated prosthesis controller 804 models the behavior of controller designs (and operates in a manner similar to the prosthesis controller 308 described an illustrated with respect to FIG. 3), while the sensorimotor controller 806 simulates the sensorimotor processing and integration of the centra! nervous system (CMS).
  • CCS centra! nervous system
  • the simulated neural excitations 808 output from this controller 806 drive a set of muscles in a musculoskeletal simulation 810, while the simulated actuator torques 812 from the prosthesis controller 804 drive the simulated prosthetic device.
  • a single physics-based environment incorporates the musculoskeletal dynamics of the subject as well as the dynamics of the prosthetic device. This architecture allows a wealth of simulation-based testing to be performed.
  • the overall efficacy of the prosthesis controller 804 can be tested, including stability and
  • the architecture will, also facilitate subject training with the prosthetic system and subject-specific tuning and customization of the system prior to any physical coupling of the subject with the prosthetic hardware.
  • FIG. 9 depicts the neural and musculoskeletal physiology associated with the motor control problem.
  • High-level motor intent 900 from the brain 900 transmitted to the central nervous system (CNS) 904, Sensorimotor integration and control results in the low-level command of motor neurons 906 innervating individual, muscles 90S (in the somatic nervous system 909) to generate coordinated musculoskeletal 910 movement.
  • Afferent proprioceptive signals 9.12 transmit sensory data from mechanoreceptors in the joints and tendons back to the CNS 904.
  • FIG. 1 depicts a system-level abstraction of FIG . 9.
  • sensorimotor controller block 806 is central to the sensorimotor controller described herein. It receives task-level command input (i.e., motor intent 902) from the human operator (or brain 900) and generates low-level neural excitations 808 for musculoskeletal simulations 1004. This is a closed-loop control process where sensory data (via proprioceptive signals 1006) is fed back to the sensorimotor controller 806, which provides continual updates to the neural excitations 808 driving the musculoskeletal simulation 1.004.
  • the sensorimotor controller 806 (simulator) operates by running musculoskeletal simulations on a musculoskeletal plant.
  • the various aspects of the seiisiromotor controller S06 are described in further detail below with respect to elements 1300, 1600, and 1700 as illustrated in PIGs. 13, 1 , and 1 7, respectively.
  • FIG. 1 1 provides an illustration of an
  • FIG. 12 is an illustration depicting a
  • neuromuscular 1200 and musculoskeletal 1202 system feed-forward path.
  • neural excitations 808 provide input to the activation dynamics 1206 and output 1208 of the ac tivation dynamics 1206 provides input to the contraction dynamics 1210.
  • output 1212 of the contraction dynamics 1210 provides input to the musculoskeletal ⁇ 202 dynamics through the joint torques.
  • the musculoskeletal 1202 system dynamics are described by the following system of equations in configuration space.
  • the neuromuscular 1202 dynamics describe the behavior of a set of r musculotendon actuators spanning the musculoskeletal system. These actuators are modeled as Hill-type acti ve state force generating units, as described by De Sapio et al. (see Literature Reference No. 18). It is assumed that the vector of r musculotendon lengths, I, can be uniquely determined from the system configuration, q. That is, — (f), As a consequence of this assumption, differential variations in i are given by,
  • f T is the vector of r tendon forces.
  • the negative sign is due to the convention of taking contractile muscle forces as positive.
  • the .matrix of moment arms is denoted St. [000101] The behavior of the musculotendon actiiators (neurom oscular
  • Activation dynamics 1208 refers to the process of muscle activation in response to neural excitation. This process can be modeled by the following equation of state written in terms of the vector of r mu where % 3 ⁇ 4
  • the contraction dynamic s 1210 of the musculotendon. unit (illustrated as element 1 100 in FIG, 1 1) can be modeled as the lumped parameter system.
  • the lumped parameter system 1 102 describes the configuration of forces.
  • the relative angle associated with the muscle fibers 1 1 10, a. is referred to as the pennation angle 1 1 1 2
  • the muscie actuated task space equations of motion can be expressed as:
  • a set of muscle activations, « is determined, which minimizes e
  • a set of neural excitations., » is computed which causes the actual activations, «, from the forward neuromuscular simulation to track , as follows:
  • FIG. 13 is an illustration of a task- level sensorimotor controller 1300 according to the principles of the present invention, where motion commands are represented in. task space.
  • the sensorimotor controller BOO generates optimal neural excitation commands 808 that minimize muscle activation 1302, and generate motion consistent with the input motion commands.
  • holonomic constraints 1400 can be incorporated into the system.
  • FIG. 14 illustrates a iunctiona! block diagram showing an abstracted representation of neural and musculoskeletal physiology.
  • the sensorimoto controlier 806 in this aspect is a goal-oriented neiuomuscuiuar feedback controller that dri ves the constrained 1400 musculoskeletal simulation by specifying low-level neural 808 excitations that execute task-level commands. 0001.10]
  • the holonomic constraints, ( ) ⁇ ® can be incorporated into the musculoskeletal dynamics.
  • the configuration space dynamics was augmented with Lagrange multipliers, 1, ⁇ TM M (tj) q- b ⁇ q s q) + g ⁇ q)— ⁇ 3 ⁇ 4 where ⁇ * is the constraint Jacobiati.
  • FIG. 15 constrained musculoskeletal system is depicted in FIG. 15. Specifically,
  • FIG. 15 is an illustration depicting a neuromuscular and constrained musculoskeletal system (feed-forward path), showing the musculoskeletal system dynamics 1500 augmented with a set of holonomic constraints 1502 and Lagrange multipliers 1504.,
  • the controller 1600 seeks a 1602, which minimizes
  • constrained d n m cs can be represented in task space as
  • FIG, 17 is an illustration of a constrained task-level sensorimotor controller 1700 based on a reformulation of the computed muscle control approach.
  • the system represents motion commands in task space and generates optimal neural excitation 808 commands that minimize muscle activation 1702. and aenerates motion consistent with the input motion commands in the presence of holonomic system constraints.

Landscapes

  • Health & Medical Sciences (AREA)
  • Cardiology (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Transplantation (AREA)
  • Engineering & Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Vascular Medicine (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Animal Behavior & Ethology (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Prostheses (AREA)

Abstract

Described is a system for controlling a torque controlled prosthetic device given motor intent inferred from neuroimaging data. The system includes at least one torque controlled prosthetic device operably connected with one or more processors. Further, the system is configured to receive neuroimaging data of a user from a neuroimaging device and decode the neuroimaging data to infer spatial motion intent of the user, where the spatial motion intent includes desired motion commands of the torque controlled prosthetic device represented in a coordinate system. The system thereafter executes, with a prosthesis controller, the motion commands as torque commands to cause the torque controlled prosthetic device to move according to the spatial motion intent of the user.

Description

SYSTEM FOR CONTROLLING BRAIN MACHINE INTERFACES AND NEURAL PROSTHETIC SYSTEMS
[0002] CROSS-REFERENCE TO RELATED APPLICATIONS
[0003] This is a non-provisional patent application of U.S. Provisional
Application No. 61/903,538, filed November 13, 2013, entitled, "A control and simulation architecture for application to brain machine interfaces and neural prosthetic systems." [0004] This is ALSO a non-provisional patent application of U.S. Provisional Application No.6.1 /903,526, filed November 13, 2013, entitled, "A goal- oriented sensorimotor controller for controlling musculoskeletal
simulations with neural excitation commands."
[0005] BACKGROUND OF THE INVENTION
[0006] ( 1 ) T ieid of invent! on
[0007] The present invention relates to a robotic control system and, more particularly, to a system for controlling robotic prosthetic devices given motor intent inferred from neuroimatiinti data.
[0008] (2) Description of Related Art
[0009] Brain machine interfaces (BMI) and neural prosthetics offer great hope for restoring function to people with spinal cord injuries and amputees, as well augmenting and enhancing the abilities of people with full motor function , There have been a number of advances in neural imaging as well as in decoding motor intent from neuroimagmg data. However, with regard to prosthetic device control there have been fewer advances, with most approaches rooted in conventional robotic control. [00 10] There is a body of work in BMI and neural prosthetics research which addresses the decoding of cortical signals for the downsiream execution of motion commands by an externa! device (see, for example, the List of Incorporated Literature References, Literature Reference Nos. I, 9, .1 15 1.2, and 15). Most of this work focuses on the neural decoding with less rigor applied to the prosthesis control hi cases where a robotic arm prosthesis is control led, the motion trajectory is typically converted into joint commands and executed using joint space control (see, for example.
Literature Reference Nos, ϊ I , ! 2, and 15),
.1] To date, no known prosthetic control system has been proposed which uses a task/posture decomposition. Task/posture decomposition has particular advantages over joint space control in B Is. It is based on an abstraction analogous to the manner in which cortical signals are encoded (e.g., eye-centered, hand-centered Cartesian coordinates). Further, it is generali abie to whole body prosthetic control involving multiple prosthetic limbs and .high degree-of-iteedom kinematics. Joint space prosthetic control is limited in this capacity due to its reliance on inverse kinematics solutions. Additionally, task/posture decomposition allows for the specification of postural behaviors based on minimizing important objective functions (e.g. power consumption, virtual muscle effort, etc.) consistent with the execution of cortical motion commands.
[00012] There has been some work; on applying neiuomorphie computing
architectures to BMls and neural prosthetic systems. For example, the work of Dethier et ai, focused on the use of artificial spiking neural networks for decoding and filtering cortical signals {(Caiman filter based decoding using endpomt kinematics as the state vector and neural spike rates as the measurement vector) rather than prosthetic control (see Literature Reference No. 7), [00013] Additionally, the work of Bougaais and Shanahan wa directed to controlling a robotic arm using artificial spiking neural networks; although their work was not applied to a neural prosthetic system (see Literature Reference No. 2).
[00014] Other researchers performed work in the simulation of neural
prosthetic limbs in conjunction with biomechanical models (see Literature
Reference Nos. 4, 5, and 8). However, while their work implements musculoskeletal models into the simulation, the locus of these efforts is on simulating and validating the control of the prosthetic device. Thus, again, there is no known system that integrates a sensorimotor controller, in addition to the prosthetic controller.
15] Therefore, a continuing need exists for a system thai allows enhanced model ing and analysis of the interface issues associated, not just, wi th the biological and engineered components., but also with the biological and engineered control systems. Further, a need exists for a system-level architecture for applying artificial spiking neural networks to the learning of motor maps, consistent with postural criteria, for controlling motion in BM I and neural prosthetic systems. [0001.6] SUMMARY OF INVENTION'
[00017] Described is a system for controlling a torque controlled prosthetic device given motor intent inferred from neoroiniagin data. The system includes one or more processors and a memory, the memory having executable instructions encoded thereon, such that upon execution of the instructions, the one or more processors perform several operations, such as recei ving neuroimaging data of a user from a neuroimaging device. The neuroimaging data is then decoded to infer spatial motion intent of the user, where the spatial motion intent includes desired motion commands of the torque controlled prosthetic device represented in a coordinate system. Thereafter, the system executes, with a prosthesis controller, the motion commands as torque commands to cause the torque controlled prosthetic device to move according to the spatial motion intent of the user.
[00018] In another aspect; the system includes at least one torque controlled prosthetic device operabiy connected with the one or more processors.
[0001 ] in yet another aspect, the system performs an operation of receiving, in the controller, sensory information regarding a current state of the prosthetic device,
[00020] Additionally, in executing the motion commands, the motion
commands are executed using a task decomposition and posture decomposition, wherein the task decomposition is a task space control and the posture decomposition is formulated as a cost potential which represents a cost, function.
[0002.1 ] Further, i executing the motion commands, the motion commands are executed as torque commands that generate a desired, task space control while minimizing the cost potential.
[00022] in yet another aspect, in executing the motion commands, the motio commands are executing using spiking neural network.
[00023] In another aspect, the prosthesis controller is a neuromorphic
prosthesis controller and further comprises: a neuromorphic spike encoder to represent the motion command as a set of neural spikes; a neuromorphic motor mapper to map the neural spikes representing Cartesian
displacements to neural spikes representing configuration (joint) space di placements; a spike decoder to decode the neural spikes representing configuration (joint) space displacements and generate a joint space command; and a joint servo to execute the joint, space command. [00024] In yet another aspect, the system performs operations of receiving a model of the prosthetic device and a musculoskeletal mode! of the user, wherein the musculoskeletal model includes musculoskeletal dynamics that include steadv state tendon forces; eeneratina, with a sensorimotor controller, simulated neural excitations given the motion commands to drive a set of muscle activations in a musculoskeletal simulation; and generating, with the prosthesis controller,, simulated actuator joint torques given the motion commands to drive a simulated prosthetic device.
[00025] Finally, the present invention also includes a computer program
product and a computer implemented method. The computer program product includes computer-readable instructions stored on a non-transitory computer-readable medium that are executable b a computer having one or more processors, such that upon execution of the instructions, the one or more processors perform the operations listed herein. Alternatively, the computer implemented method includes an act of causing a computer to execute such instructions and perform the resulting operations. [00026] BRIEF DESCRIPTIO OF THE DRAWINGS
[00027] The objects, features and advantages of the present invention will be apparent from the following detailed, descriptions of the various aspects of the invention in conjunctio with reference to the following drawings, where:
[00028] FIG. t is a block diagram depicting the components of system
according to the principles of the present invention;
[00029] FIG. 2 is an illustration of a computer program product embodying an aspect of the present invention; [00030] FIG. 3 is a process flow-chart illustrating how high-level motor intent from the brain is decocted and a task-level .motion command is
transmitted to the prosthetic device controller, which in turn executes the motion command using a task/posture decomposition;
[0003 1 ] FIG. 4A is a simplified model of a human arm actuated by 14 muscles;
[00032] FIG, 4B s a table illustrating corresponding maximum isometric
forces, . , and optimal fiber lengths, tMn of the muscles illustrated in. FiG. 4A;
[00033] FIG. 5 A is an il lustration of a redundant muscle-actuated model of the human ann used as a physiological template for controlling a prosthetic arm;
[00034] FIG. 5B is a graph illustrating time histories of joint motion, while performing a simulation run of the model as illustrated i FIG. 5 A;
[00035] FIG. 5C is a graph illustrating time histories of hand motion while performing a simulation run of the mode l as illustrated in FIG. 5A;
[00036] FIG. 5D is a graph illustrating time histories of muscle effort while performing a simulation run of the model as illustrated in FIG. 5 A; [00037] FIG. 6 A is a flow chart illustrating an artificial spiking neural network where motor map redundancy is resolved by training the network only on unique solutions of the inverse mapping given by Δ¾
[00038] FIG. 6B is a flow chart illustrating an artificial spiking neural network where motor map redundancy is resolved by explicitly inhibiting the learning of solutions where the cost is high; [00039] FIG. 7 is flow chart illustrating how high-level motor intent from the brai is decoded and a task-level motion command is transmitted, to the prosthetic device controller;
[00040] FIG. 8 is a flow chart illustrating how a decoded cortical output is sent to a simulated prosthesis controller and a simulated sensorimotor controller; [00041 ] FIG. 9 is a flow chart illustrating the neural and musculoskeletal
physiology associated with the motor control problem;
[00042] FIG. 10 is a functional block diagram depicting a system-level
abstraction of FIG. 9, showing an abstracted representation of neural and musculoskeletal physiology;
[00043] FIG. 1 1 is an illustration depicting an active state museulotendon
model, where the active contractile element and passive viscoelastic element are i paral lel and the passive elastic tendon element is in series;
[00044] FIG. 12 is an illustration depicting a neuromuscular and
musculoskeletal system (feed-forward path};
[00045] FIG. 13 is an illustration of a task-level sensorimotor controller;
[00046] FIG. 14 is a functional block diagram depicting an abstracted
representation of neural and musculoskeletal physiology;
[00047] FIG. 15 is an illustration depicting a neuromuscular and
musculoskeletal system (feed-forward path), showing the musculoskeletai system dynamics augmented with a set ofholonomic constraints and Lagrange multipliers;
[00048] FIG , 16 is an illustration of a constrained task-level sensorimotor
controller; and
[00049] F G. 17 is an illustration of a, constrained task-level sensorimotor
controller, which generates motion consistent, with the input motion commands in the presence of olonomic system constraints.
[00050] DETAILED DESCRIPTION
[00051] The present invention relates to a robotic control system and, more particularly, to a system for controlling robotic prosthetic devices given motor intent inferred from neuioimaging data. The following descriptio is presented to enable one of ordinary skill in the art to make and use the invention and to incorporate it in the context of particular applications. Various modifications, as well as a variety of uses in. different applications will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to a wide range of aspects. Thus, the present invention is not intended to be limited to the aspects presented, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
[00052] In the following detailed description, numerous specific details are set forth in order to provide a more thorough understanding of the presen t invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without necessarily being .limited to these specific details. In other instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the present invention. [00053] The reader's attention, is directed to all papers and documents winch are filed concurrently with this specification and which are open to public inspection with th is specification, and the contents of all such papers and documents are incorporated herein by reference. All the features disclosed in this specification, (including any accompanying claims, abstract, and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic series of equivalent or similar features,
[00054] Furthermore, any element in a claim that, does not explicitly state
"means for" performing a specified function, or "step for" performing a specific function, is not to be interpreted as a "means" or "step" clause as specified in 35 IJ.S.C. Section 1 12, Paragraph 6. In particular, the use of "step of or "act of in the claims herein is not intended to invoke the provisions of 35 IJ.S.C. 1 12, Paragraph 6.
[00055] Before describing the invention in detail, first a list of incorporated literature references is provided. Next, a description of various principal aspects of the present invention is provided. Subsequently, an
introduction provides the reader with a general understanding of the present in vention. Finally, details of the present invention are provided to give an understanding of the specific aspects. [00056] (i) List of ncorporated Literature References
[00057] The following references are cited throughout this application. For clarity and convenience, the references are listed herein as a central resource for the reader. The following references are hereby incorporated by reference as though fully included herein. The references are cited in the application by referring to the corresponding literature reference number. Andersen, R, A., Hwang, E. J., & Muiltken, G. I I ... (2010),
Cognitive neural prosthetics. Annual review of psychology, 61, 169.
Bouganis, A., & Shanahan, M., (2010, July), Training a spiking neural network to control a 4-DoF robotic arm based on spike timing-dependent plasticity. In Neural Networks (UCN ), The 2010 international Joint Conference on (pp. 1-8), IEEE.
Buneo, (1 A., Jarvis, M. R., Batista, A. P., & Andersen, R. A., (2002), Direct visuomotor transformati ns for reaching. Nature, 416(6881 ), 632-636.
Davoodi, R„ & Loeb, G. E., (201 1, February), MSMS software for VR simulations of neural prostheses and pati ent train ing and rehabilitation, I MMVR (pp. 156- 162).
Davoodi, R., Uratas C, Hauschild, M, Khachani, M., & Loeb, G. E„, (2007), Model-based development of neural prostheses for .movement. Biomedical Engineering, IEEE Transactions on, 54(1 1 ), 1909-1918.
De Sapio, V., Khatib, O., & Delp, S., (2008), Least action principles and their application to constrained and task-level problems in robotics and biomechanics, Multibody System
Dynamics, 19(3), 303-322,
Dethier J., Nuyujukian, P., Eliastrrsth, C, Stewart, T„ C, Elasaad, S. A,, Sbenoy, K, V., & Boahen, K. A,, (2011), A brain-machine interface operating with a real-time spiking neural network control algorithm, in Advances in Neural information Processing Systems (pp. 2213-2221 ).
Hauscbiid, M., Davoodi, R,, Sc. Loeb, G. E., (2007), A virtual reality environment for designing and fitting neural prosthetic limbs. Neural Systems and Rehabilitation Engineering, IEEE
Transactions on, 15(1 ), 9-15, Hochberg, L. ., Bacher, D., Jarosiewicz, B,, Masse, N. Y., Simera!, J. D., Vogel, I, ... & Donoghue, J. P., (2012), Reach and grasp by people with tetraplegia using a neuraliy controlled robotic arm. Nature, 485(7398), 372-375.
Kiiatib, O,, (1 95), inertia! properties in robotic manipulation: An object-level framework, The International Journal of Robotics Research,. 14(1 ), 1 -36.
Lebedev, M. A., Tate, A. I, Hanson, T. L,, Li, Z,, O'Doherty, J. E.„ Winans, 1 A., ... & Nieolelis, M. A., (201 1), Futiiie developments in brain-machine interface research. Clinics, 66, 25-32.
Naidu, D. S., Chen, C. B., Perez, A., & Schoen, M. P, (2008, August), Control strategies for smart prosthetic hand technology: An overview. In Engineering in Medicine and Biology Society, EMBS 2008. 30th Annual International Conference of the IEEE (pp. 4314-4317), IEEE.
Sabes, P. N., (2000), The planning and control of reaching
.movements. Current opinion in neurobiology, 10(6), 740-746. Srinivasa, M., & Cho, Y., (2012), Self-organizing spiking nearal model for learning fault-tolerant spatio-motor transformations. Neural Networks and Learning Systems, IEEE Transactions on, 23(10), 1526-1538.
Veiliste, M., Perel S., Spalding, M. C, Whittord, A. S., &
Schwartz, A. B.t (2008), Cortical control of a prosthetic arm for self-feeding, Nature, 453(7198), 1098- 1 101.
De Sapio, V. (201 1 ), Task-level control of motion and constraint forces in holonomicaJly constrained robotic systems, in
Proceedings of the 18th World Congress of the International.
Federation of Automatic Control.
De Sapio, V. and Park, j . (2010), Multitask constrained motion control using a mass- weighted orthogonal decomposition, Journal of Applied. Mechanics, Vol . 77, No. 4, p.0 1004, 18. De Sapio, V., Khatib, O, and Delp, S. (2005), Simulating die task- level control of human motion: a methodology and framework for implementation, The Visual Computer, Vol 21, No. 5, pp.28 - 302.
19. Thelen, D.G. and Anderson, F.C. (2006), Using computed muscle control to generate forward dynamic simulations of human walking from experimental data. Journal of Biomechanics, Vol . 39, No. 6, p.1 107.
20. Thelen, D.G., Anderson, F.C. and Delp, S.L, (2003), Generating dynamic simulations of movement using computed muscle control, journal of Biomechanics, Vol . 36, No. 3, pp.321-328.
21. De Sapio, V., Khatib, O., and Delp, S. (2006), Task-level approaches for the con trol of constrained multibody systems,
Muhibody System Dynamics, Vol. 16, No. 1 , pp. 73-102,
[00058] (2) Principal Aspects
[00059] The present invention has three "principal" aspects. The first is a
system for controlling robotic prosthetic devices gi en motor intent inferred from neuroimaging data. The system is typically in the form of a computer system operating software or in the form of a "hard-coded" instruction set. This system may be incorporated into a wide variety of devices that provide different functionalities. In one aspect, the system is incorporated into a robot having actuators and appendages or other motion operable components and any other components as may be required to provide the functionality as described herein. The second principal aspect is a method, typically in the form of software, operated using a data processing system (computer). The third principal aspect is a computer program product. The computer program product generally represents computer-readable instructions stored on a non-transitory computer- readable medium such as an optical storage device, e.g., a compact disc (CD) or digital versatile disc (DVD), or a magnetic storage device such as a floppy disk or magnetic tape. Other, non-limiting examples of
computer-readable media include hard disks, read-only memory (ROM), and flash-type memories. These aspects will be described in more detail below.
[00060] A block diagram depicting an example of a system (i.e., computer system 100) of the present invention is provided in FIG. I . The computer system 100 is configured - to perform calculations, processes, operations, and or functions associated with a program or algorithm. In one aspect, certain processes and steps discussed herein are realized as a series of instructions (e.g., software program) that reside within computer readable memory units and are executed by one or more processors of the computer system 100. When executed, the instructions cause the computer system 100 to perform specific actions and exhibit specific behavior, such as described herein.
[00061 ] The computer system 1 0 may include an address/data bus 102 that is configured to communicate information. Additionally, one or more data processing units, such as a processor 104 (or processors), are coupled with the address/data bus 10:2. The processor 104 is configured to process information and instructions. In an aspect, the processor 104 is a
microprocessor. Alternatively, the processor 104 may be a different type of processor such as a parallel processor, or a field programmable gate array.
[00062] The computer system 100 is configured to utilize one or more data storage units. The computer system 100 may include a volatile memory unit 106 (e.g., random access memory ("RAM"), static RAM, dynamic RAM, etc.) coupled with the address data bus 102, wherein a volatile memory unit 106 is configured t store information and instructions for the processor 104, The computer system 1 0 further may include a non- volatile memory unit 108 (e.g., read-only memory ("ROM"),
programmable ROM ("PROM"), erasable programmable ROM
("EPROM"},, electrically erasable programmable ROM "EEPROM"), flash memory, etc.) coupled with the address/data bus 102. wherein the nonvolatile memory unit 108 is configured to store static information and instructions for die processor 104. Alternatively, the computer system 100 may execute instructions retrieved from an online data storage unit such as in "Cloud" computing, in an aspect, the computer system 100 also may include one or more interfaces, such as an interface 110, coupled with the address/data bus 102, The one or more interfaces are configured to enable the computer system 100 to interface with other electronic devices and computer systems. The commu ication interfaces implemented by the one or more interfaces may include wireline (e.g., serial cables, modems, network adaptors, etc.) and/or wireless (e.g.. wireless modems, wireless network adaptors, etc.) communication technology. ] In one aspect, the computer system 100 may include an input device .1 12 coupled with the address/data bus 102, wherein the input device 1 12 is configured to communicate information and command selections to the processor 100. In accordance with one aspect, the input device 1 12 is an alphanumeric input device, such as a keyboard, that may include alphanumeric and/or function keys. Alternatively, the input device 1.12 may be an input device other than an alphanumeric input device, in an aspect, the computer system 100 may include a cursor control device 1 14 coupled with the address/data bus 102, wherein the cursor control device 1 14 is configured to communicate user input information and/or command selections to the processor 100. In an aspect, the cursor control device .1 14 is implemented using a device such as a mouse, a track-ball, a track-pad, an optical tracking device, or a touch screen. The foregoing
notwithstanding, in an aspect, the cursor control device 1 14 is directed and/or activated via input from the input device 112, such as in response to the use of special keys and key sequence commands associated with the input device 1 12. in an alternative aspect, the cursor control device 1. 1 is configured to be directed or guided by voice commands.
[00064] In an aspect, the computer system 100 further may include one or more optional computer usable data storage devices, such as a storage device 116, coupled with the address/data bus 102. The storage device 1 16 is configured to store information and/or computer executable instructions, in one aspect, the storage device 1 16 is a storage device such as a magnetic or optical disk drive (e.g., hard disk drive {"HDD"), floppy diskette, compact disk read only memory ("CD-ROM"), digital versatile disk ("DVD")). Pursuant to one aspect, a display device 1 18 is coupled with the address/data bus 102, wherein the display device 118 is configured to display video and/or graphics. In an aspect, the display device 118 may include a cathode ray rube ("CRT"), liquid crystal display ("LCD"), field emission display ("FED"), plasma display, or any othe display device suitable for displaying video and/or graphic images and alphanumeric characters recognizable to a user. [00065] The computer system 100 presented herein is an example computing environment in accordance with an aspect. However, the non-limiting example of the computer system 100 is not strictly limited to being a computer system. For example, an aspect provides thai the computer system 100 represents a t pe of data processing analysis that may be used in accordance with various aspects described herein. Moreover, other computing systems may also be implemented, indeed, the spirit and scope of the present technology is not limited to any single data processing environment. Thus, in an aspect, one or more operations of various aspects of the present technology are control led or implemented using computer-executable instructions, such as program modules, being executed by a computer. In one implementation, such program modules include routines, programs, objects, components and/or data structures that are configured to perform particular tasks or implement particular abstract data types. In addition, an aspect provides that one or more aspects of the present technology are implemented by utilising one or more distributed computing environments, such as where tasks are performed by remote processing devices that are linked through a communications network, or such as where various program modules are located in both local and remote computer-storage medi including memory-storage devices. [00066] An illustrative diagram of a computer program product { i.e., storage device) embodying the present invention is depicted in FIG. 2. The computer program product is depicted as floppy disk 200 or an optical disk 202 such as a CD or DVD. However, as mentioned previously, the computer program product generally represents computer-readable instructions stored o any compatible non-transitory computer-readable medium. The term "instructions" as used with respect to this invention generally indicates a set of operations to be performed on a computer, and may represent pieces of a whole program or individual, separable, software modules. Non-limitin examples of "instruction" include computer program code (source or object code) and "hard -coded" electronics (i.e. computer operations coded into a computer chip). The "instruction" is stored on any non-transitory computer-readable medium, such as in the memory of a computer or on a floppy disk, a CD-ROM, and a flash drive. In either event, the instructions are encoded on a non- transitory computer-readable medium.
[00067] (3) Introduction
[00068] Brain machine interfaces (B I) and neural prosthetics offer great hope for restoring function to people with spinal cord injuries and amputees, as wel l augmenting and enhancing the abilities of people with full motor function. There ha ve been a number of advances in neural imaging as well as in decoding motor intent from neuroiinagiiig data. However, with regard to prosthetic device control there have been fewer advances, with most approaches rooted in conventional robotic control.
[00069] Thus, the system described herein provides a system-level architecture for controlling robotic prosthetic devices given motor intent inferred from neuroimaging data. The system also provides stimulation architecture to facilitate virtual testing of the specific controller and prosthetic device design, with, respect to the human subject, prior to any clinical
intervention. Two unique control architectures will he described. One controller is based on a bio-inspired task/posture decomposition of motion intended for impiementation on conventional digital computing hardware. The other controller is based on neuroraorphic mapping of task motion to joint motion with a bio-inspired resolution of kinematic redundancy., whic can he implemented on oeuromorphic computing hardware. A unique aspect is that both approaches utilize bio-inspired abstractions for controlling prosthetic devices in a natural human-like manner rather than utilizing conventional inverse kinematic approaches. Specific
instantiations of either of these control architectures can be tested with a b.um.an~.in~the oop using a novel model-based simulation environment, !n this environment actual cortical output from neoroimaging equipment (electroencephalography (BEG), functional magnetic resonance imaging (fMRT), etc.) attached to the subject will drive both a biomechanics! model of the subject and a model of the prosthetic device to be tested. This modeling provides a technique for testing the efficacy of the prosthetic device, the controller, and the overall bio-integration of the prosthetic hardware with the subject, without requiring the subject to be physically coupled with the actual prosthetic hardware.
[00070] The ability of the simulation environment to dri ve both the motion of the prosthetic device and the motion of the simulated subject through cortical commands (i.e., BMi) is unique and can be utilized by a variety of prosthetic manufacturers and/or developers. For example, human augmentation technologies can be implemented for enhancing the capabilities of the manufacturing floor worker. These technologies include assistive exoskeletons and other devices to increase worker productivity and reduce injury. The invention offers significant capabilities for the development of next generation BMls, such as user performance enhancement, assistive robotic devices, and neural prosthetic technologies.
[00071 ] (4) Specific Details
[00072] Described are two control architectures for controlling robotic
prosthetic devices given motor intent inferred from, neuroiinaguig data. Additionally, a simulation architecture is described for human- in-the-Ioop virtual resting of bio-integration, issues associated with the prosthetic hardware. Both control architectures start with the assumption of a source of decoded cortical data from the brain which represen ts high-level motor intent This .motor intent is in the form of desired motion commands represented in different Cartesian coordinate systems (eye-centered, hand- centered, etc.) using visumotor transformations. Examples of such.
visumotor transformations were describe in Literature Reference os. 3 and 13. This desired motion command is referred to herein as ¾ a multidimensional vector of task coord inates (e.g. Cartesian coordinates associated motion of a. single hand or coordinated motion of both hands). The goal is to control the prosthetic device based on this motor intent.
[00073] Before describing the first control architecture based on bio-inspired task/posture decomposition of motion, it is necessary to define the dynam ics of the prosthetic de vice. In joint space, the dynamics are defined as follows: ϊ— {q)q - b(qfq) 4- g(q\ where is the vector of joint coordinates, r, is the vector of applied joint torques, M{q), is the joint space mass matrix, b{qtq}, is the vector of centrifugal and CorioHs forces, and g{q) is the vector of gravity forces. inverse kinematics is conventionally used to map a desired Cartesian goal into a set of joint angles, {sf 1 (¾}|at→ q}. Subsequently, given the joint level commands the prosthetic device is controlled by servoing individual joints or by using computed torque control. Given an anthropomorphic prosthetic device designed to replace a human limb (e.g., the human arm), it is necessary to address kinematic redundanc in the limb motion. That is, the device will typically possess more joint coordinates than are required by the motion command. From a robotic control perspective this implies that there are an infinite number of joint coordinate solutions associated with a given motion command.
[00074] The approach to dealing with this issue is to .represent the dynamics of the prosthetic device in task space, a more natural space for addressing the control problem since the decoded corticai motion command is expressed in this space, A task space representation of the dynamics is given by, f ~ A($)x- p(q.,q) - p(q), where x is a description of task coordinates that are to be controlled, f are the control forces that act at the task, and .f* f X and .p{q\ are the task, space mass .matrix, centrifugal and Coriolis force vector, and gravity force vector, respectively. Task space was described by hatib in Literature Reference No. .10.
[00075] The applied joint torques are given by,
* =jrf + AT where # is the Jaeobian of the task coordinates, Nf is the mill space projection matrix, and τβ is an aribitrar joint torque filtered through the null space projection matrix. The key point to note here is that control of the prosthetic device can be decomposed into a task-level component and a complementary postural component (due to the kinematic redundancy). Based on the choice of a particular generalized inverse of the Jaeobian these tw components of motion control are guaranteed to be dynamically consistent with each other and, as such, control can be synthesized to execute the cortical motion command while simultaneously achieving som postural objective for the prosthetic device. This postural objective can be based on minimizing some physiological cost criterion or it can be based on minimizing a power consumption criterion for the device. In the former case, while the prosthetic device is not necessarily actuated in a manner analogous to the biological appendage it is replacing, encoding a physiological criterion into its control can result in more human-like behavior of the device's postural motion. This would achieve the desirable effect of the device moving more consistently with how the subject's own appendage would move. For further understanding. FIG. 3 depicts the system-level architecture associated with this approach. High-level motor intent from the brain 302 is decoded (with a cortical decoder 304) and a task-level motion command 306 is transmitted to the prosthetic device controller 308. High-level motor intent is captured using a neuroimaging device or any other suitable device operable for capturing neuroimaging data 300, non-limiting examples of which include neuroimaging equipment such as
electroencephalography (EEG), and functional magnetic resonance imaging ifMRi ). The cortical decoder 304 uses a decoding algorithm to infer spatial, motion intent 306 from the nemoiraadna data 300. As an example and as noted above, this motor intent is in the form of desired motion commands represented in different Cartesian coordinate systems (eye-centered, hand-centered, etc.) using visumoior transfonmtions.
[00077] The controller 308 executes the motion command using a task/posture decomposition. Relevant physiological criteri (i.e., postural criterion) or device power consumption criteria can be used to drive the postural component of motion 310, consistent with the conical command. Sensory information 312 (e.g., joint angles of the current state of the prosthetic- device 1 } from the prosthetic device 314 is fed back to the controller 308, resulting in closed loop control The architecture also accommodates sensory information 312 from the prosthetic device 314 being sent to the brain 302 to facilitate afferent proprioception.
[00078] In formulating a postural criterion, a potential. can be specified as representing some cost function. An example of suitable cost functions was described by De Sapio in Literature 'Reference No. 6. The task-level motion command can be controlled while simultaneously controlling motion in the complementary posture space to minimize This can be accomplished by applying gradient descent in the null/posture space. That is.
, . r oil
[00079] The task space control equations can then be expressed as follow-
where "^* represents estimates of the dynamic parameters and i* is the control law, A proportional-derivaiive (PD) control law can be expressed as: f* = . Γ_{χΛ— 3.) · ί..(ά ά · 3ί„> < where Κρ and Κν are control gains. The control torque, which generates the desired task space control while minimizing the instantaneous cost potential in the posture space, is then.
dq
[00080] A physiological cost criterion, like muscle effort, can he selectively represented. For example, De Sapio, et a!., (see Literature Reference No. 6), proposed the following muscle effort criterion related to minimizing muscle activation, where g{q) is the gravity torque, M{q) is the matrix of muscle moment arms, and IT/0?} is the diagonal matrix mapping muscle acti vation, «, to muscle force. . The elements of f for a given muscle can be modeled by, where <$) is the muscle length, and fa and ½ represent, the maximum isometric muscle force and the optimal muscle fiber length respectively. 0008 J ] This effort criterion can he used to design an arm prosthesis controller based on the architecture of FIG. 3, For illustrative purposes and as a non limiting example, the characteristics of the biomechanical model of FIG, 4A are employed into an arm prosthesis controller. FIG. 4A illustrates a simplified biomechanical model of the human arm actuated by 14 muscles, while IG. 4B is a table illustrating the maximum isometric forces. /s, and optimal fiber lengths, lMf> for the biomechanical model of
FIG. 4A. The control law is defined as,
£ * ~ k«{x4— x)— and the control torque is defined as, where a dissipative term, and a gain, are added on the gradient descent term in the posture space portion of the control torque.
[00082] FIG. 5A illustrates a redundant muscle-actuated model of the human arm used as a physiological template for controlling a prosthetic arm, while PIGs. SB, 5C and 5D are graphs illustrating time histories of joint motion, hand motion, and muscle effort, respecti vely, for a simulation run using the model of FIG. 5A. The model of FIG. 5 A depicts initial and final configurations (joint angles), q{ ) and - (if), associated with gradient descent movement to a target, xf. The motion corresponds to gradient descent of the rnuscie effort, subject to the task requirement. The posture control seeks to reduce the muscle effort but is also constrained by the task requirement.
[00083] The controller achieves the final target objective while the posture control simultaneously seeks to reduce the muscle effort (consisteni with the task requirement). No compensation for the dynamics (except for gravity) was included in the control. Thus, there is no feedback linearization of the inertia! terms present in the control . Normally, perfect feedback linearization would produce straight line motion to the goal, in the absence of feedback, linearization non-straight line motion results.
[00084] I t is noted that the biomechanical model of FIG. 4A is used to generate a physiological criterion that is encoded in the prosthesis controller. The ac tuation system of the prosthetic arm need not be analogous to its biological muscle-driver* counterpart Rather, the biological muscie model defines a set of virtual muscles that direc t the postural control of the prosthetic device in a bio-inspired manner, eve if conventional robot actuators (e.g., DC motors, etc. ) are used in the prosthetic device to control joint movements. [00085] The control architecture depicted in FIG, 3 is intended to be
implemented on conventional digital computing hardware and integrate with a prosthetic device if desired. An approach that would implement neuromorphic hardware is also described, below. This involves the design of an artificial spiking neural network to learn the mapping between task- level motion commands, derived from decoded cortical signals, and joint space commands. In the approach described above, kinematic redundancy in the prosthetic device was resol ved using a. task/posture decomposition, in the case of motor map learning by an arti ficial spiking neural network , kinematic redundancy also needs to be addressed since a differential movement in x would result in an infinite space of differential movements in q.
[00086] FIG. 6.A and 6B depict artificial spiking neural networks with two different approaches for resolving kinematic redundancy. Both networks learn the inverse kinematic mapping using trajectory babbling 600, 61 1 of the prosthetic arm. Kinematic redundancy impacts the trainin such that for a given configuration (set. of joint angles), q 603, and Cartesian displacement. Ax 601, there will be multiple va lues of configuration (joint) space displacement, A*| 604 to learn. In other words, the forward
mapping, ^x ~ cannot be uniquel inverted. As shown in FIG.
6A, this can be resolved by using a specific generalized inverse of / (i.e.,
/* 602), such that ~ f*&x* Given this unique mapping, the spiking neural network of FIG. 6A trains on the output of the trajectory babbling in Cartesian space 600. The signals 603 and Ax 601 serve as the context for the training and the signal &q 604 serves as the signal to be learned with respect, to that context. Signal 603 represents the current joint angle configuration that is being sensed at any given time. ] The spikes representing these signals act as inputs to layer 1 605 of spiki g neurons. The outputs of layer 1 605 act as spike inputs to layer 2 606. An additional layer 3 60? of neurons is employed on the context side (q 603 and Ax 601 input side). In this layer each neuron codes for a single combined representation of both the configuration, q 603, and Cartesian displacement. Ax (see Literature Reference No, 14). In layer 4 60S the synaptic weights of the spiking neurons in that layer are reinforced consistent with learning the training signals 609. Spike timing-dependent plasticity (see Literature Reference Nos. 2 and 14) is utilized to modulate the synaptic weights in the layers. After sufficient training on motor babbling data the spike timing-dependent plasticity can be turned, off as the neurons in layer 4 60? will have ieamed the differential, kinematic mapping. Note that x(t+ 1 ) and q(t+ 1 ) denote a new value for a next time step. ] The generalized inverse, * 602, can be chose based on the minimization of some quadratic Form. For example, to find a solution thai minimizes I f I3 , the following solution can be used: &q ^β4Άκ ^βτ ββτ)~1Αχ.
[00089] As another example, a solution that minimizes the instantaneous
kinetic energy of the system, qTMq/2, would be given by,
q ^ jAx « Μβτ(βΜ~τ}&α.
[00090] Alternatively, a solution that minimizes the instantaneous muscle fiber displacement rate, \\l\lz ~ qTR$kT would be given by,
Δ<? * (l?^)"1 ^ ! ^^}"1/7]" ^
[00091] As shown in FIG, (SB. the redundancy problem can also be resolved by adding a layer of neurons 612 that explicitly inhibits the learning of solutions where the cost (physiological or otherwise) is high. The structure of Fig. 6B is similar to Fig, 6A, However, rather than using a generalized inverse to compute a unique value of &q to team for a given configuration, <?, and Cartesian displacement, Δχ, forward kinematics 610 is used to compute the unique forward mapping from the configuration (joint) space motor babbling 61 1 to Cartesian space. While many values of&% will satisfy this mapping for a given value of q and Δ , a layer of neurons 612 is incorporated to inhibit the learning 613 f q signals with high cost values. In this way training will be directed at learning Aq values associated with minimal cost values.
[00092] Referring again to FIGs. 6 A and 6B, the spiking neural networks
operate as nearomorphic motor mappers (illustrated as element 708 in
FIG. 7) that map Cartesian displacement, &x 601, to configuration (joint) space displacement, &q. Utilizing this neuromorphic motor mapper 708 a feedback loop 700 is specified in FIG. 7 to control the prosthetic arm. The overall system consists of output from a conical decoder, which specifies intentional mo vement 306 of the arm in Cartesian space. This signal is encoded as spikes 706 b the aeuromorphic spike encoder 702. These spikes, as well, as joint angle measurements from the prosthetic arm 714 encoded as spikes 720, act as input to the neuromorphic motor mapper 708. A spiking output representing the learned mapping to the
configuration (joint) space displacement, Δ<?. is decoded 710 and sent to a joint servo 712, which controls the motion of the prosthetic ami 718.
[00093] For further understanding, FIG. 7 depicts the system-level architecture associated with the approach depicted in FIG. 6B. in this case, the prosthetic device controller (i.e., neuromorphic prosthesis controller) 308 includes a neuromorphic spike encoder 702 (to represent the motion command 306 as a set of spikes 706), a neuromorphic motor mapper 708 (as illustrated in FIG. 6A and/or 6B) to map the neural spikes representing Cartesian displacements to neural spikes representing configuration (joint) space displacements, a spike decoder 710 (to decode the neural spikes representing configuration (joint) space displacements) and generate joint space commands, and a join t servo 712 to execute the joint space
command. The sensed joint angles 71 (of the current state of the prosthetic device 718) are fed hack to the joint servo 712 and to the motor mapper 70S (after spike encoding 720). Relevant physiological criteria or device power consumption criteria can incorporated into the neuromorphic motor mapper (as illustrated in FIG, 6A and/or 6B) to drive the postural component of motion, consistent with the cortical command.
[00094] The system can also he modified to allow human-in-the-ίοορ virtual testing of bio-integration issues associated with the prosthetic hardware. FIG. 8, for example, depicts such an architecture thai includes human in- the-loop virtual testing. The decoded cortical output 306 from
oeuroioia ing equipment. 802 (e.g., EEG, tMRI, etc.) attached to the subject is sent to a simulated prosthesis controller 804 (whereas the prosthesis controller 308 in FIG, 3 is controlling an actual piece of hardware) and a simulated sensorimotor controller 806. The simulated prosthesis controller 804 models the behavior of controller designs (and operates in a manner similar to the prosthesis controller 308 described an illustrated with respect to FIG. 3), while the sensorimotor controller 806 simulates the sensorimotor processing and integration of the centra! nervous system (CMS).
[00095] The simulated neural excitations 808 output from this controller 806 drive a set of muscles in a musculoskeletal simulation 810, while the simulated actuator torques 812 from the prosthesis controller 804 drive the simulated prosthetic device. A single physics-based environment incorporates the musculoskeletal dynamics of the subject as well as the dynamics of the prosthetic device. This architecture allows a wealth of simulation-based testing to be performed. The overall efficacy of the prosthesis controller 804 can be tested, including stability and
performance of the control system in executing cortical commands. A number of bio-integration issues can also be evaluated. These include mechanical forces and stresses on the device and the subject in various configurations, as well as reaction forces at the body-prosthetic interfaces. In addition to facilitating design optimization of the control system and prosthetic device the architecture will, also facilitate subject training with the prosthetic system and subject-specific tuning and customization of the system prior to any physical coupling of the subject with the prosthetic hardware.
[00096] The sensorimotor controller 806 component is described in further detail below with respect to FIGs. 9 through 17. FIG. 9, for example, depicts the neural and musculoskeletal physiology associated with the motor control problem. High-level motor intent 900 from the brain 900 transmitted to the central nervous system (CNS) 904, Sensorimotor integration and control results in the low-level command of motor neurons 906 innervating individual, muscles 90S (in the somatic nervous system 909) to generate coordinated musculoskeletal 910 movement. Afferent proprioceptive signals 9.12 transmit sensory data from mechanoreceptors in the joints and tendons back to the CNS 904.
[00097] FIG. 1 depicts a system-level abstraction of FIG . 9. The
sensorimotor controller block 806 is central to the sensorimotor controller described herein. It receives task-level command input (i.e., motor intent 902) from the human operator (or brain 900) and generates low-level neural excitations 808 for musculoskeletal simulations 1004. This is a closed-loop control process where sensory data (via proprioceptive signals 1006) is fed back to the sensorimotor controller 806, which provides continual updates to the neural excitations 808 driving the musculoskeletal simulation 1.004. Thus, in one aspect, the sensorimotor controller 806 (simulator) operates by running musculoskeletal simulations on a musculoskeletal plant. The various aspects of the seiisiromotor controller S06 are described in further detail below with respect to elements 1300, 1600, and 1700 as illustrated in PIGs. 13, 1 , and 1 7, respectively.
[00098] For further understanding. FIG. 1 1 provides an illustration of an
example mu sculoskeletal plant i 100 according to the principles of the present invention. As shown in FIG, 12, the musculoskeletal plant can be divided in neuromuscular 1200 dynamics and musculoskeletal 1202 dynamics. Specifically, FIG. 12 is an illustration depicting a
neuromuscular 1200 and musculoskeletal 1202 system (feed-forward path). As shown, neural excitations 808 provide input to the activation dynamics 1206 and output 1208 of the ac tivation dynamics 1206 provides input to the contraction dynamics 1210. Further, output 1212 of the contraction dynamics 1210 provides input to the musculoskeletal Ϊ202 dynamics through the joint torques. [00099] Further details regarding the neuromuscular 1200 and musculoskeletal 1202 systems are provided be!ow. The musculoskeletal 1202 system dynamics are described by the following system of equations in configuration space. τ ~ M{q)q MqA) -h &(q% where q is the vector of n generalized or joint coordinates, r, is the vector o applied joint torques, M{q is the joint space mass matrix, b(qfq)s is the vector of centrifugal and Corio!is forces, and is the vector of sraviiv forces.
[000100] The neuromuscular 1202 dynamics describe the behavior of a set of r musculotendon actuators spanning the musculoskeletal system. These actuators are modeled as Hill-type acti ve state force generating units, as described by De Sapio et al. (see Literature Reference No. 18). it is assumed that the vector of r musculotendon lengths, I, can be uniquely determined from the system configuration, q. That is, — (f), As a consequence of this assumption, differential variations in i are given by,
SI ~ |·· Sq — l(q) q.
where l(q) is the muscle Jacobiaii. From the principle of virtual work, it is concluded that,
where fT is the vector of r tendon forces. The negative sign is due to the convention of taking contractile muscle forces as positive. The .matrix of moment arms is denoted St. [000101] The behavior of the musculotendon actiiators (neurom oscular
dynamics 1200) can be divided into activation dynamics .1208 and contraction dynamics 1210. Activation dynamics 1208 refers to the process of muscle activation in response to neural excitation.. This process can be modeled by the following equation of state written in terms of the vector of r mu where % ¾ |0,i] aod t½€ [041 is the neural input. The term
time constant given bv.
(ft. — 4* 'ί-' for Us≥ it:
for ¾i< where ¾ and ¾ are the. activation and deactivation ti me constants, respectively.
[000102] The contraction dynamic s 1210 of the musculotendon. unit (illustrated as element 1 100 in FIG, 1 1) can be modeled as the lumped parameter system. Referring again to FIG. 1 1, the lumped parameter system 1 102 describes the configuration of forces. There is an acti ve element 1 104, a passive viscoelastic element, (in parallel) 1.106, and an elastic tendon element (in series) 1 108. The relative angle associated with the muscle fibers 1 1 10, a. is referred to as the pennation angle 1 1 1 2
[000103] This yields the following relations,
where ^· is the vector of muscle lengths and is the vector of tendon lengths. The following force equilibrium equations can be expressed, where fA is the vector of active forces in the muscles and fp is the vector of passive forces in the muscles. Using this force equilibrium equation as well as cooslitutive relationships describing muscle forces as a function of muscle length and contraction velocity, l > a) and ρ(ί » I^), and tendon force as a function of tendon length, T ly), the following equation of state in functional form can be expressed as,
l$$ ~ ¾ Q d* ¾A )>
[000104] So, for a system of r musculotendon actuators, the following system of
2r first-order state equations can be expressed as, a ~ a{uf a) ~ {« - a), and,
where the internal states are ½ and . This is complemented by the force relationship,
fr - fr( ) ~ ~ ½cos a - T{¾),½).
[000105] Returning to the musculoskeletal dynamics (depicted as element 1202 in FIG. 12),
τ = M{q)q - k{q,q) - g(q\ these equations of motion can be mapped into operational or task space, as follows: where | is the dynamically consistent inverse of the task Jacobian, x is a description of task coordinates that are to be controlled, f, are the control forces that act at the task, and {$), μ^,ό.), and ρ{ Χ are the task space mass matrix, centrifugal and Coriolis force vector, and gravity force vector, respectively. See the work o Khatib in Literature Reference No. 10 for a discussion of task space, which is hereby incorporated b reference as through fully set forth herein. [000106] Given a system of r musculotendon actuators, the applied joint torques are given by, which provides the following muscle actuated task, space equations of motion,
T fT™ Ax μ 1- p.
[000107] For control purposes, the muscie actuated task space equations of motion can be expressed as:
f8fT ~ Xf* p 1- p, where T represents estimates of the dynamic parameters and fs is the control Saw. Since the term f7RfT is underdetermined it offers many solutions for/y. These solutions correspond to the kinematic redundancy associated with the task and the actuator redundancy associated with the system of musculotendon force actuators, hi the computed muscle control approach of Thelen et al., (see Literature Reference Nos. 1.9 and 20), there was only muscle redundancy that needed to be resolved. This was achieved through a static optimization step. In the task-level formulation as used herein, a static optimization is also specified to resolve both muscle redundancy and kinematic redundancy. Specifically, a set of muscle activations, «, is determined, which minimizes e||2, where a are muscle activations associated with the steady state t ndon forces fT(&) (i.e.. tendon forces at. a time when the contraction, dynamics have equilibrated and the transients have decayed). Further these activations must generate steady state tendon forces that satisfy the task motion requirements, f ria} - If fi + . ] Using a linear proportional controller, a set of neural excitations., », is computed which causes the actual activations, «, from the forward neuromuscular simulation to track , as follows:
« = e } — a), where ΚΛ is the gain. This task-level input computed muscle control architecture is depicted in FK1. 13. FIG. 13 is an illustration of a task- level sensorimotor controller 1300 according to the principles of the present invention, where motion commands are represented in. task space. The sensorimotor controller BOO generates optimal neural excitation commands 808 that minimize muscle activation 1302, and generate motion consistent with the input motion commands. ] Further and as shown in FIG. 14, holonomic constraints 1400 can be incorporated into the system. Specifically, FIG. 14 illustrates a iunctiona! block diagram showing an abstracted representation of neural and musculoskeletal physiology. The sensorimoto controlier 806 in this aspect is a goal-oriented neiuomuscuiuar feedback controller that dri ves the constrained 1400 musculoskeletal simulation by specifying low-level neural 808 excitations that execute task-level commands. 0001.10] Thus and as shown in FIG. 15, the holonomic constraints, ( ) ~ ®, can be incorporated into the musculoskeletal dynamics. "The configuration space dynamics was augmented with Lagrange multipliers, 1, τ M (tj) q- b {qs q) + g {q)— Φ ¾ where Φ* is the constraint Jacobiati. The block diagram of the
constrained musculoskeletal system is depicted in FIG. 15. Specifically,
FIG. 15 is an illustration depicting a neuromuscular and constrained musculoskeletal system (feed-forward path), showing the musculoskeletal system dynamics 1500 augmented with a set of holonomic constraints 1502 and Lagrange multipliers 1504.,
[0001 11 j The approach expresses these dynamics in one of two ways. First and as shown in FIG. 16, the constrained dynamics can be represented in task space as:
Jr8r r ~ A{q" L - q) - p{q}— Τ Τ (a - p) where β* is the constraint null space projection matrix. Task space representations were described by De Sapio et al. in Literature Reference No. 21. The centrifugal and Corioiis forces, and gravity forces projected at the constraints are fif and p respectively. For neuromuscular control purposes, this can be expressed as the following muscle actuated task space equations of motion,
JT TBfT— Af * 4- β - p— |ΓΦτ(έ¾4- ),
[000112] As before, the controller 1600 seeks a 1602, which minimizes
(i.e.. magnitude squared of the activations) to generate the neural excitations 808. Further these activations must generate steady state tendon forces that satisfy the constrained task motion requirements, as follows:
1 13] Alternately and as shown in FIG, 7, constrained d n m cs can be represented in task space as
z -~ #T|r(A ir- με - p ) + Γ(ίϊ - p) - £if %— ΦΓ1, where ®* is the constraint null space projection matrix, Ae.^c pCi are the task/constraint space mass matrix, centrifugal and Coriolis force vector, and gravity force vector, respectively. Task space representations were further described in Literature Reference Nos. 1 and 17. The centrifugal and Coriolis forces, and gravity forces projected at the constraints are a and p respectively. Finally, 8« is the combined null space projection matrix, with respect to both task and constraints, and % is the control vector for the null space.
[0001 14] This representation expose the constraint forces and is useful for generating optimal neural excitations that track task motio under constraints and also minimizes specific internal joint reaction forces. For neuromuscular control purposes, this can be expressed as the following muscle actuated task space equations of motion,
MfT - Φτ1 - &l % - ®T (K * l ¾ i ¾) - #r(St [0001 15] Thus, the system now seeks H and which minimize
wa||als - ¾¾|2|2, Further these activations must generate steady state tendon forces onstrained task motion requirements. ] Thus, FIG, 17 is an illustration of a constrained task-level sensorimotor controller 1700 based on a reformulation of the computed muscle control approach. The system represents motion commands in task space and generates optimal neural excitation 808 commands that minimize muscle activation 1702. and aenerates motion consistent with the input motion commands in the presence of holonomic system constraints.

Claims

What is claimed is:
1. A system for controlling a torque controlled prosthetic device given motor intent inferred from neuroimaging data, the system comprising:
one or more processors and a memory, the memory having executable instructions encoded thereon, such that upon execution of the instructions, the one or more processors perform operations of:
receiving neuroimaging data of a user from a neuroimaging device;
decoding the neuroimaging data to infer spatial motion intent of the user, where the spatial motion intent includes desired motion commands of the torque controlled prosthetic device represented in a coordinate system; and
executing, with a prosthesis controller, the motion
commands as torque commands to cause the torque controlled prosthetic device to move according to the spatial motion intent of the user.
2. The system as set forth in Claim 1 , further comprising at least one torque controlled prosthetic device operably connected with the one or more processors.
3. The system as set fort in Claim 2, further comprising an operation of receiving, in the controller, sensory information regarding a current state of the prosthetic device.
4. The system, as set forth in Claim 3, wherein in executing the motion
commands, the motion commands are executed using a task
decomposition and posture decomposition, wherein the task
decomposition is a task space control and the posture decomposition is formulated as cost potential which represents a cost function.
5. The system as set forth in Claim 4, wherein in executing the motion commands, the motion commands are executed as torque commands that generate a desired task space control while minimizing the cost potential.
6. The system as set forth in Claim 3, wherein in executing the motion
commands, the motion commands are executin using a spiking neural network.
The system as set forth in Claim 6, wherein the prosthesis controller is a neuromorphie prosthesis controller and further comprises:
a neuromorphie spike encoder to represent the motion command as a set of neural spikes:
a neuromorphie motor mapper to map the neural spikes representing Cartesian displacements to neural spikes representing configuration (joint) space displacements;
a spike decoder to decode the neural spikes representing configuration (joint) space displacements and. generate a joint space command: and
a joint servo to execute the joint space command.
The system as set forth in Claim 1, further comprising operations of. receiving a model of the prosthetic device and a musculoskeletal model of the user, wherein the musculoskeletal model includes
musculoskeletal dynamics that include steady state tendon forces;
generating, with a sensorimotor controller, simulated neural excitations given the motion commands to drive a set of muscle activations in a musculoskeletal simulation; and
generating, with the prosthesis controller, simulated actuator joint torques given the motion commands to drive a simulated prosthetic device. A computer program product for controlling a torque controlled prosthetic device given motor intent inferred from neuroimaging data, the computer program product comprising:
a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions by one or more processors, the one or more processors perform operations of:
receiving neuroimaging data of a user from a neuroimaging device;
decoding the neuroimagmg data to infer spatial motion intent of the user, where the spatial motion intent includes desired motion commands of the torque controlled prosthetic device represented in a coordinate system; and
executing, with a prosthesis controller, the motion commands as torque commands to cause the torque controlled prosthetic device to move according to the spatial motion intent of , The computer program product as set forth, in Claim 9, further comprising instructions for controlling at least one torque controlled prosthetic device that is operahly connected with the one or more processors. , The computer program product as set forth in Claim 10, further comprising an operation of receiving, in the controller, sensory information regarding a current state of the prosthetic device. , The computer program product as set forth in Claim i 1 , wherein in executing the motion commands, the motion commands are executed using a task decomposition and posture decomposition, wherein the task decomposition is a task space control and the posture decomposition is formulated as a cost potential which represents a cost function.
13. The computer program product as set forth in Claim 12, wherein in
executing the motion commands, the motion commands are executed as torque commands thai generate a desired task space control while minimizing the cost potential.
14. The computer program product as set forth in Claim 1 1 , wherein in
executing the motion commands, the motion commands are executing using a spiking neural network.
15. The computer program product as set forth hi Claim 14, wherei the
prosthesis controller is a neuromorphic prosthesis controller and further comprises:
a neuromorphk spike encoder to represent the motion command as a set of neural spikes;
a neuromorphic motor mapper to map the neural spikes representing Cartesian displacements to neural spikes representing configur tion {joint} space displacements;
a spike decoder to decode the neural spikes representing configuration, (joint) space displacements and generate a joint space command; and
a joint servo to execute the joint space command.
16. The computer program product as set forth in Claim 9, further comprising operations of:
receiving a model of the prosthetic device and a musculoskeletal model of the user, wherein the musculoskeletal model includes musculoskeletal dynamics that include steady state tendon forces; generating, with a sensorimotor controller, simulated neural, excitations given the .motion commands to drive a set of muscle
activations in a musculoskeletal simulation; and
generating, with the prosthesis controller, simulated actuator joint torques given the motion commands to drive a simulated prosthetic device.
17. A method for controlling a torque controlled prosthetic device given motor intent inferred from neuroimaging data, the method comprising an act of executing instructions encoded on a memory, such that upon execution of the instructions, one or more processors perform operations of:
receiving neuroimaging data of a user from a neoroimaging device;
decoding the neuroimaging data to infe spatial motion intent of the user, where the spatial motion intent includes desired motion commands of the torque controlled prosthetic device represented in a coordinate system; and
executing, with a prosthesis controller, the motion commands as torque commands to cause the torque controlled prosthetic device to move according to the spatial motion intent of the user,
1 S, The method as set. forth in Claim 17, further comprising an act of
controlling at least one torque controlled prosthetic device that is operably connected with the one or more processors.
19, The method as set forth in Claim IS, further comprising an operation of receiving, in the controller, sensory information regarding a current state of the prosthetic device.
20. The method as set forth hi Claim 1 , wherein in executing the motion commands, the motion commands are executed using a task
decomposition and posture decomposition, wherein the task
decomposition is a task space control and the posture decomposition is formulated as a cost potential which represents a cost function.
21. The method as set forth in Claim 20, wherein in executing the motion commands, the motion commands are executed as torque commands that generate a desired task space control while minimizing the cost potential.
22. The method as set forth in Claim 1 , wherein in executing the motion commands, the motion commands are executing using a spiking neural network.
23. The method as set forth in Claim 22, wherein the prosthesis controller is a neuromorphic prosthesis controller and further comprises:
a neuromorphic spike encoder to represent the motion command as a set of neural spikes;
a neuroiTiorphic motor mapper to map the neural spikes representing Cartesian, displacements to neural spikes representing configuration (joint) space displacements;
a spike decoder to decode the neural spikes representing configuration (joint ) space displacements and generate a joint space command; and
a joint servo to execute the joint space command.
24. The method as set forth in Claim 17, further comprising operations of:
receiving a model of the prosthetic device and a musculoskeletal model of the user, wherein the musculoskeletal model includes musculoskeletal dynamics that include steady state tendon forces; generating, with a sensorimotor controller, simulated neural, excitations given the .motion commands to drive a set of muscle activations in a musculoskeletal simulation; and
generating, with the prosthesis controller, simulated actuator joint torques given the motion commands to drive a simulated prosthetic device.
EP14861977.8A 2013-11-13 2014-11-13 System for controlling brain machine interfaces and neural prosthetic systems Pending EP3068349A4 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201361903538P 2013-11-13 2013-11-13
PCT/US2014/065537 WO2015073713A1 (en) 2013-11-13 2014-11-13 System for controlling brain machine interfaces and neural prosthetic systems

Publications (2)

Publication Number Publication Date
EP3068349A1 true EP3068349A1 (en) 2016-09-21
EP3068349A4 EP3068349A4 (en) 2017-12-13

Family

ID=53058028

Family Applications (1)

Application Number Title Priority Date Filing Date
EP14861977.8A Pending EP3068349A4 (en) 2013-11-13 2014-11-13 System for controlling brain machine interfaces and neural prosthetic systems

Country Status (3)

Country Link
EP (1) EP3068349A4 (en)
CN (1) CN105722479B (en)
WO (1) WO2015073713A1 (en)

Families Citing this family (41)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10042422B2 (en) 2013-11-12 2018-08-07 Thalmic Labs Inc. Systems, articles, and methods for capacitive electromyography sensors
US11921471B2 (en) 2013-08-16 2024-03-05 Meta Platforms Technologies, Llc Systems, articles, and methods for wearable devices having secondary power sources in links of a band for providing secondary power in addition to a primary power source
US10188309B2 (en) 2013-11-27 2019-01-29 North Inc. Systems, articles, and methods for electromyography sensors
US9880632B2 (en) 2014-06-19 2018-01-30 Thalmic Labs Inc. Systems, devices, and methods for gesture identification
US10409371B2 (en) 2016-07-25 2019-09-10 Ctrl-Labs Corporation Methods and apparatus for inferring user intent based on neuromuscular signals
EP3487457B1 (en) 2016-07-25 2023-09-06 Facebook Technologies, LLC. Adaptive system for deriving control signals from measurements of neuromuscular activity
US10489986B2 (en) 2018-01-25 2019-11-26 Ctrl-Labs Corporation User-controlled tuning of handstate representation model parameters
US11216069B2 (en) 2018-05-08 2022-01-04 Facebook Technologies, Llc Systems and methods for improved speech recognition using neuromuscular information
US11331045B1 (en) 2018-01-25 2022-05-17 Facebook Technologies, Llc Systems and methods for mitigating neuromuscular signal artifacts
EP3487395A4 (en) 2016-07-25 2020-03-04 CTRL-Labs Corporation Methods and apparatus for predicting musculo-skeletal position information using wearable autonomous sensors
EP3487595A4 (en) 2016-07-25 2019-12-25 CTRL-Labs Corporation System and method for measuring the movements of articulated rigid bodies
EP3697297A4 (en) 2017-10-19 2020-12-16 Facebook Technologies, Inc. Systems and methods for identifying biological structures associated with neuromuscular source signals
EP3743901A4 (en) 2018-01-25 2021-03-31 Facebook Technologies, Inc. Real-time processing of handstate representation model estimates
US10970936B2 (en) 2018-10-05 2021-04-06 Facebook Technologies, Llc Use of neuromuscular signals to provide enhanced interactions with physical objects in an augmented reality environment
WO2019147996A1 (en) 2018-01-25 2019-08-01 Ctrl-Labs Corporation Calibration techniques for handstate representation modeling using neuromuscular signals
US11493993B2 (en) 2019-09-04 2022-11-08 Meta Platforms Technologies, Llc Systems, methods, and interfaces for performing inputs based on neuromuscular control
US10937414B2 (en) 2018-05-08 2021-03-02 Facebook Technologies, Llc Systems and methods for text input using neuromuscular information
US10504286B2 (en) 2018-01-25 2019-12-10 Ctrl-Labs Corporation Techniques for anonymizing neuromuscular signal data
US11907423B2 (en) 2019-11-25 2024-02-20 Meta Platforms Technologies, Llc Systems and methods for contextualized interactions with an environment
WO2019147956A1 (en) 2018-01-25 2019-08-01 Ctrl-Labs Corporation Visualization of reconstructed handstate information
US11150730B1 (en) 2019-04-30 2021-10-19 Facebook Technologies, Llc Devices, systems, and methods for controlling computing devices via neuromuscular signals of users
US11481030B2 (en) 2019-03-29 2022-10-25 Meta Platforms Technologies, Llc Methods and apparatus for gesture detection and classification
US11961494B1 (en) 2019-03-29 2024-04-16 Meta Platforms Technologies, Llc Electromagnetic interference reduction in extended reality environments
CN112005198A (en) 2018-01-25 2020-11-27 脸谱科技有限责任公司 Hand state reconstruction based on multiple inputs
US10592001B2 (en) 2018-05-08 2020-03-17 Facebook Technologies, Llc Systems and methods for improved speech recognition using neuromuscular information
CN112469469A (en) 2018-05-25 2021-03-09 脸谱科技有限责任公司 Method and apparatus for providing sub-muscular control
EP3801216A4 (en) 2018-05-29 2021-04-14 Facebook Technologies, LLC. Shielding techniques for noise reduction in surface electromyography signal measurement and related systems and methods
US10970374B2 (en) 2018-06-14 2021-04-06 Facebook Technologies, Llc User identification and authentication with neuromuscular signatures
US11045137B2 (en) 2018-07-19 2021-06-29 Facebook Technologies, Llc Methods and apparatus for improved signal robustness for a wearable neuromuscular recording device
WO2020027904A1 (en) * 2018-07-31 2020-02-06 Hrl Laboratories, Llc Enhanced brain-machine interfaces with neuromodulation
EP3836836B1 (en) 2018-08-13 2024-03-20 Meta Platforms Technologies, LLC Real-time spike detection and identification
EP4241661A1 (en) 2018-08-31 2023-09-13 Facebook Technologies, LLC Camera-guided interpretation of neuromuscular signals
WO2020061451A1 (en) 2018-09-20 2020-03-26 Ctrl-Labs Corporation Neuromuscular text entry, writing and drawing in augmented reality systems
WO2020069181A1 (en) 2018-09-26 2020-04-02 Ctrl-Labs Corporation Neuromuscular control of physical objects in an environment
US11797087B2 (en) 2018-11-27 2023-10-24 Meta Platforms Technologies, Llc Methods and apparatus for autocalibration of a wearable electrode sensor system
US10905383B2 (en) 2019-02-28 2021-02-02 Facebook Technologies, Llc Methods and apparatus for unsupervised one-shot machine learning for classification of human gestures and estimation of applied forces
WO2021002993A1 (en) * 2019-07-02 2021-01-07 Hrl Laboratories, Llc System and method for continual decoding of brain states to multi-degree-of-freedom control signals in hands free devices
US20210018896A1 (en) * 2019-07-16 2021-01-21 Carnegie Mellon University Methods and Systems for Noninvasive Mind-Controlled Devices
CN111568615A (en) * 2020-04-16 2020-08-25 南方科技大学 Electric artificial limb system and electric artificial limb control method
US11868531B1 (en) 2021-04-08 2024-01-09 Meta Platforms Technologies, Llc Wearable device providing for thumb-to-finger-based input gestures detected based on neuromuscular signals, and systems and methods of use thereof
CN115282431B (en) * 2022-07-06 2023-06-30 电子科技大学 Brain electricity-based light control intelligent lamp stimulation method and device

Also Published As

Publication number Publication date
CN105722479A (en) 2016-06-29
WO2015073713A1 (en) 2015-05-21
EP3068349A4 (en) 2017-12-13
CN105722479B (en) 2018-04-13

Similar Documents

Publication Publication Date Title
WO2015073713A1 (en) System for controlling brain machine interfaces and neural prosthetic systems
US9566174B1 (en) System for controlling brain machine interfaces and neural prosthetic systems
Zhu et al. Robot learning from demonstration in robotic assembly: A survey
Kang et al. Design, modeling and control of a pneumatically actuated manipulator inspired by biological continuum structures
Calinon Learning from demonstration (programming by demonstration)
Mataric Getting humanoids to move and imitate
Kane et al. The use of Kane's dynamical equations in robotics
Mick et al. Reachy, a 3D-printed human-like robotic arm as a testbed for human-robot control strategies
Mansouri et al. A platform for dynamic simulation and control of movement based on OpenSim and MATLAB
Calinon et al. Active teaching in robot programming by demonstration
Felis et al. Synthesis of full-body 3-d human gait using optimal control methods
Caggiano et al. MyoSuite--A contact-rich simulation suite for musculoskeletal motor control
De Sapio et al. Simulating the task-level control of human motion: a methodology and framework for implementation
Nguiadem et al. Motion planning of upper-limb exoskeleton robots: a review
US10899017B1 (en) System for co-adaptation of robot control to human biomechanics
Roller et al. Optimal control of a biomechanical multibody model for the dynamic simulation of working tasks
Ingram et al. Modelling of the human shoulder as a parallel mechanism without constraints
Trivedi et al. Biomimetic approaches for human arm motion generation: literature review and future directions
De Sapio et al. The control of kinematically constrained shoulder complexes: Physiological and humanoid examples
De Sapio An approach for goal-oriented neuromuscular control of digital humans in physics-based simulations
Liu et al. Computational modeling: Human dynamic model
US10409928B1 (en) Goal oriented sensorimotor controller for controlling musculoskeletal simulations with neural excitation commands
Heremans et al. Bio-inspired balance controller for a humanoid robot
Schearer et al. Identifying inverse human arm dynamics using a robotic testbed
Fischer et al. From bot to bot: Using a chat bot to synthesize robot motion

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20160609

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAX Request for extension of the european patent (deleted)
A4 Supplementary search report drawn up and despatched

Effective date: 20171115

RIC1 Information provided on ipc code assigned before grant

Ipc: A61F 2/70 20060101ALI20171109BHEP

Ipc: A61F 2/68 20060101AFI20171109BHEP

Ipc: A61F 2/76 20060101ALI20171109BHEP

Ipc: A61F 2/72 20060101ALI20171109BHEP

Ipc: G06N 3/04 20060101ALI20171109BHEP

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20191018

GRAJ Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted

Free format text: ORIGINAL CODE: EPIDOSDIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

INTC Intention to grant announced (deleted)
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20201109

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230525

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20231211

GRAJ Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted

Free format text: ORIGINAL CODE: EPIDOSDIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTC Intention to grant announced (deleted)
INTG Intention to grant announced

Effective date: 20240517