EP3068349A1 - System for controlling brain machine interfaces and neural prosthetic systems - Google Patents

Abstract

Described is a system for controlling a torque controlled prosthetic device given motor intent inferred from neuroimaging data. The system includes at least one torque controlled prosthetic device operably connected with one or more processors. Further, the system is configured to receive neuroimaging data of a user from a neuroimaging device and decode the neuroimaging data to infer spatial motion intent of the user, where the spatial motion intent includes desired motion commands of the torque controlled prosthetic device represented in a coordinate system. The system thereafter executes, with a prosthesis controller, the motion commands as torque commands to cause the torque controlled prosthetic device to move according to the spatial motion intent of the user.

Description

SYSTEM FOR CONTROLLING BRAIN MACHINE INTERFACES AND NEURAL PROSTHETIC SYSTEMS

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This is a non-provisional patent application of U.S. Provisional

Application No. 61/903,538, filed November 13, 2013, entitled, "A control and simulation architecture for application to brain machine interfaces and neural prosthetic systems." [0004] This is ALSO a non-provisional patent application of U.S. Provisional Application No.6.1 /903,526, filed November 13, 2013, entitled, "A goal- oriented sensorimotor controller for controlling musculoskeletal

simulations with neural excitation commands."

[0005] BACKGROUND OF THE INVENTION

[0006] ( 1 ) T ieid of invent! on

[0007] The present invention relates to a robotic control system and, more particularly, to a system for controlling robotic prosthetic devices given motor intent inferred from neuroimatiinti data.

[0008] (2) Description of Related Art

[0009] Brain machine interfaces (BMI) and neural prosthetics offer great hope for restoring function to people with spinal cord injuries and amputees, as well augmenting and enhancing the abilities of people with full motor function , There have been a number of advances in neural imaging as well as in decoding motor intent from neuroimagmg data. However, with regard to prosthetic device control there have been fewer advances, with most approaches rooted in conventional robotic control. [00 10] There is a body of work in BMI and neural prosthetics research which addresses the decoding of cortical signals for the downsiream execution of motion commands by an externa! device (see, for example, the List of Incorporated Literature References, Literature Reference Nos. I, 9, .1 1₅ 1.2, and 15). Most of this work focuses on the neural decoding with less rigor applied to the prosthesis control hi cases where a robotic arm prosthesis is control led, the motion trajectory is typically converted into joint commands and executed using joint space control (see, for example.

Literature Reference Nos, ϊ I , ! 2, and 15),

.1] To date, no known prosthetic control system has been proposed which uses a task/posture decomposition. Task/posture decomposition has particular advantages over joint space control in B Is. It is based on an abstraction analogous to the manner in which cortical signals are encoded (e.g., eye-centered, hand-centered Cartesian coordinates). Further, it is generali abie to whole body prosthetic control involving multiple prosthetic limbs and .high degree-of-iteedom kinematics. Joint space prosthetic control is limited in this capacity due to its reliance on inverse kinematics solutions. Additionally, task/posture decomposition allows for the specification of postural behaviors based on minimizing important objective functions (e.g. power consumption, virtual muscle effort, etc.) consistent with the execution of cortical motion commands.

[00012] There has been some work; on applying neiuomorphie computing

architectures to BMls and neural prosthetic systems. For example, the work of Dethier et ai, focused on the use of artificial spiking neural networks for decoding and filtering cortical signals {(Caiman filter based decoding using endpomt kinematics as the state vector and neural spike rates as the measurement vector) rather than prosthetic control (see Literature Reference No. 7), [00013] Additionally, the work of Bougaais and Shanahan wa directed to controlling a robotic arm using artificial spiking neural networks; although their work was not applied to a neural prosthetic system (see Literature Reference No. 2).

[00014] Other researchers performed work in the simulation of neural

prosthetic limbs in conjunction with biomechanical models (see Literature

Reference Nos. 4, 5, and 8). However, while their work implements musculoskeletal models into the simulation, the locus of these efforts is on simulating and validating the control of the prosthetic device. Thus, again, there is no known system that integrates a sensorimotor controller, in addition to the prosthetic controller.

15] Therefore, a continuing need exists for a system thai allows enhanced model ing and analysis of the interface issues associated, not just, wi th the biological and engineered components., but also with the biological and engineered control systems. Further, a need exists for a system-level architecture for applying artificial spiking neural networks to the learning of motor maps, consistent with postural criteria, for controlling motion in BM I and neural prosthetic systems. [0001.6] SUMMARY OF INVENTION^'

[00017] Described is a system for controlling a torque controlled prosthetic device given motor intent inferred from neoroiniagin data. The system includes one or more processors and a memory, the memory having executable instructions encoded thereon, such that upon execution of the instructions, the one or more processors perform several operations, such as recei ving neuroimaging data of a user from a neuroimaging device. The neuroimaging data is then decoded to infer spatial motion intent of the user, where the spatial motion intent includes desired motion commands of the torque controlled prosthetic device represented in a coordinate system. Thereafter, the system executes, with a prosthesis controller, the motion commands as torque commands to cause the torque controlled prosthetic device to move according to the spatial motion intent of the user.

[00018] In another aspect; the system includes at least one torque controlled prosthetic device operabiy connected with the one or more processors.

[0001 ] in yet another aspect, the system performs an operation of receiving, in the controller, sensory information regarding a current state of the prosthetic device,

[00020] Additionally, in executing the motion commands, the motion

commands are executed using a task decomposition and posture decomposition, wherein the task decomposition is a task space control and the posture decomposition is formulated as a cost potential which represents a cost, function.

[0002.1 ] Further, i executing the motion commands, the motion commands are executed as torque commands that generate a desired, task space control while minimizing the cost potential.

[00022] in yet another aspect, in executing the motion commands, the motio commands are executing using spiking neural network.

[00023] In another aspect, the prosthesis controller is a neuromorphic

prosthesis controller and further comprises: a neuromorphic spike encoder to represent the motion command as a set of neural spikes; a neuromorphic motor mapper to map the neural spikes representing Cartesian

displacements to neural spikes representing configuration (joint) space di placements; a spike decoder to decode the neural spikes representing configuration (joint) space displacements and generate a joint space command; and a joint servo to execute the joint, space command. [00024] In yet another aspect, the system performs operations of receiving a model of the prosthetic device and a musculoskeletal mode! of the user, wherein the musculoskeletal model includes musculoskeletal dynamics that include steadv state tendon forces; eeneratina, with a sensorimotor controller, simulated neural excitations given the motion commands to drive a set of muscle activations in a musculoskeletal simulation; and generating, with the prosthesis controller,, simulated actuator joint torques given the motion commands to drive a simulated prosthetic device.

[00025] Finally, the present invention also includes a computer program

product and a computer implemented method. The computer program product includes computer-readable instructions stored on a non-transitory computer-readable medium that are executable b a computer having one or more processors, such that upon execution of the instructions, the one or more processors perform the operations listed herein. Alternatively, the computer implemented method includes an act of causing a computer to execute such instructions and perform the resulting operations. [00026] BRIEF DESCRIPTIO OF THE DRAWINGS

[00027] The objects, features and advantages of the present invention will be apparent from the following detailed, descriptions of the various aspects of the invention in conjunctio with reference to the following drawings, where:

[00028] FIG. t is a block diagram depicting the components of system

according to the principles of the present invention;

[00029] FIG. 2 is an illustration of a computer program product embodying an aspect of the present invention; [00030] FIG. 3 is a process flow-chart illustrating how high-level motor intent from the brain is decocted and a task-level .motion command is

transmitted to the prosthetic device controller, which in turn executes the motion command using a task/posture decomposition;

[0003 1 ] FIG. 4A is a simplified model of a human arm actuated by 14 muscles;

[00032] FIG, 4B s a table illustrating corresponding maximum isometric

forces, . , and optimal fiber lengths, t_Mn of the muscles illustrated in. FiG. 4A;

[00033] FIG. 5 A is an il lustration of a redundant muscle-actuated model of the human ann used as a physiological template for controlling a prosthetic arm;

[00034] FIG. 5B is a graph illustrating time histories of joint motion, while performing a simulation run of the model as illustrated i FIG. 5 A;

[00035] FIG. 5C is a graph illustrating time histories of hand motion while performing a simulation run of the mode l as illustrated in FIG. 5A;

[00036] FIG. 5D is a graph illustrating time histories of muscle effort while performing a simulation run of the model as illustrated in FIG. 5 A; [00037] FIG. 6 A is a flow chart illustrating an artificial spiking neural network where motor map redundancy is resolved by training the network only on unique solutions of the inverse mapping given by Δ¾

[00038] FIG. 6B is a flow chart illustrating an artificial spiking neural network where motor map redundancy is resolved by explicitly inhibiting the learning of solutions where the cost is high; [00039] FIG. 7 is flow chart illustrating how high-level motor intent from the brai is decoded and a task-level motion command is transmitted, to the prosthetic device controller;

[00040] FIG. 8 is a flow chart illustrating how a decoded cortical output is sent to a simulated prosthesis controller and a simulated sensorimotor controller; [00041 ] FIG. 9 is a flow chart illustrating the neural and musculoskeletal

physiology associated with the motor control problem;

[00042] FIG. 10 is a functional block diagram depicting a system-level

abstraction of FIG. 9, showing an abstracted representation of neural and musculoskeletal physiology;

[00043] FIG. 1 1 is an illustration depicting an active state museulotendon

model, where the active contractile element and passive viscoelastic element are i paral lel and the passive elastic tendon element is in series;

[00044] FIG. 12 is an illustration depicting a neuromuscular and

musculoskeletal system (feed-forward path};

[00045] FIG. 13 is an illustration of a task-level sensorimotor controller;

[00046] FIG. 14 is a functional block diagram depicting an abstracted

representation of neural and musculoskeletal physiology;

[00047] FIG. 15 is an illustration depicting a neuromuscular and

musculoskeletal system (feed-forward path), showing the musculoskeletai system dynamics augmented with a set ofholonomic constraints and Lagrange multipliers;

[00048] FIG , 16 is an illustration of a constrained task-level sensorimotor

controller; and

[00049] F G. 17 is an illustration of a, constrained task-level sensorimotor

controller, which generates motion consistent, with the input motion commands in the presence of olonomic system constraints.

[00050] DETAILED DESCRIPTION

[00051] The present invention relates to a robotic control system and, more particularly, to a system for controlling robotic prosthetic devices given motor intent inferred from neuioimaging data. The following descriptio is presented to enable one of ordinary skill in the art to make and use the invention and to incorporate it in the context of particular applications. Various modifications, as well as a variety of uses in. different applications will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to a wide range of aspects. Thus, the present invention is not intended to be limited to the aspects presented, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[00052] In the following detailed description, numerous specific details are set forth in order to provide a more thorough understanding of the presen t invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without necessarily being .limited to these specific details. In other instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the present invention. [00053] The reader's attention, is directed to all papers and documents winch are filed concurrently with this specification and which are open to public inspection with th is specification, and the contents of all such papers and documents are incorporated herein by reference. All the features disclosed in this specification, (including any accompanying claims, abstract, and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic series of equivalent or similar features,

[00054] Furthermore, any element in a claim that, does not explicitly state

"means for" performing a specified function, or "step for" performing a specific function, is not to be interpreted as a "means" or "step" clause as specified in 35 IJ.S.C. Section 1 12, Paragraph 6. In particular, the use of "step of or "act of in the claims herein is not intended to invoke the provisions of 35 IJ.S.C. 1 12, Paragraph 6.

[00055] Before describing the invention in detail, first a list of incorporated literature references is provided. Next, a description of various principal aspects of the present invention is provided. Subsequently, an

introduction provides the reader with a general understanding of the present in vention. Finally, details of the present invention are provided to give an understanding of the specific aspects. [00056] (i) List of ncorporated Literature References

[00057] The following references are cited throughout this application. For clarity and convenience, the references are listed herein as a central resource for the reader. The following references are hereby incorporated by reference as though fully included herein. The references are cited in the application by referring to the corresponding literature reference number. Andersen, R, A., Hwang, E. J., & Muiltken, G. I I ... (2010),

Cognitive neural prosthetics. Annual review of psychology, 61, 169.

Bouganis, A., & Shanahan, M., (2010, July), Training a spiking neural network to control a 4-DoF robotic arm based on spike timing-dependent plasticity. In Neural Networks (UCN ), The 2010 international Joint Conference on (pp. 1-8), IEEE.

Buneo, (1 A., Jarvis, M. R., Batista, A. P., & Andersen, R. A., (2002), Direct visuomotor transformati ns for reaching. Nature, 416(6881 ), 632-636.

Davoodi, R„ & Loeb, G. E., (201 1, February), MSMS software for VR simulations of neural prostheses and pati ent train ing and rehabilitation, I MMVR (pp. 156- 162).

Davoodi, R., Urata_s C, Hauschild, M, Khachani, M., & Loeb, G. E„, (2007), Model-based development of neural prostheses for .movement. Biomedical Engineering, IEEE Transactions on, 54(1 1 ), 1909-1918.

De Sapio, V., Khatib, O., & Delp, S., (2008), Least action principles and their application to constrained and task-level problems in robotics and biomechanics, Multibody System

Dynamics, 19(3), 303-322,

Dethier J., Nuyujukian, P., Eliastrrsth, C, Stewart, T„ C, Elasaad, S. A,, Sbenoy, K, V., & Boahen, K. A,, (2011), A brain-machine interface operating with a real-time spiking neural network control algorithm, in Advances in Neural information Processing Systems (pp. 2213-2221 ).

Hauscbiid, M., Davoodi, R,, Sc. Loeb, G. E., (2007), A virtual reality environment for designing and fitting neural prosthetic limbs. Neural Systems and Rehabilitation Engineering, IEEE

Transactions on, 15(1 ), 9-15, Hochberg, L. ., Bacher, D., Jarosiewicz, B,, Masse, N. Y., Simera!, J. D., Vogel, I, ... & Donoghue, J. P., (2012), Reach and grasp by people with tetraplegia using a neuraliy controlled robotic arm. Nature, 485(7398), 372-375.

Kiiatib, O,, (1 95), inertia! properties in robotic manipulation: An object-level framework, The International Journal of Robotics Research,. 14(1 ), 1 -36.

Lebedev, M. A., Tate, A. I, Hanson, T. L,, Li, Z,, O'Doherty, J. E.„ Winans, 1 A., ... & Nieolelis, M. A., (201 1), Futiiie developments in brain-machine interface research. Clinics, 66, 25-32.

Naidu, D. S., Chen, C. B., Perez, A., & Schoen, M. P, (2008, August), Control strategies for smart prosthetic hand technology: An overview. In Engineering in Medicine and Biology Society, EMBS 2008. 30th Annual International Conference of the IEEE (pp. 4314-4317), IEEE.

Sabes, P. N., (2000), The planning and control of reaching

.movements. Current opinion in neurobiology, 10(6), 740-746. Srinivasa, M., & Cho, Y., (2012), Self-organizing spiking nearal model for learning fault-tolerant spatio-motor transformations. Neural Networks and Learning Systems, IEEE Transactions on, 23(10), 1526-1538.

Veiliste, M., Perel S., Spalding, M. C, Whittord, A. S., &

Schwartz, A. B._t (2008), Cortical control of a prosthetic arm for self-feeding, Nature, 453(7198), 1098- 1 101.

De Sapio, V. (201 1 ), Task-level control of motion and constraint forces in holonomicaJly constrained robotic systems, in

Proceedings of the 18th World Congress of the International.

Federation of Automatic Control.

De Sapio, V. and Park, j . (2010), Multitask constrained motion control using a mass- weighted orthogonal decomposition, Journal of Applied. Mechanics, Vol . 77, No. 4, p.0 1004, 18. De Sapio, V., Khatib, O, and Delp, S. (2005), Simulating die task- level control of human motion: a methodology and framework for implementation, The Visual Computer, Vol 21, No. 5, pp.28 - 302.

19. Thelen, D.G. and Anderson, F.C. (2006), Using computed muscle control to generate forward dynamic simulations of human walking from experimental data. Journal of Biomechanics, Vol . 39, No. 6, p.1 107.

20. Thelen, D.G., Anderson, F.C. and Delp, S.L, (2003), Generating dynamic simulations of movement using computed muscle control, journal of Biomechanics, Vol . 36, No. 3, pp.321-328.

21. De Sapio, V., Khatib, O., and Delp, S. (2006), Task-level approaches for the con trol of constrained multibody systems,

Muhibody System Dynamics, Vol. 16, No. 1 , pp. 73-102,

[00058] (2) Principal Aspects

[00059] The present invention has three "principal" aspects. The first is a

system for controlling robotic prosthetic devices gi en motor intent inferred from neuroimaging data. The system is typically in the form of a computer system operating software or in the form of a "hard-coded" instruction set. This system may be incorporated into a wide variety of devices that provide different functionalities. In one aspect, the system is incorporated into a robot having actuators and appendages or other motion operable components and any other components as may be required to provide the functionality as described herein. The second principal aspect is a method, typically in the form of software, operated using a data processing system (computer). The third principal aspect is a computer program product. The computer program product generally represents computer-readable instructions stored on a non-transitory computer- readable medium such as an optical storage device, e.g., a compact disc (CD) or digital versatile disc (DVD), or a magnetic storage device such as a floppy disk or magnetic tape. Other, non-limiting examples of

computer-readable media include hard disks, read-only memory (ROM), and flash-type memories. These aspects will be described in more detail below.

[00060] A block diagram depicting an example of a system (i.e., computer system 100) of the present invention is provided in FIG. I . The computer system 100 is configured - to perform calculations, processes, operations, and or functions associated with a program or algorithm. In one aspect, certain processes and steps discussed herein are realized as a series of instructions (e.g., software program) that reside within computer readable memory units and are executed by one or more processors of the computer system 100. When executed, the instructions cause the computer system 100 to perform specific actions and exhibit specific behavior, such as described herein.

[00061 ] The computer system 1 0 may include an address/data bus 102 that is configured to communicate information. Additionally, one or more data processing units, such as a processor 104 (or processors), are coupled with the address/data bus 10:2. The processor 104 is configured to process information and instructions. In an aspect, the processor 104 is a

microprocessor. Alternatively, the processor 104 may be a different type of processor such as a parallel processor, or a field programmable gate array.

[00062] The computer system 100 is configured to utilize one or more data storage units. The computer system 100 may include a volatile memory unit 106 (e.g., random access memory ("RAM"), static RAM, dynamic RAM, etc.) coupled with the address data bus 102, wherein a volatile memory unit 106 is configured t store information and instructions for the processor 104, The computer system 1 0 further may include a non- volatile memory unit 108 (e.g., read-only memory ("ROM"),

programmable ROM ("PROM"), erasable programmable ROM

("EPROM"},, electrically erasable programmable ROM "EEPROM"), flash memory, etc.) coupled with the address/data bus 102. wherein the nonvolatile memory unit 108 is configured to store static information and instructions for die processor 104. Alternatively, the computer system 100 may execute instructions retrieved from an online data storage unit such as in "Cloud" computing, in an aspect, the computer system 100 also may include one or more interfaces, such as an interface 110, coupled with the address/data bus 102, The one or more interfaces are configured to enable the computer system 100 to interface with other electronic devices and computer systems. The commu ication interfaces implemented by the one or more interfaces may include wireline (e.g., serial cables, modems, network adaptors, etc.) and/or wireless (e.g.. wireless modems, wireless network adaptors, etc.) communication technology. ] In one aspect, the computer system 100 may include an input device .1 12 coupled with the address/data bus 102, wherein the input device 1 12 is configured to communicate information and command selections to the processor 100. In accordance with one aspect, the input device 1 12 is an alphanumeric input device, such as a keyboard, that may include alphanumeric and/or function keys. Alternatively, the input device 1.12 may be an input device other than an alphanumeric input device, in an aspect, the computer system 100 may include a cursor control device 1 14 coupled with the address/data bus 102, wherein the cursor control device 1 14 is configured to communicate user input information and/or command selections to the processor 100. In an aspect, the cursor control device .1 14 is implemented using a device such as a mouse, a track-ball, a track-pad, an optical tracking device, or a touch screen. The foregoing

notwithstanding, in an aspect, the cursor control device 1 14 is directed and/or activated via input from the input device 112, such as in response to the use of special keys and key sequence commands associated with the input device 1 12. in an alternative aspect, the cursor control device 1. 1 is configured to be directed or guided by voice commands.

[00064] In an aspect, the computer system 100 further may include one or more optional computer usable data storage devices, such as a storage device 116, coupled with the address/data bus 102. The storage device 1 16 is configured to store information and/or computer executable instructions, in one aspect, the storage device 1 16 is a storage device such as a magnetic or optical disk drive (e.g., hard disk drive {"HDD"), floppy diskette, compact disk read only memory ("CD-ROM"), digital versatile disk ("DVD")). Pursuant to one aspect, a display device 1 18 is coupled with the address/data bus 102, wherein the display device 118 is configured to display video and/or graphics. In an aspect, the display device 118 may include a cathode ray rube ("CRT"), liquid crystal display ("LCD"), field emission display ("FED"), plasma display, or any othe display device suitable for displaying video and/or graphic images and alphanumeric characters recognizable to a user. [00065] The computer system 100 presented herein is an example computing environment in accordance with an aspect. However, the non-limiting example of the computer system 100 is not strictly limited to being a computer system. For example, an aspect provides thai the computer system 100 represents a t pe of data processing analysis that may be used in accordance with various aspects described herein. Moreover, other computing systems may also be implemented, indeed, the spirit and scope of the present technology is not limited to any single data processing environment. Thus, in an aspect, one or more operations of various aspects of the present technology are control led or implemented using computer-executable instructions, such as program modules, being executed by a computer. In one implementation, such program modules include routines, programs, objects, components and/or data structures that are configured to perform particular tasks or implement particular abstract data types. In addition, an aspect provides that one or more aspects of the present technology are implemented by utilising one or more distributed computing environments, such as where tasks are performed by remote processing devices that are linked through a communications network, or such as where various program modules are located in both local and remote computer-storage medi including memory-storage devices. [00066] An illustrative diagram of a computer program product { i.e., storage device) embodying the present invention is depicted in FIG. 2. The computer program product is depicted as floppy disk 200 or an optical disk 202 such as a CD or DVD. However, as mentioned previously, the computer program product generally represents computer-readable instructions stored o any compatible non-transitory computer-readable medium. The term "instructions" as used with respect to this invention generally indicates a set of operations to be performed on a computer, and may represent pieces of a whole program or individual, separable, software modules. Non-limitin examples of "instruction" include computer program code (source or object code) and "hard -coded" electronics (i.e. computer operations coded into a computer chip). The "instruction" is stored on any non-transitory computer-readable medium, such as in the memory of a computer or on a floppy disk, a CD-ROM, and a flash drive. In either event, the instructions are encoded on a non- transitory computer-readable medium.

[00067] (3) Introduction

[00068] Brain machine interfaces (B I) and neural prosthetics offer great hope for restoring function to people with spinal cord injuries and amputees, as wel l augmenting and enhancing the abilities of people with full motor function. There ha ve been a number of advances in neural imaging as well as in decoding motor intent from neuroiinagiiig data. However, with regard to prosthetic device control there have been fewer advances, with most approaches rooted in conventional robotic control.

[00069] Thus, the system described herein provides a system-level architecture for controlling robotic prosthetic devices given motor intent inferred from neuroimaging data. The system also provides stimulation architecture to facilitate virtual testing of the specific controller and prosthetic device design, with, respect to the human subject, prior to any clinical

intervention. Two unique control architectures will he described. One controller is based on a bio-inspired task/posture decomposition of motion intended for impiementation on conventional digital computing hardware. The other controller is based on neuroraorphic mapping of task motion to joint motion with a bio-inspired resolution of kinematic redundancy., whic can he implemented on oeuromorphic computing hardware. A unique aspect is that both approaches utilize bio-inspired abstractions for controlling prosthetic devices in a natural human-like manner rather than utilizing conventional inverse kinematic approaches. Specific

instantiations of either of these control architectures can be tested with a b.um.an~.in~the oop using a novel model-based simulation environment, !n this environment actual cortical output from neoroimaging equipment (electroencephalography (BEG), functional magnetic resonance imaging (fMRT), etc.) attached to the subject will drive both a biomechanics! model of the subject and a model of the prosthetic device to be tested. This modeling provides a technique for testing the efficacy of the prosthetic device, the controller, and the overall bio-integration of the prosthetic hardware with the subject, without requiring the subject to be physically coupled with the actual prosthetic hardware.

[00070] The ability of the simulation environment to dri ve both the motion of the prosthetic device and the motion of the simulated subject through cortical commands (i.e., BMi) is unique and can be utilized by a variety of prosthetic manufacturers and/or developers. For example, human augmentation technologies can be implemented for enhancing the capabilities of the manufacturing floor worker. These technologies include assistive exoskeletons and other devices to increase worker productivity and reduce injury. The invention offers significant capabilities for the development of next generation BMls, such as user performance enhancement, assistive robotic devices, and neural prosthetic technologies.

[00071 ] (4) Specific Details

[00072] Described are two control architectures for controlling robotic

prosthetic devices given motor intent inferred from, neuroiinaguig data. Additionally, a simulation architecture is described for human- in-the-Ioop virtual resting of bio-integration, issues associated with the prosthetic hardware. Both control architectures start with the assumption of a source of decoded cortical data from the brain which represen ts high-level motor intent This .motor intent is in the form of desired motion commands represented in different Cartesian coordinate systems (eye-centered, hand- centered, etc.) using visumotor transformations. Examples of such.

visumotor transformations were describe in Literature Reference os. 3 and 13. This desired motion command is referred to herein as ¾ a multidimensional vector of task coord inates (e.g. Cartesian coordinates associated motion of a. single hand or coordinated motion of both hands). The goal is to control the prosthetic device based on this motor intent.

[00073] Before describing the first control architecture based on bio-inspired task/posture decomposition of motion, it is necessary to define the dynam ics of the prosthetic de vice. In joint space, the dynamics are defined as follows: ϊ— {q)q - b(q_fq) 4- g(q\ where is the vector of joint coordinates, r, is the vector of applied joint torques, M{q), is the joint space mass matrix, b{q_tq}, is the vector of centrifugal and CorioHs forces, and g{q) is the vector of gravity forces. inverse kinematics is conventionally used to map a desired Cartesian goal into a set of joint angles, {sf ¹ (¾}|at^■→ q}. Subsequently, given the joint level commands the prosthetic device is controlled by servoing individual joints or by using computed torque control. Given an anthropomorphic prosthetic device designed to replace a human limb (e.g., the human arm), it is necessary to address kinematic redundanc in the limb motion. That is, the device will typically possess more joint coordinates than are required by the motion command. From a robotic control perspective this implies that there are an infinite number of joint coordinate solutions associated with a given motion command.

[00074] The approach to dealing with this issue is to .represent the dynamics of the prosthetic device in task space, a more natural space for addressing the control problem since the decoded corticai motion command is expressed in this space, A task space representation of the dynamics is given by, f ~ A($)x- p(q.,q) - p(q), where x is a description of task coordinates that are to be controlled, f are the control forces that act at the task, and .f_* f X and .p{q\ are the task, space mass .matrix, centrifugal and Coriolis force vector, and gravity force vector, respectively. Task space was described by hatib in Literature Reference No. .10.

[00075] The applied joint torques are given by,

* =j^rf + AT where # is the Jaeobian of the task coordinates, N^f is the mill space projection matrix, and τ_β is an aribitrar joint torque filtered through the null space projection matrix. The key point to note here is that control of the prosthetic device can be decomposed into a task-level component and a complementary postural component (due to the kinematic redundancy). Based on the choice of a particular generalized inverse of the Jaeobian these tw components of motion control are guaranteed to be dynamically consistent with each other and, as such, control can be synthesized to execute the cortical motion command while simultaneously achieving som postural objective for the prosthetic device. This postural objective can be based on minimizing some physiological cost criterion or it can be based on minimizing a power consumption criterion for the device. In the former case, while the prosthetic device is not necessarily actuated in a manner analogous to the biological appendage it is replacing, encoding a physiological criterion into its control can result in more human-like behavior of the device's postural motion. This would achieve the desirable effect of the device moving more consistently with how the subject's own appendage would move. For further understanding. FIG. 3 depicts the system-level architecture associated with this approach. High-level motor intent from the brain 302 is decoded (with a cortical decoder 304) and a task-level motion command 306 is transmitted to the prosthetic device controller 308. High-level motor intent is captured using a neuroimaging device or any other suitable device operable for capturing neuroimaging data 300, non-limiting examples of which include neuroimaging equipment such as

electroencephalography (EEG), and functional magnetic resonance imaging ifMRi ). The cortical decoder 304 uses a decoding algorithm to infer spatial, motion intent 306 from the nemoiraadna data 300. As an example and as noted above, this motor intent is in the form of desired motion commands represented in different Cartesian coordinate systems (eye-centered, hand-centered, etc.) using visumoior transfonmtions.

[00077] The controller 308 executes the motion command using a task/posture decomposition. Relevant physiological criteri (i.e., postural criterion) or device power consumption criteria can be used to drive the postural component of motion 310, consistent with the conical command. Sensory information 312 (e.g., joint angles of the current state of the prosthetic- device 1 } from the prosthetic device 314 is fed back to the controller 308, resulting in closed loop control The architecture also accommodates sensory information 312 from the prosthetic device 314 being sent to the brain 302 to facilitate afferent proprioception.

[00078] In formulating a postural criterion, a potential. can be specified as representing some cost function. An example of suitable cost functions was described by De Sapio in Literature ^'Reference No. 6. The task-level motion command can be controlled while simultaneously controlling motion in the complementary posture space to minimize This can be accomplished by applying gradient descent in the null/posture space. That is.

, . r oil

[00079] The task space control equations can then be expressed as follow-

where ^"^^* represents estimates of the dynamic parameters and i^* is the control law, A proportional-derivaiive (PD) control law can be expressed as: f* = . Γ_{χ_Λ— 3.) · ί..(ά ά · 3ί„> _< where Κ_ρ and Κ_ν are control gains. The control torque, which generates the desired task space control while minimizing the instantaneous cost potential in the posture space, is then.

dq

[00080] A physiological cost criterion, like muscle effort, can he selectively represented. For example, De Sapio, et a!., (see Literature Reference No. 6), proposed the following muscle effort criterion related to minimizing muscle activation, where g{q) is the gravity torque, M{q) is the matrix of muscle moment arms, and IT_/0?} is the diagonal matrix mapping muscle acti vation, «, to muscle force. . The elements of _f for a given muscle can be modeled by, where <$) is the muscle length, and f_a and ½ represent, the maximum isometric muscle force and the optimal muscle fiber length respectively. 0008 J ] This effort criterion can he used to design an arm prosthesis controller based on the architecture of FIG. 3, For illustrative purposes and as a non limiting example, the characteristics of the biomechanical model of FIG, 4A are employed into an arm prosthesis controller. FIG. 4A illustrates a simplified biomechanical model of the human arm actuated by 14 muscles, while IG. 4B is a table illustrating the maximum isometric forces. /_s, and optimal fiber lengths, l_Mf> for the biomechanical model of

FIG. 4A. The control law is defined as,

£ ^{* ~} k_«{x₄— x)— and the control torque is defined as, where a dissipative term, and a gain, are added on the gradient descent term in the posture space portion of the control torque.

[00082] FIG. 5A illustrates a redundant muscle-actuated model of the human arm used as a physiological template for controlling a prosthetic arm, while PIGs. SB, 5C and 5D are graphs illustrating time histories of joint motion, hand motion, and muscle effort, respecti vely, for a simulation run using the model of FIG. 5A. The model of FIG. 5 A depicts initial and final configurations (joint angles), q{ ) and - (i_f), associated with gradient descent movement to a target, x_f. The motion corresponds to gradient descent of the rnuscie effort, subject to the task requirement. The posture control seeks to reduce the muscle effort but is also constrained by the task requirement.

[00083] The controller achieves the final target objective while the posture control simultaneously seeks to reduce the muscle effort (consisteni with the task requirement). No compensation for the dynamics (except for gravity) was included in the control. Thus, there is no feedback linearization of the inertia! terms present in the control . Normally, perfect feedback linearization would produce straight line motion to the goal, in the absence of feedback, linearization non-straight line motion results.

[00084] I t is noted that the biomechanical model of FIG. 4A is used to generate a physiological criterion that is encoded in the prosthesis controller. The ac tuation system of the prosthetic arm need not be analogous to its biological muscle-driver* counterpart Rather, the biological muscie model defines a set of virtual muscles that direc t the postural control of the prosthetic device in a bio-inspired manner, eve if conventional robot actuators (e.g., DC motors, etc. ) are used in the prosthetic device to control joint movements. [00085] The control architecture depicted in FIG, 3 is intended to be

implemented on conventional digital computing hardware and integrate with a prosthetic device if desired. An approach that would implement neuromorphic hardware is also described, below. This involves the design of an artificial spiking neural network to learn the mapping between task- level motion commands, derived from decoded cortical signals, and joint space commands. In the approach described above, kinematic redundancy in the prosthetic device was resol ved using a. task/posture decomposition, in the case of motor map learning by an arti ficial spiking neural network , kinematic redundancy also needs to be addressed since a differential movement in x would result in an infinite space of differential movements in q.

[00086] FIG. 6.A and 6B depict artificial spiking neural networks with two different approaches for resolving kinematic redundancy. Both networks learn the inverse kinematic mapping using trajectory babbling 600, 61 1 of the prosthetic arm. Kinematic redundancy impacts the trainin such that for a given configuration (set. of joint angles), q 603, and Cartesian displacement. Ax 601, there will be multiple va lues of configuration (joint) space displacement, A*| 604 to learn. In other words, the forward

mapping, ^^x ~ cannot be uniquel inverted. As shown in FIG.

6A, this can be resolved by using a specific generalized inverse of / (i.e.,

/* 602), such that ^~ f^*&x* Given this unique mapping, the spiking neural network of FIG. 6A trains on the output of the trajectory babbling in Cartesian space 600. The signals 603 and Ax 601 serve as the context for the training and the signal &q 604 serves as the signal to be learned with respect, to that context. Signal 603 represents the current joint angle configuration that is being sensed at any given time. ] The spikes representing these signals act as inputs to layer 1 605 of spiki g neurons. The outputs of layer 1 605 act as spike inputs to layer 2 606. An additional layer 3 60? of neurons is employed on the context side (q 603 and Ax 601 input side). In this layer each neuron codes for a single combined representation of both the configuration, q 603, and Cartesian displacement. Ax (see Literature Reference No, 14). In layer 4 60S the synaptic weights of the spiking neurons in that layer are reinforced consistent with learning the training signals 609. Spike timing-dependent plasticity (see Literature Reference Nos. 2 and 14) is utilized to modulate the synaptic weights in the layers. After sufficient training on motor babbling data the spike timing-dependent plasticity can be turned, off as the neurons in layer 4 60? will have ieamed the differential, kinematic mapping. Note that x(t+ 1 ) and q(t+ 1 ) denote a new value for a next time step. ] The generalized inverse, * 602, can be chose based on the minimization of some quadratic Form. For example, to find a solution thai minimizes I f I³ , the following solution can be used: &q ^β⁴Άκ ^β^τ ββ^τ)^~1Αχ.

[00089] As another example, a solution that minimizes the instantaneous

kinetic energy of the system, q^TMq/2, would be given by,

q ^ jAx ^« Μ^~χβ^τ(βΜ^~*β^τ}^~ί&α.

[00090] Alternatively, a solution that minimizes the instantaneous muscle fiber displacement rate, \\l\l^z ~ q^TR$k^T would be given by,

Δ<? * (l?^)^"1 ^ ! ^^}^"1/⁷]^" ^

[00091] As shown in FIG, (SB. the redundancy problem can also be resolved by adding a layer of neurons 612 that explicitly inhibits the learning of solutions where the cost (physiological or otherwise) is high. The structure of Fig. 6B is similar to Fig, 6A, However, rather than using a generalized inverse to compute a unique value of &q to team for a given configuration, <?, and Cartesian displacement, Δ^χ, forward kinematics 610 is used to compute the unique forward mapping from the configuration (joint) space motor babbling 61 1 to Cartesian space. While many values of&% will satisfy this mapping for a given value of q and Δ , a layer of neurons 612 is incorporated to inhibit the learning 613 f q signals with high cost values. In this way training will be directed at learning Aq values associated with minimal cost values.

[00092] Referring again to FIGs. 6 A and 6B, the spiking neural networks

operate as nearomorphic motor mappers (illustrated as element 708 in

FIG. 7) that map Cartesian displacement, &x 601, to configuration (joint) space displacement, &q. Utilizing this neuromorphic motor mapper 708 a feedback loop 700 is specified in FIG. 7 to control the prosthetic arm. The overall system consists of output from a conical decoder, which specifies intentional mo vement 306 of the arm in Cartesian space. This signal is encoded as spikes 706 b the aeuromorphic spike encoder 702. These spikes, as well, as joint angle measurements from the prosthetic arm 714 encoded as spikes 720, act as input to the neuromorphic motor mapper 708. A spiking output representing the learned mapping to the

configuration (joint) space displacement, Δ<?. is decoded 710 and sent to a joint servo 712, which controls the motion of the prosthetic ami 718.

[00093] For further understanding, FIG. 7 depicts the system-level architecture associated with the approach depicted in FIG. 6B. in this case, the prosthetic device controller (i.e., neuromorphic prosthesis controller) 308 includes a neuromorphic spike encoder 702 (to represent the motion command 306 as a set of spikes 706), a neuromorphic motor mapper 708 (as illustrated in FIG. 6A and/or 6B) to map the neural spikes representing Cartesian displacements to neural spikes representing configuration (joint) space displacements, a spike decoder 710 (to decode the neural spikes representing configuration (joint) space displacements) and generate joint space commands, and a join t servo 712 to execute the joint space

command. The sensed joint angles 71 (of the current state of the prosthetic device 718) are fed hack to the joint servo 712 and to the motor mapper 70S (after spike encoding 720). Relevant physiological criteria or device power consumption criteria can incorporated into the neuromorphic motor mapper (as illustrated in FIG, 6A and/or 6B) to drive the postural component of motion, consistent with the cortical command.

[00094] The system can also he modified to allow human-in-the-ίοορ virtual testing of bio-integration issues associated with the prosthetic hardware. FIG. 8, for example, depicts such an architecture thai includes human in- the-loop virtual testing. The decoded cortical output 306 from

oeuroioia ing equipment. 802 (e.g., EEG, tMRI, etc.) attached to the subject is sent to a simulated prosthesis controller 804 (whereas the prosthesis controller 308 in FIG, 3 is controlling an actual piece of hardware) and a simulated sensorimotor controller 806. The simulated prosthesis controller 804 models the behavior of controller designs (and operates in a manner similar to the prosthesis controller 308 described an illustrated with respect to FIG. 3), while the sensorimotor controller 806 simulates the sensorimotor processing and integration of the centra! nervous system (CMS).

[00095] The simulated neural excitations 808 output from this controller 806 drive a set of muscles in a musculoskeletal simulation 810, while the simulated actuator torques 812 from the prosthesis controller 804 drive the simulated prosthetic device. A single physics-based environment incorporates the musculoskeletal dynamics of the subject as well as the dynamics of the prosthetic device. This architecture allows a wealth of simulation-based testing to be performed. The overall efficacy of the prosthesis controller 804 can be tested, including stability and

performance of the control system in executing cortical commands. A number of bio-integration issues can also be evaluated. These include mechanical forces and stresses on the device and the subject in various configurations, as well as reaction forces at the body-prosthetic interfaces. In addition to facilitating design optimization of the control system and prosthetic device the architecture will, also facilitate subject training with the prosthetic system and subject-specific tuning and customization of the system prior to any physical coupling of the subject with the prosthetic hardware.

[00096] The sensorimotor controller 806 component is described in further detail below with respect to FIGs. 9 through 17. FIG. 9, for example, depicts the neural and musculoskeletal physiology associated with the motor control problem. High-level motor intent 900 from the brain 900 transmitted to the central nervous system (CNS) 904, Sensorimotor integration and control results in the low-level command of motor neurons 906 innervating individual, muscles 90S (in the somatic nervous system 909) to generate coordinated musculoskeletal 910 movement. Afferent proprioceptive signals 9.12 transmit sensory data from mechanoreceptors in the joints and tendons back to the CNS 904.

[00097] FIG. 1 depicts a system-level abstraction of FIG . 9. The

sensorimotor controller block 806 is central to the sensorimotor controller described herein. It receives task-level command input (i.e., motor intent 902) from the human operator (or brain 900) and generates low-level neural excitations 808 for musculoskeletal simulations 1004. This is a closed-loop control process where sensory data (via proprioceptive signals 1006) is fed back to the sensorimotor controller 806, which provides continual updates to the neural excitations 808 driving the musculoskeletal simulation 1.004. Thus, in one aspect, the sensorimotor controller 806 (simulator) operates by running musculoskeletal simulations on a musculoskeletal plant. The various aspects of the seiisiromotor controller S06 are described in further detail below with respect to elements 1300, 1600, and 1700 as illustrated in PIGs. 13, 1 , and 1 7, respectively.

[00098] For further understanding. FIG. 1 1 provides an illustration of an

example mu sculoskeletal plant i 100 according to the principles of the present invention. As shown in FIG, 12, the musculoskeletal plant can be divided in neuromuscular 1200 dynamics and musculoskeletal 1202 dynamics. Specifically, FIG. 12 is an illustration depicting a

neuromuscular 1200 and musculoskeletal 1202 system (feed-forward path). As shown, neural excitations 808 provide input to the activation dynamics 1206 and output 1208 of the ac tivation dynamics 1206 provides input to the contraction dynamics 1210. Further, output 1212 of the contraction dynamics 1210 provides input to the musculoskeletal Ϊ202 dynamics through the joint torques. [00099] Further details regarding the neuromuscular 1200 and musculoskeletal 1202 systems are provided be!ow. The musculoskeletal 1202 system dynamics are described by the following system of equations in configuration space. τ ~ M{q)q MqA) -h &(q% where q is the vector of n generalized or joint coordinates, r, is the vector o applied joint torques, M{q is the joint space mass matrix, b(q_fq)_s is the vector of centrifugal and Corio!is forces, and is the vector of sraviiv forces.

[000100] The neuromuscular 1202 dynamics describe the behavior of a set of r musculotendon actuators spanning the musculoskeletal system. These actuators are modeled as Hill-type acti ve state force generating units, as described by De Sapio et al. (see Literature Reference No. 18). it is assumed that the vector of r musculotendon lengths, I, can be uniquely determined from the system configuration, q. That is, — (f), As a consequence of this assumption, differential variations in i are given by,

SI ~ |·· Sq — l(q) q.

where l(q) is the muscle Jacobiaii. From the principle of virtual work, it is concluded that,

where f_T is the vector of r tendon forces. The negative sign is due to the convention of taking contractile muscle forces as positive. The .matrix of moment arms is denoted St. [000101] The behavior of the musculotendon actiiators (neurom oscular

dynamics 1200) can be divided into activation dynamics .1208 and contraction dynamics 1210. Activation dynamics 1208 refers to the process of muscle activation in response to neural excitation.. This process can be modeled by the following equation of state written in terms of the vector of r mu where % ¾ |0,i] aod t½€ [041 is the neural input. The term

time constant given bv.

(ft. — 4* 'ί-' for Us≥ it_:

for ¾i_< where _¾ and ¾ are the. activation and deactivation ti me constants, respectively.

[000102] The contraction dynamic s 1210 of the musculotendon. unit (illustrated as element 1 100 in FIG, 1 1) can be modeled as the lumped parameter system. Referring again to FIG. 1 1, the lumped parameter system 1 102 describes the configuration of forces. There is an acti ve element 1 104, a passive viscoelastic element, (in parallel) 1.106, and an elastic tendon element (in series) 1 108. The relative angle associated with the muscle fibers 1 1 10, a. is referred to as the pennation angle 1 1 1 2

[000103] This yields the following relations,

where ^· is the vector of muscle lengths and is the vector of tendon lengths. The following force equilibrium equations can be expressed, where f_A is the vector of active forces in the muscles and fp is the vector of passive forces in the muscles. Using this force equilibrium equation as well as cooslitutive relationships describing muscle forces as a function of muscle length and contraction velocity, l > a) and _ρ(ί » I^), and tendon force as a function of tendon length, _T l_y), the following equation of state in functional form can be expressed as,

l$$ ~ ¾ Q d* ¾A )>

[000104] So, for a system of r musculotendon actuators, the following system of

2r first-order state equations can be expressed as, a ~ a{u_f a) ~ {« - a), and,

where the internal states are ½ and . This is complemented by the force relationship,

fr - fr( ) ~ ^~ ½cos a - _T{¾),½).

[000105] Returning to the musculoskeletal dynamics (depicted as element 1202 in FIG. 12),

τ = M{q)q - k{q,q) - g(q\ these equations of motion can be mapped into operational or task space, as follows: where | is the dynamically consistent inverse of the task Jacobian, x is a description of task coordinates that are to be controlled, f, are the control forces that act at the task, and {$), μ^,ό.), and ρ{ Χ are the task space mass matrix, centrifugal and Coriolis force vector, and gravity force vector, respectively. See the work o Khatib in Literature Reference No. 10 for a discussion of task space, which is hereby incorporated b reference as through fully set forth herein. [000106] Given a system of r musculotendon actuators, the applied joint torques are given by, which provides the following muscle actuated task, space equations of motion,

T f_T™ Ax μ 1- p.

[000107] For control purposes, the muscie actuated task space equations of motion can be expressed as:

f8f_T ~ Xf^* p 1- p, where T represents estimates of the dynamic parameters and f^s is the control Saw. Since the term f⁷Rf_T is underdetermined it offers many solutions for/y. These solutions correspond to the kinematic redundancy associated with the task and the actuator redundancy associated with the system of musculotendon force actuators, hi the computed muscle control approach of Thelen et al., (see Literature Reference Nos. 1.9 and 20), there was only muscle redundancy that needed to be resolved. This was achieved through a static optimization step. In the task-level formulation as used herein, a static optimization is also specified to resolve both muscle redundancy and kinematic redundancy. Specifically, a set of muscle activations, «, is determined, which minimizes e||², where a are muscle activations associated with the steady state t ndon forces f_T(&) (i.e.. tendon forces at. a time when the contraction, dynamics have equilibrated and the transients have decayed). Further these activations must generate steady state tendon forces that satisfy the task motion requirements, f ria} - If fi + . ] Using a linear proportional controller, a set of neural excitations., », is computed which causes the actual activations, «, from the forward neuromuscular simulation to track , as follows:

« = e } — a), where Κ_Λ is the gain. This task-level input computed muscle control architecture is depicted in FK1. 13. FIG. 13 is an illustration of a task- level sensorimotor controller 1300 according to the principles of the present invention, where motion commands are represented in. task space. The sensorimotor controller BOO generates optimal neural excitation commands 808 that minimize muscle activation 1302, and generate motion consistent with the input motion commands. ] Further and as shown in FIG. 14, holonomic constraints 1400 can be incorporated into the system. Specifically, FIG. 14 illustrates a iunctiona! block diagram showing an abstracted representation of neural and musculoskeletal physiology. The sensorimoto controlier 806 in this aspect is a goal-oriented neiuomuscuiuar feedback controller that dri ves the constrained 1400 musculoskeletal simulation by specifying low-level neural 808 excitations that execute task-level commands. 0001.10] Thus and as shown in FIG. 15, the holonomic constraints, ( ) ~ ®, can be incorporated into the musculoskeletal dynamics. ^"The configuration space dynamics was augmented with Lagrange multipliers, 1, τ^™ M (tj) q- b {q_s q) + g {q)— Φ ¾ where Φ* is the constraint Jacobiati. The block diagram of the

constrained musculoskeletal system is depicted in FIG. 15. Specifically,

FIG. 15 is an illustration depicting a neuromuscular and constrained musculoskeletal system (feed-forward path), showing the musculoskeletal system dynamics 1500 augmented with a set of holonomic constraints 1502 and Lagrange multipliers 1504.,

[0001 11 j The approach expresses these dynamics in one of two ways. First and as shown in FIG. 16, the constrained dynamics can be represented in task space as:

J^r8^r r ^~ A{q^" L - q) - p{q}— ^{Τ Τ} (a - p) where ^β* is the constraint null space projection matrix. Task space representations were described by De Sapio et al. in Literature Reference No. 21. The centrifugal and Corioiis forces, and gravity forces projected at the constraints are fif and p respectively. For neuromuscular control purposes, this can be expressed as the following muscle actuated task space equations of motion,

J^{T T}Bf_T— Af * 4- β - p— |^ΓΦ^τ(έ¾4- ),

[000112] As before, the controller 1600 seeks a 1602, which minimizes

(i.e.. magnitude squared of the activations) to generate the neural excitations 808. Further these activations must generate steady state tendon forces that satisfy the constrained task motion requirements, as follows:

1 13] Alternately and as shown in FIG, 7, constrained d n m cs can be represented in task space as

z -~ #^T|^r(A ir- μ_ε - p ) + ^Γ(ίϊ - p) - £if %— Φ^Γ1, where ®^* is the constraint null space projection matrix, A_e.^_c p_Ci are the task/constraint space mass matrix, centrifugal and Coriolis force vector, and gravity force vector, respectively. Task space representations were further described in Literature Reference Nos. 1 and 17. The centrifugal and Coriolis forces, and gravity forces projected at the constraints are a and p respectively. Finally, 8« is the combined null space projection matrix, with respect to both task and constraints, and % is the control vector for the null space.

[0001 14] This representation expose the constraint forces and is useful for generating optimal neural excitations that track task motio under constraints and also minimizes specific internal joint reaction forces. For neuromuscular control purposes, this can be expressed as the following muscle actuated task space equations of motion,

Mf_T - Φ^τ1 - &l % - ®^T (K * l ¾ i ¾) - #^r(St [0001 15] Thus, the system now seeks H and which minimize

w_a||al^s - ¾¾|2|², Further these activations must generate steady state tendon forces onstrained task motion requirements. ] Thus, FIG, 17 is an illustration of a constrained task-level sensorimotor controller 1700 based on a reformulation of the computed muscle control approach. The system represents motion commands in task space and generates optimal neural excitation 808 commands that minimize muscle activation 1702. and aenerates motion consistent with the input motion commands in the presence of holonomic system constraints.

Claims

What is claimed is:

1. A system for controlling a torque controlled prosthetic device given motor intent inferred from neuroimaging data, the system comprising:

one or more processors and a memory, the memory having executable instructions encoded thereon, such that upon execution of the instructions, the one or more processors perform operations of:

receiving neuroimaging data of a user from a neuroimaging device;

decoding the neuroimaging data to infer spatial motion intent of the user, where the spatial motion intent includes desired motion commands of the torque controlled prosthetic device represented in a coordinate system; and

executing, with a prosthesis controller, the motion

commands as torque commands to cause the torque controlled prosthetic device to move according to the spatial motion intent of the user.

2. The system as set forth in Claim 1 , further comprising at least one torque controlled prosthetic device operably connected with the one or more processors.

3. The system as set fort in Claim 2, further comprising an operation of receiving, in the controller, sensory information regarding a current state of the prosthetic device.

4. The system, as set forth in Claim 3, wherein in executing the motion

commands, the motion commands are executed using a task

decomposition and posture decomposition, wherein the task

decomposition is a task space control and the posture decomposition is formulated as cost potential which represents a cost function.

5. The system as set forth in Claim 4, wherein in executing the motion commands, the motion commands are executed as torque commands that generate a desired task space control while minimizing the cost potential.

6. The system as set forth in Claim 3, wherein in executing the motion

commands, the motion commands are executin using a spiking neural network.

The system as set forth in Claim 6, wherein the prosthesis controller is a neuromorphie prosthesis controller and further comprises:

a neuromorphie spike encoder to represent the motion command as a set of neural spikes:

a neuromorphie motor mapper to map the neural spikes representing Cartesian displacements to neural spikes representing configuration (joint) space displacements;

a spike decoder to decode the neural spikes representing configuration (joint) space displacements and. generate a joint space command: and

a joint servo to execute the joint space command.

The system as set forth in Claim 1, further comprising operations of. receiving a model of the prosthetic device and a musculoskeletal model of the user, wherein the musculoskeletal model includes

musculoskeletal dynamics that include steady state tendon forces;

generating, with a sensorimotor controller, simulated neural excitations given the motion commands to drive a set of muscle activations in a musculoskeletal simulation; and

generating, with the prosthesis controller, simulated actuator joint torques given the motion commands to drive a simulated prosthetic device. A computer program product for controlling a torque controlled prosthetic device given motor intent inferred from neuroimaging data, the computer program product comprising:

a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions by one or more processors, the one or more processors perform operations of:

receiving neuroimaging data of a user from a neuroimaging device;

decoding the neuroimagmg data to infer spatial motion intent of the user, where the spatial motion intent includes desired motion commands of the torque controlled prosthetic device represented in a coordinate system; and

executing, with a prosthesis controller, the motion commands as torque commands to cause the torque controlled prosthetic device to move according to the spatial motion intent of , The computer program product as set forth, in Claim 9, further comprising instructions for controlling at least one torque controlled prosthetic device that is operahly connected with the one or more processors. , The computer program product as set forth in Claim 10, further comprising an operation of receiving, in the controller, sensory information regarding a current state of the prosthetic device. , The computer program product as set forth in Claim i 1 , wherein in executing the motion commands, the motion commands are executed using a task decomposition and posture decomposition, wherein the task decomposition is a task space control and the posture decomposition is formulated as a cost potential which represents a cost function.

13. The computer program product as set forth in Claim 12, wherein in

executing the motion commands, the motion commands are executed as torque commands thai generate a desired task space control while minimizing the cost potential.

14. The computer program product as set forth in Claim 1 1 , wherein in

executing the motion commands, the motion commands are executing using a spiking neural network.

15. The computer program product as set forth hi Claim 14, wherei the

prosthesis controller is a neuromorphic prosthesis controller and further comprises:

a neuromorphk spike encoder to represent the motion command as a set of neural spikes;

a neuromorphic motor mapper to map the neural spikes representing Cartesian displacements to neural spikes representing configur tion {joint} space displacements;

a spike decoder to decode the neural spikes representing configuration, (joint) space displacements and generate a joint space command; and

a joint servo to execute the joint space command.

16. The computer program product as set forth in Claim 9, further comprising operations of:

receiving a model of the prosthetic device and a musculoskeletal model of the user, wherein the musculoskeletal model includes musculoskeletal dynamics that include steady state tendon forces; generating, with a sensorimotor controller, simulated neural, excitations given the .motion commands to drive a set of muscle

activations in a musculoskeletal simulation; and

generating, with the prosthesis controller, simulated actuator joint torques given the motion commands to drive a simulated prosthetic device.

17. A method for controlling a torque controlled prosthetic device given motor intent inferred from neuroimaging data, the method comprising an act of executing instructions encoded on a memory, such that upon execution of the instructions, one or more processors perform operations of:

receiving neuroimaging data of a user from a neoroimaging device;

decoding the neuroimaging data to infe spatial motion intent of the user, where the spatial motion intent includes desired motion commands of the torque controlled prosthetic device represented in a coordinate system; and

executing, with a prosthesis controller, the motion commands as torque commands to cause the torque controlled prosthetic device to move according to the spatial motion intent of the user,

1 S, The method as set. forth in Claim 17, further comprising an act of

controlling at least one torque controlled prosthetic device that is operably connected with the one or more processors.

19, The method as set forth in Claim IS, further comprising an operation of receiving, in the controller, sensory information regarding a current state of the prosthetic device.

20. The method as set forth hi Claim 1 , wherein in executing the motion commands, the motion commands are executed using a task

decomposition and posture decomposition, wherein the task

decomposition is a task space control and the posture decomposition is formulated as a cost potential which represents a cost function.

21. The method as set forth in Claim 20, wherein in executing the motion commands, the motion commands are executed as torque commands that generate a desired task space control while minimizing the cost potential.

22. The method as set forth in Claim 1 , wherein in executing the motion commands, the motion commands are executing using a spiking neural network.

23. The method as set forth in Claim 22, wherein the prosthesis controller is a neuromorphic prosthesis controller and further comprises:

a neuromorphic spike encoder to represent the motion command as a set of neural spikes;

a neuroiTiorphic motor mapper to map the neural spikes representing Cartesian, displacements to neural spikes representing configuration (joint) space displacements;

a spike decoder to decode the neural spikes representing configuration (joint ) space displacements and generate a joint space command; and

a joint servo to execute the joint space command.

24. The method as set forth in Claim 17, further comprising operations of:

receiving a model of the prosthetic device and a musculoskeletal model of the user, wherein the musculoskeletal model includes musculoskeletal dynamics that include steady state tendon forces; generating, with a sensorimotor controller, simulated neural, excitations given the .motion commands to drive a set of muscle activations in a musculoskeletal simulation; and

generating, with the prosthesis controller, simulated actuator joint torques given the motion commands to drive a simulated prosthetic device.