WO2020061213A1 - Virtual reality training tasks used for physical therapy and physical rehabilitation - Google Patents

Virtual reality training tasks used for physical therapy and physical rehabilitation Download PDF

Info

Publication number
WO2020061213A1
WO2020061213A1 PCT/US2019/051760 US2019051760W WO2020061213A1 WO 2020061213 A1 WO2020061213 A1 WO 2020061213A1 US 2019051760 W US2019051760 W US 2019051760W WO 2020061213 A1 WO2020061213 A1 WO 2020061213A1
Authority
WO
WIPO (PCT)
Prior art keywords
challenge
physical
user
motion
physical challenge
Prior art date
Application number
PCT/US2019/051760
Other languages
French (fr)
Inventor
Yang Shen
Jacob Rosen
Original Assignee
The Regents Of The University Of California
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by The Regents Of The University Of California filed Critical The Regents Of The University Of California
Publication of WO2020061213A1 publication Critical patent/WO2020061213A1/en

Links

Classifications

    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09BEDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B5/00Electrically-operated educational appliances

Definitions

  • the present invention generally relates to providing training tasks for physical rehabilitation and, more specifically, dynamically adjusting training tasks for rehabilitation.
  • PT physical therapy
  • Orthopedic PT (48%), Neurological PT (6%), Pediatric PT(8%) Sport PT (6%); Geriatric Physical Therapy (9%) General PT (9%), Other specialty PT (3%).
  • PT may be provided for a limited time in which the patient regain partial or complete functionally or in chronic conditions has to be provided for the rest of the patient’s life.
  • An example of a life-long need for PT is in the case of stroke. The majority of stroke survivors experience hemiparesis and severe deficiency of the motor control system. It is often difficult and expensive to provide rehabilitation care due to the extensive time required for training and to the uniqueness of each patient’s response to therapy.
  • One embodiment includes a method for providing training tasks through a virtual environment.
  • the method includes steps for providing a first physical challenge for a user within the virtual environment, determining whether the user successfully completed the first physical challenge, modifying a challenge level associated with the virtual environment, and providing a second physical challenge based on the modified challenge level.
  • the first physical challenge requires a first range of motion
  • the challenge level is increased
  • the second physical challenge requires a second range of motion greater than the first range of motion
  • the first physical challenge requires the performance of a unilateral motion, when the user successfully completes the first physical challenge, the challenge level is increased, and the second physical challenge requires a bilateral motion.
  • the method further includes steps for providing a set of diagnostic tasks for setting an initial challenge level for the first physical challenge.
  • the virtual environment is provided in virtual reality.
  • modifying the challenge level includes increasing the speed of an object presented in the second physical challenge.
  • modifying the challenge level comprises determining a state of the user, and identifying a first action based on the state of the user, wherein the first action includes a modification of a plurality of modifications that can be made to the challenge level of the first physical challenge to provide the second physical challenge, wherein the process further comprises computing a reward for the second physical challenge based on the user’s performance of the second physical challenge, computing a new state of the user based on the user’s performance of the second physical challenge, and identifying a second action based on reward and the new state of the user.
  • providing the first physical challenge includes displaying an object, wherein the first action includes at least one of modifying a speed of the object, modifying a trajectory of the object, modifying a placement of the object, and modifying a size of the object.
  • One embodiment includes a non-transitory machine readable medium containing processor instructions for providing training tasks through a virtual environment, where execution of the instructions by a processor causes the processor to perform a process that comprises providing a first physical challenge for a user within the virtual environment, determining whether the user successfully completed the first physical challenge, modifying a challenge level associated with the virtual environment, and providing a second physical challenge based on the modified challenge level.
  • One embodiment includes a training device comprising a set of one or more processors, and a non-transitory machine readable medium containing processor instructions for providing training tasks through a virtual environment, where execution of the instructions by the set of processors causes the set of processors to perform a process that comprises providing a first physical challenge for a user within the virtual environment, determining whether the user successfully completed the first physical challenge, modifying a challenge level associated with the virtual environment, and providing a second physical challenge based on the modified challenge level.
  • Figure 1 is an illustration of a system of various devices that may perform one or more processes to provide training tasks for physical rehabilitation in accordance with various embodiments of the invention.
  • Figure 2 illustrates a training element for providing training tasks in accordance with an embodiment of the invention.
  • Figure 3 illustrates a training application for providing training tasks in accordance with an embodiment of the invention.
  • Figure 4 is a flowchart that conceptually illustrates a process for providing training tasks in accordance with an embodiment of the invention.
  • Figure 5 is a flowchart that conceptually illustrates a process for providing training tasks using reinforcement learning in accordance with an embodiment of the invention.
  • Training tasks in accordance with many embodiments of the invention are provided in a virtual (or augmented) reality gaming environment, allowing a user to perform a set of motions with their bodies that can be evaluated and measured for scoring within the gaming environment.
  • Many different types of games can be provided in accordance with different embodiments of the invention.
  • Network 100 includes a communications network 160.
  • the communications network 160 is a network such as the Internet that allows devices connected to the network 160 to communicate with other connected devices.
  • Server systems 1 10, 140, and 170 are connected to the network 160.
  • Each of the server systems 1 10, 140, and 170 is a group of one or more server computer systems communicatively connected to one another via internal networks that execute processes that provide cloud services to users over the network 160.
  • cloud services are one or more applications that are executed by one or more server systems to provide data and/or executable applications to devices over a network.
  • the server systems 1 10, 140, and 170 are shown each having three servers connected via an internal network. However, the server systems 1 10, 140 and 170 may include any number of servers and any additional number of server systems may be connected to the network 160 to provide cloud services including, but not limited to, virtualized server systems.
  • processes for providing training tasks and virtual gaming environments are provided by one or more software applications executing on a single server system and/or a group of server systems communicating over network 160.
  • server systems in accordance with certain embodiments of the invention can communicate and save user performance data at the clinic of all the user treated by a clinic, communicate and save the entire database of all the users utilizing the services regardless of the clinic that they are associated with for off line data analysis and/or to primarily store and retrieval of data as well from the cloud as well as distributing software updates and local executable software.
  • Systems in accordance with a variety of embodiments of the invention can operate in a variety of different settings.
  • a patient and therapist can share the same space.
  • the therapist can operate his/her own device allowing him/her to change internal parameters of the tasks in the virtual environment (e.g., size of the object, speed of the object, location of the object, etc.) for various physical therapy tasks.
  • systems in accordance with a variety of embodiments of the invention can be used in a remote setting (e.g., a patient’s home). In such a setting, a patient can be primarily alone or with his care giver and away from the clinic and the physical therapist.
  • tasks can change primarily by an artificial intelligent (Al) agent controlling settings for the various challenges presented to the user.
  • Al artificial intelligent
  • remote interaction with the physical therapist can be enabled, in which the physical therapist may monitor a patient’s progress, change the internal parameters of tasks in conjunction with the Al agent, and/or conduct an audiovisual teleconference with specific users.
  • peer to peer remote communication can be enabled in which two users can interact by playing a game one against each other.
  • Al agents in accordance with a number of embodiments of the invention may intervene by adjusting the internal parameters for each user according to their one performance, even if their peer challenges them beyond their performance envelope.
  • the tasks along with the Al agent are agnostic to the display as well as the modality used for tracking the user motion.
  • Visual information can be displayed to the user through various displays, such as (but not limited to) an immersed head mounted display (HMD), goggles, as well as any kind of screen (TV, monitor, projector).
  • HMD immersed head mounted display
  • TV monitor, projector
  • tracking user motion and presenting an avatar can be performed with various motion capturing systems including (but not limited to) markers, optic method, electromagnetic sensors, and/or robotic systems.
  • the personal devices 180 are shown as desktop computers that are connected via a conventional“wired” connection to the network 160.
  • the personal device 180 may be a desktop computer, a laptop computer, a smart television, an entertainment gaming console, or any other device that connects to the network 160 via a“wired” or“wireless” network connection.
  • the mobile device 120 connects to network 160 using a wireless connection.
  • a wireless connection is a connection that uses Radio Frequency (RF) signals, Infrared signals, or any other form of wireless signaling to connect to the network 160.
  • RF Radio Frequency
  • the mobile device 120 is a mobile telephone.
  • mobile device 120 may be a mobile phone, Personal Digital Assistant (PDA), a tablet, a smartphone, a virtual reality headset, an augmented reality headset, a mixed reality headset or any other type of device that connects to network 160 via wireless connection without departing from this invention.
  • the processes for providing training tasks are performed by the user device.
  • an application being executed by the user device may capture or obtain the two or more input images and transmit the captured image(s) to a server system that performs the processes for providing training tasks.
  • the user device may include a camera or some other image capture device that captures the image.
  • Training elements in accordance with many embodiments of the invention can include (but are not limited to) one or more of mobile devices, motion capture systems, exoskeleton robots, cameras, and computers.
  • Training element 200 includes processor 205, motion capture device 210, network interface 215, and memory 220.
  • Training applications in accordance with a number of embodiments of the invention can generate separate graphical user interfaces (GUI) for display to a user and a therapist.
  • GUI graphical user interfaces
  • User GUIs in accordance with various embodiments of the invention can depict an avatar representing the user in virtual reality along with a specific task.
  • each task is presented with a set of numerical indicators regarding the status of the task and the user’s performance.
  • therapist GUIs can allow a human therapist to control various aspect of the task (e.g., the location of the task with respect of the operator, the speed and size of the ball or any target object, the time duration or the number of repetitions of each task, and/or the position and orientation of the virtual camera presenting the task to the user).
  • the processor 205 can include (but is not limited to) a processor, microprocessor, controller, or a combination of processors, microprocessor, and/or controllers that performs instructions stored in the memory 220 to manipulate data stored in the memory. Processor instructions can configure the processor 205 to perform processes in accordance with certain embodiments of the invention.
  • Motion capture device 210 can include any of a variety of components for capturing motion, such as (but not limited to) motion sensors, exoskeleton robots, cameras, accelerometers, and gyroscopes.
  • Network interface 215 allows training element 200 to transmit and receive data over a network based upon the instructions performed by processor 205.
  • Memory 220 includes a training application 225, user data 230, model data 235, and virtualization engine 240.
  • Training applications in accordance with several embodiments of the invention are used provide training tasks (e.g., through a video game or other medium) to assist a user during rehabilitation and/or physical training.
  • training applications can provide training tasks through virtual reality systems.
  • An example of a training application in accordance with an embodiment of the invention is described in greater detail below.
  • training applications can operate with patient data to determine a reasonable challenge level for a user, such that the training tasks are neither too simple, nor too difficult for the user, allowing the user to be engaged with the tasks, without causing excessive levels of frustration or anxiety.
  • training applications can constantly change the operational point to challenge a user’s skill by increasing the challenge automatically.
  • User data in accordance with some embodiments of the invention can include diagnostic data that is gathered during a preliminary session that measures a patient’s current capabilities.
  • patient data can include (but is not limited to) medical history data, demographic data, and/or medical diagnoses.
  • model data can include (but is not limited to) parameters, weights, and/or data structures for models.
  • Models in accordance with certain embodiments of the invention can be trained and used to provide an Al agent that can intelligently adjust task settings for challenges based on a user’s performance.
  • Training applications in accordance with various embodiments of the invention can incorporate training tasks into a challenging and entertaining game based on a patient’s capabilities that is provided via a virtualization engine.
  • Virtualization engines in accordance with certain embodiments of the invention can include a game engine that can provide games in a variety of formats such as (but not limited to) virtual reality, console gaming, and augmented reality.
  • training element 200 Although a specific example of a training element 200 is illustrated in Figure 2, any of a variety of training elements can be utilized to perform processes similar to those described herein as appropriate to the requirements of specific applications in accordance with embodiments of the invention.
  • Training application 300 includes motion analysis engine 305, scoring engine 310, challenge adaptation engine 315, and output engine 320.
  • motion evaluation applications operate on mobile devices, improving accessibility, reducing costs, and allowing a user to quickly and efficiently evaluate motion of an individual.
  • motion analysis engines can be used to evaluate a user’s performance of specified tasks in the context of a game.
  • Motion analysis engines in accordance with various embodiments of the invention can perform a variety of processes to evaluate a user’s performance based on the data collected from the motion capturing system.
  • Motion analysis engines in accordance with a number of embodiments of the invention evaluate various characteristics of a user’s performance of a set of training tasks, such as (but not limited) one or more of the joints’ range of motion, angular and linear velocity, angular and linear acceleration, angular and linear jerk (third time derivative of the position or the angle) as well as higher derivatives, and/or completion time of a user’s motions.
  • motion analysis engines can assess motor control functionality utilizing know clinical measures and/or administer the evaluation and assess the quantitative results autonomously.
  • Scoring engines in accordance with various embodiments of the invention can be used to score the performance of a user’s motions based on the results of a motion analysis engine. Scoring engines in accordance with various embodiments of the invention can generate scores based on various types of data. For example, scoring engines in accordance with many embodiments of the invention can use data collected during an established and well-defined clinical protocol (e.g., the Fugl-Meyer protocol), where numerical evaluations can be assigned to each one of the motions, and a total score is a sum of all the individual motions scores. In a variety of embodiments, scores can be generated based on data collected from the tasks themselves that are not necessarily standardized as the well-established clinical measures.
  • an established and well-defined clinical protocol e.g., the Fugl-Meyer protocol
  • Scoring engines in accordance with numerous embodiments of the invention can measure performance based on a variety of factors, including (but not limited to) the timing of the performance of the task with objectives provided in a game, a user’s range of motion in relation to an expected range of motion, and/or a user’s speed in performing a set of actions.
  • Challenge adaptation engines in accordance with many embodiments of the invention can be used to adapt the challenges provided to a user based on a user’s medical history, initial assessment, and/or previous performance accomplished by using specific tasks, etc. Challenges can change during the therapeutic session or between the sessions.
  • challenge adaptation engines can be trained using reinforcement learning.
  • Reinforcement learning trains an agent (or model) to perform actions that maximize rewards in a particular environment, given a particular situation (or state).
  • a reinforcement learning problem can be defined in terms of a state space (i.e. , the set of possible states) and an action space (i.e., the set of possible actions).
  • the goal is to repetitively train the neural network, so that information about the current state can be fed into the model, along with the previous reward and action, in order to output an optimal action that leads to a maximized "cumulative" reward in the long run future.
  • challenge adaptation engines in accordance with various embodiments of the invention can be considered an agent, with the human user being treated as the environment.
  • State in accordance with various embodiments of the invention can include information about the user’s current situation, such as (but not limited to) previous performance, expected range of motion, energy levels, etc.
  • Actions in accordance with numerous embodiments of the invention can include the various tasks for the user to perform and/or ways in which a challenge adaptation engine can modify the difficulty of a task.
  • rewards can include scores or measures of the challenge adaptation engines performance.
  • challenge adaptation engines in accordance with several embodiments of the invention can be rewarded when a user performs an action that exceeds their expected range and/or when the user performs a challenge successfully.
  • challenge adaptation engines can be penalized when a user is unable to perform a challenge (an indication that the difficulty level is too challenging), when a user succeeds at too many challenges (an indication that the difficulty level is too easy), and/or when a user quits.
  • a reinforcement learning process is described below.
  • a user arm's range of motion (ROM) x.
  • Challenge adaptation engine performs a first action a1 to change a setting in the game to increase the speed of a moving object in the game.
  • the adaptation engine can receive a reward (e.g., +5).
  • the state s2 has now been updated the ROM to x’.
  • Challenge adaptation engine then performs a second action a2 to change a setting in the game to move the object farther away from the user. In this case, the user fails to reach the object, so the adaptation engine can receive a negative reward (e.g., -2).
  • This places the environment in a new state s3, where the ROM x", which is adjusted based on the successful performance of the first challenge and the failed attempt at the second challenge.
  • Output engines in accordance with several embodiments of the invention can provide a variety of outputs to a user, including (but not limited to) physical therapy exercises and/or providing a virtual environment (e.g., in virtual reality, augmented reality, on a personal computer, mobile device, etc.) with scores and/or objectives.
  • output engines provide a set of graphical user interfaces (GUIs) to allow an administrator (e.g., a physical therapist, engineer, doctor, etc.) to manually modify and monitor parameters for individual games, tasks, and/or patients.
  • GUIs graphical user interfaces
  • a specific task could be selected by clicking the button next to its name; then arm moving mode is selected in“ARMS CONFIG” frame:“unilateral” (use single arm to accomplish the task), or“bilateral” (use two arms simultaneously, in a mirror-image fashion), while in the GUI for engineers, exoskeleton parameters as well as the exoskeleton-PC communication could be set and monitored.
  • Training tasks in accordance with many embodiments of the invention can be provided by a human therapist or through software (e.g., through an artificial intelligence (Al) agent).
  • Process 400 provides (405) a physical challenge to a user.
  • processes can provide an initial set of tasks to generate a baseline diagnostic to measure and visualize a user’s capabilities, such as (but not limited to) their range of motion (ROM) of various degrees of freedom (DOFs) in each joint for one or more target body parts (e.g., arm, hand, leg, foot, spine, etc.).
  • Baseline diagnostics in accordance with certain embodiments of the invention can be used to initialize the parameters for the physical challenge.
  • baseline diagnostics can be performed once for each patient, or at each therapeutic session.
  • Processes in accordance with numerous embodiments of the invention can calculate a reachable workspace within a three-dimensional space, which can be used to adjust and/or select tasks (e.g., the placement, trajectory, and/or motion of objects within space) presented to a user as part of training or physical therapy.
  • the workspace for a particular individual may be a subset of the operational workspace of a healthy individual due to injury or some other disorder.
  • Physical challenges can include any of a variety of different challenges to focus on the use and coordination of various user motions.
  • a human arm has generally seven (7) degrees of freedom (DOFs): shoulder abduction/adduction, shoulder flexion/extension, shoulder internal/external rotation, elbow flexion/extension, elbow rotation (supination/pronation), wrist flexion/extension and wrist ulnar/radial deviation.
  • physical challenges can be provided through a series of events in a virtual environment, providing goals and purpose for each motion. Training tasks (and associated game elements) in accordance with a number of embodiments of the invention are used to for measure and improve the performance of such tasks.
  • training tasks can be developed to map a patients’ high-dimensional movements to a training task with a lower-dimension requirement, in order to train single joint mobility and alleviate abnormal multiple joint musculoskeletal synergies.
  • training tasks are categorized on a variety of characteristics, which can be used to measure different elements of a user’s progression.
  • some training tasks can be static or dynamic, depending on whether a target object (e.g., a ball that a user must touch) is moving or not. Users can then be instructed to interact with such objects in a variety of ways, including (but not limited to) touching, reaching, grasping, translating, placing, releasing, kicking and/or creating any virtual body contact with the object.
  • Training tasks in accordance with some embodiments of the invention are also classified by dimensionality (e.g., 1 D, 2D, and 3D motion), depending on how many dimensions the end effector is asked to move in, in order to accomplish the task.
  • dimensionality e.g., 1 D, 2D, and 3D motion
  • training tasks can be categorized as unilateral or bilateral, based on whether one or both sides of the body (e.g., arms and/or legs) are required to perform an action.
  • bilateral training tasks can further be developed based on whether the limbs are used in a mirror-image fashion or independently of each other.
  • Process 400 determines (410) whether the user successfully performs the physical challenge.
  • processes can determine successful performance through one or more sensors that can measure the positions and range of various body parts over time. Determining successful performance in accordance with many embodiments of the invention can be performed using any of a variety of systems, including (but not limited to) exoskeleton robots, image-based processing, and mobile devices including sensors such as, but not limited to, accelerometers, gyroscopes, and cameras.
  • process 400 determines that the user was unable to successfully perform the physical challenge, process 400 calculates (415) a“miss”, or an error factor, for the physical challenge.
  • Process 420 modifies a challenge level for the physical challenge accordingly.
  • modifying a challenge can include (but is not limited to) one or more of increasing/decreasing the speed of a moving object, changing the static location of an object, modifying the trajectory of a dynamic object with respect to the operational workspace of a user, and/or increasing/decreasing the size of an object.
  • the challenge level is increased.
  • Modified physical challenges in accordance with a number of embodiments of the invention can include (but are not limited to) changing the required range for successful completion of a challenge, changing the requisite speed for performing a challenge, modifying a complexity of a challenge (e.g., whether an action is to be performed with unilaterally or bilaterally, whether bilateral actions are performed in a mirror image fashion or with independent motions of both arms, etc.)
  • Process 500 determines (505) the state of a user. State in accordance with various embodiments of the invention can include information about the user’s current situation, such as (but not limited to) previous performance, expected range of motion, energy levels, etc.
  • Process 500 identifies (510) an action to perform based on the state.
  • Actions (or training tasks) in accordance with numerous embodiments of the invention can include different challenge types and/or modified settings for the challenges to be presented to the user.
  • Modified settings can include (but are not limited to) the placement of an object in relation to a reachable workspace, modifying the speed of a moving object, and/or decreasing a size of an object.
  • Process 500 presents (515) a challenge to the user based on the identified action.
  • Challenges in accordance with a variety of embodiments of the invention can include various games that can be performed in virtual reality settings, where the type of challenge presented and/or the settings (e.g., speed, range, difficulty, etc.) for the challenges can be varied based on the identified action.
  • challenges can be designed to get a user to perform any of a variety of actions including (but not limited to) one or more of touching, reaching, grasping, translating, placing, releasing, kicking and/or creating any virtual body contact with an object.
  • Process 500 computes (520) a reward based on a user’s performance of the presented challenge.
  • rewards can include scores or measures of the challenge adaptation engines performance.
  • processes in accordance with several embodiments of the invention can compute a positive reward when a user performs an action that exceeds their expected range and/or when the user performs a challenge successfully.
  • processes can compute a negative reward when a user is unable to perform a challenge, when a user succeeds at too many challenges, and/or when a user quits.
  • Process 500 computes (525) a new state based on the user’s performance of the presented challenge. Updates to state can include, but are not limited to, changes in a user’s range of motion (ROM), flexibility, quickness, etc. Process 500 then returns to 510 to compute a next action based on the updated state and the previous reward.
  • Updates to state can include, but are not limited to, changes in a user’s range of motion (ROM), flexibility, quickness, etc.
  • Process 500 then returns to 510 to compute a next action based on the updated state and the previous reward.
  • steps may be executed or performed in any order or sequence not limited to the order and sequence shown and described.
  • some of the above steps may be executed or performed substantially simultaneously where appropriate or in parallel to reduce latency and processing times.
  • one or more of the above steps may be omitted.

Landscapes

  • Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Physics & Mathematics (AREA)
  • Educational Administration (AREA)
  • Educational Technology (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Rehabilitation Tools (AREA)

Abstract

Systems and methods for providing training tasks for physical rehabilitation are described herein. Training tasks can be provided in a virtual (or augmented) reality environment, allowing a user to perform a set of motions with their bodies that can be evaluated and measured for scoring within the virtual environment. Many different types of games can be used to incorporate training tasks. In a variety of embodiments, patient performance can be assessed using clinical protocol as well as kinematic and dynamics data collected by a motion capture system. Challenge levels of a task in accordance with many embodiments of the invention can be changed to trigger an improvement of the patient's skill level.

Description

Virtual Reality Training Tasks Used For Physical Therapy and Physical
Rehabilitation
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] The current application claims the benefit of and priority under 35 U.S.C. 1 19(e) to U.S. Provisional Patent Application Serial No. 62/732,736, entitled“Virtual Reality Training Tasks Used For Stroke Rehabilitation”, filed September 18, 2018, the contents of which are incorporated herein by reference in its entirety for all purposes.
FIELD OF THE INVENTION
[0002] The present invention generally relates to providing training tasks for physical rehabilitation and, more specifically, dynamically adjusting training tasks for rehabilitation.
BACKGROUND
[0003] Approximately 23,000,000 individuals in the US alone suffer from physical disability that required physical therapy. Excluding Speech and Auditory therapy (10%), the distribution of the remaining physical therapy (PT) services (90%) defining the target population of this inventions includes: Orthopedic PT (48%), Neurological PT (6%), Pediatric PT(8%) Sport PT (6%); Geriatric Physical Therapy (9%) General PT (9%), Other specialty PT (3%). PT may be provided for a limited time in which the patient regain partial or complete functionally or in chronic conditions has to be provided for the rest of the patient’s life. An example of a life-long need for PT is in the case of stroke. The majority of stroke survivors experience hemiparesis and severe deficiency of the motor control system. It is often difficult and expensive to provide rehabilitation care due to the extensive time required for training and to the uniqueness of each patient’s response to therapy.
SUMMARY OF THE INVENTION
[0004] Systems and methods for providing and modifying challenge tasks in accordance with embodiments of the invention are illustrated. One embodiment includes a method for providing training tasks through a virtual environment. The method includes steps for providing a first physical challenge for a user within the virtual environment, determining whether the user successfully completed the first physical challenge, modifying a challenge level associated with the virtual environment, and providing a second physical challenge based on the modified challenge level.
[0005] In a further embodiment, the first physical challenge requires a first range of motion, when the user successfully completes the first physical challenge, the challenge level is increased, and the second physical challenge requires a second range of motion greater than the first range of motion.
[0006] In still another embodiment, the first physical challenge requires the performance of a unilateral motion, when the user successfully completes the first physical challenge, the challenge level is increased, and the second physical challenge requires a bilateral motion.
[0007] In a still further embodiment, the method further includes steps for providing a set of diagnostic tasks for setting an initial challenge level for the first physical challenge.
[0008] In yet another embodiment, the virtual environment is provided in virtual reality.
[0009] In a yet further embodiment, modifying the challenge level includes increasing the speed of an object presented in the second physical challenge.
[0010] In another additional embodiment, modifying the challenge level comprises determining a state of the user, and identifying a first action based on the state of the user, wherein the first action includes a modification of a plurality of modifications that can be made to the challenge level of the first physical challenge to provide the second physical challenge, wherein the process further comprises computing a reward for the second physical challenge based on the user’s performance of the second physical challenge, computing a new state of the user based on the user’s performance of the second physical challenge, and identifying a second action based on reward and the new state of the user.
[0011] In a further additional embodiment, providing the first physical challenge includes displaying an object, wherein the first action includes at least one of modifying a speed of the object, modifying a trajectory of the object, modifying a placement of the object, and modifying a size of the object.
[0012] One embodiment includes a non-transitory machine readable medium containing processor instructions for providing training tasks through a virtual environment, where execution of the instructions by a processor causes the processor to perform a process that comprises providing a first physical challenge for a user within the virtual environment, determining whether the user successfully completed the first physical challenge, modifying a challenge level associated with the virtual environment, and providing a second physical challenge based on the modified challenge level.
[0013] One embodiment includes a training device comprising a set of one or more processors, and a non-transitory machine readable medium containing processor instructions for providing training tasks through a virtual environment, where execution of the instructions by the set of processors causes the set of processors to perform a process that comprises providing a first physical challenge for a user within the virtual environment, determining whether the user successfully completed the first physical challenge, modifying a challenge level associated with the virtual environment, and providing a second physical challenge based on the modified challenge level.
[0014] Additional embodiments and features are set forth in part in the description that follows, and in part will become apparent to those skilled in the art upon examination of the specification or may be learned by the practice of the invention. A further understanding of the nature and advantages of the present invention may be realized by reference to the remaining portions of the specification and the drawings, which forms a part of this disclosure.
BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The description and claims will be more fully understood with reference to the following figures and data graphs, which are presented as exemplary embodiments of the invention and should not be construed as a complete recitation of the scope of the invention.
[0016] Figure 1 is an illustration of a system of various devices that may perform one or more processes to provide training tasks for physical rehabilitation in accordance with various embodiments of the invention.
[0017] Figure 2 illustrates a training element for providing training tasks in accordance with an embodiment of the invention.
[0018] Figure 3 illustrates a training application for providing training tasks in accordance with an embodiment of the invention. [0019] Figure 4 is a flowchart that conceptually illustrates a process for providing training tasks in accordance with an embodiment of the invention.
[0020] Figure 5 is a flowchart that conceptually illustrates a process for providing training tasks using reinforcement learning in accordance with an embodiment of the invention.
DETAILED DESCRIPTION
[0021] Turning now to the drawings, systems and methods for providing training tasks for rehabilitation in accordance with embodiments of the invention are described below. Training tasks in accordance with many embodiments of the invention are provided in a virtual (or augmented) reality gaming environment, allowing a user to perform a set of motions with their bodies that can be evaluated and measured for scoring within the gaming environment. Many different types of games can be provided in accordance with different embodiments of the invention.
Systems for Providing Training Tasks
Training System
[0022] A system that may perform one or more processes to provide training tasks for physical rehabilitation in accordance with some embodiments of the invention is shown in Figure 1 . Training tasks in accordance with various embodiments of the invention can be provided through such a system locally or remotely (e.g., Telehealth). Network 100 includes a communications network 160. The communications network 160 is a network such as the Internet that allows devices connected to the network 160 to communicate with other connected devices. Server systems 1 10, 140, and 170 are connected to the network 160. Each of the server systems 1 10, 140, and 170 is a group of one or more server computer systems communicatively connected to one another via internal networks that execute processes that provide cloud services to users over the network 160. For purposes of this discussion, cloud services are one or more applications that are executed by one or more server systems to provide data and/or executable applications to devices over a network. The server systems 1 10, 140, and 170 are shown each having three servers connected via an internal network. However, the server systems 1 10, 140 and 170 may include any number of servers and any additional number of server systems may be connected to the network 160 to provide cloud services including, but not limited to, virtualized server systems. In accordance with various embodiments of this invention, processes for providing training tasks and virtual gaming environments are provided by one or more software applications executing on a single server system and/or a group of server systems communicating over network 160. For example, server systems in accordance with certain embodiments of the invention can communicate and save user performance data at the clinic of all the user treated by a clinic, communicate and save the entire database of all the users utilizing the services regardless of the clinic that they are associated with for off line data analysis and/or to primarily store and retrieval of data as well from the cloud as well as distributing software updates and local executable software.
[0023] Systems in accordance with a variety of embodiments of the invention can operate in a variety of different settings. For example, in a clinical setting, a patient and therapist can share the same space. The therapist can operate his/her own device allowing him/her to change internal parameters of the tasks in the virtual environment (e.g., size of the object, speed of the object, location of the object, etc.) for various physical therapy tasks. In another example, systems in accordance with a variety of embodiments of the invention can be used in a remote setting (e.g., a patient’s home). In such a setting, a patient can be primarily alone or with his care giver and away from the clinic and the physical therapist. In numerous embodiments, tasks can change primarily by an artificial intelligent (Al) agent controlling settings for the various challenges presented to the user. In numerous embodiments, remote interaction with the physical therapist can be enabled, in which the physical therapist may monitor a patient’s progress, change the internal parameters of tasks in conjunction with the Al agent, and/or conduct an audiovisual teleconference with specific users. In numerous embodiments, peer to peer remote communication can be enabled in which two users can interact by playing a game one against each other. Al agents in accordance with a number of embodiments of the invention may intervene by adjusting the internal parameters for each user according to their one performance, even if their peer challenges them beyond their performance envelope. [0024] In numerous embodiments, the tasks along with the Al agent are agnostic to the display as well as the modality used for tracking the user motion. Visual information can be displayed to the user through various displays, such as (but not limited to) an immersed head mounted display (HMD), goggles, as well as any kind of screen (TV, monitor, projector). Furthermore, tracking user motion and presenting an avatar (a virtual representation of the user in VR) can be performed with various motion capturing systems including (but not limited to) markers, optic method, electromagnetic sensors, and/or robotic systems.
[0025] Users may use personal devices and that connect to the network to perform processes for providing training tasks in accordance with various embodiments of the invention. In the illustrated embodiment, the personal devices 180 are shown as desktop computers that are connected via a conventional“wired” connection to the network 160. However, the personal device 180 may be a desktop computer, a laptop computer, a smart television, an entertainment gaming console, or any other device that connects to the network 160 via a“wired” or“wireless” network connection. The mobile device 120 connects to network 160 using a wireless connection. A wireless connection is a connection that uses Radio Frequency (RF) signals, Infrared signals, or any other form of wireless signaling to connect to the network 160. In Figure 1 , the mobile device 120 is a mobile telephone. However, mobile device 120 may be a mobile phone, Personal Digital Assistant (PDA), a tablet, a smartphone, a virtual reality headset, an augmented reality headset, a mixed reality headset or any other type of device that connects to network 160 via wireless connection without departing from this invention. In accordance with some embodiments of the invention, the processes for providing training tasks are performed by the user device. In several other embodiments, an application being executed by the user device may capture or obtain the two or more input images and transmit the captured image(s) to a server system that performs the processes for providing training tasks. In accordance with a number of embodiments where one or more of the images is captured by the user device, the user device may include a camera or some other image capture device that captures the image.
[0026] As can readily be appreciated the specific computing system used to capture images and/or processing images to provide training tasks in a virtual environment is largely dependent upon the requirements of a given application and should not be considered as limited to any specific computing system(s) implementation.
Training Element
[0027] An example of a processing system in a device that executes instructions to perform processes that provide interaction with other devices connected to the network as shown in Figure 1 and/or for providing training tasks in accordance with various embodiments of the invention is shown in Figure 2. Training elements in accordance with many embodiments of the invention can include (but are not limited to) one or more of mobile devices, motion capture systems, exoskeleton robots, cameras, and computers. Training element 200 includes processor 205, motion capture device 210, network interface 215, and memory 220. Training applications in accordance with a number of embodiments of the invention can generate separate graphical user interfaces (GUI) for display to a user and a therapist. User GUIs in accordance with various embodiments of the invention can depict an avatar representing the user in virtual reality along with a specific task. In many embodiments, each task is presented with a set of numerical indicators regarding the status of the task and the user’s performance. In various embodiments, therapist GUIs can allow a human therapist to control various aspect of the task (e.g., the location of the task with respect of the operator, the speed and size of the ball or any target object, the time duration or the number of repetitions of each task, and/or the position and orientation of the virtual camera presenting the task to the user).
[0028] One skilled in the art will recognize that a particular training element may include other components that are omitted for brevity without departing from this invention. The processor 205 can include (but is not limited to) a processor, microprocessor, controller, or a combination of processors, microprocessor, and/or controllers that performs instructions stored in the memory 220 to manipulate data stored in the memory. Processor instructions can configure the processor 205 to perform processes in accordance with certain embodiments of the invention. Motion capture device 210 can include any of a variety of components for capturing motion, such as (but not limited to) motion sensors, exoskeleton robots, cameras, accelerometers, and gyroscopes. Network interface 215 allows training element 200 to transmit and receive data over a network based upon the instructions performed by processor 205.
[0029] Memory 220 includes a training application 225, user data 230, model data 235, and virtualization engine 240.Training applications in accordance with several embodiments of the invention are used provide training tasks (e.g., through a video game or other medium) to assist a user during rehabilitation and/or physical training. In several embodiments, training applications can provide training tasks through virtual reality systems. An example of a training application in accordance with an embodiment of the invention is described in greater detail below. In numerous embodiments, training applications can operate with patient data to determine a reasonable challenge level for a user, such that the training tasks are neither too simple, nor too difficult for the user, allowing the user to be engaged with the tasks, without causing excessive levels of frustration or anxiety. In some embodiments, training applications can constantly change the operational point to challenge a user’s skill by increasing the challenge automatically.
[0030] User data in accordance with some embodiments of the invention can include diagnostic data that is gathered during a preliminary session that measures a patient’s current capabilities. In certain embodiments, patient data can include (but is not limited to) medical history data, demographic data, and/or medical diagnoses.
[0031] In a number of embodiments, model data can include (but is not limited to) parameters, weights, and/or data structures for models. Models in accordance with certain embodiments of the invention can be trained and used to provide an Al agent that can intelligently adjust task settings for challenges based on a user’s performance.
[0032] Training applications in accordance with various embodiments of the invention can incorporate training tasks into a challenging and entertaining game based on a patient’s capabilities that is provided via a virtualization engine. Virtualization engines in accordance with certain embodiments of the invention can include a game engine that can provide games in a variety of formats such as (but not limited to) virtual reality, console gaming, and augmented reality.
[0033] Although a specific example of a training element 200 is illustrated in Figure 2, any of a variety of training elements can be utilized to perform processes similar to those described herein as appropriate to the requirements of specific applications in accordance with embodiments of the invention.
Training Application
[0034] A training application for providing training tasks in accordance with an embodiment of the invention is illustrated in Figure 3. Training application 300 includes motion analysis engine 305, scoring engine 310, challenge adaptation engine 315, and output engine 320. In many embodiments, motion evaluation applications operate on mobile devices, improving accessibility, reducing costs, and allowing a user to quickly and efficiently evaluate motion of an individual.
[0035] In a variety of embodiments, motion analysis engines can be used to evaluate a user’s performance of specified tasks in the context of a game. Motion analysis engines in accordance with various embodiments of the invention can perform a variety of processes to evaluate a user’s performance based on the data collected from the motion capturing system. Motion analysis engines in accordance with a number of embodiments of the invention evaluate various characteristics of a user’s performance of a set of training tasks, such as (but not limited) one or more of the joints’ range of motion, angular and linear velocity, angular and linear acceleration, angular and linear jerk (third time derivative of the position or the angle) as well as higher derivatives, and/or completion time of a user’s motions. In a number of embodiments, motion analysis engines can assess motor control functionality utilizing know clinical measures and/or administer the evaluation and assess the quantitative results autonomously.
[0036] Scoring engines in accordance with various embodiments of the invention can be used to score the performance of a user’s motions based on the results of a motion analysis engine. Scoring engines in accordance with various embodiments of the invention can generate scores based on various types of data. For example, scoring engines in accordance with many embodiments of the invention can use data collected during an established and well-defined clinical protocol (e.g., the Fugl-Meyer protocol), where numerical evaluations can be assigned to each one of the motions, and a total score is a sum of all the individual motions scores. In a variety of embodiments, scores can be generated based on data collected from the tasks themselves that are not necessarily standardized as the well-established clinical measures. In some such embodiments, a percentage value with respect to health performance can be provided along with more detailed indicators. Scoring engines in accordance with numerous embodiments of the invention can measure performance based on a variety of factors, including (but not limited to) the timing of the performance of the task with objectives provided in a game, a user’s range of motion in relation to an expected range of motion, and/or a user’s speed in performing a set of actions.
[0037] Challenge adaptation engines in accordance with many embodiments of the invention can be used to adapt the challenges provided to a user based on a user’s medical history, initial assessment, and/or previous performance accomplished by using specific tasks, etc. Challenges can change during the therapeutic session or between the sessions.
[0038] In certain embodiments, challenge adaptation engines can be trained using reinforcement learning. Reinforcement learning trains an agent (or model) to perform actions that maximize rewards in a particular environment, given a particular situation (or state). A reinforcement learning problem can be defined in terms of a state space (i.e. , the set of possible states) and an action space (i.e., the set of possible actions). In many cases, the goal is to repetitively train the neural network, so that information about the current state can be fed into the model, along with the previous reward and action, in order to output an optimal action that leads to a maximized "cumulative" reward in the long run future.
[0039] For example, challenge adaptation engines in accordance with various embodiments of the invention can be considered an agent, with the human user being treated as the environment. State in accordance with various embodiments of the invention can include information about the user’s current situation, such as (but not limited to) previous performance, expected range of motion, energy levels, etc. Actions in accordance with numerous embodiments of the invention can include the various tasks for the user to perform and/or ways in which a challenge adaptation engine can modify the difficulty of a task. In several embodiments, rewards can include scores or measures of the challenge adaptation engines performance. For example, challenge adaptation engines in accordance with several embodiments of the invention can be rewarded when a user performs an action that exceeds their expected range and/or when the user performs a challenge successfully. In a variety of embodiments, challenge adaptation engines can be penalized when a user is unable to perform a challenge (an indication that the difficulty level is too challenging), when a user succeeds at too many challenges (an indication that the difficulty level is too easy), and/or when a user quits.
[0040] An example of a reinforcement learning process is described below. From an initial state s1 , a user arm's range of motion (ROM) = x. Challenge adaptation engine performs a first action a1 to change a setting in the game to increase the speed of a moving object in the game. When the user successfully touches the object, the adaptation engine can receive a reward (e.g., +5). Since the user successfully touched the object, the state s2 has now been updated the ROM to x’. Challenge adaptation engine then performs a second action a2 to change a setting in the game to move the object farther away from the user. In this case, the user fails to reach the object, so the adaptation engine can receive a negative reward (e.g., -2). This places the environment in a new state s3, where the ROM = x", which is adjusted based on the successful performance of the first challenge and the failed attempt at the second challenge.
[0041] Output engines in accordance with several embodiments of the invention can provide a variety of outputs to a user, including (but not limited to) physical therapy exercises and/or providing a virtual environment (e.g., in virtual reality, augmented reality, on a personal computer, mobile device, etc.) with scores and/or objectives. In some embodiments, output engines provide a set of graphical user interfaces (GUIs) to allow an administrator (e.g., a physical therapist, engineer, doctor, etc.) to manually modify and monitor parameters for individual games, tasks, and/or patients. For example, in a GUI for therapists in accordance with many embodiments of the invention, a specific task could be selected by clicking the button next to its name; then arm moving mode is selected in“ARMS CONFIG” frame:“unilateral” (use single arm to accomplish the task), or“bilateral” (use two arms simultaneously, in a mirror-image fashion), while in the GUI for engineers, exoskeleton parameters as well as the exoskeleton-PC communication could be set and monitored.
[0042] Although a specific example of a training application is illustrated in Figure 3, any of a variety of training applications can be utilized to perform processes similar to those described herein as appropriate to the requirements of specific applications in accordance with embodiments of the invention.
Process for Providing Training Tasks
[0043] A process for providing training tasks in accordance with an embodiment of the invention is conceptually illustrated in Figure 4. Training tasks in accordance with many embodiments of the invention can be provided by a human therapist or through software (e.g., through an artificial intelligence (Al) agent). Process 400 provides (405) a physical challenge to a user.
[0044] In some embodiments, prior to providing physical challenges within a game to a user, processes can provide an initial set of tasks to generate a baseline diagnostic to measure and visualize a user’s capabilities, such as (but not limited to) their range of motion (ROM) of various degrees of freedom (DOFs) in each joint for one or more target body parts (e.g., arm, hand, leg, foot, spine, etc.). Baseline diagnostics in accordance with certain embodiments of the invention can be used to initialize the parameters for the physical challenge. In numerous embodiments, baseline diagnostics can be performed once for each patient, or at each therapeutic session. Processes in accordance with numerous embodiments of the invention can calculate a reachable workspace within a three-dimensional space, which can be used to adjust and/or select tasks (e.g., the placement, trajectory, and/or motion of objects within space) presented to a user as part of training or physical therapy. The workspace for a particular individual may be a subset of the operational workspace of a healthy individual due to injury or some other disorder.
[0045] Physical challenges can include any of a variety of different challenges to focus on the use and coordination of various user motions. For example, a human arm has generally seven (7) degrees of freedom (DOFs): shoulder abduction/adduction, shoulder flexion/extension, shoulder internal/external rotation, elbow flexion/extension, elbow rotation (supination/pronation), wrist flexion/extension and wrist ulnar/radial deviation. In many embodiments, physical challenges can be provided through a series of events in a virtual environment, providing goals and purpose for each motion. Training tasks (and associated game elements) in accordance with a number of embodiments of the invention are used to for measure and improve the performance of such tasks. In some embodiments, training tasks can be developed to map a patients’ high-dimensional movements to a training task with a lower-dimension requirement, in order to train single joint mobility and alleviate abnormal multiple joint musculoskeletal synergies.
[0046] In certain embodiments, training tasks are categorized on a variety of characteristics, which can be used to measure different elements of a user’s progression. For example, some training tasks can be static or dynamic, depending on whether a target object (e.g., a ball that a user must touch) is moving or not. Users can then be instructed to interact with such objects in a variety of ways, including (but not limited to) touching, reaching, grasping, translating, placing, releasing, kicking and/or creating any virtual body contact with the object.
[0047] Training tasks in accordance with some embodiments of the invention are also classified by dimensionality (e.g., 1 D, 2D, and 3D motion), depending on how many dimensions the end effector is asked to move in, in order to accomplish the task. In many embodiments, training tasks can be categorized as unilateral or bilateral, based on whether one or both sides of the body (e.g., arms and/or legs) are required to perform an action. In certain embodiments, bilateral training tasks can further be developed based on whether the limbs are used in a mirror-image fashion or independently of each other.
[0048] Process 400 then determines (410) whether the user successfully performs the physical challenge. In certain embodiments, processes can determine successful performance through one or more sensors that can measure the positions and range of various body parts over time. Determining successful performance in accordance with many embodiments of the invention can be performed using any of a variety of systems, including (but not limited to) exoskeleton robots, image-based processing, and mobile devices including sensors such as, but not limited to, accelerometers, gyroscopes, and cameras.
[0049] When process 400 determines that the user was unable to successfully perform the physical challenge, process 400 calculates (415) a“miss”, or an error factor, for the physical challenge. Process 420 then modifies a challenge level for the physical challenge accordingly. In some embodiments, modifying a challenge can include (but is not limited to) one or more of increasing/decreasing the speed of a moving object, changing the static location of an object, modifying the trajectory of a dynamic object with respect to the operational workspace of a user, and/or increasing/decreasing the size of an object. In many embodiments, when a physical challenge is performed successfully, the challenge level is increased. When a user is unable to perform a physical challenge successfully, processes in accordance with several embodiments of the invention reduce the challenge level based on the calculated“miss” or error factor. Process 400 then returns to step 405 to provide a modified physical challenge based on the modified challenge level. Modified physical challenges in accordance with a number of embodiments of the invention can include (but are not limited to) changing the required range for successful completion of a challenge, changing the requisite speed for performing a challenge, modifying a complexity of a challenge (e.g., whether an action is to be performed with unilaterally or bilaterally, whether bilateral actions are performed in a mirror image fashion or with independent motions of both arms, etc.)
[0050] A process for providing training tasks using reinforcement learning in accordance with an embodiment of the invention is conceptually illustrated in Figure 5. Process 500 determines (505) the state of a user. State in accordance with various embodiments of the invention can include information about the user’s current situation, such as (but not limited to) previous performance, expected range of motion, energy levels, etc.
[0051] Process 500 identifies (510) an action to perform based on the state. Actions (or training tasks) in accordance with numerous embodiments of the invention can include different challenge types and/or modified settings for the challenges to be presented to the user. Modified settings can include (but are not limited to) the placement of an object in relation to a reachable workspace, modifying the speed of a moving object, and/or decreasing a size of an object.
[0052] Process 500 presents (515) a challenge to the user based on the identified action. Challenges in accordance with a variety of embodiments of the invention can include various games that can be performed in virtual reality settings, where the type of challenge presented and/or the settings (e.g., speed, range, difficulty, etc.) for the challenges can be varied based on the identified action. In certain embodiments, challenges can be designed to get a user to perform any of a variety of actions including (but not limited to) one or more of touching, reaching, grasping, translating, placing, releasing, kicking and/or creating any virtual body contact with an object.
[0053] Process 500 computes (520) a reward based on a user’s performance of the presented challenge. In several embodiments, rewards can include scores or measures of the challenge adaptation engines performance. For example, processes in accordance with several embodiments of the invention can compute a positive reward when a user performs an action that exceeds their expected range and/or when the user performs a challenge successfully. In a variety of embodiments, processes can compute a negative reward when a user is unable to perform a challenge, when a user succeeds at too many challenges, and/or when a user quits.
[0054] Process 500 computes (525) a new state based on the user’s performance of the presented challenge. Updates to state can include, but are not limited to, changes in a user’s range of motion (ROM), flexibility, quickness, etc. Process 500 then returns to 510 to compute a next action based on the updated state and the previous reward.
[0055] While specific implementations of systems and processes for providing training tasks for rehabilitation have been described above with respect to Figures 1 -5, there are numerous configurations of systems and processes in accordance with a variety of embodiments of the invention that would be apparent to one skilled in the art as appropriate to the requirements of a given application. In certain embodiments, steps may be executed or performed in any order or sequence not limited to the order and sequence shown and described. In a number of embodiments, some of the above steps may be executed or performed substantially simultaneously where appropriate or in parallel to reduce latency and processing times. In some embodiments, one or more of the above steps may be omitted. Although many of the examples are described with reference to rehabilitation, one skilled in the art will recognize that similar systems and methods can be used in a variety of applications, including (but not limited to) physical training, physical therapy, and/or gaming, without departing from this invention.
[0056] Although specific methods of providing training tasks in virtual reality are discussed above, many methods can be implemented in accordance with many different embodiments of the invention. It is therefore to be understood that the present invention may be practiced in ways other than specifically described, without departing from the scope and spirit of the present invention. Thus, embodiments of the present invention should be considered in all respects as illustrative and not restrictive. Accordingly, the scope of the invention should be determined not by the embodiments illustrated, but by the appended claims and their equivalents.

Claims

WHAT IS CLAIMED IS:
1. A method for providing training tasks through a virtual environment, the method comprising:
providing a first physical challenge for a user within the virtual environment;
determining whether the user successfully completed the first physical challenge; modifying a challenge level associated with the virtual environment; and providing a second physical challenge based on the modified challenge level.
2. The method of claim 1 , wherein:
the first physical challenge requires a first range of motion,
when the user successfully completes the first physical challenge, the challenge level is increased, and
the second physical challenge requires a second range of motion greater than the first range of motion.
3. The method of claim 1 , wherein:
the first physical challenge requires the performance of a unilateral motion, when the user successfully completes the first physical challenge, the challenge level is increased, and
the second physical challenge requires a bilateral motion.
4. The method of claim 1 further comprising providing a set of diagnostic tasks for setting an initial challenge level for the first physical challenge.
5. The method of claim 1 , wherein the virtual environment is provided in virtual reality.
6. The method of claim 1 , wherein modifying the challenge level comprises increasing the speed of an object presented in the second physical challenge.
7. The method of claim 1 , wherein modifying the challenge level comprises: determining a state of the user; and
identifying a first action based on the state of the user, wherein the first action comprises a modification of a plurality of modifications that can be made to the challenge level of the first physical challenge to provide the second physical challenge;
wherein the process further comprises:
computing a reward for the second physical challenge based on the user’s performance of the second physical challenge;
computing a new state of the user based on the user’s performance of the second physical challenge; and
identifying a second action based on reward and the new state of the user.
8. The method of claim 7, wherein providing the first physical challenge comprises displaying an object, wherein the first action comprises at least one of modifying a speed of the object, modifying a trajectory of the object, modifying a placement of the object, and modifying a size of the object.
9. A non-transitory machine readable medium containing processor instructions for providing training tasks through a virtual environment, where execution of the instructions by a processor causes the processor to perform a process that comprises:
providing a first physical challenge for a user within the virtual environment;
determining whether the user successfully completed the first physical challenge; modifying a challenge level associated with the virtual environment; and providing a second physical challenge based on the modified challenge level.
10. The non-transitory machine readable medium of claim 9, wherein:
the first physical challenge requires a first range of motion,
when the user successfully completes the first physical challenge, the challenge level is increased, and
the second physical challenge requires a second range of motion greater than the first range of motion.
1 1 . The non-transitory machine readable medium of claim 9, wherein:
the first physical challenge requires the performance of a unilateral motion, when the user successfully completes the first physical challenge, the challenge level is increased, and
the second physical challenge requires a bilateral motion.
12. The non-transitory machine readable medium of claim 9, wherein the process further comprises providing a set of diagnostic tasks for setting an initial challenge level for the first physical challenge.
13. The non-transitory machine readable medium of claim 9, wherein the virtual environment is provided in virtual reality.
14. The non-transitory machine readable medium of claim 9, wherein modifying the challenge level comprises increasing the speed of an object presented in the second physical challenge.
15. A training device comprising:
a set of one or more processors; and
a non-transitory machine readable medium containing processor instructions for providing training tasks through a virtual environment, where execution of the instructions by the set of processors causes the set of processors to perform a process that comprises:
providing a first physical challenge for a user within the virtual environment; determining whether the user successfully completed the first physical challenge;
modifying a challenge level associated with the virtual environment; and providing a second physical challenge based on the modified challenge level.
16. The training device of claim 15, wherein: the first physical challenge requires a first range of motion,
when the user successfully completes the first physical challenge, the challenge level is increased, and
the second physical challenge requires a second range of motion greater than the first range of motion.
17. The training device of claim 15, wherein:
the first physical challenge requires the performance of a unilateral motion, when the user successfully completes the first physical challenge, the challenge level is increased, and
the second physical challenge requires a bilateral motion.
18. The training device of claim 15, wherein the process further comprises providing a set of diagnostic tasks for setting an initial challenge level for the first physical challenge.
19. The training device of claim 15, wherein the virtual environment is provided in virtual reality.
20. The training device of claim 15, wherein modifying the challenge level comprises increasing the speed of an object presented in the second physical challenge.
PCT/US2019/051760 2018-09-18 2019-09-18 Virtual reality training tasks used for physical therapy and physical rehabilitation WO2020061213A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201862732736P 2018-09-18 2018-09-18
US62/732,736 2018-09-18

Publications (1)

Publication Number Publication Date
WO2020061213A1 true WO2020061213A1 (en) 2020-03-26

Family

ID=69887981

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2019/051760 WO2020061213A1 (en) 2018-09-18 2019-09-18 Virtual reality training tasks used for physical therapy and physical rehabilitation

Country Status (1)

Country Link
WO (1) WO2020061213A1 (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114694798A (en) * 2020-12-31 2022-07-01 浙江凡聚科技有限公司 Virtual reality-based joint injury rehabilitation training system and method
CN118016250A (en) * 2024-04-09 2024-05-10 北京智精灵科技有限公司 Mental disorder rehabilitation training system based on virtual person

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140188009A1 (en) * 2012-07-06 2014-07-03 University Of Southern California Customizable activity training and rehabilitation system
US20170365101A1 (en) * 2016-06-20 2017-12-21 Magic Leap, Inc. Augmented reality display system for evaluation and modification of neurological conditions, including visual processing and perception conditions

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140188009A1 (en) * 2012-07-06 2014-07-03 University Of Southern California Customizable activity training and rehabilitation system
US20170365101A1 (en) * 2016-06-20 2017-12-21 Magic Leap, Inc. Augmented reality display system for evaluation and modification of neurological conditions, including visual processing and perception conditions

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114694798A (en) * 2020-12-31 2022-07-01 浙江凡聚科技有限公司 Virtual reality-based joint injury rehabilitation training system and method
CN118016250A (en) * 2024-04-09 2024-05-10 北京智精灵科技有限公司 Mental disorder rehabilitation training system based on virtual person

Similar Documents

Publication Publication Date Title
US20220338761A1 (en) Remote Training and Practicing Apparatus and System for Upper-Limb Rehabilitation
KR20220033507A (en) Electronic arrangements and related methods for therapeutic intervention utilizing virtual or augmented reality
EP2726164B1 (en) Augmented-reality range-of-motion therapy system and method of operation thereof
US10111603B2 (en) Apparatus, method and system for pre-action therapy
US20130123667A1 (en) Systems, apparatus and methods for non-invasive motion tracking to augment patient administered physical rehabilitation
Montagner et al. A pilot clinical study on robotic assisted rehabilitation in VR with an arm exoskeleton device
Lupu et al. Virtual reality based stroke recovery for upper limbs using leap motion
EP2502169A2 (en) Remote physical therapy apparatus
EP3761916B1 (en) Grasp assistance system and method
US11090528B2 (en) Robotic physical therapy systems and data analysis of user interactions
Karime et al. A fuzzy-based adaptive rehabilitation framework for home-based wrist training
Vogiatzaki et al. Serious games for stroke rehabilitation employing immersive user interfaces in 3D virtual environment
Agyeman et al. Design and implementation of a wearable device for motivating patients with upper and/or lower limb disability via gaming and home rehabilitation
WO2021163334A1 (en) Adaptive virtual rehabilitation
WO2020061213A1 (en) Virtual reality training tasks used for physical therapy and physical rehabilitation
Zaher et al. A framework for assessing physical rehabilitation exercises
Palaniappan et al. Adaptive virtual reality exergame for individualized rehabilitation for persons with spinal cord injury
Li et al. Sensor fusion-based virtual reality for enhanced physical training
Lin et al. An upper extremity rehabilitation system using virtual reality technology
Kolsanov et al. Augmented Reality application for hand motor skills rehabilitation
GB2575299A (en) Method and system for directing and monitoring exercise
Kaluarachchi et al. Virtual games based self rehabilitation for home therapy system
Esfahlani et al. Intelligent physiotherapy through procedural content generation
Chiensriwimol et al. Frozen shoulder rehabilitation: exercise simulation and usability study
Prahm et al. Extending Mirror Therapy into Mixed Reality—Design and Implementation of the Application PhantomAR to Alleviate Phantom Limb Pain in Upper Limb Amputees

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19862396

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19862396

Country of ref document: EP

Kind code of ref document: A1