EP3649633A1 - Systèmes et procédés de formation d'habileté de mouvement pilotée par des données - Google Patents

Systèmes et procédés de formation d'habileté de mouvement pilotée par des données

Info

Publication number
EP3649633A1
EP3649633A1 EP18829049.8A EP18829049A EP3649633A1 EP 3649633 A1 EP3649633 A1 EP 3649633A1 EP 18829049 A EP18829049 A EP 18829049A EP 3649633 A1 EP3649633 A1 EP 3649633A1
Authority
EP
European Patent Office
Prior art keywords
movement
training
skill
patterns
task
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP18829049.8A
Other languages
German (de)
English (en)
Other versions
EP3649633A4 (fr
Inventor
Bérénice METTLER MAY
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
ICUEMOTION LLC
Original Assignee
ICUEMOTION LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ICUEMOTION LLC filed Critical ICUEMOTION LLC
Publication of EP3649633A1 publication Critical patent/EP3649633A1/fr
Publication of EP3649633A4 publication Critical patent/EP3649633A4/fr
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B19/00—Teaching not covered by other main groups of this subclass
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
    • A63B24/00—Electric or electronic controls for exercising apparatus of preceding groups; Controlling or monitoring of exercises, sportive games, training or athletic performances
    • A63B24/0062—Monitoring athletic performances, e.g. for determining the work of a user on an exercise apparatus, the completed jogging or cycling distance
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
    • A63B24/00—Electric or electronic controls for exercising apparatus of preceding groups; Controlling or monitoring of exercises, sportive games, training or athletic performances
    • A63B24/0003—Analysing the course of a movement or motion sequences during an exercise or trainings sequence, e.g. swing for golf or tennis
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
    • A63B24/00—Electric or electronic controls for exercising apparatus of preceding groups; Controlling or monitoring of exercises, sportive games, training or athletic performances
    • A63B24/0075—Means for generating exercise programs or schemes, e.g. computerized virtual trainer, e.g. using expert databases
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
    • A63B69/00—Training appliances or apparatus for special sports
    • A63B69/40—Stationarily-arranged devices for projecting balls or other bodies
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
    • A63B71/00—Games or sports accessories not covered in groups A63B1/00 - A63B69/00
    • A63B71/06—Indicating or scoring devices for games or players, or for other sports activities
    • A63B71/0619—Displays, user interfaces and indicating devices, specially adapted for sport equipment, e.g. display mounted on treadmills
    • A63B71/0622—Visual, audio or audio-visual systems for entertaining, instructing or motivating the user
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B21/00—Measuring arrangements or details thereof, where the measuring technique is not covered by the other groups of this subclass, unspecified or not relevant
    • G01B21/02—Measuring arrangements or details thereof, where the measuring technique is not covered by the other groups of this subclass, unspecified or not relevant for measuring length, width, or thickness
    • G01B21/04—Measuring arrangements or details thereof, where the measuring technique is not covered by the other groups of this subclass, unspecified or not relevant for measuring length, width, or thickness by measuring coordinates of points
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/23—Recognition of whole body movements, e.g. for sport training
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B19/00—Teaching not covered by other main groups of this subclass
    • G09B19/003—Repetitive work cycles; Sequence of movements
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B19/00—Teaching not covered by other main groups of this subclass
    • G09B19/003—Repetitive work cycles; Sequence of movements
    • G09B19/0038—Sports
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H20/00—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
    • G16H20/30—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
    • A63B24/00—Electric or electronic controls for exercising apparatus of preceding groups; Controlling or monitoring of exercises, sportive games, training or athletic performances
    • A63B24/0062—Monitoring athletic performances, e.g. for determining the work of a user on an exercise apparatus, the completed jogging or cycling distance
    • A63B2024/0065—Evaluating the fitness, e.g. fitness level or fitness index
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
    • A63B24/00—Electric or electronic controls for exercising apparatus of preceding groups; Controlling or monitoring of exercises, sportive games, training or athletic performances
    • A63B24/0062—Monitoring athletic performances, e.g. for determining the work of a user on an exercise apparatus, the completed jogging or cycling distance
    • A63B2024/0071—Distinction between different activities, movements, or kind of sports performed
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
    • A63B71/00—Games or sports accessories not covered in groups A63B1/00 - A63B69/00
    • A63B71/06—Indicating or scoring devices for games or players, or for other sports activities
    • A63B71/0619—Displays, user interfaces and indicating devices, specially adapted for sport equipment, e.g. display mounted on treadmills
    • A63B71/0622—Visual, audio or audio-visual systems for entertaining, instructing or motivating the user
    • A63B2071/0625—Emitting sound, noise or music
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
    • A63B2220/00—Measuring of physical parameters relating to sporting activity
    • A63B2220/10—Positions
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
    • A63B2220/00—Measuring of physical parameters relating to sporting activity
    • A63B2220/40—Acceleration
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
    • A63B2220/00—Measuring of physical parameters relating to sporting activity
    • A63B2220/50—Force related parameters
    • A63B2220/54—Torque
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
    • A63B2220/00—Measuring of physical parameters relating to sporting activity
    • A63B2220/50—Force related parameters
    • A63B2220/56—Pressure
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
    • A63B2220/00—Measuring of physical parameters relating to sporting activity
    • A63B2220/80—Special sensors, transducers or devices therefor
    • A63B2220/803—Motion sensors
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
    • A63B2220/00—Measuring of physical parameters relating to sporting activity
    • A63B2220/80—Special sensors, transducers or devices therefor
    • A63B2220/806—Video cameras
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
    • A63B2230/00—Measuring physiological parameters of the user
    • A63B2230/60—Measuring physiological parameters of the user muscle strain, i.e. measured on the user

Definitions

  • Movement performance relies on a broad range of functions (e.g., sensory, perceptual, planning, cognition).
  • Many movement skills within the category of complex movement are unnatural and therefore require adaptation of innate movement skills to accommodate the specific task requirements.
  • Complex movement also involve the coordination of large numbers of muscles and body segments. They may take place over short time-frames, with critical phases spanning 10th to 100th of a few millisecond. They often need to be adapted during performance and synchronized with external events or elements.
  • Movements are typically learned by trial and error, mostly by using some outcomes as feedback for corrections. Due to these complexities, the specific details regarding movement organization are stored in procedural memory and therefore are only known implicitly. Explicit knowledge surrounding movement details are typically not used during practice and execution. The fact that complex movements often unfold quickly and involve many dimensions make them hard or impossible to fully be perceived let alone comprehended. For example, just the path of a piece of equipment, such as a tennis racket, already involves three translational and rotational variables (e.g., six degrees of freedom) with their additional kinematic (speeds and angular rates) as well as dynamic (accelerations) characteristics.
  • translational and rotational variables e.g., six degrees of freedom
  • Movement skills also depend on a perceptual understanding of the external task elements. These characteristics are much harder to assess from observations of the movement performance. They manifest indirectly in the performance. A good instructor will call attention to important perceptual cues and how these can be used to inform the movement response characteristics.
  • FITBIT® activity tracker available from Fitbit, Inc.
  • JAWBONE® fitness tracker available from Aliphcom doing business as Jawbone
  • fitness trackers include devices for counting steps and tracking distance covered. More advanced capabilities can be found in devices that are specialized for a particular sport. Tennis, badminton, and golf represent the largest market segments (see, e.g., BABOLAT PLAYTM (from Babolat, France), ZEPP® tennis swing analyzer (available from Zepp US, Inc.), and the Smart Tennis Sensor (available from Sony)). These products aim to provide a description of players' technical performance. Typical features include tracking the type of actions; reconstructing movements, such as the path of the tennis racket during a stroke; tracking select outcome variables of actions such as the racket head speed, the distribution of impacts on the string bed, and the amount of spin.
  • the outputs of these assessments are typically provided after a training or play session.
  • the data is presented as summaries of session performance, as well as time.
  • the data is also aggregated to provide statistical trends.
  • the main shortcoming of these products is that the analysis is based on outcome variables (referred to as knowledge of results in the human skill literature) and thus does not provide actionable information that can be leveraged directly for training.
  • Skill acquisition follows an incremental process; therefore, most people's skills can be considered at some intermediate level that could be further developed.
  • Each successive iteration along a path to improving skills involves increasingly complex mental representations and their supportive functions such as movement coordination, and perception (Ericsson, 2009). It seems that skill acquisition stabilizes along successive skill levels.
  • Data-driven movement skill training systems are disclosed herein.
  • the systems may use movement skill assessment and diagnostics at distinct levels of the human movement system hierarchy to specify training goals.
  • the systems may then provide different forms of augmentations synthesized to help pursue the training goals.
  • the system may also include a system to track and/or manage the learning process.
  • FIG. 1 is an illustration of a human augmentation system for movement skill training or rehabilitation according to an embodiment.
  • FIG. 2 is an illustration of an interaction between a stroke motion and task and environment elements, including ball trajectory relative to a court, the impact of the ball, and its bouncing before the interception with the racket trajectory.
  • FIG. 2 also illustrates the gaze of the player along different points of the ball trajectory and court locations, and shows a ball machine as an apparatus that can be programmed to enable different forms of interactions.
  • FIG. 3A is an illustration of a general movement trajectory envelope delineating the movement phases that typically arise from biomechanical and neuromotor constraints.
  • FIG. 3B is an illustration of the finite-state model representation for the system shown in FIG. 3A, where each state represents a movement phase.
  • FIG. 4 is an illustration of the primary movement unit for six movement activities, along with corresponding phase segments.
  • the figure also highlights primary outcome quantities as vectors (e.g., effect of a racket or club on a ball for tennis or golf, effect of arm coordination on hand placement in rehabilitation, propulsive force generated by a foot strike for running, transversal acceleration used in turning while skiing, and propulsive force generated by the pull phase in swimming).
  • vectors e.g., effect of a racket or club on a ball for tennis or golf, effect of arm coordination on hand placement in rehabilitation, propulsive force generated by a foot strike for running, transversal acceleration used in turning while skiing, and propulsive force generated by the pull phase in swimming.
  • FIG. 5 is an illustration of the progression across different tennis stroke architectures corresponding to different skill levels. The architecture is shown in terms of its constituent movement phases.
  • FIG. 6 is an illustration of an overview of the movement processing
  • FIG. 7 is an illustration of different outcome levels for tennis and some of the outcome measurements: 1) stroke technique and racket impact; 2) stroke primary outcomes; 3) shot trajectory and type; 4) shot placement (relative to the court and opponent).
  • FIG. 7 also shows trajectories for two types of strokes e.g., flat (FL) and topspin (TS).
  • FL flat
  • TS topspin
  • FIG. 8 is an illustration of two players' respective ground impact distributions from tennis shots, describing the discretization of the task environment associated with the ball-environment interactions.
  • the skills at the shot level manifest as different resolutions and precision in the interactions with the task environment.
  • FIG. 8 also shows court landmarks that are relevant to the players' task environment perception and the players' court motion.
  • FIG. 9 is an illustration of three interception types: 1) on the descent; 2) at the apex; 3) on the rise of the trajectory following the ground impact.
  • Fig. 9 also illustrates the racket string bed associated with the corresponding impact conditions and examples of the return shot outcomes.
  • FIG. 10 is an illustration of the levels of assessment highlighting the elements and outcomes for tennis and summarizing the assessment and diagnostic components across the different levels.
  • FIG. 11 is an illustration of the movement acquisition as an evolutionary process during which movement patterns are learned either from scratch or through differentiation of existing patterns.
  • FIG. 12 is an illustration of movement pattern clusters based on features extracted from movement data.
  • FIG. 13 is a tree diagram illustrating the evolutionary relationship between movement patterns.
  • FIG. 14 is an illustration of the state space X (measured or estimated dimensions derived from performance data that are relevant to describe the movement behavior), highlighting the classes associated with the movement patterns, and the mapping associated with the outcome space or other attributes used in the skill assessment. The figure also shows the embedding from V into a subspace W that produces meaningful outcome categories (semantic interpretation).
  • FIG. 15 is an illustration of a generic outcome-movement pattern map showing the movement patterns in terms of associated outcomes dimensions.
  • FIG. 16 is an illustration of a generic repertoire map showing the movement pattern classes arranged in terms of outcome dimensions that have been rescaled such as through the embedding g: V -> W.
  • FIG. 17 is an illustration of the skill profile of two subjects A, B resulting from their repertoire of movement patterns or skill elements, highlighting the difference in skill profile that results in an overall skill gap as well as a gap in the repertoire range.
  • FIG. 18 is an illustration of population subgroups based on skill attributes with level lines associated with a performance or skill objective function.
  • FIG. 19 is an illustration of distribution for an entire population group described by the group distribution highlighting a member (subject A, described by skill element distribution (ei, e 2 )), and the tiers (low, medium, high, very high) associated with an outcome function for the entire population subgroup (ei,c , e 2 ,G).
  • FIG. 20 is an illustration of distribution of motion patterns produced by subject A described by two features (fi, f 2 ), showing the center of the ellipsoid ( ⁇ , ⁇ ) and the axes given by the eigenvalues (ei, e 2 ), and highlighting the level lines of some outcome tiers (low, medium, high, very high).
  • FIG. 21 is an illustration of a data-driven closed-loop training system including its primary processes organized according to three primary feedback loops.
  • FIG. 22 is an illustration of a Human Augmentation System.
  • the system encompasses three primary tiers of augmentation that leverage the human information processing hierarchy: real-time feedback (cue stimuli and activity interactions), intermittent feedback, and visualization and instructions.
  • FIG. 23 is an illustration of the augmented perception-action loop associated with the feedback cueing system. Low-level signal and cues are emphasized.
  • FIG. 24 is an illustration of the main components used for feedback
  • FIG. 25 is an illustration of process flow along the training process shows an activity across the stack of processes of an assessment and training loop (e.g., data acquisition and processing, motion model, skill model, training goals, augmentation laws) as a function of time. During each session, activity data is collected and processed.
  • an assessment and training loop e.g., data acquisition and processing, motion model, skill model, training goals, augmentation laws
  • FIG. 26 is an illustration of a diagnostic system building on the assessment system.
  • FIG. 27 is an illustration of a diagnostic system, which combines a knowledge representation, observations, and an inference mechanism to produce a diagnostic of the movement performance.
  • FIG. 28 is an illustration of the factors influencing stroke quality (categorized as observations, uncertain factors, or hypotheses) and their relationships.
  • FIG. 29 is an illustration of population analysis and player or performer profiling.
  • FIG. 30 is an illustration of an assessment, diagnostics, and training goals across the skill-model hierarchy, incorporating player profile information..
  • FIG. 31 is an illustration of an assessment including a) different levels of assessment, b) elements that describe each level, c) criteria and quantities that can be used to determine the skill characteristics at a given level, d) analysis or diagnostics to identify the critical characteristics, e) the drivers and mechanisms used to produce training interventions, and f) the intervention or feedback form that can be used.
  • FIG. 32 is an illustration of the primary outcome characteristics (i.e., pace and spin) for a player's groundstroke repertoire with overall reference ranges from population analysis (gray background tiles).
  • FIG. 33 is an illustration of spin envelope for the groundstrokes (solid line) divided into forehand and backhand with reference ranges from population analysis (dashed lines).
  • FIG. 34 is an illustration of a leaderboard for population analysis based on the global score shown as a percentile rank from highest to lowest computed from the skill profile of a population of players.
  • FIG. 35 is an illustration of play activity summation over a calendar period showing sets and sessions.
  • FIG. 36 is an illustration of movement outcome trends for a specific motion pattern class with overall reference ranges from population analysis (gray background tiles). The vertical bands delineate the sets.
  • FIG. 37 is an illustration of the forward swing phase stroke profile for forehand topspin stroke class.
  • FIG. 38 is an illustration of selected components of a skill element including outcome, attributes, and other characteristics forming a composite skill element score. Two polygons are superposed to provide a comparison.
  • FIG. 39 is an illustration of the trends in movement patterns and movement outcomes for an activity session delineated in individual sets.
  • FIG. 40 is an illustration of the skill profile as a bar graph for the values from a composite score across a repertoire of groundstrokes.
  • FIG. 41 is an illustration of the acquisition stage for the strokes in the groundstroke repertoire.
  • FIG. 42 is an illustration of impact timing statistics for a player's groundstroke repertoire with overall reference ranges from population analysis (gray background tiles).
  • FIG. 43 is an illustration of the integrated perspective on the assessment and diagnostic process organized in terms of the assessment levels (i.e., physical, pattern, task, and competitive).
  • FIG. 44A is an illustration of a skill status screen showing the skill elements arranged according to their acquisition stage (Patterns to Form, Patterns to Consolidate, and Patterns to Optimize).
  • FIG. 44B is an illustration of a skill status screen showing how training activity over several training sessions (Set 1-3) lead to a change in the skill status of skill elements.
  • FIG. 45A is an illustration of a training list showing selected training elements ei and their associated training goals g k i expressed in terms of attributes a; for an epoch k.
  • the list is indexed according to relevant criteria, such as user preference or importance of the element to the activity or to the skill acquisition process.
  • FIG. 45B is an illustration of a training schedule.
  • the training session is subdivided into sets (Set 1 ... Set N). Each set focuses on one or more training element e;
  • FIG. 46 is an illustration of a state machine showing the active training element and the criteria for the issuance of notifications to the performer.
  • FIG. 47 is an illustration of a trend plot displaying the progress along training goals (gi, g2, and g 3 ) over a specified time range shown here as seven sessions.
  • FIG. 48 is an illustration of a learning curve associated with the data driven training process.
  • the learning curve shows the incremental improvement in some relevant attribute ai of a skill element ei over the training activity (sets and sessions).
  • FIG. 49 is a flow diagram illustrating a data-driven training process according to an embodiment.
  • FIG. 50 is a flow diagram illustrating the movement modeling processes of FIG. 49.
  • FIG. 51 is a flow diagram illustrating the processes of the skill modeling and assessment of FIG. 49.
  • FIG. 52A is a flow diagram illustrating the skill assessment processes of FIG. 49.
  • FIG. 52B is a flow diagram illustrating the skill status process of FIG. 49.
  • FIG. 53 is a flow diagram illustrating the training goals and feedback synthesis processes of FIG. 49.
  • FIG. 54A is a flow diagram illustrating the training goal computation process accounting for skill status of FIG. 49.
  • FIG. 54B is a flow diagram illustrating the feedback synthesis processes of FIG. 49.
  • FIG. 55A is a flow diagram illustrating the instructions synthesis of FIG. 49.
  • FIG. 55B is a flow diagram illustrating the feedback and cueing laws synthesis processes of FIG. 49.
  • FIG. 56 is a flow diagram illustrating the activity management and monitoring processes of FIG. 49.
  • FIG. 57A is a flow diagram illustrating the system configuration processes of FIG. 49.
  • FIG. 57B is a flow diagram illustrating the activity monitoring process including the notification and user input of FIG. 49.
  • FIG. 58 is illustration of a temporal structure and organization of a typical activity session.
  • GMP general motor program
  • Movement phases are usually formed to support various functional characteristics, such as biomechanical constraints, task structure and various sensory interactions with the environment.
  • Movement segments can be conceptualized as a movement directed towards a sub-goal, each with its particular biomechanical and sensory-motor constraints. This structure allows to breakdown complex movements into simpler movement elements. It can also help in the acquisition of complex movement skills, and support the flexibility and adaptability needed to operate in dynamic and uncertain environments.
  • the human bandwidth limitation for closed-loop feedback involving perceptual motor control is somewhere between 0.5 and 2Hz, depending on the task. Above that bandwidth, intermittent closed-loop control can be used. Movement phases typically represent open-loop segments. Corrections can be implemented at specific phase transition. These phase transitions are also associated with functional features, such as when specific elements of information are available. For example, in a tennis stroke, an advanced player already has an idea of the intended outcome and anticipates the conditions of the oncoming ball, at the initiation of the stroke. At the end of the backswing phase, and before the initiation of the forward swing, the player makes adjustments based on the up-to-date information available from the oncoming ball trajectory.
  • FIGS. 1 and 2 illustrate the interactions of a tennis player's racket with a delineation of the racket stroke trajectory
  • FIG. 9 illustrates the interception conditions that the performer has to accommodate to best control the ball trajectory.
  • the perceptual system usually provides cues that are used to select the type of motion pattern from the repertoire of learned movement patterns. Signals from the sensory or perceptual system are used to modulate specific aspects of the pattern, such as the timing of the stroke phases based on a tennis ball's perceived speed.
  • Training movement skills therefore, involves acquiring a comprehensive set of mechanisms. Movements are not simply programs to steer body segments; they encompass numerous mechanisms and capabilities to support the interactions and adaptation to conditions. Therefore, skill acquisition also includes learning how to extract relevant signs or cues from the task environment, and developing plans for sequencing individual movement patterns. The basic motor learning concepts are introduced next.
  • Motor skills require integration of both sensory information and motor responses to attain a particular goal.
  • Goal-directed, deliberate, instrumental, or intentional movements are movements characterized by forethought with reference to the consequences they produce. The outcome to be obtained is clear to the performer and determines how they organize their movement pattern. Such deliberate movements contrast reflexes or fixed action patterns.
  • Motor skills are categorized on a continuum defined by the dynamics of the task and environment condition. On one end of the continuum are the open skills, which take place in temporally and spatially changing conditions; on the other end are the closed skills, which take place under fixed, unchanging environmental conditions.
  • a new movement formed to respond to a new aspect of the task environment may either originate as a variation of an existing pattern, or as a new movement that is formed as a unique new pattern (see FIG. 11), albeit the new pattern may be reusing components of the original pattern. Therefore, in open skills, the user develops a repertoire of movement patterns that match the range of environmental conditions and task requirements. On the other hand, in closed skills, as the user learns to master the task, the movement performance converges over time to a fixed movement pattern that optimizes the outcome in relationship to the task requirement.
  • the term "user” may refer to a user of the data-driven training system, an agent using the system, a subject to whom the system is applied, or a combination thereof.
  • the movement segments that compose most complex movements result from how the subject exploits the large number of degrees of freedom (DOF).
  • DOF degrees of freedom
  • the high DOF in human motion result in redundant movement solutions.
  • racket swinging can be achieved through various combinations of joint motions such as wrist, elbow, shoulder, hips, etc.
  • Each DOF has its own specific displacement range as well as other constraints such as speed or torque. Different executions of the same general movement will cause saturations at different stages of the overall trajectory and will result in a different sequence of movement phases.
  • a deliberate movement is needed to produce a particular outcome or change in the environment.
  • Many skilled movements involve the control of an end effector such as the hand, foot, or a piece of equipment or instrument.
  • Another class of skilled motions are characterized by controlling the dynamics of interactions with an environment such as in skiing or surfing. These interaction behaviors involve the performance of particular maneuvers to allow deliberate control of motion. Examples of maneuvers include different turning techniques (stem, parallel, carve) as well as other maneuvers such as rapid stopping, jumping, etc.
  • maneuvers are movement units that can be used to interact with the environment under different conditions or purposes. Movement skill acquisition can be defined as the process used by an individual to best change or maintain either their own state, or the state of objects, in space.
  • end effector motions encompass a variety of different movement behaviors including reaching motions, such as those used to grab an object or touch something, or interception and throwing or hitting motions. All of these motions guide the end effector along a path to a particular location in space. Most of the reaching motions involve stationary end conditions. Interception and hitting involve more dynamic end conditions. Most skillful end-effector motions involve the precise control of its state at various instances or phases of the movement (apex, contact, interception, or throw) (see FIG. 5).
  • Skiers use gravitational forces and body biomechanics to generate a turning motion to steer and control their path. These coordinated movements represent the primary unit of motion. While they may not be a distinct goal state such as in tennis or other swing sports, they often have a movement phase such as the apex of a turn, which together with the local environment interaction determine the primary outcome of the movement pattern. Skilled human movements, such as the tennis stroke, involve the sequencing of complex coordinated motions that are executed based on internal states and external cues.
  • the scope of motion analysis can encompass multiple levels. For example, it could focus on low-level neuro-motor aspects, the movement technique and structure, the optimization of outcomes, all the way up to tactical and strategic levels (see FIG. 31).
  • the range of motion sensors available either embedded or deployed in the environment, can provide measurements of broad aspects of the movement dynamics surrounding the users, actors and their equipment. However, data alone is not sufficient to produce useful and actionable insights.
  • a forward stroke will be a rudimentary movement including a forward swinging motion implemented from the shoulder joint.
  • the brain will learn to better take advantage of their physical potential, range of coordination of their body segments, and other movement system components (FIG. 5).
  • Skilled behavior relies on organized strategies and builds on the well-defined hierarchical organization of neurological processes.
  • the instances of observed movements belong to specific classes of movement patterns that are used to support interactions needed for a task performance. Therefore, capturing movements and aggregating them within classes provides a solution to systematic analysis even in the face of variations. These classes of movement correspond to the movement units.
  • the core technology focuses on decoding movement data to extract relevant movement elements that can be used for skill analysis.
  • the relevant elements in natural speech processing are the units of organization of speech production, known as phonemes.
  • the decoded phonemes can then be used to identify words and eventually the meaning of a sound bite.
  • these units similar to speech, have to be related to the process used for movement production.
  • the result of this type of analysis can then be more readily translated into instructions and used to synthesize augmentation systems.
  • a data-driven skill augmentation environment requires a system infrastructure to operationalize the various processes.
  • the basis of the infrastructure is a data structure derived from the movement units that support efficient handling, processing, tracking, and managing of motion skill data.
  • the data structure allows codification of skill components and their functional characteristics to design feedback mechanisms that target precise aspects of the movement skill performance and learning.
  • the proposed modeling language and skill model and accompanying technology infrastructure can accommodate the nuances that naturally occur in human performance, and build on the structural features inherent to the human movement system and its various functional and learning mechanisms. Moreover, the methods capture both the global skill components that give users its versatile performance in an activity domain, and the specific skill components needed for performance and adaptation to the specific task elements and conditions. And finally, it can be generalized to different activities and scaled to larger populations.
  • FIG. 4 shows examples of movement architecture for the primary movement unit for other movement activities (tennis 441, golf 442, rehabilitation 443, skiing 444, running 445, and swimming 446).
  • the drawings also highlight the movement phases and the primary outcome.
  • Behaviors are produced through a process of selection of a response (movement behavior), which is typically based on the observable environment state. A successful outcome of a behavior therefore depends on both the correct selection of the behavior type and its correct execution. Learning is defined as a change in behavior that results from experience. Learning is typically improved through feedbacks that reinforce correct behavior (Law of Effect).
  • the learning process therefore depends on availability of signals that inform the subject of the success of its movement behavior. Moreover, for complex behavior, information about the outcome alone, or so-called knowledge of result, may not be sufficient. For complex movements, it can be helpful to combine an understanding of the movement technique—i.e., cognitive level— with feedback on specific aspects of that technique during and/or after the execution.
  • FIG. 21 illustrates a data-driven closed-loop training system including its primary processes organized according to three primary feedback loops.
  • the assessment loop 200 is configurable to have five components.
  • An extractor 201 extracts motion elements from a target motion.
  • the extracted motion elements can be directed from an augmentation loop 202 which collects information from user training or play.
  • the augmentation loop 202 can have a feedback loop between a movement process 222 and a cueing system 224.
  • the augmentation loop 202 can receive information from an instruction module 203.
  • the instruction module 203 may receive a set of target skills 204 from a user or a trainer.
  • Session data 226 can be provided to the extractor 201.
  • the extractor 201 output generates a motion model 205 which can then be used for skill assessment and diagnostics 206 based on reference skill data 207.
  • a measurement process can be provided that maps aspects of behavior or movement into one or more measurement signals.
  • the system operationalizes the training process and creates a systematic schedule that builds skills in following logical development, consistent with human learning principles.
  • the training starts from a user's existing motor skills and proceeds by shaping these skills towards the specified goal skills.
  • the Assessment Loop corresponds to the process of data acquisition and processing associated with modeling subjects' movement technique and skills, skill diagnosis, and the organization of knowledge, for example in training lists and training schedules/plans, as well as the synthesis of augmentation laws.
  • the Training Loop corresponds to interactions associated with the management and organization of the training activity, including reviewing the skill status, learning about the movement technique, selecting training elements and goals for the session, scheduling a training or performance session, and finally tracking the progress of the training process.
  • the inner most loop is the Feedback Augmentation Loop (FL), which corresponds to actual performance of the movement activity and includes the effect of feedback cues communicated to the subject during the performance.
  • the cueing system 124 can include two components: a cue processor and a cue generator.
  • the cue processor translates movement data into cue signals.
  • the cue processor implements a finite state estimator and a cueing law calculator.
  • the finite-state estimator is an approximation of the user's movement model (which is itself represented as a finite-state machine).
  • the cue generator translates cue signals into physical stimuli; the system operates in real-time to provide feedback as the user participates in an activity.
  • the cueing law calculator takes the state estimate and the motion data and operates on them to calculate if a cue will be delivered and what the cue should communicate.
  • the feedback synthesis model determines how the cueing law calculator operates, whereas the finite-state estimator is defined by the user's current movement model.
  • the cue generator takes the cue signal and translates it into feedback stimuli generated by a transducer (audio, visual, haptic, symbolic, or other type).
  • the form of transducer is determined by the platform implementation details, user characteristics, equipment parameters, environment status, and/or other concerns.
  • the system receives input from a user's physical movement that takes place during a use or play session.
  • the measurements can capture a range of movement behavior that was performed to complete the activity (e.g., all the motion associated with a tennis stroke, all the motion associated with a golf swing, etc.), associated task conditions, as well as the elements relevant to the broader functional components such as perception of task elements.
  • FIG. 1 illustrates one embodiment of a human augmentation system 101 applied to movement skill training or rehabilitation.
  • the system in this example, combines existing devices such as a smart phone 102, a smart watch 103, or other processor in wired or wireless communication with a motion tracking device 104 attached to or embedded in the tennis racket 105.
  • the device 104 streams motion measurements to the smart watch 102 and/or phone 103 or other processor.
  • Motion measurements are typically obtained from MEMS IMUs (e.g., available from ST Microelectronics and InvenSense), which usually include 6-axes acceleration and angular rates and 3-axes magnetometers, which are often used to estimate absolute orientation in space (Attitude and Heading Reference System or AHRS).
  • MEMS IMUs e.g., available from ST Microelectronics and InvenSense
  • 6-axes acceleration and angular rates and 3-axes magnetometers which are often used to estimate absolute orientation in space (Attitude and Heading Reference System or AHRS).
  • the motion data is processed at different levels in this system to render useful information for the subject's training or rehabilitation.
  • the processing is distributed across typical internet of things (IoT) components, such as the wearable/embeddable devices, smart devices and cloud infrastructure.
  • IoT internet of things
  • the segregation of these processes depend on the temporal requirements, such as acceptable delays or latencies, the required computational capacity, the availability of data, such as subjects' history and even larger population data and meta-date. Other factors include the streaming bandwidth and power requirements. All of these factors combine to determine the best network topology, data structure and management, as well as hardware selection.
  • Movement characteristics can be represented as geometrical and topological properties, which can be related to specific aspects of movement organization and skill. For example, movement characteristics can be observed in movement phase portraits such as that of the racket angular rate. Ensembles of movement data can be analyzed for patterns (e.g., using principle component analysis, phase-space analysis, and nonlinear time series analysis techniques such as state-space embedding). In addition, machine learning techniques can be applied to analyze the distribution of features and
  • one or more motion sensors can be used with the system to provide measurements of movement dynamics encompassing one or more users, actors, and their associated equipment (if any).
  • the scope of motion analysis can be conducted at multiple levels.
  • the illustration in FIG. 2 delineates between different categories of measured or captured quantities.
  • the output side includes behavioral quantities (movement such as the end effector, body segments; visual attentions; muscle activation); task and environment elements and objects.
  • On the input side are motion tracking cameras 70, Gaze Tracking/ AR device 80, and other sensor input.
  • Analysis of the intrinsic movement structure of the movement technique and functional characteristics can be used for skill analysis. This analysis can be formalized by focusing on the interactions of the movement with the environment and task elements. Operators or agents such as a tennis player organize their behavior in relationship to environment and task elements.
  • FIG. 7 shows the different outcome levels, using tennis as an example, and some of the outcome measurements and FIG. 8 shows how these interactions produce the repertoire of strokes and their associated shot distributions.
  • a particular distinctive characteristic of human behavior which contrasts with robots and other engineered system, is that human behavior can be considered relational, i.e., movement behavior is produced through the action-perception loop and therefore is often anchored in a particular environment features and elements.
  • the stroke which can be considered the primary movement unit, is directed at specific target areas in the court environment.
  • FIG. 2 illustrates an exemplar augmented activity for tennis.
  • the primary interaction is the tennis stroke, driving the tennis racket 20 toward a ball impact 30.
  • the activity environment elements include the tennis court environment 50, with a net 52, as well as marking on the court 51.
  • One or more motion tracking cameras 70 and/or other acoustic or RF motion sensors 90 can be used to track the subject's motion on the court environment 50, including the details of the individual body segments 15, the ball 30 and the racket 20.
  • Other measurements can include the subject's visual gaze 81, which direction changes depending on the focus of visual attention, when tracking different visual cues, including the ball's ground impact 32, or net crossing 31 as well as desired court placement.
  • the apparatus 40 shown in the same figure can be programmed to enable different forms of interactions.
  • the apparatus 40 is a ball machine that can be programmed to support the development of specific stroke patterns and therefore can be programmed in conjunction with the cueing system.
  • the systems and devices disclosed herein augment movement skills at several levels, for example: 1) providing users feedback for training, including providing signals during the performance; 2) enhancing the athletic experience during performance to help focus; 3) providing protection from injury by helping users engage in optimal techniques; and 4) developing training protocols which are directed to developing skills related to the training.
  • Patterning characteristics are expected in many movement activities.
  • the same general stroke pattern can be used to generate different amounts of top spin or pace.
  • distinct patterns have to be formed to fully exploit the biological capabilities.
  • a stroke for a top spin or slice has characteristic features in the temporal and spatial arrangement of movement phases. Movement patterning is due to how changes in movement outcomes or task conditions affect movement technique within a particular operating region of the state-space. As the desired outcome or task conditions change beyond a certain threshold, the biomechanics and motor-control organize differently to best take advantage of the system's capabilities. From a trajectory optimization perspective, the changes in outcome and condition alter the system's "operating point" and result in activation of a different set of constraints.
  • Patterning corresponds to a tendency for the trajectories in each movement class of behavior to stay close together in spatial and temporal terms. This closeness can be described formally using techniques from nonlinear time series analysis. Using these techniques, measurement data describing racket state trajectories during a tennis stroke can be aggregated and clustered to identify different stroke patterns, and subsequently analyzed to determine their functional properties and characteristics.
  • Such performance data for an activity taken in its totality results in a repertoire of distinct movement patterns.
  • This repertoire of distinct movement patterns is the result of the optimization of movements technique, i.e., achieving the range of outcomes and conditions required to be proficient in the particular activity.
  • an individual will develop a repertoire of different strokes to optimize the desired outcomes (e.g., type and amount of spin, strength, etc.) and accommodate the range of impact conditions (ball height, speed, etc.; see FIG. 9).
  • This repertoire essentially plays the role of a vocabulary of motion pattern that an individual can call upon when engaged in a particular activity.
  • FIG. 8 illustrates distributions of shots associated with different strokes and the effect of skills on the accuracy and granularity of the discretization of the task environment, which in this case is the tennis court.
  • the movement patterning and organization in repertoire therefore, have implications for the assessment of skills.
  • the skills of a particular tennis player can be assessed by: 1) extracting characteristics about the entire repertoire of strokes, e.g., how well they collectively achieve the range of outcomes and conditions in the activity domain, 2) determining how well and how consistently each class of strokes in the repertoire achieves associated outcomes, and 3) determining how well the strokes adapt to the impact conditions.
  • the first analysis provides a comprehensive assessment, and the last two emphasize the technical implementation of the motion skills.
  • the following disclosure addresses the general question of how to improve movement learning using information technology, machine learning, and wearable devices.
  • the disclosure also addresses specific questions including how to formulate training goals; how to manage the larger training process, in particular how to break up larger training goals into a sequence of goals; and how to dynamically update these goals based on data from the training activity such as skill acquisition stage and trends.
  • the system determines what type of feedback to use to augment the experience and accelerate the learning process, when to present the feedback, how to determine the best type of feedback given the learning stage, and how to distinguish between different skill elements.
  • the disclosure also addresses how to best represent information to augment a subject's training experience.
  • the resulting system takes into account what is learned by the subject as they make progress in an activity domain, what aspects of behavior to emphasize depending on learning stage, and also accounts for the characteristics of human information processing to provide feedback and information that can be processed and assimilated efficiently.
  • the central requirement for deliberate training is the specification of training goals and management of the training process using these goals. These processes are usually handled by human coaches or physical therapists.
  • the contributions of this disclosure are the algorithms and system that enable training to be operationalized following a computational, data-driven process.
  • the disclosure addresses two central capabilities: the computation of training goals, and scheduling and management of the training process.
  • the general approach is to use movement data to assess skill and identify deficiencies, followed by specification of training goals to address these deficiencies.
  • training process management the general approach is: i) leveraging the natural structure and organization of the human skill learning process; ii) using information from both individual subjects as well as from a larger population to extract knowledge to guide that process while accounting for individual characteristics.
  • the structure of the skill acquisition processes refers to the type of changes taking place over time as a result of activity (training or experience), which manifest as sequence of learning patterns characterized by specific changes in movement skill attributes and task performance.
  • FIG. 11 illustrates the development of movement patterns over time. It is expressed as the differentiation of existing patterns as well as the formation of new patterns.
  • the model encompasses the repertoire of movement patterns, and the movement structure associated with the movement patterns used in the interactions with the task of environment.
  • the specific Movement Functional Structure (MFS) also makes it possible to extract the wide range of movement skill attributes across the levels of organization of the movement system and the task structure.
  • Movement patterns that correspond to the primary movement units are typically associated with primary interactions found in an activity, some of those interactions produce specific outcomes on the environment or task elements, and hence can be characterized by their range of outcome and operating conditions. Therefore, the motion patterns associated with these primary movement units can be considered as the basic unit of skill, or skill element.
  • FIG. 6 gives an overview of the movement processing starting from the extraction of movement units, their classification, the movement model for each classes, following with the skill model that is used to determine relevant skill attributes used in the skill assessment and diagnostics. The figure also shows how these skill elements are then aggregated to produce the repertoire, which provides the basis for a subject's skill profile that can then be used for the analysis of the skill development (learning curve) and the population analysis.
  • the quantitative definition of a unit of skill also provides the foundations to proceduralize training under an iterative learning scheme, which specifies how skill assessment, diagnostics and training goals are computed and updated over time.
  • the system also incorporates the movement performance augmentations defined in U.S. Patent Application Publication No. 2017/0061817 (FIGS. 22 and 23) that are used to help induce changes in movement technique.
  • the first capability includes the precise and comprehensive assessment of an individual's movement skills, and more generally, data-driven training includes tracking various attributes of these skill elements.
  • data-driven training includes tracking various attributes of these skill elements.
  • Using motion patterns as unit of skill enables the formulation of quantifiable, incremental change in movement technique, and its associated effect on measurable outcomes, as a result of experience or training.
  • the sum of all changes in skill elements also ultimately produce incremental changes in some overall skill level that captures the larger impact of skill on the activity or task performance.
  • the skill element in the skill model represents the basic unit of skill acquisition. It is defined as the primary outcome associated with a particular class of movement pattern, and the associated attributes, that describe the relevant movement characteristics.
  • These skill elements are derived from the movement system hierarchy specified in U.S. Patent Application Publication No. 2017/0061817. They encompass: (a) the repertoire of movement pattern classes, where each class is described by a movement pattern which is decomposed into phases; (b) the movement phases, which are the manifestation of the movement functional structure determined by the biomechanical constraints and other constraints arising from the properties of the environment interactions.
  • the skill elements can be combined to form a subject's comprehensive skill profile, which captures skill attributes associated with the skill elements.
  • An individual's skill profile can be precisely and comprehensively characterized by the skill element attributes that can be derived from the hierarchical movement model and the functional structures underlying all movement patterns in a repertoire used in a domain of activity.
  • Task performance metrics can be computed from attributes of the repertoire of movement patterns.
  • Simple metrics can, for example, be determined from the use frequency of the various movement patterns.
  • More detailed models for higher-level assessment can be determined from the temporal sequence of movement patterns. Spatiotemporal patterns at the level of the repertoire, i.e., what movement patterns are used where and when, also enable the description of the high-level decision-making processes associated with planning and strategy which represent cognitive functions.
  • This extended task performance analysis provides tools to compare players or performers, i.e., support the analysis competitive level performance. They can also be extended using population analysis (see concept of player profile).
  • Target Skills can be used to drive skill or performance attributes at different levels from features of the movement technique used to optimize outcomes to higher-level attributes such as success rates of tennis shots in specific areas of the tennis court.
  • Training goals are a quantitative specification of a subject's target changes in the movement that will produce the desired increment in skill level.
  • the training goal targets actionable characteristics in movement technique and therefore represents the drivers to achieve the larger skill level targets.
  • the goals typically combine expected changes in movement outcomes with the associated movement characteristic (functional element).
  • the training goals can be augmented by a range of instructions and feedback cues as defined in U.S. Patent Application Publication No. 2017/0061817, which can encompass different components of the information processing levels to best target the various attributes of the movement functional model.
  • training goals should be: actionable, sufficiently broad in scope, effective, and realistic. By fulfilling these requirements, training goals enable subjects to train deliberately and achieve predictable, quantifiable changes in technique that result in improvements in skill level, relative to the existing skill level, but also provide a path for the long-term development of skills needed to attain the desired level of proficiency.
  • the training goals have to encompass the various characteristics in movement behavior engaged while operating in a particular activity or task. This is achieved by comprehensive assessment enabled by the hierarchical model and the movement functional structure.
  • the training goals have to provide actionable milestones that lead to an incremental improvement in skill towards the next tier, and are aligned with the larger developmental or skill acquisition path. This is achieved by accounting for the larger skill development process.
  • the present disclosure describes how training goals are identified and subsequently specified.
  • the training goals are specified as target values of skill element attributes.
  • the target skill values used to formulate training goals are computed from the individual's performance data, and extended by population data.
  • the general approach is based on a statistical model describing the individual's skill elements and skill profile.
  • FIG. 19 shows the distribution for some example skill attributes. Skill levels are captured through some objective function which is shown in terms of its level lines (shown here as low, medium, high, very high). The information specifies the direction the attributes have to be changed to achieve a higher skill level.
  • the tiers can be derived from the individual's data or the data obtained from a larger population.
  • the present disclosure further describes the computational framework needed to determine training goals and manage the training process.
  • the framework is based on a skill development or acquisition process model and, as already discussed, builds on the movement and skill model elaborated in U.S. Patent Application Publication No.
  • This training process model accounts for the development of the skill as the acquisition of a repertoire of skill elements. This process extends over larger periods of time and is influenced by a broad range of factors. Characterizing the skill development as a sequence of formations of movement patterns, i.e., skill elements, it is possible to analyze the acquisition process, and actually apply the gained knowledge to optimize an individual's skill acquisition process.
  • the present disclosure extends the skill model to account for the skill acquisition process.
  • This process is formalized as series of transformations in movement technique, which describe, the longitudinal development or acquisition stages for each skill element (characterizing the brain's and motor system's natural learning process for the formation and consolidation of the movement patterns), and how these manifest into the movement functional structure, and overall skill profile.
  • Typical learning processes are described by learning curves. However, these don't capture the details of structural changes associated with learning complex movements.
  • the overall goal is to evolve a subject's MFS along the larger skill development process following stages that are best suited for an individual, and their overall performance or skill goals.
  • the latter depends on a broad range of factors, including desire/motivation, needs (e.g., for professionals), as well as the various individual factors that are determined by biological and health conditions.
  • the specific sequence of acquisition can, on the one hand, be determined by the task requirements, specified by interactions (outcomes and conditions) that can be relevant to the performance of the task, and on the other hand, the individual's factors that determine what is feasible given, for example, the current skill level, the neuro- motor and physical factors involved in the development of coordination.
  • Learning process can be characterized in terms of the skill acquisition stage, which provides the information to determine the best type of intervention, drivers, and activity to pursue the training goal.
  • Population data is able to capture the larger set of factors and therefore provides useful information to help orient and schedule this process, and at the same time account for these individual factors, i.e., how different body types, injuries, or health conditions affect the skill acquisition process, skill profile, and overall performance.
  • the details of the larger skill acquisition process are determined based on movement data collected from a population of performers.
  • the population data provides understanding about the global characteristics of the movement skill acquisition that emerges when taking into account the broad range of factors expressed in the population that affect this process. In essence, it makes the extracted information actionable by contextualizing it.
  • the general idea therefore, is that learned global population characteristics can help support individualization of training and rehabilitation.
  • the individualization is supported by providing reference data that relates an individual's skill attributes and skill profile to the larger population.
  • This data provides both local reference about the skill attributes, e.g., how much specific attributes have to be improved to gain in skill level.
  • It also provides more global reference about the longitudinal skill development from that local skill status, e.g., what aspects of the movement skills have to be optimized and in what order to produce favorable long-term development (e.g., faster progress in skill level and lower incidence of injuries). Therefore, information extracted from a larger population can help direct both the local performance and the more global, long-term training process (what aspects to focus on first, etc.).
  • the present disclosure also details a computational model of the skill assessment and diagnostics specific for population analysis and the extended task performance level.
  • the population analysis builds on the skill profile derived from the movement hierarchical model in U.S. Patent Application Publication No. 2017/0061817.
  • the skill analysis from the population perspective is defined under the concept of a player profile which describes the skill attributes in the context of a larger population to capture the type of player based on the type or style of movement technique.
  • the player profile can also encompass the higher-level characteristics such as game strategy that captures how the movement patterns are utilized or exploited in the settings of a task or activity.
  • FIG. 30 describes the assessment and diagnostic process incorporating the player profile which is applied across the movement system hierarchy (see FIG. 6).
  • the main components are: (a) determination of the movement classes in the repertoire that are in formation, consolidation, or optimization stage (based on the individual's skill acquisition stage); (b) determination of how these patterns are used in the performance of the task (e.g., based on use frequency and game or performance strategy); (c) identification of which aspects of the skill element needs to improved, e.g., the quality of the primary outcomes, which can be achieved by interventions at different levels of the motor control hierarchy, from task level attention to deeper movement technique (based on attributes).
  • the skill analysis incorporating the player profile, enhanced by the reference values derived from the population analysis makes it possible to account for a broad set of factors needed to support individualized training.
  • the present disclosure also delves deeper into the system architecture that supports data-driven augmented training.
  • the delineation between the different modalities of augmentation instructions, cues, apparatus
  • their deployment across the human information processing system and the data and information management infrastructure.
  • FIG. 22 depicts the main elements of the augmentation system architecture delineating the augmented activity (with feedback cueing and/or apparatus interactions), the human system augmentation loop (with communications and UI systems), and the training management and configuration loop driven by the training agent system (not shown, which performs the modeling, assessment, and diagnostics to identify training elements that can be activated as training goals).
  • FIG. 22 also highlights the primary tiers of augmentation that leverage human's natural information processing levels.
  • These include "cognitive level” information, which is communicated symbolically, verbally or visually (here as instructions or notifications provided by a visual UI or natural language such as a smart phone or smart watch, eyeglasses, etc.). Instructions and other forms of information such as notifications are provided by a communication system that can include a visual display for text and graphical objects and natural language processing system. Instructions are typically designed to help subjects' understanding of their technique and performance.
  • the "feedback cue level” describes information communicated via some cueing system (here an audible signal) but can also include visual or haptic systems. And the
  • signal level which includes both cueing signals and activity interactions (e.g., ball machine) that are typically delivered concurrently with the movement performance.
  • FIG. 23 illustrates the augmented human performance associated with the feedback cueing system emphasizing the low-level signal and cues within a typical perception-action loop.
  • the movement data is processed in real-time by the cueing system to compute cue stimuli designed to help performer improve a specific aspect of the movement e.g., by acting as reinforcement signal.
  • cueing can also be used to help focus attention to relevant elements of the task environment including, for example, the location of a task object (tennis ball) or features of that object (ball trajectory) or features of the adversary's movement that can be used to anticipate the result of the adversary's movement.
  • Anticipatory information can for example help the subject select the movement pattern.
  • feedback cues also include cue signals that can be used by the subject to time the execution of the movement.
  • the various forms of feedback augmentations are computed by algorithms that have been synthesized based on the subject movement and skill model for the current epoch and historical records, and can also include reference data from larger population.
  • the training process is formalized within a computational framework with similarities to iterative learning.
  • the framework describes the management of data sets used to support skill assessment and diagnostics, which include motion and skill model (skill profile and the player profile), and the training goals and synthesis of
  • the data management process encompasses: i) the creation of data sets, models, baselines; ii) tracking their validity; iii) and updating these quantities to support an effective augmented training process.
  • FIG. 49 illustrates the top-level logic diagram of this process and
  • FIG. 25 illustrates the process flow diagram, highlighting the activity across the stack of processes of the assessment and training loop (data acquisition and processing, motion model, skill model, training goals, augmentation laws) as a function of time. During each session, activity data is collected and processed.
  • the motion model, skill model, training goals, and augmentation laws are typically updated based on their validity with respect to the new session data.
  • the processing stack for the feedback augmentation (cueing system) isn't shown here. For example at n-3 a full update is implemented following changes in motion architecture. At session n-2 the motion model is still valid and the remaining parameters don't need updating.
  • the skill model is validated and progress on training goals prompts an update in training goal and augmentation laws.
  • the skill model is updated (skill status) and new training goals are determined along with augmentation laws.
  • the training process is delineated in sets, sessions, and epochs.
  • the former two are time periods needed to organize activity and training (see FIG. 58).
  • the epochs correspond to time periods that correspond to the use and updating of the data sets supporting the computation and processing of quantities.
  • a new epoch starts when the movement technique and performance has evolved beyond the validity of the current models.
  • Each epoch typically encompasses a set of training goals for the range of movement classes in a repertoire that will drive the next increment in skill level.
  • the more recent sets of movement data are used to create new motion and skill model baselines, all the parameters used in the movement processing algorithms (e.g., classification), and the other algorithms supporting the computation of skill attributes, training goals, and synthesis of various feedbacks.
  • the temporal structure introduced by this system provides the basis for the management of the training process.
  • the structural patterns in the acquisition process can inform how to compute trends, generate reference data, as well as other critical capabilities of the data-driven training system.
  • the motion data, model, skill profile, and all the training elements, when extracted over time can also be used for bootstrapping recovery or rehabilitation following injury or other causes of interruptions in training or practice.
  • Movement skill can be categorized into two primary groups. The first, the so-called closed motor skills, involve a stable environment where the performer initiates the action or movement. These conditions allow selecting the best movement or action to achieve the task objective. Closed motor skills therefore can typically be learned and perfected in a systematic way by identifying conditions and training movements in these corresponding conditions.
  • Open motor skills involve a dynamic environment with changing conditions and require responding to the task and environmental conditions. These conditions also require a broader range of movements and actions to adapt and achieve the task goals. Open motor skills typically involve learning a large repertoire of sensory-motor behaviors and associated perceptual mechanisms, as well as planning mechanisms. The broad range in system state and task conditions makes it difficult to understand what movement patterns to train. The performance under dynamic environments and conditions also makes it harder to create meaningful training task conditions. Furthermore, it is difficult to predict the specific range of conditions that need to be trained because the behavior results from the dynamic interactions between the performer and his or her environment.
  • the neural system supporting motor control is organized hierarchically to enable efficient encoding and programming of the movements.
  • a central theory in motor control is that to mitigate the complexity associated with the large amount of degrees of freedoms (DOF) (resulting from the numerous muscles and joints), movement patterns take advantage of so-called muscle synergies (see Bernstein, 1979).
  • the synergies encode the coordination between groups of muscles and joints and thereby reduce the DOF that need to be controlled. They represent the functional elements of a hierarchical and modular representation that can be efficiently employed by the central nervous system to program and execute complex movements.
  • the tennis stroke is an action directed at returning a ball that is itself moving relative to the player and court, and the execution of a tennis stroke also depends on the body's motion relative to the court and the ball trajectory. Accounting for all these dimensions results in a potentially intractably large amount of information that needs to be extracted and encoded.
  • one aspect of operating effectively in skilled movement tasks is the automation of processes associated with environment perception and organization of behavior to exploit the natural structure of flow of information and behavior dynamics, respectively.
  • Such a strategy minimizes the amount of information that needs to be explicitly processed from sensory signals, information that needs to be programmed for the motor actions, and information that needs to be stored.
  • Affordances can take a broad range of forms. They can be static, such as chairs affording a person to sit, or dynamic, such as stairs affording climbing.
  • researchers have developed and adapted the concept of affordances to specific domains. Norman, for example, adapted it to the domain of human computer interfaces (HCI) where good interfaces convey action possibilities in forms that are readily perceivable by users (Norman, 1999).
  • HCI human computer interfaces
  • the affordances are specified based on the dynamics of the agent-environment system.
  • Such systems are typically complex, high- dimensional nonlinear systems, with numerous components interacting through their processes and physical components, including body segments.
  • Complex, nonlinear dynamic systems are characterized by emergent behaviors (see Davids 2008 for emergent behavior in human movement).
  • the physical system and the muscle and sensory-motor supporting movement coordination, along with the various processes needed to interact with the task and environment elements, form a complex system. Therefore, the overall evolution of the movement patterns and their properties are emergent phenomenon.
  • Interaction patterns are agent-environment dynamics that are exploited to achieve efficient learning and programing of motion behavior. They have been shown to represent behavioral invariants that satisfy properties of equivalence relations (Kong & Mettler, 2013). Therefore, they provide an efficient decomposition of the complex, high-dimensional agent-environment dynamics into small sets of behaviors. Similar to muscle synergies in body coordination, but here describing the coordination of agent behavior relative to its environment.
  • interaction patterns can be exploited by humans or animals, and provide functional capabilities needed to achieve adaptive and robust performance in complex environments. Besides helping with the organization of behavior, where they play the role of unit of organization, the interaction patterns are manifestations of the functional structure of sensory-motor functions. Therefore, the interaction patterns also represent a type of functional unit that helps with the organization of the system- wide integration between different processes (control, perception, and planning) (Mettler, 2017).
  • affordances can be formalized as emergent properties of a complex dynamic system.
  • the understanding of behaviors as interaction patterns emerging from the agent-environment dynamics provides additional insights about what is learned, and therefore helps determine how this implicit knowledge acquired in a domain of activity can be modeled.
  • Sensory-motor skills condition the interactions between the agent dynamics and the task and environment elements, and therefore, viewed from larger perspective they determine the affordances available to the operator or agent.
  • the human motor system has evolved to manage a variety of movement tasks that involve interactions with environment elements, while efficiently handling uncertainties, disturbances, and contingencies arising during the performance. While the human movement system has tremendous potential, systematic and dedicated training is required for high levels of motor facility. This requirement for training is similar in any domain of activity, such as athletics, music, or vocational tasks. Movement task constraints can be divided into extrinsic and intrinsic factors. Extrinsic factors include the interactions with the environment such as the foot strike or impact of the ball on the racket. Intrinsic factors include the biomechanics, human motor control, and effects arising from the manipulated equipment's dynamics. Most skilled behaviors are so-called deliberate behaviors that are directed at achieving specific outcomes.
  • learning movement skills involves changes in the cortex as a result of neuroplasticity. These changes, however, follow a specific process that is dictated by the organization of the various cortical structures (cerebellum, parietal cortex, pre-motor and motor cortices, and the prefrontal cortex). As a result, movement skills are best acquired early in life when the brain is still developing.
  • motion primitives are related to the concept of "motor equivalence" which has been identified as one of the fundamental characteristics of motor behavior. The idea is that the same movement behavior can be repeated in various contexts and without changing the overall form of the motion. Therefore, segmentation of human movement behavior into motion primitives has been most successful from invariant characteristics in the performance that arise from symmetries and equivalences in the problem space.
  • the next level of primitive represents the segments that can be combined sequentially to compose movements. This is due to the brain's efficient encoding which exploits principles of modularity.
  • the last level of decomposition is related to the so-called muscle synergies, which represent the movement components that describe the parallel combination of different muscle activations and the associated body segments displacements.
  • the top-level primitives are considered the primary motion units, which support the interactions with the task and environment elements, and the lower two levels, the movement phases that specify the movement architecture and the synergies, provide understanding of the functional properties in relationship to the environment interactions and the biomechanical constraints.
  • Muscle synergies can be obtained from factorization methods (e.g., principle component analysis or non-negative matrix factorization).
  • factorization methods e.g., principle component analysis or non-negative matrix factorization.
  • the general idea is that many movements can be described as variations of a general model and once the general category of movement is specified, some of the mechanisms needed to achieve robust movement performance are those that allow adapting to those movement pattern to changes in conditions and transferring them to different contexts in a similar task or activity.
  • Feedback types can be delineated in terms of their temporal activity and the specific levels of the control hierarchy at which they operate: Real-time feedback, taking place during performance; feedback immediately following an action, such as based on information from the movement outcome; and feedback at the end of a training set or session.
  • Inherent feedback associated with the feel, look, sounds, etc., of movement performance, as well as the movement outcome and interactions with the task and environment, can provide large amounts of information that can be used to assess performance and help train. Individuals, however, have to learn to recognize and evaluate those sources of information.
  • Natural feedback describes feedback signals at each of these forms that are inherently present in the task-environment and movement associated with an activity.
  • FIG. 24 illustrates the natural and augmented feedback based on cues and interactions.
  • the cueing system operates by augmenting the natural cues that are available to the performer, e.g., from the movement outcomes, task environment (task elements and objects, adversary's movement, etc.).
  • Task interactions are produced by an apparatus that is coupled with the activity and possibly with the subject.
  • Augmented feedback is information that is supplementary to inherent information about the task or movement.
  • the two major categories of augmented feedback are recognized in the literature: knowledge of result (KR) and knowledge of performance (KP).
  • KR represents post-performance information about the outcome or goal achieved. It is sometimes called reinforcement. Note, however, that not all movements have an outcome that is separable from the movement performance.
  • KP represents information about the movement technique and patterning. This information is useful for the acquisition of complex movement skills, such as those requiring high- dimensional spatial and temporal coordination. Previously, it was difficult to measure and track performance in many activities. The advent of MEMS movement sensors has created a wide range of possibilities for using information about movement kinematics and dynamics (kinetics) from measurements.
  • At the center of the neuro-motor system is a specialized system that deals with the formation of complex movement patterns, especially the chunking and sequencing of movement phases.
  • Feedback mechanisms use information from cues extracted by visual, auditory, and haptic sources.
  • the task of this system is to fine tune and synchronize behavior with external tasks and environment elements, such as adapting timing of movement phases, or modulating phase profiles.
  • the phases are typically part of a sequence generated by the cortical circuits.
  • the highest structure is the cortical system used for perception, planning and execution.
  • This system combines the various sources of sensory and perceptual information to build representation that can be used to generate plans and monitor the performance and outcomes of the behavior.
  • This system can also handle abstract information such as that in verbal or written form.
  • the human information processing model helps provide an understanding of what type of feedback information is most useful, and for which components of movement behavior these feedbacks apply.
  • TABLE A summarizes the type of signals, cues/signs, and symbols in tennis as an example.
  • knowledge-based behavior corresponds to the type of stroke and body positioning, etc. to use given the information about the overall situational awareness, such as adversary behaviors gained from exteroceptive information.
  • cues trigger behavior.
  • signals are used to modulate muscle responses.
  • the rule-based behavior involves determining which pattern to activate based on the signs or cues typically obtained from the exteroceptive information.
  • cues are used for time movement execution. For example, the particular state of the ball extracted visually, such as the impact, may be used to signal the instant to initiate the backswing or the forward stroke, and modulate the strength of the initial acceleration.
  • the skill-based behavior corresponds to movement patterns. Signals are primarily the proprioceptive information.
  • the delays and time constants of the sensory-motor system are too large to provide continuous feedback corrections for fast-paced skilled movements.
  • the neuromuscular time constant time from the signal to go from the motor cortex to the muscle response
  • the response time from visual or auditory stimuli to a physical response is of the order of about 200msecs.
  • Motion skills assuming training within specific known outcomes, primarily involve the skill-based and rule-based behaviors.
  • the symbol level is relevant to form mental models, for example movement architecture and functional characteristics including the environment cues. However, it is primarily relevant at the level of task and competitive performance, such as planning and strategy.
  • FIG. 24 describes the main components used for real-time feedback
  • the feedback augmentation comprises two primary forms of augmentation: feedback cues and interactions.
  • the cueing system achieves its effect by augmenting the natural cues that are available to the performer, e.g., from the analysis of the movement outcomes, the real-time analysis of the movement technique, or event generating cues that pertain to the task environment such as the behavior of task elements and objects, opponents' movement or actions.
  • One consideration is that the natural feedback environment is usually very sparse. Not many relevant quantities are directly observable by the subject or operator. Therefore, augmentation can be conceived as the supplementation of the useful signals and cues that the brain can take advantage of to improve movement performance and learning.
  • the augmented cue environment is designed to help the human perform and train for the task. Task interactions are produced by an apparatus that is coupled with the activity and possibly with the subject to enhance the scope of conditions.
  • the apparatus can also include an assistive device that mechanically augments human movement. Note also that this configuration also applies to settings where humans operate in teleoperation such as a surgical robot, where the subject interacts with the system through a visual display and haptic interface, or even in the context of the operation of prosthetics.
  • KP feedback that are most useful are those that contribute to the understanding of the task or movement. This explains why providing a type of normative reference trajectory, e.g., to model after, is not necessarily useful. In that sense, KR has the advantage that it provides objective information about the implicit correctness of a movement.
  • KP feedback contributes to understanding the task or movement. This can be achieved by using movement kinematic and dynamic measurements that produce KP that is connected to the movement outcomes, as well as organized in terms of timing and form, etc. in ways that are consistent with the movement's functional dimensions, including biomechanics, motor control, and sensing or perception mechanisms.
  • Movement analysis includes at least three components.
  • the first component involves decomposing the movement into primary movement elements or units. Units are typically associated with subtasks or subgoals that depend on the elements of the task and environment giving rise to the task stages. These units manifest as movement patterns that emerge from the functional characteristics of the movement interactions with some elements of the task or environment within a task stage, and therefore these units are also named movement patterns in this disclosure.
  • Second is segmenting these movement units into the sequence of movement phases.
  • third is decompose into components that can be combined in parallel to achieve the coordination of the body segments and muscles, i.e., muscle synergies.
  • the movement profiles there are three primary levels of movement organization, including i) the movement profiles and their associated outcomes.
  • This level corresponds, for example, to task level description and represents the overall movement element or unit such as a tennis stroke in tennis.
  • the movement profiles are usually composed of series of multiple phases.
  • This level corresponds to the biomechanical implementation, i.e., the coordination of the limb segments and joints to achieve a complex movement.
  • the movement phase profiles can then be decomposed into muscle synergies.
  • This level corresponds to the neuromuscular implementation, i.e., how the profiles are achieved by superposition of muscle units.
  • the muscle synergies represent muscle activation patterns.
  • the first organizational level corresponds to the building blocks developed by the brain through interactions with the environment and task elements to partition the workspace efficiently, and achieve a range of outcomes relevant to a task. It is related to what could be considered the semantic characteristics, i.e., the meaning of the movement elements in relationship to the task goals, elements, contingencies, and the range of conditions.
  • the second level, the phase segmentation corresponds to the functional structure of the movement, and is related to the strategy used by the nervous system to achieve the particular outcome given the available neuro-muscular system.
  • the third level describes how the various muscles are activated to achieve the movement profile at the phase level.
  • the synergies typically provide spatial and temporal components that can be combined to achieve a variety of movement. Therefore, it is expected that same set of synergies can be reused by other movements. Yet, for example, in tennis the arm segments configuration can be very different at different stroke phases, therefore it is likely that different sets of synergies are used in each phase.
  • complex human movements are high-dimensional, i.e., their description requires large numbers of state variables (position, velocities, angles).
  • the representational complexity is in part due to 3-dimensional (3D) space which involves six degrees of freedom for the linear and angular motions.
  • control and trajectory optimization framework provide useful tools for the conceptualization and analysis of movement. For example, it is possible to define cost functions that characterize human trajectories, such as energy or more general physical performance. Furthermore, the calculus of variation used in trajectory optimization make it possible to investigate relationships between variations in trajectory and outcomes of the trajectory.
  • Movement measurements such as from wearable motion sensors or optical motion capture systems, are typically given in the form of time series. Since these time series typically originate from nonlinear dynamic processes, their analysis relies on an understanding of the structural characteristics of the underlying dynamics. These structural characteristics are associated with the architecture of the movement, such as the movement phases in a tennis stroke or golf swing. Insights can be gained using computational visualization tools such as phase space; however, the states may have too many dimensions to be practical. Therefore, the data should be reduced.
  • the behavioral data captured from the available measurements results in a high-dimensional state space.
  • the dynamics driving the behavior may be lower-dimensional.
  • Dimensionality reduction is a class of unsupervised learning techniques that can be used to discover the state dimension of the underlying behavior. The goal is to transform the original movement data time series which are described in terms of the high-dimensional time series x t into a lower dimensional description that preserves the geometric characteristics of the underlying nonlinear movement dynamics. This can be done, for example, using Taken' s embedding theory. Examples of recent applications of dimensionality reduction for movement analysis include gait analysis.
  • the movement architecture can be analyzed by focusing on the low
  • Using a nonlinear dynamic systems formulation gives access to analysis and modeling tools that, under certain conditions, can reconstruct the pattern dynamics from measurements of the behavior. The reconstructed dynamics can then be analyzed to determine the underlying structure and geometry, which can then be used to determine useful abstractions or models.
  • F t is a map
  • x t E W 1 is the state vector at discrete time t E M
  • e t is a time-dependent noise.
  • a continuous time representation could also be used.
  • the nonlinear model of a movement pattern therefore can be described by a map F that captures the combined effects of the biomechanics, sensory, and motor- control processes.
  • This model assumes that the learned movements result in deterministic dynamics.
  • ODE differential equation
  • x f(x(t), e(t)) > which describes a vector field and is typically called the flow.
  • the set of initial conditions which result in the same asymptotic behavior are referred to as the basin of attraction.
  • Such nonlinear dynamic models can describe a broad range of phenomena. The model could be decomposed into subcomponents, giving access to the various contributing systems and processes.
  • a property of movement at the highest level is referred to as "motor equivalence.”
  • the fact that the brain generates movements that are equivalent in terms of their accomplished outcomes underscores the idea that at the highest level the brain encodes outcomes and their relationship with task goals.
  • the planning and monitoring functions associated with goals are part of the brain' s executive system. For example, in tennis, the player selects a stroke type based on the desired outcome and the conditions (ball state including expected impact height, velocity, and spin of ball). Even within the continuum of conditions and outcomes, it is possible to recognize distinct classes of strokes.
  • the invariant characteristics in movement features enables the delineation between movement classes, e.g., movements within one particular class can be related through some smooth transformation such as rigid-body translation and rotation, i.e., they are invariant under this class of transformation.
  • the overall movement class can be subdivided into subclasses. For example, a hierarchical decomposition would group movements based on relative similarity.
  • the overall stroke class can be subdivided into dozens of subclasses based on movement where the levels represent different types of features.
  • a top hierarchical level is called the category level. It differentiates between groundstroke, volleys, serves, etc.
  • the distinction between stroke categories is made primarily based on the height of the impact point. Further, subcategories can be created based on the side of the impact, i.e., forehand or backhand. Even further subclasses can be delineated based on the outcome (topspin, flat, slice), and strength. Beyond these common classes, finer distinctions can then be added based on additional aspects of stroke technique, such as open or closed stance. Most of the stroke characteristics can be determined entirely from the racket trajectory and therefore do not require additional measurements such as the position of the player on the court.
  • Each movement pattern class in a repertoire has different geometrical characteristics and their domain may occupy a different subspace of state-space (see FIG. 14).
  • the shape and dimension are a result of the dynamics, which is given by the transition map F.
  • the repertoire is the collection of these shapes or patterns.
  • the precise geometrical characteristics of the movement patterns can be described via embedding theory. The idea is to determine the subspace of DOF that fully describes the movement.
  • the dimensionality of the system and the geometry of the manifold that contains the trajectory describe the movement class structure.
  • the state transition map F (the dynamics), the output map h, and the dimensionality of the state vector n are not known.
  • Techniques of nonlinear time series analysis can (assuming deterministic dynamics F and smooth output map h) estimate the dynamics associated with a movement pattern from time series obtained from measurements of the behavior.
  • Movement classification has been used in other applications unrelated to skill modeling, such as activity detection or gesture recognition.
  • Gesture recognition is a growing aspect of natural human-machine interfaces.
  • the general goal in the latter application is to determine motion primitives that provide a low-dimensional description of the various movements that can occur in that domain.
  • the primitives can then be used to classify the movements.
  • the library of primitives can then be used by other agents to identify the intent of a human or robotic agent and, for example, allow collaboration between agents.
  • the emphasis of gesture classification is the identification of semantic characteristics.
  • the goal is classification based on
  • the higher categories of the stroke classification can be considered in a semantic sense (e.g., groundstroke vs. volley or backhand vs. forehand), and the lower level classes are related to different techniques and conditions (see FIG. 9).
  • a particular ensemble or repertoire of patterns in a domain of activity arises through the effects of biomechanical, neuro-muscular constraints, as well as task-related constraints.
  • the patterns describe how an individual's movement techniques are used to achieve an outcome.
  • One aspect of the movement characteristics is how they are broken down into phases. Overall movement pattern characteristics, therefore, are the result of the phase structure, and those can be used to classify motion patterns.
  • the serial order in behavior i.e., the task stages, and the movement phase structure are usually distinct.
  • the serial order is associated with the activity level, for example, characteristics related to the activity constraints such as process stages, rules, etc.
  • the movement phases are associated with the movement technique and are related to characteristics and constraints of the movement system and its interactions with the environment and task elements.
  • the movement stages associated with the serial-order of behavior include serving, then moving to the anticipated return position, making adjustments in the positioning as the ball returns, setting up for the stroke and engaging the ball using the stroke type required for the desired outcome (see FIG. 7-9).
  • Each stage can be parsed to extract the primary movement unit, and these movement patterns can then be analyzed to determine the movement functional characteristics, i.e., how the movement produces its specific outcomes while at the same time adapting to conditions.
  • the functional analysis is facilitated by further decomposing the patterns into movement phases.
  • the phase structure of the primary movement patterns defines the topological characteristics of the manifold, while the dynamics that drive the phases define its geometrical characteristics.
  • Phase structuring of patterns typically arise from the intrinsic movement constraints (biomechanics), some aspects of task constraints, as well as functional factors related to motor-control and decision mechanisms as discussed elsewhere. For example, in gait, distinct phases are associated with the basic leg biomechanics and mechanics of ground interactions.
  • the general goal of the user is to return an oncoming ball and further control the trajectory of that ball (see FIG. 7). This is accomplished by imparting precise linear and angular momentum to the ball with the racket.
  • the user controls the ball primarily by modulating the amount of momentum imparted to the ball and selecting the precise interception point and time as the ball enters the half-court (FIG. 9).
  • the overall tennis stroke motion encompasses the kinetic chain formed by the legs, hips, shoulder and elbow, and wrist. These segments are coordinated to form a continuous movement starting from the backswing all the way to the follow through and recovery. At closer inspection, distinct phases can be recognized.
  • phase characteristics depend heavily on skill level. beginner players primarily swing the racket from the shoulder without very precise coordination with the rest of the body segments. Advanced players exploit the entire body kinematics to maximize the outcome. Ultimately, the phase characteristics reflect the combination of the body segments' biomechanics and neuro-motor strategies, including the muscle synergies that achieve the highest outcome reliability with best use of the physical capabilities. Different phases are associated with different biomechanical functions. For example, in walking, synergies that are activated at specific phases of the gait cycle (e.g., forward propulsion, swing initiation, deceleration, etc.) have been identified.
  • phases of the gait cycle e.g., forward propulsion, swing initiation, deceleration, etc.
  • trajectory segments are related to the concept of singular arcs, which correspond to segments where different sets of constraints are activated by the trajectory.
  • these systems are best controlled using switched control laws.
  • the control law is determined based on a partitioning of the system' s state. As the system is driven by the control action, and travels through the different partitions of the state-space, the control strategy switches to best account for the local characteristics of the dynamics.
  • trajectory phasing can be described mathematically as a sequence of dynamic models F 1 , F 2 , ... , F N .
  • the overall trajectory is obtained by a series of initial values and asymptotic behaviors, where the next set of initial values corresponds to the terminal values of the previous phases (FIG. 3 A).
  • the dynamics associated with each phase result from different joint and limb segment configuration and force fields.
  • Each dynamic model Fj can therefore be assigned a state-space region specified by an initial state set and a goal or subgoal set.
  • the initial dynamics F 1 will take the state to its subgoal set X t , and from there, assuming the state satisfies the next initial state conditions for the next dynamics F 2 , and the dynamics are triggered, the system dynamics will switch to the next phase, where it will evolve to the next subgoal, etc.
  • This process can be cyclic, where the state transitions form a loop, such as for periodic movements (see e.g., running 445 or swimming 446 depicted in FIG. 4). In other activities such as tennis or skiing, the behavior can be quasi-cyclic, where for example the same general sequence of movement phase continues after a pause (see FIG. 3B).
  • the dynamics can also switch between patterns that have different phase segments, such as a different stroke type or gait type, or altogether different movement patterns, such as in skiing when switching from a periodic turning sequence to a stopping maneuver, or switching between different stroke patterns in tennis.
  • the dynamics associated witch each phase result from different joint and limb segment configurations and force patterns.
  • These force patterns are determined by the spatio-temporal muscle activation patterns, i.e., muscle synergies.
  • the force patterns are specified in open loop, therefore the dynamics are specified by the force fields associated with the muscle synergies.
  • the brain ostensibly learns to compensate for variations in initial conditions by adapting these force fields. This makes it possible to produce fast corrections in movement without relying on feedback.
  • Feedback can be used intermittently, e.g., during phase transitions or during specific movement phases which can accommodate such effects, e.g., because of the slower dynamics and availability of sensory information.
  • the muscle synergies describe the coordination between the different muscle groups and limbs segments that are used to implement movements.
  • the synergies are a type of motor primitive which is typically reserved for the neuro-muscular coordination.
  • various movement profiles observed in an activity can be obtained through the combination of such primitives.
  • Decomposition into synergies therefore can help determine the set of biomechanical and neurological components that participate in movement skill. In turn, this information can be used to gain understanding about the biological components, and could be useful for physical performance and injury prevention.
  • Some movements have an explicit outcome or goal.
  • the ball impact is the primary goal or outcome of the stroke, this phase is not the actual end of the movement.
  • the movement phase following the impact, the follow through is one part of the overall movement pattern.
  • Most complex movements involve many body segments or degrees of freedom. Therefore, the state trajectory is a multidimensional state vector and it can be helpful to add distinctions between the different state trajectories that participate in the action.
  • Focal and corollary movements are distinguishable; the focal movement is, for example, in a piano performance, the finger movement that hits the key; the corollary movement is, for example, the motion of all other fingers that are part of the overall kinematic pattern involved in the task of hitting the key.
  • the absolute optimal trajectory is the global optimal solution for a given outcome, while the local optimal trajectory corresponds to a given phase structure.
  • the latter for example, represents situations where due to a lack of flexibility or skills, or the presence of an injury, only a limited set of configurations and/or force fields is achievable. Therefore, movement phase characteristics provide valuable information for injury prevention and generally also for rehabilitation.
  • perturbation of the initial value leads to neighboring optimal trajectories. This is guaranteed if the initial value is within the so-called basin of attraction of the system.
  • a similar idea can be used for perturbations in the dynamics F.
  • FIG. 3A illustrates the trajectory envelope 113 for a hypothetical movement pattern delineating the movement phases that typically arise from biomechanical and neuromotor constraints. The figure also highlights a primary outcome and its associated phase (shown as a goal phase). It also shows an optimal trajectory across the movement phases, and different envelopes (optimal, admissible, feasible) resulting from the various movement constraints.
  • the trajectory envelope delineates a region of the state-space over time and highlights the feasible envelope and the envelope of admissible trajectories as well as the region for the optimal trajectory's initial conditions (x*oi), and the optimal trajectory (x*(t)).
  • the structure of the movement both in terms of patterning and the phase segmentation are given by its spatio-temporal characteristics. Movement characteristics are defined by the geometry and dimension of the manifold containing the trajectory.
  • phase 1 movement initiation
  • phase 2 an intermediate goal phase
  • a follow-on phase recovery phase
  • these phases correspond to the stroke initiation, backswing, back loop, forward swing, impact, follow through and recovery.
  • the goal phase in tennis represents the impact phase, which is the phase during which the primary outcome is produced.
  • movement pattern characteristics are usually determined from the topology of the movement pattern manifold obtained from analyzing the nonlinear time series.
  • a user may choose "admissible movements” that belong to the same movement pattern and still reach the goal conditions or outcome. This could happen due to changes in movement goal conditions (impact height and velocity), or imperfect initiation of the movement.
  • the suboptimal trajectories can still reach the desired end state or outcome; however, they will typically require more physical effort, may cause stress in some of the muscles or joints, or other undesirable effects.
  • the physical performance can be described through models of the musculoskeletal system and cost functions such as for energy consumption.
  • Movements belonging to the same pattern can therefore be related through perturbations relative to a nominal trajectory.
  • the trajectory perturbations also result in perturbations in the primary outcome and any other secondary outcome characteristic such as the different phase outcomes.
  • This information provides a quantitative basis to generate skill characteristics, such as what aspects of the technique contributes favorably to the outcomes and vice-versa what aspects are detrimental to good outcomes. This knowledge in turn can be used for training and eventually help synthesize feedback laws for real time cueing.
  • FIG. 3B is an illustration of the finite-state model representation 114 for the system shown in FIG. 3A.
  • the pattern dynamics can be abstracted as a finite- state model (see FIG. 3B and FIG. 5).
  • the finite states are the individual phase dynamics , which take the system from initial value x,o to the next subgoal state xn. More generally, the initial and subgoal states are represented by sets to account for the variations and disturbances that are typically expected in human behavior. With this model, the overall motion behavior is then given by some finite-state automata which gets triggered from the initial state and initial movement phase.
  • the motion behavior combines both continuous dynamics and discrete variables that capture phase transitions and mode switching which may be associated with discrete decision variables.
  • Hybrid models can be used in many modern engineering applications including robotics such as for autonomous systems, as well as human-machine systems. Once the structure of the motion is characterized, it can be described by finite-state models.
  • Statistical models in contrast to deterministic models where the current state uniquely determines the evolution of the system (i.e., within the disturbance or model uncertainties), describe the evolution of the probability density of future states.
  • Statistical models such as Dynamic Bayesian Networks have become increasingly popular in data- driven approaches. Popular applications in the movement domain are identification of human activities. These approaches typically require learning the phase of activities based on statistical pattern analysis; subsequently using this knowledge to discretize the state space into discrete states; and finally determining the state-transition probabilities.
  • a common model is the Hidden Markov Model (HMM). Most of the notational systems focus on the discrete game structure and can be used to analyze game plans but currently do not reach down to the actual movement skill level.
  • HMM Hidden Markov Model
  • Real-time movement phase estimation can be implemented by someone trained in the art.
  • a multi-layer HMM application to movement could be based on similar models to those used for real time speech recognition.
  • Decoding sound recording for speech recognition typically proceeds on multiple levels. Most of those are associated with the levels of organization of the speech production system.
  • the units of decomposition of speech is based on phones which combine to form the phonemes.
  • the phonemes are the basic building blocks used to form words.
  • the phones are related to features of the vocal movements.
  • This model for movement corresponds to having, at the top level, a movement phase model which describes the probability distribution over possible sequences of movement phases. At the midlevel, a phase model that describes the composition of the movement phases in terms of movement components (c.f.
  • the movement model that describes the movement components based on features in the available measurements (IMU unit or other sensors).
  • Chunks have been extensively studied in domains that involve static and discrete quantities, such as the perception or memorization of chessboard configuration.
  • Early chunking theory has been studied as part of human perception and more generally information processing in (Miller, 1956). Many activities are described by a complex spatial and temporal structure. Later, the chunking theory has also been applied to improve our understanding of motor learning and more generally skill acquisition. There exist fewer investigations in the sensory-motor domain. In that domain, chunking is primarily associated with the concept of "serial order in behavior" introduced by Lashley (Lashley, 1951), and the general hierarchical learning theory.
  • the hierarchical models conceive complex skills as a "hierarchy of habits.” This model was introduced by Bryan and Harter (1897) studying Morse code learning. In that example, the telegrapher learns letters first, followed by sequences of letters to form syllables and words, and then phrases. This model applies to many motor skill domains. In most movement skills such as tennis, the elementary actions are movement phases (muscle synergies) that can be combined to form gross movements. Learning such skills, therefore, involves learning elementary movement units, and combining those into larger movement elements that are themselves nested into actions.
  • Lashley' s serial order in behavior was a response to the linear sequencing that was suggested based on association learning theory (Terrace, 2001). Instead of a serial sequence, Lashley argues that skilled behaviors are planned, and plans have a hierarchical organization which combine multiple units of behavior into larger units. Some units are related to a movement's biomechanical and functional constraints, and others are related to task constraints (e.g., subgoals).
  • Chunks are usually not made of arbitrary segments but have a functional purpose. Chunks, therefore, combine specific sensory and motor patterns that relate to the task environment interactions, as well as the constraints of the organism.
  • major behavioral chunks can include the "ready state,” “reposition,” “preparation,” and “stroke execution.” Each chunk can be described by a set of movement patterns with their associated perceptual process.
  • the ready state the player orients himself or herself, extracting cues from the environment needed for court positioning, observes the motion of the ball and the opponent, etc. This information allows prediction of the location of anticipated ball interception selecting the desired outcome and planning the sequence of actions to achieve the desired outcome of the stroke.
  • the player acquires the new court position and may start to bring the racket back (backswing).
  • the player adjusts his or her posture and extracts updated information about the ball and opponent needed to fine-tune posture and prime the stroke execution. Just before the stroke execution, the player obtains final ball trajectory information for the interception. The execution of the forward swing is synchronized with the arriving ball. Finally, after the execution of the stroke, the player returns to a ready state.
  • the behavioral chunks forming the larger program are typically subdivided into smaller sensory-motor units, starting with the elements such as muscle synergies that are combined to form larger movement patterns.
  • the stroke is composed of a sequence of body and arm movements (described elsewhere).
  • extracting information involves a type of perceptual chunking which describe how the various sensory stimuli are integrated to form the cues that can be used to predict the intentions of the opponent, anticipate the ball trajectory, and select and initiate the appropriate stroke type.
  • Proficient individuals are able to focus attention on task-relevant information, which enable better planned, more systematically organized behavior with fewer extraneous movements and smoother movement execution that takes advantage of the subject's physical performance.
  • the chunking theory of learning also provides additional understanding of the learning process. For example, it has been used to explain the so-called Power Law of Learning (see Newell, 1981). This law describes the improvement in skill (measured as response time) as a function of training and has been validated in many domains besides perceptual motor tasks, hence it is widely accepted as a universal law. However, the law has received criticisms, in particular that it does not explain qualitative changes in movement dynamics with practice (see Newell, 1991). As described in that reference, these may be due to the limited tasks used in studies (few degrees of freedom and limited perceptual environment).
  • the cognitive stage (also called verbal stage) is characterized by a conscious effort required to understand and control the movements. As a result, in this stage, movements are slow, they lack dynamic coordination, and have low success rate.
  • Problem solving, by way of cognitive processes is a critical aspect for the development of mental models, or representations that could be used to support this stage (Ericsson 2009).
  • the movements are partly automated. Conscious efforts are fewer but are still required to monitor and improve performance.
  • movements are stored in procedural memory which allows automatic execution. Movement in this stage may still require visual inputs to ensure accurate and consistent execution. However, these inputs are also automated and focus on very specific elements, i.e., cues.
  • the type of knowledge gained by subjects as they learn to be proficient in a task is directly related to the structure of the task, and the structure of the interactions between the movement and the task and environment elements.
  • the critical aspect is the structure of the interaction between the subject and the task environment and elements (see e.g., the interaction patterns in Mettler 2015).
  • Cues can be viewed as sparse sensory stimuli such as, for example, in Tau theory (see Lee 1998).
  • the repertoire represents a library of sensory-motor patterns that is stored in the brain' s long- term memory.
  • the structure associated with the task and interaction between the movement and task elements suggests that sensory-motor patterns are grouped hierarchically.
  • the top sensory-motor chunks define larger categories of behavior, such as ground strokes and volleys; the intermediate level, which include the various stroke classes in a category; and at the lower level of the hierarchy are components of behavior which include muscle synergies and are shared by different classes.
  • the hierarchical and modular encoding has been known from early studies of the neural visual processing and encoding, and has been verified in the domain of movement encoding and control (Poggio 2004). For example, movement patterns within related movement classes (e.g., tennis forehand slice and top spin) share similar sub- movements. The movement phases result from the activation of muscle synergies that are encoded in part in the spinal circuits. Multiple studies have demonstrated the modular encoding of movement (Mussa-Ivaldi 1999).
  • Each class of movement learned has some operating range that defines the range of validity of the learned patterns.
  • movement skill acquisition results from the need to adapt to the task and environment, and thus learning proceeds incrementally with exposure and experience performing a task. Therefore, it is possible to conceive skill acquisition as an evolutionary process (see FIG. 11).
  • the specific skill elements are classes of movement patterns that are evolving with their usage in the task or activity. Learning and perfecting skills are the result of an iterative process that takes place as these elements are repeated under different conditions, and modified based on the observed outcomes and effectiveness to the overall task goals and performance.
  • the acquisition process can thus be described as the evolution that involves two primary dimensions: 1) the diversification of the movement patterns to respond to the range of requirements and conditions called for by open motor tasks; 2) the refinement and optimization of individual movement patterns, which corresponds to the changes in those movements over the stages of acquisition.
  • the process therefore can be analyzed by tracking the movement repertoire over time.
  • an individual's skills are described by a repertoire with one or more classes of movement patterns (FIG. 11).
  • the repertoire reflects both aspects of how the individual deals with the task and environmental structure, as well as the individual's perceptual and motor control abilities.
  • Each pattern class can be at a different stage of acquisition.
  • Sensory-motor patterns serve as units of behavior used for organizing and planning the behavior toward the larger task goals (see Mettler, 2015). Identifying the repertoire of patterns therefore also provides the elements needed to analyze the skills at the planning level. CHALLENGES IN MOVEMENT ACQUISITION AND TRAINING
  • movement skill training has depended on the expertise of a coach.
  • the patient depends on the availability of a physical therapist.
  • the traditional role of a coach is to help focus training efforts on correct technique, and attend to relevant aspects of the task and performance.
  • expert coaches are subject to limitations in perceptual and information processing.
  • Most skilled movements involve coordinating many degrees of freedom that take place over short time scales (hundredth to even tens of milliseconds). These movements, such as a tennis stroke or golf swing, are highly dynamic behaviors that combine temporal and spatial dimensions into complex patterns.
  • motion patterns depend on complex biomechanical constraints and muscle synergies. These depend on musculoskeletal constraints, as well as the physical fitness and general health of an individual. Therefore, the training approach should be able to account for individual characteristics both in the movement technique and in the longitudinal skill development process. It takes great experience for a coach to be able to analyze movement and identify relevant characteristics of these patterns while taking into account the individual's constraints.
  • open motor skills In contrast to closed motor skills, in which the conditions can be controlled, open motor skills require a broad movement repertoire in order to accommodate the varying conditions associated with the task and environment and produce the range of outcomes that help to control and pursue the task goals. In addition, not all movements associated with a task have the same importance to the task performance. Some movements are part of a basic repertoire that cover the general performance and conditions, and other movements are more specialized and allow actions in more specific conditions.
  • Skill acquisition is a parallel process where at any given time, a subject's repertoire will contain multiple movement patterns, each at different stages of development.
  • the two primary directions in the skill acquisition process are: 1) the development of a sufficiently broad repertoire to cover the task requirements and conditions, and 2) the refinement of the movement technique within each class of the repertoire to achieve better outcomes and/or movement performance as well as adjust to conditions.
  • These two directions are referred in this document as the longitudinal and vertical dimensions of skill acquisition.
  • the longitudinal dimension represents the stage of development or acquisition, which is determined by skill characteristics in specific classes of movement.
  • the vertical dimension represents the aspects of movement skills that have to be developed to cover the task conditions.
  • the training can be directed at refining a movement or diversifying the repertoire.
  • the two dimensions are typically interrelated.
  • the differentiation of the repertoire in the vertical direction often develops from the longitudinal process of refinement of an existing movement.
  • human skills rely on multiple levels of human information processing, including signals, cues, and knowledge.
  • the knowledge level supports reasoning about technique, such as particular details of the movement's spatial configuration. It also supports game strategy, taking into account the environment and task elements, etc.
  • the cue level supports the efficient processing of information; for example, the visual perceptual system learns to focus on aspects of the scene and action that provide the most valuable information for the performance.
  • the signal level typically encompasses the information used by brain processes to control movement such as proprioception or actual visual stimuli.
  • Technology can play a role in several areas of skill acquisition.
  • Technology provides a means of collecting comprehensive information about human behavior that exceeds humans' sensory processes' spatial and temporal resolution.
  • the combination of distributed sensors in the form of wearable, implantable, and remote sensors can capture comprehensive dimensions of movement performance. This includes the movement of an end effector such as a piece of equipment, an individual body segment, muscle activity, as well as subjects' visual attention and task relevant quantities (see FIGS. 2 and 24).
  • Information technology enables the deployment of analytical and computational resources beyond humans' information processing capabilities. Algorithms can be designed to estimate various unmeasurable quantities, which can be used to provide feedback on outcomes ("knowledge of results"), as well as more complex aspects of performance such as those involved with the fast and high-dimensional dynamics, and coordination with the environment and task elements. This functional understanding can be used to design feedback augmentations that target movement technique ("knowledge of performance"). Information technology enables scalable deployment of analytical and computational resources across larger populations, where it can be deployed to identify patterns in movement technique and skill acquisition processes that can take into account broad range of individual factors. However, to be effective, these different augmentations and feedbacks should be provided within a system that is compatible with the natural movement mechanisms and learning process.
  • Open motor skills require the development of a variety of movement patterns to produce desired outcomes under changing task and environment conditions. These movements and their associated sensory-perceptual mechanisms are acquired from experience in a task domain. Depending on the task or activity complexity, learning motor skills can take several years.
  • the central idea of this technology is that the movement performance at its various levels can be assessed computationally, i.e. it can be computed, and then further diagnosed to identify deficiencies at the various levels of the movement hierarchy, which are needed to determine training goals.
  • the training goals can then be pursued through targeted training activities that can be augmented by various feedback modalities.
  • the following provides a technical description of the capabilities needed to support comprehensive data-driven skill assessment, and diagnostic and training intervention for open motor skills. It introduces definitions of the relevant quantities and processes that will be formalized subsequently.
  • the fundamental element of movement behavior are the set of movement patterns that support the relevant interactions with the environment and task elements. These are also called primary movement units or skill elements. Most movement patterns are directed at producing an outcome or action toward the activity or task goal(s). The various movement patterns used by a subject in a task can be identified and classified.
  • the quality of the skill assessment depends on the ability to extract relevant movement patterns that characterize relevant interactions in the task, and to classify these patterns according to their intrinsic characteristics, i.e., the movement technique and movement phases, and their relevance to the task, i.e., the movement outcomes and the task conditions.
  • This is in particular critical for open motor skills, since the subject acquires a repertoire of movement patterns to produce a broad range of outcomes under a range of conditions.
  • performance can be contextualized, which may include identifying what movement technique is used under which conditions and to produce what outcome.
  • FIG. 11 illustrates the acquisition and evolution of movement patterns over time, highlighting the formation of movement patterns either from scratch or through a process of differentiation.
  • training or performance history shown as stages SO, SI, (7) the movement skills can be described as a repertoire of movement patterns (e.g., at S2 patterns Pl-A, Pl-B, P2-A, P2-B).
  • the width of the branches in FIG. 11 indicate the variability in movement characteristics in a given pattern.
  • beginner subjects tend to employ similar techniques to achieve a range of outcomes and conditions.
  • experience subjects learn to perfect their control over the task conditions and can develop movement techniques that are more specialized and yield higher performance (more efficient, higher outcomes, more extreme conditions). Therefore the general trend is for a subject to start with a repertoire of a few movement patterns with fewer capabilities, and with experience and training develop a larger repertoire of more differentiated movement patterns.
  • a new pattern can form through differentiation of an existing pattern (i.e., core pattern), shown here as a dashed line that indicates the beginning of the differentiation process (e.g., differentiation of PI into Pl-A and Pl-B at SI).
  • the pattern can form "de novo" such as illustrated for P3 at S3 in FIG. 11.
  • Newly differentiated patterns next go through a consolidation stage (shows as the bifurcation point at the end of the dashed line, e.g., Pl-A and Pl-B at S2) where they each become distinct patterns.
  • a consolidation stage shows as the bifurcation point at the end of the dashed line, e.g., Pl-A and Pl-B at S2
  • FIG. 12 shows several classes of movement patterns as clusters for some parameterization such as features from the measurement time histories.
  • the clusters capture the pattern differentiation that takes place as the individual improves their skills.
  • the example is based on the patterns at stage S3 in FIG. 11.
  • the patterns that form following differentiation typically appear as a mixture of two patterns, such as shown for Pl-Al and P1-B2 in the original pattern Pl-B. Patterns in an early stage of consolidation show distinct features such as P2-A and P2-B.
  • FIG. 13 shows the family tree highlighting the evolutionary relationship between movement patterns. Since some patterns form through differentiation, it is possible to track the based on features or attributes that are inherited.
  • core pattern to refer to the pattern that inherits the main attributes in the development of the new patterns.
  • the non-core patterns differentiate to create new attributes that are distinct from the core pattern.
  • Movement pattern classification is typically based on movement profile features (e.g., racket angular rate or acceleration). Movement outcomes are a consequence of the movement performance and conditions, and therefore a function of the movement characteristics (see FIGS. 3A and 3B). Consequently, some movement profile features can be used to predict or estimate the movement outcome. Viewed abstractly, therefore, the classification task corresponds to identifying the structure of the extended state-space X in FIG. 14.
  • the state-space associated with the entire human or system performance combines the typical states of the systems, such as needed to describe the subject's or agent's movement, as well as states that are associated with the task and environment elements that participate in defining the conditions in which a particular movement performance or pattern takes place. Classification therefore can be conceived as the mapping from the extended state space into its co-domain V.
  • FIG. 14 shows the mapping between the movement performance state (state- space X) and the movement outcomes and other attributes fi in V.
  • the state space highlights the partitions associated with various movement pattern classes.
  • Movement patterns are typically associated with features that originate from domain characteristics (such as geometrical characteristics of the manifold associated with task dynamics, interactions and various constraints).
  • the classification maps the state-space features to the movement attribute space.
  • Movement attributes include results or outcomes (e.g., spin, pace, etc.), as well as other attributes that can be used to assess the movement technique (consistency, timing, smoothness, etc.), or performance (energy, etc.). These attributes can be computed via analytical functions, estimated statistically, generated using neural networks or even directly measured (e.g., ball spin using computer vision).
  • Each pattern has a range of values for the particular outcome metric shown as a partition.
  • a "stroke map” can be used to depict the different stroke classes (forehand, backhand) as a function of the outcome: the spin level (slice, flat, top spin) and pace (low, med, high) imparted on the ball.
  • the dimension 01 could represent stroke intensity and dimension 02 spin imparted on the ball.
  • FIG. 16 illustrates the relationship between the movement patterns and their outcomes quantized based on ranges defined by On, O12, O13 for dimension Oi, and O21, ⁇ 22, ⁇ 23 for dimension O2. Such relationships can be determined by embedding V into a subspace W that produces meaningful outcome categories (semantic interpretation), as illustrated in FIG. 14.
  • movement pattern classes can be represented as a combination of outcomes and conditions.
  • the performer has to compensate for effects of conditions or even exploit these conditions to their advantage in order to produce the desired outcome.
  • the ball comes into the court with varying amounts of pace and spin.
  • FIG. 9 shows three interception types that are characterized by the impact conditions.
  • More advanced players are generally more conscious about the conditions, since they will try to exploit the conditions to help improve the outcomes, for example in FIG. 9 where backing off from an oncoming ball affords the choice to intercept it on the descent, which is advantageous for producing top spin.
  • the subject can also decide to intercept the ball on the rise or at the apex depending on the desired outcomes at the different levels of the task (e.g., producing a shallow power shot deep in the court and down the line, or clearing the player at the net). Therefore, extended repertoire representation can include conditions as well as outcomes to provide a more complete understanding of the subject's skills, which in turn can be used to determine more complete and precise training interventions.
  • Pattern formation represents the first stage of skill acquisition, the so- called cognitive stage.
  • the subject forms a model of the movement such as the outline of the movement spatial configuration.
  • the movement at this stage cannot be performed reliably because it relies on conscious guidance and visual feedback, required to ensure that the movement conforms to the model.
  • Pattern consolidation refers to the process of consolidation of the movement pattern from spatial configuration (e.g., based on visual demonstration or verbal description) into sensory-motor patterns that can be executed dynamically without conscious effort.
  • the movement patterns are encoded as motor programs that can be performed in an open loop (e.g., without visual feedback). This corresponds to the acquisition of procedural memory.
  • Pattern optimization refers to the stage where a given movement pattern undergoes further differentiation or refinement (e.g., by fine-tuning technique and perceptual mechanisms), as well as developing physical performance.
  • the acquisition stages manifest in movement characteristics that are captured by the skill model. Therefore, the acquisition stage can be assessed from statistics associated with the skill attributes. MOVEMENT PATTERNS OPTIMIZATION
  • movement skill acquisition can be assessed and modeled by tracking the evolution of the movement architecture, i.e., the sequence of movement phases that makes up each movement pattern.
  • Movement patterns may go through several generations of acquisition, each generation characterized by a specific movement architecture and its associated functional characteristics (see evolutionary process in FIG. 11). Within each generation, movement patterns may progress through the stages of formation, consolidation, and optimization. Newly acquired physical strength, or other changes in constraints, can also prompt a new iteration in movement architecture that will typically have to go through the formation, consolidation, and optimizations stages.
  • One factor that drives the evolution of the movement pattern architecture is the opportunity to make movement more efficient. Efficiency is determined by how well a performer is able to use his or her biomechanics while protecting the body from wear and injury. Typically, evolution of the movement pattern architecture follows a development that proceeds from proximal to distal body segments. Therefore, the architecture usually evolves to involve the superposition of an increasing number of body segment motions.
  • the early generation of stroke pattern is characterized by a simple backswing and forward swing 441 (see FIG. 4).
  • the pattern is then refined as a performer learns to exploit the multiple degrees of freedom afforded by their body (legs, hip, torso, shoulders, elbow, wrist).
  • the overall pattern composed of multiple movement phases can be represented by a finite-state machine (FIG. 5).
  • a typical evolution in the stroke is starting from relatively simple, lower-dimensional motions that exploit the basic biomechanical capabilities, for example a basic backswing and forward swing phases (see e.g., 4- state system in FIG. 5), to learning to exploit and coordinate the larger degrees of freedom, for example using a more elaborate backswing, with a back loop that transitions optimally into the forward swing phase (e.g., 8-state system in FIG. 5).
  • the process of movement pattern refinement simultaneously exposes the body to new and larger displacements with the potential to create undesirable stresses on the joints, ligaments, tendons, and muscles. Therefore, it is possible to conceive acquisition of more advanced movement patterns as a process that's geared at maximizing outcomes while minimizing strain and more generally injury risks. Increased loads are also drivers for the development of physical strength as well as musculoskeletal structure.
  • FIG. 13 illustrates the evolutionary relationship between movement patterns. Each movement pattern is identified in terms of its ancestor(s) or parent(s) (shown in bold). The patterns shown correspond to the ones in FIG. 11, ordered by which stage it was formed along the evolutionary process (S1-S5).
  • each movement also can be assigned a degree of significance to the task that specifies how relevant the pattern is to the task performance and goals, and is indicated as primary, secondary, tertiary, etc.
  • movement patterns can either be formed de novo, or through differentiation from an existing pattern. In the former case, the new pattern typically fills a new need for the task performance, such as a volley. In the latter case, new patterns typically form to expand the range of outcomes or conditions. For example, in tennis, a generic forehand stroke can evolve into several subclasses to achieve specific ball spin and pace such as to better control the outcome of the stroke (see FIGS. 11 and 12).
  • FIG. 11 Movement specialization or differentiation at the pattern level is illustrated in FIG. 11 at time SI, where the PI patterns start its differentiation into two distinct patterns Pl-A and Pl-B. At the early stage of this differentiation process, the movements still have overlap in their characteristics, as shown as mixture in FIG. 12 for P2-B.
  • the two patterns begin to be sufficiently differentiated to represent distinct movements in terms of their technique.
  • the movement technique is formed by sequencing movement phases that build on muscle synergies. The development of movement technique, therefore, also relies on the development of physical strength along with motor coordination.
  • a subject can begin optimizing their movement.
  • Pl-B differentiates into more specialized patterns. Patterns can be further differentiated as a result of ongoing refinement or optimization of the technique. For example, S4 shows the optimization of pattern Pl-A. Optimization requires narrowing down on operating conditions and technique; therefore, the patterns begin to have more restricted domains of operation which leads to two new sub-patterns Pl-Al and P1-A2.
  • Epoch refers to time periods associated with a data set that is associated with a particular model (see assessment loop described later).
  • Learning/acquisition stage refers to time periods associated with transitions in a subject's neurological learning process for a particular movement pattern (formation, consolidation, and optimization).
  • Developmental stage refers to time periods associated with evolutionary milestones in the development of a player's larger movement pattern repertoire.
  • Generations refer to time periods associated with differentiation in a player's overall skill profile (e.g., as it relates to other player subgroups), based on the aggregate contribution of skill, technique, etc. This information can be captured by the player subgroups through population analysis described later.
  • CP core pattern
  • FIG. 13 the core patterns are shown by a solid edge to underscore that they inherit the main attributes in the development of the new patterns.
  • the non-core patterns, linked by dashed edges in FIG. 13 differentiate to create new attributes that are distinct from the core pattern.
  • the core pattern often corresponds to the predominant technique in that class, for example that is further consolidated in procedural memory. Under challenging conditions, the subject may tend to fall back on that pattern. The core pattern may also be more difficult to change because of its long-standing history.
  • This conceptualization of movement learning as an evolutionary process combining the development of new patterns through differentiation, as well as formation of patterns de novo, is useful to assess the longitudinal skill acquisition process. This involves relating the patterns through features that are inherited as they differentiate and tracking the stage of learning of the patterns through the differentiation process.
  • the hierarchical classification of patterns can determine the hierarchical relationships between the classes. These structural characteristics can be exploited to design training interventions, and plan and manage the training process. For example, interventions that help new patterns form through differentiation and consolidation.
  • Proficient movement technique and overall task proficiency rely on the formation and optimization of perceptual mechanisms, for example the ability to recognize the state of an incoming ball and adjust the stroke to these conditions. For example: a fast, high bouncing ball is returned with a slice, enabling a more reliable but less offensive return.
  • a player can extract early cues to estimate the return location (e.g., from an opponent's body and racket swing) they can control the point by selecting the return and positioning the body to precisely intercept the ball in the strike zone to achieve the desired return trajectory.
  • a broad movement repertoire allows a subject to select the best actions needed to control the state of the activity based on the task state and conditions. For example, a tennis player may take advantage of a slower, shorter return to intercept the ball earlier and produce large top-spin and pace as a way to surprise the adversary with a deep return in the open side of the court. Alternatively, under an offensive return from the adversary, the player has less time to prepare a stroke and uses a slice to gain time before the adversary's return. These changes reflect the subject's ability to assess the situation and use this information to control the task and achieve its goal, while at the same time adapting to the environment and conditions.
  • FIG. 7 shows relevant interactions in the larger system and illustrates the following outcome levels, which are also shown in FIG. 2:
  • Shot trajectory and type relative to the environment elements e.g., net clearance, curvature, velocity, spin.
  • stroke-impact and primary stroke outcome are most directly related to motor control processing (body coordination, and ball interception).
  • body coordination, and ball interception The performer can assess these outcomes through proprioception, including how the racket "feels" at impact, and the resulting shot. But the latter does not provide as much information about the movement technique or knowledge of performance.
  • the shot trajectory and placement are most directly related to the planned shot and game strategy but also depend on the performer' s performance and control of the ball and conditions.
  • the performer assesses these by perceiving the ball trajectory relative to the court and the opponent and the impact on the game. Information from this level helps improve the positioning and shot selection and game strategy. However, training at this level relies on sufficient facility to control the ball and achieve sufficiently precise outcomes at levels l)-3).
  • the strategy level is concerned with the negotiating the environment and conditions. This requires reading the terrain and conditions, and planning the deployment of a sequence of movement patterns.
  • Training relies on the ability to 1) assess motor skills, which corresponds to the description of the movement outcomes and characteristics in relationship to task requirements, and to 2) diagnose skill, which corresponds to the identification of specific aspects of the movement technique that are deficient and reduce performance in the task through their effect on critical outcomes (diagnostics).
  • diagnosis skill which corresponds to the identification of specific aspects of the movement technique that are deficient and reduce performance in the task through their effect on critical outcomes (diagnostics).
  • the gained knowledge can subsequently be used to determine adequate interventions that address the specific skill deficiencies and lead to a higher skill level and hence task proficiency.
  • Skill assessment is responsible for characterizing the movement performance. Producing an assessment is essentially the challenge of defining metrics and features from collected movement data that provide a concise and useful description of a subject's performance outcome and technique (knowledge of result and performance). For example: "The ball spin produced by the impact is too low for the forehand top-spin high-strength (FHTSH) class.”
  • FHTSH top-spin high-strength
  • Skill diagnostic is responsible for identifying the causes of movement and task performance characteristics. It typically focuses on the deficiencies that need to be addressed or corrected to improve the skills towards a task performance. For the previous example: "Racket height at forward stroke initiation is too high and racket roll rate profile is too shallow.”
  • FIG. 10 illustrates the relationship between the levels of the movement hierarchy and the task hierarchy. It defines the following levels of assessment:
  • Physical performance level The assessment at this level focuses on the physical details of how the movement is produced. This level is best analyzed at the level of movement phase segments, including considerations such as the movement phases and relationship with muscle synergies, the musculoskeletal constraints, and sensory and perceptual processes used to execute and deploy the movement in respective task conditions.
  • Pattern performance level The assessment at this level focuses on how well the movement pattern associated with the primary movement unit support the task and environment interactions, and more specifically produce outcomes that contribute to the task goals and adapt or take advantage of conditions. This level is best analyzed through the movement pattern and outcomes, for example in tennis the stroke and shot relative to the court, as well as the oncoming shot and conditions (see FIG. 9).
  • Task performance level The assessment at this level focuses on the relationship between the acquired skill elements and the task requirements. This level of assessment is best analyzed through the repertoire. It includes considerations such as what types of patterns have been acquired to support critical task interactions such as producing the range of outcomes and adapting to conditions, and how these outcomes and interactions collectively contribute to the task or activity performance. In analogy to robotics or trajectory planning, this level corresponds to the assessment of the discretization of the task space, i.e., how the overall range of outcomes and conditions are quantized into distinct patterns that collectively provide the skill elements to perform the task proficiently.
  • Assessment components refers to the different perspectives that can be taken on the movement performance and skills and follow from the assessment level analysis that was just discussed and is summarized in FIG. 10. The following components can be considered:
  • Outcome characteristics The outcome assessment corresponds to the traditional knowledge of result and performance. The outcomes capture specific qualities of the movement pattern, their effects on the task environment, and the associated conditions in which they are executed. Outcomes are defined and analyzed at different levels of the movement system such as the different outcome levels defined in FIGS. 7 and 8.
  • Functional characteristics The assessment focuses on the underlying mechanisms of the movement pattern classes and their effect on the task.
  • the functional analysis is usually connected to the various outcome quantities and the range of conditions required for a task.
  • functional analysis at the pattern level considers how the movement phases combine to produce the movement pattern that support the interaction with the task and environment level, and produce the primary outcome for the task.
  • Functional analysis also encompasses the perceptual mechanisms, for example those used to support synchronization with the environment and task elements.
  • the functional characteristics can encompass the details of biomechanics and muscle activation (muscle synergy).
  • Perceptual characteristics This assessment highlights the quantities that can drive the subject's behavior across the different assessment levels. For example, at the physical performance level, perceptual quantities correspond to the proprioceptive features of the movement phases that are critical to the execution of a particular movement pattern. Perceptual mechanisms are part of the functional characteristics, they are separated as a component to emphasize their potential role as part of cueing for example.
  • Movement characteristics and skill level depend on the movement's acquisition stage, which refers to specific milestones associated with the brain's learning process. This assessment focuses on the identification of the learning stage of a movement pattern, which can help better select diagnostic tools and training interventions, such as cueing to reinforce sensory-motor patterns, or visualizations that can help form mental models.
  • FIG. 10 illustrates the different levels of assessment highlighting the representative elements 280 of the model at each level for the tennis example.
  • the figure summarizes the assessment and diagnostic components 290 that are applied across the different levels.
  • the illustration also conveys how the different levels are nested into one another, going from the movement segments at the bottom, which are used to form the stroke patterns; then, how these patterns enable the shot interaction with the court environment; next, how the different strokes and shots collectively discretize the task space; and finally, the decision making and strategy driving the task competitive performance.
  • FIG. 31 provides a different perspective with a description of the: a) the levels of assessment, b) the central elements that describe that level, c) the criteria and quantities that can be used to determine the skill characteristics at that level, d) the analysis or diagnostics to identify the critical characteristics, and finally, e) the drivers and mechanisms used to produce training interventions.
  • outcomes represent the primary result of the movement, as viewed from the perspective of the task. As already discussed and described in FIG. 10, outcomes can be defined at different levels of the movement system hierarchy and task structure hierarchy. Outcomes are quantities that provide relevant information for the task performance and skill assessment. They are typically designated based on the task requirements and available measurements.
  • Success and success rate can be determined at different outcome levels (see FIGS. 2 and 7). For example, in tennis the success at the racket-ball interaction level (Outcome 1) is determined by the racket impact location and outcome level for the particular class such as spin and pace. At the court interaction level (Outcome 3) it is determined by the court impact location and state (see FIG. 8).
  • Every stroke class is characterized by the range of values that characterize the functional model, which includes states at phase transitions such as the racket states at the beginning of the forward swing, or the racket orientation, the racket angular rate, etc. at impact, etc. These characteristics can be used to determine outcomes at different levels, including the ball spin and pace, but also the shot trajectory. With the additional information about the player position and orientation, it is also possible to predict and estimate the ball impact location on the court.
  • An example of this in tennis includes a comprehensive motion capture system that measures the task object (tennis ball) relative to the task space in addition to the subject's body segments, body pose, motion of the equipment, etc. It is possible to more directly assess the outcome of the subject's motion. In addition to the quality of the outcome, another attribute is the success rate of movements for each specific movement class.
  • FIG. 8 shows respective shot placements based on ground impact distributions for a player and an opponent.
  • the skills at the shot level manifest as different resolutions and precision in interactions with the task environment.
  • the task level performance which will be described next, while depending on the stroke, puts more emphasis on the shot outcome level such as how the stroke used by the player can control the ball relative to the court and opponent (see FIG. 7).
  • the skill profile represents a holistic description of the subject's skill, which can be used to compare players as well as track how skill evolves over time.
  • the movement class technique assessment looks at the overall characteristics of the movement pattern. As described earlier, each movement pattern can be described by a so-called core pattern (CP). The idea is that the movement follows a motor program which has a template with specific variability due to disturbances and adjustments made to adapt to conditions. The CP therefore describes the nominal movement performance.
  • CP core pattern
  • Deviation from a CP can therefore be used to assess technique and other attributes such as adaptability. Even under perturbation, movement pattern should be distributed around the nominal range of CP, i.e., normal range of variations. Movements that exceed the normal range can represent poor execution or may also be of a secondary pattern that may be due to differentiation of the core pattern as part of the normal skill learning.
  • Movement differentiation can be detected from the presence of a secondary pattern grouping, distinct from the CP, within a hierarchical movement pattern class. This type of differentiation is especially likely in the early stages of skill acquisition, when new patterns are derived from existing patterns.
  • FIG. 2 illustrates an interaction between a stroke motion and the task and environment elements, including the ball trajectory relative to the court, the impact of the ball, and its bouncing before the interception with the racket trajectory.
  • the figure also illustrates the gaze of the player along different points of the ball trajectory and environment elements, and shows a ball machine as an apparatus that can be programmed to enable different forms of interactions.
  • FIG. 2 also illustrates details associated with the functional characteristics of the stroke pattern and the interaction with the environment to produce a desired outcome (e.g., Outcomes 1-3).
  • the interactions include for example adapting to conditions, such as the timing of the movement phases relative to the ball state following ground impact 32 (see also FIG. 9).
  • FIG. 2 also shows examples of visual cues that are used to control the movement execution, such as the ball trajectory curvature, the magnitude and angle of the bounce or impact 32.
  • the figure depicts the visual attention based on the gaze vector 81 to some of these cues, as well as the elements that are relevant for the Outcomes 1-3 indicated by labels 33-35.
  • FIGS. 3 A and 3B illustrate the movement as a sequence of phases and highlights phase transition characteristics, and phase profile characteristics.
  • the phase profile characteristics refer to the dynamics during the phase segment. These characteristics are associated with the coordination of the movement segment and muscle synergies.
  • the figure also shows the feasible envelope that results from musculoskeletal and other constraints, an admissible envelope that represents movements that produce acceptable outcomes but are suboptimal, and an optimal envelope that represents the range of motions that produce the best outcomes with the best use of the biological system.
  • the figure also introduces the concept of a goal phase, which represents the phase associated with the primary interaction relevant to the production of an outcome and environment and task element interactions.
  • the goal phase in tennis is the movement segment that corresponds to the ball impact and extends throughout the ball interaction or contact. This phase is critical in the production of the outcome.
  • Transition characteristics are determined by the movement configuration, including the state of the body segments and end effector such as the racket. These conditions also include timing characteristics, such as the synchronization with the environment elements. For example, in tennis, a relevant timing is the synchronization between the tennis stroke initiation phase and the tennis ball state, which itself can be delineated into different phases, such as the net crossing, ground impact, and various phases before the ball impact (see conditions in FIG. 9). Next, the timing of the forward swing phase initiation (phase 2) is determined similarly by ball state and the anticipated impact conditions but closer to the impact time. This synchronization and modulation of the movement phases are instrumental in achieving an accurate interception of the ball and producing the desired impact conditions (target phase) that will lead to a successful outcome. Note that similar considerations can be made about the rest of the body segments and configuration.
  • the skill element therefore can be defined formally in terms of these primary interactions and the skill characteristics and determined from the various attributes of these interactions, including: the movement functional characteristics (described by the movement phase characteristics and perceptual and motor interactions), the musculoskeletal characteristics, physical performance, and different levels of the task and motor system hierarchy.
  • the following movement acquisition stages can be defined from the three learning states described earlier: movement formation, movement consolidation, and movement refinement/optimization.
  • the movement acquisition stages manifest in movement characteristics and can be described as follows:
  • Patterns to form are missing from the repertoire or exist in unreliable form.
  • the missing patterns typically are due to the lack of differentiation among existing motion patterns. For example, in tennis, the absence of subclasses in backhand topspin represent gaps in possible operating regimes and possible outcomes such as pace or spin. These gaps in movement repertoire preclude flexible production of outcome and adaptation to conditions and therefore manifest in task performance.
  • Patterns to consolidate (e.g,. FIG. 52B, step 323): Movement phases are not yet sufficiently defined and integrated in the movement pattern to allow reliable execution under dynamic conditions. For example, the muscle synergies associated with the phases are not yet fully automated and their transition are not smooth. These deficiencies manifest as unreliable outcomes, variability in movement pattern, lack of smoothness, inefficient movement performance, and do not have sufficient flexibility to deal with changing conditions.
  • patterns undergo automatization and refinement in their structure. These changes reflect the brain's learning mechanisms (e.g., procedural memory). The automatization allows repeatability and reliability. The refinement of the pattern structure is guided by the functional requirements, including achieving better outcomes and physical efficiency, as well as effectiveness with the task and environment constraints and conditions.
  • Patterns to optimize (e.g., FIG. 53B, step 324): Movement patterns do not achieve the outcomes efficiently and do not adapt sufficiently to environment or task conditions. For example, movement phases do not make optimal use of the subject's biomechanics. These deficiencies for example, may result in excessive use of force when seeking an increase in outcome.
  • Skill acquisition stages also manifest in physical changes, including gaining sufficient strength and endurance to sustain good technique over time.
  • the acquisition stage is captured by the concept of skill status. For each existing class of movement patterns in the repertoire, it is possible to assign a skill acquisition.
  • the acquisition stage can be determined based on quantitative criteria or metrics. For example:
  • Missing patterns can be determined by the repertoire completeness, i.e., how well the movements in the repertoire cover the performance requirements associated with the task objectives and environment conditions.
  • Typical pattern analysis tools such as clustering combined with similarity measure (e.g., dendrogram) can be used to identify new patterns within an existing movement class. The degree of differentiation of a pattern relative to other existing patterns can provide a measure of its development.
  • Patterns to consolidate can be identified by success rates, variability in technique, and outcomes within a given class. At this stage, movements also tend to display particular physical performance characteristics, such as high jerk, lack of smoothness, and timing variability. These patterns can also be identified by inconsistency in movement phase structure, smoothness of phase transitions, as well as unreliable timing of some movement phases (e.g., forward swing acceleration profile). Finally, patterns at this acquisition stage can also be identified from the lack of flexibility in adapting outcomes to the range of conditions and outcomes.
  • Patterns to improve or optimize are already formed, but the movement structure does not utilize the subject's biomechanical potential efficiently, and does not achieve the theoretical range of outcome and level of flexibility helpful to deal with the range of conditions. Patterns to optimize are primarily analyzed from the functional characteristics (feature analysis described elsewhere), which provide a detailed understanding of the relationship between the movement technique and its relationship to outcomes. Frequently also relevant is movement efficiency, i.e., the work required to produce an outcome.
  • One goal of movement optimization is refining movement technique to use the least energy and produce the least strain on the musculoskeletal system.
  • FIG. 41 provides an example of the acquisition stage assignment for the skill elements in the groundstroke repertoire.
  • Population analysis is valuable to understand the contribution of the broad range of factors intervening in the skill acquisition process.
  • Population analysis can be used to determine player types based on skill levels and a variety of other factors such as body type, health, etc.
  • the player type or profiling makes it possible to generate appropriate reference outcome values by accounting for groups of players with similar technique types and skill levels.
  • the player profiling at the same time enables identification of the player characteristics or attributes, i.e., what skill attributes and other factors such as development stage, are characteristic traits of a particular player group.
  • the player profile information can be used, for example, to determine weights in the composite scores that determine the larger player characteristics.
  • FIG. 29 illustrates the process of generating population groups based on the performance and skill data from the hierarchical movement model.
  • the information extracted from the population analysis makes it possible to determine performer profiles.
  • FIG. 30 illustrates assessment across the skill-model hierarchy, incorporating player profile information to generate reference attribute values used to assess the skills at the different levels of the movement system and performance hierarchy.
  • the reference values can be used to provide contextual information to determine what training interventions to pursue.
  • the movement physical implementation describes how each movement is composed from distinct phase segments, where each segment is typically associated with the coordination of a specific set of body segments driven by so-called muscle synergies.
  • the performance criteria at this level includes how the biomechanical system supports the phase segments, for example, which muscles and joints are involved in the motion as a function of segment profile dynamics and phase transitions.
  • the analysis at the movement phase level is based on identifying the components of motion such as muscle synergies and other musculoskeletal quantities. There are overlaps between the segment level analysis and the functional movement analysis, in particular when it comes to the critical movement phases, such as the forward swing in the tennis stroke.
  • the individual movement segments combine to form the entire movement pattern.
  • This pattern represents the basic skill element supporting the various task interactions.
  • the primary movement performance criteria are the movement outcomes relevant to the task performance, as well as how movement adapts to the task conditions.
  • the analysis focuses on identifying features or attributes that explain the relevant qualities of the outcomes and conditions (e.g., using sensitivity analysis). These features provide the quantities that can be manipulated through training interventions to optimize movement technique.
  • One question is to determine the most actionable features or attributes and synthesize feedback cueing or other augmentations such as instructions that can be used to produce effective training intervention.
  • Other criteria relevant to movement technique and performance can be considered, such as movement efficiency or injury risks.
  • Movements involve the spatial and temporal coordination of multiple motion degrees of freedom.
  • Detailed skill models focus on the functional aspects of movement characteristics that support the performance of the movement outcomes, the interaction with the relevant task and environment elements, and the adaptation to conditions.
  • More detailed assessments at the level of movement technique can be performed by decomposing the patterns into segments.
  • the analysis of the movement's functional characteristics for example, determines movement efficiency in producing specific outcomes, synchronization with task and environment event, and the ability to compensate for conditions.
  • An example of a functional skill model for a stroke is the coordination between racket roll and swing rate during the forward swing phase, which describes the technique of the subject that may be important for top spin.
  • the model can be used to identify a subject's "spin envelope," in a particular movement class (see details the following sections, see also FIG. 33).
  • Similar models can be derived for other characteristics of the forward swing and other phases of the stroke.
  • the racket motion results from the superposition of several components of body motions including the trunk, the shoulder, the forearm, and the wrist.
  • the ability of the subject to achieve the desired impact conditions and result, as well as compensate for conditions, depends on the proper timing and coordination of the body segments. With sufficient measurements, it is possible to estimate the contribution of the movement components of the different body segments and determine outcome variables that characterize the biomechanical performance based spatial or temporal profiles.
  • FIG. 9 illustrates the interception and impact conditions and primary stroke and shot outcomes. These conditions affect the outcomes but also represent characteristics of the movement pattern classes, since these patterns are fundamental to the interactions in the task. Therefore, movement classes are typically characterized by the movement technique (stroke type), the stroke outcomes, and the conditions under which the action takes place (interception and impact conditions). Notice that the interception conditions are determined by the players' movement on the court and their ability to anticipate and plan their actions.
  • Temporal characteristics are also critical to the movement performance (see analysis of ping pong stroke timing in Bootsma 1990). For the tennis example, two timing characteristics are included for the assessment of the forward stroke: the instant of peak racket angular rate relative to the impact, and the time of the forward stroke initiation relative to the impact (see details in subsequent sections, see also FIG. 42).
  • the outcomes are related to how movement patterns change the state of the task and adapt to conditions and various contingencies that can arise in a task; for example, producing shot placements that drive the game and adapt to the opponent shots. Therefore, the repertoire of movement patterns describes the movements or actions available to a subject in an activity domain. Every subject acquires their own specific repertoire, encompassing a particular range and quality of movement patterns.
  • the elementary assessment at the task performance level is based on assessing how complete the repertoire is relative to the task requirements.
  • Task requirements define the outcomes of actions helpful for the task.
  • skill analysis primarily focuses on identifying gaps in the repertoire.
  • the absence of outcome and associated motion patterns in a relevant task area or condition can be used to identify "unformed patterns.” For example, in tennis, this may manifest as the absence of a high- strength backhand top-spin class.
  • the completeness of a repertoire is determined by the extent to which it achieves a sufficient discretization of the task space.
  • the movement patterns become more precise and therefore enable a more granular discretization of the task environment (see FIG. 8). As the discretization level increases, more optimal levels of task performance can be achieved.
  • the range of outcomes and actions can depend on the style of play or even the personality of the individual performer.
  • More advanced task performance analysis and assessment takes a more comprehensive perspective and is achieved by tracking attributes of entire sequence of actions.
  • relevant attributes include sequence of strokes, the length of the rallies, what type of strokes are used, how they relate to the other player's actions, including movement on the court, and the overall performance of the activity or task.
  • Statistical analysis of the sequence movement patterns can also be used to provide relevant information about the individual's skills and strategies, such as the frequency distribution of which movement patterns the subject uses in a task over a session, provides a signature of the activity and the subject's strategy.
  • FIG. 8 also shows the type of quantities, such as player's court motion and positioning, which can be used to model and assess the subject's game strategy, i.e., how the player can position the ball according to the opponent' s pattern of play and position.
  • player's court motion and positioning can be used to model and assess the subject's game strategy, i.e., how the player can position the ball according to the opponent' s pattern of play and position.
  • These high-level skills also depend on perception of the court, and anticipation of the opponent' s behavior.
  • These dynamic characteristics of how movement patterns are used can be modelled by using techniques for learning temporal relationships and
  • HMM Hidden Markov Models
  • RNN recurrent neural networks
  • the primary goal is to assess competitive performance, which is typically performed at the level of a population.
  • the criteria therefore represent what can determine a form of population fitness such as actual performer rankings obtained from competitions. These may not always be available; therefore it is also possible to compute rankings based on player skill profiles, which can also take into account population groupings. When available, competitive rankings can be used to calibrate the ranking based on skill profile.
  • the movement skill attributes characteristics include those included in the skill profile, how performers relate in terms of their individual characteristics, and how these contribute to their competitive performance. Analysis of the competitive performance provides information about what aspect of the skill profile (skill element and attributes) can be improved to make someone more competitive in a task.
  • the skill profile is designed to capture the comprehensive, composite characteristics of the individual's movement skills: account for the performance
  • FIG. 17 shows the skill profile as a line graph with the contribution of the different skill components, i.e., movement patterns to the composite score.
  • FIG. 40 which will be discussed later provides an illustration of the skill profile for groundstroke repertoire. Different objective functions can be used to emphasize different aspects of the performance. For example, task performance, efficiency, long-term injury risks.
  • the illustration in FIG. 17 also highlights gaps in the repertoire, and the difference between two subjects (C and A) as a skill profile gap.
  • the skill profile and composite cost can be analyzed to determine which aspect of an individual's movement behavior, or skill attribute, has the largest impact on the overall performance. Since type of sensitivity information can be used as a guide to determine what training elements to focus on.
  • the different objective functions that can be used for a skill profile provide a way to look at training from different perspectives (performance, efficiency, injury). They can also be combined in a multi-objective analysis to find the tradeoffs, such as performance vs. injury. In contrast to the analysis at the movement technique level, this analysis provides a more holistic perspective on these questions.
  • the skill profile is a static assessment in the sense that it does not account for the dynamics, i.e., how these skill elements are deployed as a task or activity unfolds, or in response to an adversary.
  • Usage frequency of motion patterns provide simple model to assess strategy.
  • the next level corresponds to the statistics that describe the sequence of patterns such as conditional probabilities of a pattern given the previous pattern or the opponent's pattern.
  • a more complete competitive analysis accounts for the dynamics of the activity performance, i.e., the transition between actions and their associated events, e.g., pattern X is used when return from opponent is of a particular stroke type and ground impact conditions.
  • Such a model takes into account the chain of events in the outcomes, which corresponds to determining a causal model.
  • the dynamics capture the complete task performance or game strategy building on the skill elements and underlying details.
  • the player or performer attributes provide information to characterize player type.
  • Player groups can be determined by clustering the player attributes as illustrated in FIG. 18. Within groups of performers that share similar characteristics, it is then possible to analyze movement performance and skills across a broader range of conditions and identify subtle variations in technique that influence an individual's level within that group.
  • FIG. 18 also indicates the relationship between the groups and some skill level such as determined by the skill profile. This information can then be used to determine the player profile (see FIG. 29).
  • FIG. 19 shows the distribution in attributes associated with a score or cost function for an entire population group described by the group distribution highlighting a member (subject A, described by distribution (ei , e 2 )), and the tiers ( ⁇ low, medium, high, very high ⁇ ) associated with the composite scoring function for the entire population subgroup (ei,c , e 2 ,G).
  • the population analysis enables performing absolute assessments.
  • the values obtained for the various skill attributes relative to a larger group of performers help contextualize a subject's performance. This allows more objective comparison between the skill profiles of groups of players (FIG. 17) and can be used to determine reference values.
  • the reference values from population analysis can be incorporated in the assessment and diagnostic of the skill elements and extended to the various levels of assessment.
  • the composite score used to capture skill elements can be normalized by reference ranges associated with the attributes for the subject's subgroup.
  • Information from the population analysis can also be used to rank players or performers, such as through leaderboards, which in turn can provide additional source of incentives for training.
  • the leaderboard also enables the determination of which attributes in the composite and profile cost function differentiate players or performers. This corresponds to the competitive assessment (see FIG. 31). Therefore, this information, for example, describes which skill element and attributes have the largest impact on the ranking, and can be also used to prioritize training.
  • the combination of the population analysis makes it possible to find larger patterns in movement technique, performance, and even skill acquisition.
  • One aspect of the assessment based on population data is the profiling of the player or performer.
  • Player profile can be determined to characterize the player' s performance or skills relative to the larger population. This profiling can include ranking of a player, e.g., based on different skill profile composites, as well as relating subgroups of performers with different but related movement technique.
  • FIG. 31 gives an overview of the holistic understanding required for systematic skill training.
  • FIG. 10 illustrate some of these quantities in the tennis use case.
  • the vertical arrows going up indicate the bottom-up aggregation of the information and characteristics that participate in the formation of the characteristics at the next level, where additional elements also come into play. For example, at the functional performance level, the movement phases combine into a movement pattern that interacts with a task element to produce an outcome. These characteristics are critical in understanding the learning process, and therefore can be used to determine what movement characteristics have to be developed first (e.g., difficulty rating: basic, intermediate, and high level).
  • the downward arrows indicate the top-down influence of the higher-level assessment on informing the focus of the lower level assessments.
  • the higher levels can provide top-down information to determine which specific assessments and
  • the skill profile characteristics drive training.
  • the skill profile characteristics provide understanding of which skill element and attribute have the most effects on the current profile level. Therefore, acting on this element and attribute will produce the most effect on the performance at the profile level.
  • Skill status provides the basis for selection of skill elements that should be exercised during training, in what order these elements are trained, which goals are achievable, and what forms of feedback augmentation are most appropriate for training (see FIG. 22). Therefore, the skill status comprehension describes an individual's skills and can be viewed as the state of the skill acquisition process.
  • the determination of the acquisition stage also makes it possible to more precisely analyze the progress someone is making in an activity domain, which specific aspects are improving, and which ones are more resistant to change.
  • the criteria applied for the acquisition stage provides specific information about the skill element that can be used to measure progress toward their improvement.
  • Skill acquisition is a process that unfolds over time as a function of exposure to the task.
  • it is also beneficial to be able to analyze the trends of the different skill elements.
  • the trends provide information about the stability or susceptibility of these elements to improve under a given training activity.
  • Motion patterns or skill elements have varying degrees of stability. Some patterns are deeply solidified in a subject's procedural memory, and therefore will show less variations from session to session; other patterns are more malleable. Furthermore, due to variability in human performance, movement patterns will occasionally achieve superior outcomes and techniques. Therefore, the skill analysis method should be able to capture such changes that are inherent to movement behavior, be able to understand which features are associated with improvements, and finally, have feedback techniques to reinforce these features.
  • Time windowing techniques can be used to highlight skill status and trends at different times or epochs in an individual's training history. Skill trends can be analyzed for different time scales, e.g., within sets, from session to session, etc.
  • the longer-term trends can measure the physical characteristics associated with movement skills such as strength, effects of wear, injury (both development and recovery).
  • the medium-term trends (weeks to months) can measure the assimilation of the training goals and the consolidation in procedural memory of the refinement and optimization of movement patterns.
  • Short-term trends days to week
  • Micro trends can measure the effectiveness of new instructions and the effectiveness of feedback cues.
  • FIG. 47 shows a plot displaying the progress along several training goals over a specified time range.
  • the progress in the figure is described as a normalized gap w.r.t. training goal (e.g., improving the top spin or consistency, success rate, etc.).
  • the system When the current training goal for a training element is attained (shown as a star), the system generates a new goal (shown as a square).
  • the trend plots can be superposed for all the active training goals or a specific subset (e.g., what a subject is currently focusing on).
  • the characteristics can be used to help identify which skill elements to prioritize. For example, more focused effort can be put on aspects that are difficult to improve, or on a training goal that is close to completion to get it done and move to a new training goal.
  • the trend can also show comprehensive skill elements as described by their associated metrics (outcome, technique, performance).
  • the information from the skill status can be converted to a numerical score or grade to provide a summative assessment of skill and its evolution over time. Furthermore, it is possible to decompose the total score into their respective components, including outcome, technique, and performance.
  • FIG. 30 gives an overview of how target skills are generated across the levels of the hierarchy. Target skills are used to determine training goals that provide actionable drivers for training or rehabilitation.
  • FIG. 31 provides an overview of the integration of assessment and diagnostics across the levels of the movement system organization. It gives a description of the following: a) levels of assessment, b) the central elements that describe that level, c) criteria and quantities that can be used to determine the skill characteristics at that level, d) analysis or diagnostics applied to identify the critical characteristics to specify the training goals, e) the drivers and mechanisms used to produce training interventions, and f) the feedback modalities that can be used to augment the training intervention. TRAINING GOALS
  • the skill assessment attributes and metrics and the skill status and trends provide the main elements to support the quantitative, data-driven approach to training.
  • the relevant step to render an assessment actionable is to determine a training goal, and preferably some specifications for the pursuit of that goal.
  • diagnostics are typically performed based on some causal models.
  • the causal models are derived from the functional component of the assessment.
  • the functional components explain how the outcomes are produced at the different levels of the movement and task structure organization.
  • the specification of training goals is also directly connected to the synthesis and selection of appropriate feedbacks (instructions and real-time cueing).
  • a skill element becomes a training element once it gets assigned one or more training goals.
  • Training goals can target any attribute across the movement model hierarchy (see e.g., FIG. 30). Training goals provide a way to direct and drive training activity, as well as basic element needed for the planning as well as the continued assessment and managements of the training process.
  • FIG. 48 shows the learning curve associated with the data driven training process.
  • the learning curve shows the incremental improvement in some relevant attribute a; of a skill element e; over the training activity (sets and sessions).
  • multiple skill elements can be improved concurrently in one training epoch.
  • the training goals are expressed as a target change in attribute ai of the skill element ei.
  • new baseline data is generated and the training goal is updated.
  • the figure also illustrates the acceleration of the learning curve provided by an update to models and augmentations, etc. As the model parameters are tracked and incrementally updated, the training goals and associated augmentations drive the learning process to achieve best efficiency.
  • the training goals are identified based on the assessment and diagnostics, which can include both the various skill and performance attributes as well as the skill status.
  • the various sources of information from the assessment and diagnostics determine the forms of augmentation that are most effective for training the skill element (see FIG. 31).
  • the skill status provides relevant information for specifying general training goals. For example:
  • the training goal for unformed patterns is directed at helping subjects develop new movement patterns that help produce desired outcomes, taking into account each subject's physical and health status.
  • the training goal for pattern formation is directed at helping subjects differentiate the existing movement into separate patterns that can each better respond to task requirements (outcomes and conditions). It can take into account the existing pattern landscape in a class, e.g., the core pattern and the newly differentiating patterns to help guide and reinforce the desired attributes.
  • the selection of which patterns to form may also depend on a subject's physical and health status, e.g., patterns that are causing stress or contribute to an injury.
  • the training goal for pattern consolidation is directed at helping subjects refine movement patterns and create procedural memory to enable automatic and repeatable execution.
  • the training goal for pattern optimization is directed at helping subjects maximize outcomes, improve efficiency, and improve ability to adapt to conditions.
  • the development stage also provides information to help select appropriate augmentation forms and determine which movement characteristic to emphasize.
  • the augmentations in particular real-time feedback or apparatus, allow more effective learning and therefore influence the training goals specification.
  • training goals can be determined based on a functional analysis of a subject's own existing performance.
  • the variability in performance ensures that there is a range of performance level and associated attributes contained in the data.
  • a general approach for the training system is to identify the best performance within the individual's range of data, and then help the subject consolidate or optimize their technique so that they operate at this new level. Incrementally, with new data available from subsequent sessions, this process can be pursued and the subject's performance therefore can be incrementally improved.
  • FIG. 30 illustrates example assessment, diagnostics, and training goals across the skill- model hierarchy, incorporating player profile information to generate reference values for the attributes used to assess the skills at each level of the movement system and performance hierarchy.
  • Training goals take different forms depending on the level in the hierarchy (see assessment levels in FIG. 10). For example at the physical level, the training goal synthesized for the improvement of an outcome can be encoded as a change in features of movement technique that has been shown to produce improvement in a specific outcome. [0501] At the pattern performance level, the training goal could consist of improving movement technique in the deployment of the stroke such as producing more precise court shot placement. The training goal is specified in terms of skill attributes that have been shown to produce improvement of shot level outcomes, such as timing (FIG. 42).
  • the target values for the quantitative specification of training goals can be determined from the statistical analysis.
  • the training goals to improve outcomes can be determined from functional feature analysis at the movement physical level.
  • FIG. 37 which illustrates key features for the forward swing phase along with some example stroke phase profiles.
  • FIG. 20 shows a model of the statistical distribution for two technique features, which can be used to analyze the forward swing phase.
  • the features could be the angle of attack or phase length shown in FIG. 37, and the outcome could be the topspin imparted on the ball.
  • the level lines in FIG. 20 can be computed based on percentile ranking from the individual's data.
  • a similar analysis can be conducted at higher levels, such as taking into account any skill attribute that is relevant for the task or activity performance.
  • the training goal can be set to achieve the next performance tier, or a fraction of the existing variation in performance (see the ellipsoid ei, e 2 ) in FIG. 19, which illustrates the relationship between attributes distribution and some measure of performance that is shown as level lines in the context of a larger population or some selected subgroup based on player profile information.
  • the level lines in this case can be computed based on percentile ranking from population data.
  • FIG. 43 gives an overview of integrated perspective on the system' s main components based on the tennis use case, organized in terms of the levels of assessment (physical 510, pattern 520, task 530, and competitive 540), how the criteria can be expressed with cost functions (512, 522, 532, and 542), and how these elements relate across the different levels. Together with FIG. 10, they highlight some of the key elements and quantities that can be used to drive the diagnostic and ultimately the training process. In particular see the assessment criteria at each level and the diagnostic components shown in FIG. 10.
  • the diagnostics are concerned about how movement patterns are deployed across a larger task environment (see FIG. 8 and 10).
  • the functional model at the task performance level can be formulated to describe conditions that can be exploited to produce the desired outcomes, including proper positioning on the court to control the impact conditions (shown in FIG. 9).
  • the training goals at that level therefore can be specified based on deficiencies in these functional characteristics.
  • the diagnostics and training goal specifications can also include perceptual aspects, such as extracting the cues from the environment and elements (court landmarks and ball trajectory) needed to anticipate the oncoming ball, as well as generating targets for the shots across desired court areas. Similarly, they can include aspects of memory/learning, also shown in FIG. 10, such as mental representations of these environment elements (see FIG. 8) and corresponding movement patterns.
  • the specification of training goals at the competitive level follow a similar logic but focus on the dynamic characteristics, i.e., the temporal sequence of shots driving the game.
  • the functional models at that level can for example be formulated using Dynamic Bayesian Networks or Hidden Markov Models. These models can then be used to assess the individual's strategy from the temporal patterns in shots, and identify deficiencies that are responsible for, for example the loss of points in a game. This understanding can then be used to generate training goal specifications that address these types of strategic or tactical deficiencies.
  • the skill status typically includes a repertoire of movement patterns, each in one of three learning stages.
  • the potentially large number of movement types and the variety of challenges specific to learning stage can make assessment and training challenging.
  • a combination of training goals is usually beneficial to effectively drive skill training, including training goals for forming new patterns, consolidating patterns, and optimizing patterns.
  • Training should follow a systematic process that accounts for the relative importance of the various skill elements to the movement activity and, at the same time, accounts for the natural skill acquisition process, i.e., how the brain naturally forms, consolidates, and refines movements.
  • the training process should be able to distinguish between what aspects of skill to preserve and build on, what aspects of skill to eliminate, and when to form and consolidate new movement patterns.
  • Training activity can be planned using the following criteria:
  • FIG. 11 and evolutionary relationship in FIG. 13). Based on this consideration, training should emphasize patterns that are fundamental to repertoire
  • the training elements can, for example, be arranged in a list sorted in the order of priority that takes into account the above criteria.
  • the training list (see FIG. 45A) is a list of training elements ordered by priority.
  • the training list serves as a type of "working memory" for the skill elements that a user wants to focus on and track at a given period in a training activity.
  • the elements of the training list can also be arranged in a training schedule (see FIG. 45B).
  • a typical schedule is defined by time units such as a session subdivided into sets, and each set is assigned with one or more training goals.
  • the training schedule makes it possible to organize the training activity for a session.
  • the sequence of training elements can be determined based on the acquisition process, i.e., how the skill elements build on one another and their respective acquisition stage.
  • the first set focuses on warming up, during which movement patterns that are technically less challenging and emphasize the range of motion and timing. Once warmed up, the subsequent sets can focus on specific technical aspects.
  • players can play freely or play points, which acts as a test for how well the focused training activity is translated into the task or activity performance. For each training goal in a set, relevant aspects of the performance can be monitored and augmented.
  • Planning can be done manually, with the assistance of an expert, or by an algorithm.
  • the user can select training goal(s) to pursue based on skill status, trend, and overall goals.
  • a coach can use their domain expertise in combination with the skill status and other quantities to help select training goals.
  • an algorithm training agent
  • FIG. 46 shows the state machine showing the active training element and the criteria for the issuance of notifications to the performer. Also shown are stopping conditions for the training element, including, the number of strokes performed, the time elapsed, the incremental (e.g., percentage) progress toward the associated training goals. Typically, the subject is notified of the incremental progress milestones and notified when the stopping criteria of the training goal has been attained. At that point, the next training element can be initiated.
  • FIG. 44A shows the skill status with elements ranked by order of priority within each training stage category.
  • the lists in each acquisition stage category can be ordered based on the contribution to the overall skill profile (based on the skill element composite score).
  • FIG. 44B illustrates an example of skill status that shows how training activity over several training sessions (e.g., Set 1-3) lead to a change in the skill status of skill elements.
  • BHTSH increases its ranking within the pattern to form (from 6 th to 4 th ).
  • the top skill element in the "patterns to form” BHSLH improves and gets re- staged to "patterns to consolidate.”
  • another skill element BHFLM is upgraded from “patterns to consolidate” to "patterns to optimize.” (Note that the training effect is exaggerated for purpose of illustration.)
  • characteristics and assessments essentially provides a rich data set that can be processed using a variety of other analytical techniques, in particular statistical modeling and learning, including neural networks.
  • the systems approach taken here was motivated by the need to identify the different components of a data-driven system and the various forms of assessment and information. It is conceivable to generate these quantities using statistical learning techniques, which can even help discover additional skill attributes from patterns in the performance data.
  • FIG. 26 shows an example of a diagnostic system building on the assessment system.
  • the assessment system used to extract the various skill attributes can be used to drive such a system.
  • Such diagnostic networks can be configured to generate the types of assessments presented herein (skill status, skill profiles), as well as training goals and even feedbacks and instructions and the configuration of the augmentation (cueing and apparatus interaction laws).
  • Typical diagnostic expert systems reason backward, through Bayesian inference, from observations to determine probable causes of specific phenomena.
  • Traditional expert systems are built around a production system which provides the mechanisms to support user interactions.
  • the core component of these mechanisms are rules (e.g., expressed using propositional logic), which are typically deterministic.
  • FIG. 27 shows details of the diagnostic system. It combines a knowledge representation, observations, and an inference mechanism to produce a diagnostic of the movement performance.
  • the domain knowledge from an expert e.g., tennis stroke motion and game
  • a representation e.g., Bayesian Network
  • An inference algorithm uses the Bayesian Network and the observations to determine the most likely explanation for the observations, i.e., diagnostic.
  • FIG. 28 shows an example of an influence diagram for tennis. The diagram captures various factors across the different levels of the movement system hierarchy, including perceptual processes, court motion and positioning, stroke technique, and ball impact. The observations correspond to the example metrics detailed in the
  • the diagram can be structured as a Bayesian Belief Network and used as part of the diagnostic system. Note that the observations can also include general features.
  • the diagnostic system can combine expert knowledge, such as shown in the influence diagram in FIG. 28, with detailed movement functional analysis and direct diagnostics based on assessments. Note also that while some features—for example, the skill attributes illustrated for the tennis use case— are deterministic, movement in the real world usually involves more complex interactions such as adaptation to conditions.
  • An instruction generator converts the diagnostic results to verbal or visual communications (FIG. 27).
  • Information from the diagnostic system when applied to the larger control hierarchy, can also be used to analyze games or task performance and even be used in real-time to recommend actions; for example, which strokes to choose and which locations to target on the court, given the current states of the system conditions.
  • TABLE 3 summarizes the primary elements of feedback and instructions at different levels of the skill hierarchy.
  • DNN multi-layered deep neural networks
  • the main components of such as DNN may include: At the lowest level, delineating between movement phases to produce movement functional structural that would allow detailed characterization of the skills and task performance. Next, learning the movement and broader performance features (conditions and contingencies associated with contextual details) that are associated with the pattern classes and explain the repertoire structure and characteristics that describe a player's performance. Furthermore, higher level layers can identify the technique features that best delineate between movement classes and outcomes at the task level to predict player task performance. Finally, learning structured relationships between features and other factors or conditions that explain a subject's skill and performance at the task and competitive levels, which includes the temporal relationships characterizing task dynamics and for example game strategy.
  • the final category of capabilities for comprehensive data-driven training are the augmentation methods described in FIGS. 22-24.
  • the general purpose of augmentation is to produce various forms of feedbacks (instructions, cues, and signals) and interactions that enhance the subject's performance and maximize training effectiveness for a given set of training goals.
  • the augmentations achieve these effects by: 1) providing information to the subject that help them assimilate the knowledge and/or learning process associated with a training goal (e.g., forming new mental models); 2) providing reinforcements that help induce specific changes in movement characteristics; and 3) creating or extending interactions with the task or activity performance that drive the operational envelope associated with the range of conditions under which a subject can successfully produce an outcome.
  • the former is typically achieved through instructions, the second through feedback cueing, and the third through the use of an apparatus or cues in the task environment.
  • the human augmentation ideally follows an architecture that builds on our knowledge of human information processing (see e.g., Rasmussen 1983).
  • Feedback augmentation can operate at any of the three primary information processing levels (see FIG. 22): the knowledge, rule, and the signal level.
  • the knowledge level includes instructions that explain training elements and training goals, bringing attention to specific movement characteristics and explaining what and how to correct these characteristics. This level of information is typically communicated verbally, in writing, or through visual representations. It helps form representations needed to monitor and correct performance.
  • the rule level includes feedback cue stimuli that encode information to help select the correct movement, or the timing of a specific movement phase, and/or focus attention on relevant aspects of the performance or environment. This level of feedback is typically communicated through visual, audio, or haptic signals.
  • the signal level includes continuous feedback, such as sonifications of the movement based on specific parameters that can be used to communicate relevant aspects or features of the movement profile.
  • This type of feedback can also include extraneous physical effects such as a force field produced by an exoskeleton or often robotic device. They may also include functional muscle stimulation.
  • Signal-level feedbacks are typically generated concurrently with movement execution.
  • feedback creates interactions that can stimulate subjects' learning process and/or assist in the movement performance. It is useful to distinguish between feedback that is produced about the subject's movement performance, and feedback that is produced about the task environment and its elements.
  • the latter includes the interactions enabled by an apparatus, e.g., robot manipulator in rehabilitation or a ball-machine in tennis.
  • Action selection e.g., stroke selection
  • Instructions operate at the cognitive information processing level, and are associated with the symbolic encoding of information. Instructions can help contribute to the formation of mental models or representations that support the skill acquisition process. Instructions are typically communicated verbally or visually.
  • Graphical instructions include plots, schematics describing the spatial outline of a movement, maps, etc.
  • a repertoire map shows the distribution of different movement classes depicted relative to their primary outcomes (e.g., pace and spin imparted on the ball).
  • the graphical description can be distilled based on a given set of movement pattern classes (see FIG. 16).
  • the repertoire of ground strokes can be shown as a stroke map that highlights attributes such as the use frequency of the movement pattern during a session; number of movement executions; and statistics about outcome, success rate, etc.
  • This information can be extracted and displayed for different time periods such as the current set or session. Additional information can be communicated in the stroke map, such as the relevance of the movement class to the task or the difficulty of the movement pattern, which can be determined from the evolutionary relationship shown in FIG. 13, as well as from the complexity of the movement architecture (see e.g., the number of states of the finite-state model in FIG. 5).
  • phase profile plots for a particular movement class to highlight relevant movement characteristics such as phase transition features.
  • FIG. 37 shows an example of the forward swing phase highlighting features associated with the spin outcome including trajectory curvature at the beginning of the phase and angle of attack at impact.
  • verbal instructions would include validation of an outcome or instructions describing which phase transition feature to focus on. Or, it could walk through the movement phases describing features that are critical to performance.
  • Textual instructions also include information layered on graphical instructions or displayed on the screen of a smart watch to display outcome information and progress toward training goals. Instructions and notifications are communicated on a display such as a smartwatch, smartphone, or tablet. It is also possible to use verbal communication via a natural language processor.
  • the training agent determines when and what type of information is presented to the subject.
  • Training activity operates as an autonomous (or semi-autonomous) program.
  • the system determines training goals and schedules, then tracks and updates the training goals and schedule based on the progress and trends.
  • Notifications and instructions are used to communicate information to run such a program, provide instructions about the active training goals, how they are pursued through the activity (e.g., drills), and when to switch training goals, etc.
  • the training goals and schedules are updated dynamically.
  • the disclosure builds on the real-time augmentation technology for movement training described in U.S. Patent Application Publication No. 2017/0061817.
  • the three primary categories of augmentation forms that can be used to help induce changes in movement technique specified by training goals include:
  • Outcome validation Signals provide instantaneous assessment of the overall movement performance and outcomes. Validation cues are generated immediately following the action to indicate a successful outcome. The outcome validation is not limited to movement outcomes, but can be used to reinforce other relevant aspects of movement performance including smoothness, timing, etc., and those captured by performance criteria.
  • Alerts Alerts augment the natural proprioceptive signals to enhance the subject's sense of movement with respect to specific training goals. They can also be used to implement injury prevention using the relationship between movement characteristics and biomechanics.
  • a central aspect of learning good movement technique is learning the sensory consequence of correct performance.
  • Real-time cueing therefore, can provide validation signals that augment the natural signals to reinforce learning the sensory consequence
  • real-time augmentations can be designed to help with:
  • Training movement architecture Real-time feedback assists in the formation of new movement structure through the use of visuals (e.g., simulation), as well as real-time cues e.g., that signal the conformance of the pattern to a template.
  • Forming anticipatory perception Provides signals to learn to identify critical environment and task cues that are used to anticipate critical states and conditions, such as timing of the movement phases that enable synchronization of movement behavior with the task elements or objects.
  • Real-time feedback augmentations are communicated by the cueing system (see FIGS. 22 and 23) and include audible, visual, and haptic signals.
  • the natural variations in a training environment combined with the variability in a subject's performance may not be sufficient to expose the subject to all relevant conditions that help drive skill acquisition. Particularly for deeply solidified patterns, highlighting erroneous features in a movement or providing feedback cues may not sufficiently change the movement pattern. In these situations, it may be more effective to actively produce new training conditions and thereby force the subject to acquire new movement patterns.
  • An apparatus can also be used to help form new movement patterns by physically guiding the movement. This technique is already used in robotic movement rehabilitation.
  • the training system is derived from the understanding of human movement learning and movement organization and performance, the training system can be implemented for a broad range of movement domains, including sports such as tennis (described in detail), rehabilitation, as well as professional activities such as surgery. Most of the concepts and quantities such as movement repertoire, their outcomes, etc. are derived from the theory of open motor skills acquisition.
  • the training system can also be used for various forms of human-machine systems, including tele- robotics, humans equipped with prosthetics, or other forms of physical augmentations such as exoskeletons.
  • the augmented training system can be conceived as an integral part of such HM systems.
  • the robotic surgery system such as the da Vinci is an example of such a HM system.
  • Many of the relevant quantities (operator inputs, manipulator or tool motion, visual gaze, etc.) are measured and recorded; therefore, the training system can be incorporated into the surgical robot's operating system.
  • a data-driven skill assessment and training system integrated into such a robotic system can fulfill many functions, including: 1) train surgeons for new procedures, where they would benefit from accurate tracking of their skill learning process and feedback to help that process; 2) opportunities to formalize the certifications of surgeon training for different procedures, etc.
  • FIGS. 32-39 give a sample of the processed performance data. Starting with FIG. 35, which illustrates the activity data for a time period, highlighting sessions and sets over a calendar period. FIG. 39 then provides a close-up into a specific session and shows the event diagram that displays select stroke types ST used over the session timeframe (12:13 to 12:50). It also displays the pace SP and spin S outcomes as time histories TH to visualize trends in those outcomes over the play duration.
  • FIG. 36 then gives a more detailed look at the period of activity on a stroke-by- stroke basis 381. Additional outcome quantities are illustrated first, including the impact variability 382 and success cumulative progress 383. Below, it includes the separate time histories for the pace 384 and spin 385. The time histories are filtered to smooth out the stroke-by- stroke variations that can make these plots more difficult to read. Note, however, that since there is no inherent continuity in outcomes such as spin from one stroke to the next, the filtering can create artifacts. The plots in FIG. 36 also highlight the reference tiers 360-364 for the outcome quantities to help their interpretation
  • FIG. 37 illustrates details of the functional analysis at the level of the stroke pattern. It displays the forward swing movement segment phase for the forehand topspin medium (FHTSM) stroke class, highlighting the path 710 of the racket relative to the origin or impact point 720.
  • the stroke analysis based on this phase segment allows for the identification of features such as the angle of attack 730, the curvature of the path at the beginning of the forward swing 740 (transition from back loop phase), and the length of the swing phase 750.
  • the figure also illustrates sets of segments corresponding to the core pattern 760 of this stroke class, and a sub-pattern set 770 that represents the subclass of strokes with the highest spin outcomes.
  • FIG. 42 shows impact timing for the different groundstroke classes GC, which is defined as the timing relationship between the impact time and the time of the peak acceleration (or angular rate) of the forward swing movement phase T. Impact timing depends on the movement technique, motor coordination, as well as proper anticipation of the impact point and the player' s preparation for the stroke. Therefore, it provides critical information to diagnose stroke technique.
  • FIG. 33 shows an aggregate view of the relationship between the swing rate R (horizontal axis) and spin S (vertical axis) produced for an ensemble of strokes in topspin, flat, and slice classes C for a particular subject.
  • the quantities define the so- called spin envelope SE, which describes the range of spin S that can be produced by the subject as a function of the racket swing rate R.
  • the spin represents the outcome and the swing rate represents a movement technique attribute, which in this case can be considered as the effort applied by the subject to produce the outcome.
  • the spin envelope is parameterized based on the slope of the two linear boundaries (k max MX and kmin MN), each are depicted along with reference lines corresponding to low, medium, high, and very high ranges, which again can be computed from a population.
  • FIG. 38 depicts the composite score for a specific stroke class (skill element) as a radar chart, which is an illustration of the skill-element composite score. It shows individual cost components based on extracted performance and skill attributes (impact precision IP, consistency CC, impact SR, efficiency EF, smoothness SS).
  • the composite score which can be visualized as the area covered by the polygon PG, represents the overall assessment of the skill element (stroke class). This polygon compared to the less opaque polygon CP illustrate what could be a comparison between two players, or between different skill elements, or the same skill element at different times in a subject training history.
  • FIG. 40 then takes a more comprehensive view and depicts the overall skill profile as a bar graph of composite scores CS for the groundstroke repertoire GR. This chart makes it possible to assess the overall repertoire strength and weaknesses (see FIG. 17). Similar to the skill element composite score, this skill profile can be used for comparisons between different players or between different times in a subject's training history. As already discussed, different composite costs can be used to emphasize different characteristics relevant to a task performance.
  • FIG. 41 displays the acquisition stage of the strokes in the groundstroke repertoire based on the criteria described in TABLE 1 and TABLE 2.
  • FIG. 34 shows the leaderboard, which synthesizes the entire assessment at the population level.
  • these data visualizations are a sample of quantities described in this disclosure, and are used here to illustrate the types of quantities that can be used for the assessment and diagnostics for different levels and components, and how they can be used in conjunction with reference ranges to support the identification of training goals and eventually the feedback synthesis.
  • These visualizations can then also be used for tracking progress and for updating the training elements and cueing laws, etc. as someone's skills evolve relative to their own history as well as to that of a larger population.
  • FIG. 43 gives an integrated perspective on the system's main components, organized in terms of the levels of assessment (physical 510, pattern 520, task 530 and competitive 540).
  • the figure highlights some elements and quantities that drive the training process, in particular it highlights examples of assessment criteria at each level, and how the criteria relate across levels.
  • the stroke forward swing phase profile with features depicts an example of a skill model that can be used to analyze a subject's movement technique, taking into account the different assessment components (outcome, biomechanical, functional, perceptual, memory, and learning). Each component can be used to generate attributes for assessment and characterization of the skill element (i.e., the stroke class).
  • the efficiency attribute EA captures the relationship between the spin outcome and forward- swing energy.
  • the attributes can be formally captured by a cost function 512, 522, 532, 542.
  • FIG. 37 emphasizes the model describing the spin outcome and relevant functional characteristics. Similar models for the biomechanical characteristics, for example to identify features that can predict joint loads or muscle strain, can be developed and then converted to an attribute, e.g., injury index, that can be included for the skill element composite score (see FIG. 38).
  • FIG. 43 shows how different attributes associated with the assessment components contribute to create the overall skill element score (see FIG. 38).
  • FIG. 43 shows how the skill elements contribute to create the subject skill profile, highlighting the forehand top-spin medium stroke FTSM depicted in levels 510 and 520. It also shows how the skill profile is obtained through a composite cost function 532 combining the skill elements in the stroke repertoire.
  • FIG. 43 shows how an individual's skill compares at the level of a population.
  • the comparison is based on percentile rank computed from the skill profile composite score.
  • the figure highlights how the individual's skill profile ranks SPR are relative to the population PP.
  • FIGS. 32-43 The material illustrated in FIGS. 32-43 can be embedded within a web-based or mobile app reporting system to allow a subject to navigate their skill elements and characteristics.
  • the content below is organized into three sections:
  • Activity Session Report provides a description of the movement activity for a given session in terms of the skill elements, how these are used throughout the activity period, and various performance and skill attributes.
  • the session report can also include training elements in the training list.
  • the knowledge also provides the data to generate training goals and to plan and schedule training activities.
  • Pattern Class Report is a class-by-class detailed description of the various assessments, including pattern level assessment, as well as functional analysis and diagnostics at the level of the skill elements.
  • the assessment can also include historical trends of how different outcomes and attributes of the individual skill elements evolved over the subject's recorded activity history.
  • the class-by-class description can also provide information about active training elements as well as suggested training goals.
  • the activity session report focuses on the overall description of the movement performance in a given session, focusing on the activity performance characteristics.
  • the purpose of the session report is to convey understanding of high-level patterns in the activity performance, such as the evolution of various attributes over the period of the session; the use of particular movement patterns; and the trend in their outcomes, such as energy and success rate.
  • the session report can enable the identification of the onset of fatigue or loss of concentration. This information can, for example, be used to help improve the training session, or even fitness or physical strength.
  • the activity summary for a session can be presented as a table that describes statistics and trends for attributes of the most frequently used movement patterns in the recorded session.
  • the statistics for the tennis use case can include: a) Pattern usage frequency (%); b) Impact success rate; c) Pace (m/s); and Spin (rpm).
  • a trend symbol (up, down, or equal) and a trend value can be appended next to each metric to highlight the change in the respective metric for the session or relative to a selected time period.
  • a similar table can be used to summarize the activity for the training elements currently in a training list.
  • the table may include the activity level for each training element during the session, when the element was created, the progress toward the goal during the last session or relative to a selected time period, etc. This information can be used to verify the effectiveness of previous training goals, training lists, and training schedules, and to help update the subsequent training plans. These summaries can be linked to visualizations of the session activity that enable more detailed insights into trends of select attributes of skill elements or training elements .
  • FIG. 39 depicts a time history TH of player movement pattern usage.
  • the usage trend plot depicts movement patterns on a stroke-by-stroke basis, where each vertical line L is a stroke occurrence.
  • the movement class membership of a stroke, representing a skill element, is indicated by the vertical position of each line L.
  • This example uses a subset of six stroke classes 30 to describe the main movement pattern trends in this activity.
  • This data used for the usage trend can also be analyzed to identify rally segment statistics, such as the average stroke counts or rate of return for each class used during the rally.
  • the rate of return describes the probability of the opponent making a return. This probability can be computed for a specific pattern class.
  • rally ending strokes that lead to either points or losses, it is possible to identify the strong or deficient pattern classes, which can be used to identify deficiencies in the repertoire.
  • the movement outcome trends in this section shown in FIG. 39 focus on the evolution in primary stroke outcomes across the different movement patterns during an activity session (pace SP and spin S).
  • the session report gives the breakdown of how the subject used their time during a session, and therefore provides a composite view of the activity in the session. This information can reveal patterns in the technique and outcomes associated with the activity performance at different stages, such as during the warm up, while training on a specific training element.
  • the information in this chart can enable automatic identification of the type of the sets in a session.
  • Sets can be identified by the intermittent rest, and for example, the deliberate training sets will have specific features such as concentration of strokes belonging to the same movement pattern, or the stroke pattern transitions forming a repeating pattern.
  • the chart and the information underlying it can convey information about the intensity or even the competitivity or competitiveness of the play in a set.
  • the information can also can reveal patterns within a set, or across sets, that are related to physiological or psychological processes, such as the onset of fatigues, or deterioration in concentration.
  • activity at the task and competitive level can be further analyzed and assessed using statistical algorithms, such as a Hidden Markov Model.
  • a Hidden Markov Model For example, such techniques can be used to build a state-machine that represents the most likely transitions between movement patterns based on various factors including a player' s own prior activity. It can also include information from opponent activity performance, and be set up to capture the extended temporal patterns encompassing the task and environment elements.
  • the pattern class report is organized at the level of the individual skill element or movement pattern. It tracks the multivariate attributes and characteristics for each movement pattern, and therefore can provide insight into the skill acquisition process of each movement pattern, and help identify the specific deficiencies, which in turn can be used to help determine training goals.
  • the play activity of the pattern class is presented as the stroke counts by set, by session, and across the entire recorded history (see FIG. 35).
  • the bottom histogram 351 in FIG. 35 shows the stroke counts by date over the entire recorded activity history of a player.
  • the shaded bar 354 on the histogram can be moved by the user to select a set of consecutive dates to be presented in top chart 355.
  • sets are shown as stacked shaded bars grouped by date 352. (This chart can also be used in the Play Activity Summation section in the player report, with stacked bars representing sets or movement pattern classes.)
  • the stroke counts 353 of the specific movement pattern indicates how frequently the pattern has been used. If use frequency is correlated with a decline in outcomes, for example, it prompts the diagnosis to identify causes which in turn could be used to formulate a training goal.
  • Movement outcome trends for a specific class are shown in FIG. 36. It focuses on the longitudinal dimension of the movement pattern development process by presenting the trends of a selection of select movement outcomes and attributes (for example: pace 384, spin 385, cumulative success progress 383, and impact variability 382) across the entire recorded activity history (see FIG. 35).
  • the plot background shades 70 in the x-axis delineates the different sets.
  • the plot background shades 360 in the y-axis encode the information about the reference ranges or tiers (e.g., very high 361, high 362, medium 363, and low 364).
  • the success rate trend plot depicts the cumulative summation of the impact success variable.
  • the trend plot takes the form of a stair function 370 (up one step for a successful impact, down one for a missed impact).
  • the dashed line 371 provides a reference for 100% success rate trend; a horizontal trend line would correspond to a 50% success rate. A subject can easily determine success rate trend by looking at the slope and contour of the trend line.
  • Impact variability is one of the class ensemble statistics. It is calculated for every set and presented as a staircase function across sets 377.
  • the other trend plots depict the evolution of the movement outcomes over time on a stroke-by- stroke basis. However, the time history can be smoothed to remove large variations that can make the interpretation more difficult.
  • this visualization can be used to verify the effectiveness of previous training goals, training lists, and training schedules, and help update the subsequent training plans.
  • the movement functional analysis focuses on the details of movement technique used by the player to achieve their outcomes across the various movement patterns or skill elements. It also encompasses other relevant mechanisms that are used to modulate the outcomes or adapt to conditions. Functional analysis at the level of movement phases provides detailed insights into the movement technique that is valuable for the determination of training goals. This is illustrated in FIG. 37 for the forehand topspin medium (FHTSM) stroke class.
  • FHTSM forehand topspin medium
  • the forward-swing phase occurring immediately before the target phase of impact, contributes to the realization of desired movement outcomes of the motion pattern. Therefore, it provides both information about the outcome, and the more general organization of the movement.
  • This phase lasts about 100ms, which means that most of this movement segment is executed in open-loop, i.e., without opportunities for corrections. Therefore, its success depends on the motor program stored in so-called procedural memory. This program encodes the coordination and perceptual cues, the muscle synergies that support the physical execution, and the correct movement phase initiation and configuration (see FIG. 3A).
  • FIG. 37 presents the stroke trajectory profile of the forward-swing phase of the forehand topspin class.
  • the figure compares core -pattern strokes with a subset of pattern strokes identified as having the highest movement spin outcomes.
  • several features can be extracted for this movement phase (angle of attack, curvature of the path at the beginning of the forward swing, and the length and the elevation of the swing phase).
  • the forward swing profile also provides a visual description of the movement technique that can be used to generate visual instructions, such as a target movement profile shape.
  • Real-time feedback cues can be generated to reinforce the desired features.
  • the efficacy of these cues can be enhanced by combining them with visual descriptions of the target profile shape, which serves as a template for a mental model.
  • the integration between the sensory-motor and cognitive levels can accelerate the consolidation.
  • FIG. 33 compares the overall spin envelope SE, which is defined by the racket swing rate R and imparted spin S.
  • the spin envelope describes the efficacy of the stroke technique as ratio or spin/swing rate. A larger angle for the line delineating the envelope indicates that a player can achieve a higher spin outcome with an equal racket swing rate.
  • the dashed lines DD correspond to the reference ranges from the population analysis.
  • the spin envelope helps identify the deficiencies in stroke technique; as shown here the cause of the spin deficiency is due to an insufficient racket roll rate at impact. Generating larger roll rate at impact requires optimizing movement coordination, i.e., the movement architecture between the backswing and impact phase.
  • Timing metric is the timing of the impact during the forward swing.
  • the timing metric is defined as the relation between the instant of the peak racket swing rate and the impact. Correct timing of the forward swing depends on the player's anticipation of the interception, as well as other factors such as anticipation of the ball trajectory, footwork, and preparation.
  • This section integrates the attributes statistics of the movement pattern to determine a composite skill score using a cost function, e.g., the weighted sum of the attributes:
  • a radar chart as shown in FIG. 38, enables an intuitive interpretation of the multivariate contributions of attributes to each skill element.
  • the figure shows a subset of select attributes depicted as a dimension 10-50.
  • the total area of the polygon 60 formed by the outcome or attribute values can be viewed as a description of the composite score of the movement pattern.
  • the composite skill score can be used to rank the movement patterns and can be combined across patterns to form the player skill profile (see FIG. 40 and FIG. 17), which provides an overview of the repertoire, enabling the identification of player strengths and weaknesses.
  • This representation also enables the comparison of skill elements over different time periods, or between different skill elements.
  • the two polygons 60, 70 shown in the chart can represent the statistics of the current epoch versus that of the entire recorded history, or the statistics of the player versus that of a subgroup that the player belongs to.
  • the example player report combines the different assessments to create an overview of a player's overall skill status and skill development progress.
  • the player report is organized at the level of the repertoire. It includes the following four sections:
  • the play summation presents a player's activity statistics, which is a summary of performance activity over the subject's entire recorded history in terms of the following: 1) total number of sets, 2) total number of sessions, 3) total time duration, 4) last time of play, and 5) overall success rate.
  • the repertoire level skill assessment focuses on how complete the repertoire is relative to the task requirements.
  • the repertoire completeness can be determined from use frequency (stroke counts) and the overall movement outcomes of a performer's repertoire relative to a nominal repertoire of motion patterns for the task.
  • the nominal groundstroke repertoire is defined by a fixed number of groundstrokes expressed in terms of the spin and pace. Each of these outcomes are discretized in three levels (“slice,” “flat,” and “topspin” for the spin imparted on the ball and "low,” “medium,” and “high” for the pace) (see FIG. 16 and FIG. 32).
  • the impact success rate defined based on sweet spot area
  • impact location variability are evaluated to measure the impact quality.
  • FIG. 32 illustrates the overall movement outcomes using pace and spin as example. Movement patterns are divided into backhand and forehand, and sorted by the average outcome values. The data is visualized as a histogram chart. The lighter color bars correspond to the movement patterns without sufficient stroke counts and low statistical significance.
  • FIG. 32 Background shades in FIG. 32 indicate different tiers/reference ranges (e.g., low, med, high, and very high). These reference ranges can either be determined based on the player' s own statistics or derived from the population analysis that extracts the statistics from a subgroup of players sharing similar movement techniques and skill level. In this example, common reference ranges for all movement pattern classes are depicted since the emphasis is the overall repertoire. A more precise assessment can be achieved by extracting reference ranges that are specific to different pattern classes, including other relevant factors such as impact conditions. The more detailed contextual information is available, the more precise and actionable assessments can be achieved.
  • tiers/reference ranges e.g., low, med, high, and very high.
  • Skill status captures the skill acquisition stages of the movement patterns in the repertoire.
  • FIG. 41 illustrates an example for the groundstroke repertoire. Each movement pattern is determined to be at one of the three stages: pattern formation, pattern consolidation, and pattern optimization. The qualitative characteristics and quantitative criteria that can be used to identify the acquisition stages are listed in TABLE 1 and TABLE 2 respectively. Skill status can be presented as a table with acquisition stages as columns, and movement pattern classes, or skill elements, are arranged in a sorted order (see FIGS. 44A and 44B).
  • PLAYER SKILL PROFILE The information of use frequency, movement outcomes, and skill status of all the movement patterns can then be used to determine the player's skill profile as a histogram of sorted scores of motion patterns (see FIG. 17).
  • the skill profile provides the information to build a leaderboard (see FIG. 33) and the larger population analysis.
  • this section also presents the rally statistics, such as the average number of strokes in a rally and the cadence (number of strokes per minutes). This provides information to identify the player style in the gameplay.
  • the disclosure includes a system to help individuals train or rehabilitate movement through and using targeted augmentations designed to stimulate learning through feedbacks and interactions. These augmentations are further adapted to the specific skill deficiencies that occur at different stages of the movement learning process, and account for the human information processing hierarchy.
  • the system builds on movement sensing, skill modeling and diagnosis, and feedback synthesis, which are described previously described in U.S. Patent Application Publication No.
  • the general goal of training augmentation is to help guide the development of skills by providing feedback during training or performance. Since skill learning is an ongoing, dynamic process, a valuable feature of systematic data-driven skill training is the capability to model and diagnose skills in a way that captures the longitudinal and vertical dimensions of skill development. Recall, the longitudinal skill dimensions refer to the process of skill acquisition over time, through transformations of existing skill elements, and the vertical skill dimensions refer to formation of new skill elements.
  • the augmented skill platform is configurable to create an integrated
  • FIG. 2 illustrates the elements of the augmented tennis activity environment that serves as a use case for this disclosure.
  • Any task can be described by environment elements EE, and task elements TE.
  • a person manipulates a device (e.g., tennis racket), end effector or piece of equipment, to interact with the task elements TE (e.g., tennis ball).
  • the task elements TE e.g., tennis ball
  • miscellaneous accessories Z such as shoes or clothing that may be relevant for the description of the activity.
  • the workspace W is contained in the environment and is specified by various constraints and rules that characterize the task's success and performance (e.g., the tennis court and tennis game).
  • the person is the player (or players); the task environment is the tennis court; the task element is the tennis ball; and the equipment is the tennis racket, and the accessories Z are the shoes and other pieces of attire such as an arm or head band.
  • output devices can be included, including graphical displays (e.g., LCD, OLED, etc.), haptic devices (e.g., embedded in the racket grip), speakers.
  • input devices including touch sensitive display (user interface), keyboard, etc.
  • the input and output devices may be integrated in the form of a smart watch, tablet, or a wearable device that can be worn by the person.
  • the overall elements, agents and other components used, including the measurement, input and output devices, are referred to as the augmented human system or simply the system S.
  • Other examples of systems that have this general setup include a robotic system, a cybernetic system (e.g., a human fitted with a prosthetic), and a human- machine system (human operating a robot through tele-operation).
  • a robotic surgical system such as the DaVinci® Surgical System (available from Intuitive Surgical, Inc.) is a robot that is an example of an integrated augmented movement skill system.
  • Measurements y that contribute to the recorded performance data can be acquired from different components of the human actors, equipment, or system.
  • instrumentation is designed to obtain measurements that encompass relevant variables for the particular level of analysis. For example, as illustrated in FIG. 2 in the analysis of human tennis stroke path 25 and performance, the states, or a subset of the racket motion may be sufficient. However, to enable a complete analysis of the movement on the court, the footwork, or the body motion such as the kinematic chain or other movement units, additional measurements about the environment and body segments 15 (e.g., arm, legs, feet, etc.) can be added.
  • body segments 15 e.g., arm, legs, feet, etc.
  • IMU inertial measurement unit
  • optical tracking systems etc. Examples include the use of video cameras 70 that capture the broader agent behavior and the task environment 50. Vision processing can also be used to extract information about the motion of individual body segments 15.
  • a gaze tracking system 80 to measure the visual attention.
  • a user 10 or player or other subject holding a tennis racket 20 which impacts a ball 30 during the swing or stroke of the racket 25.
  • One or more motion tracking or video cameras can be attached to the performer, such as integrated with the gaze tracking system.
  • These so-called first-person cameras capture data related to the interaction of the subject 10, the tennis racket 20, the ball 30, as well as the motion of other participants such as the opponent 53, and other relevant environment elements such as the court 51 and net 52.
  • video cameras on the subjects and/or environment make it possible to determine which elements or events the performers are attending to at any given time, or at specific instants of the performance such as during specific movement phases or phase transitions 26, 27 on the path 25, opponent behaviors, or task elements such as related to the ball trajectory 36.
  • Inertial sensors 21 or similar measurement units can be embedded or affixed to the equipment; worn by the user, subject, or other agent 10 to measure the movement of body or body segment 62; or even placed on the user's, subject's, or agent's skin or implanted in the body to measure muscle activity or neural signals involved in the control of muscles 15.
  • gaze follows the ball trajectory 36, which has several notable events during the motion, such as the ground impact 32, the racket impact 30, and the interception by the opponent.
  • the gaze (described by gaze vector 81) also typically can fixate on target areas on the court (outcome 3, ref 35), in between the court (outcome 2, ref 34), as well as anticipated racket impact or post- impact location (outcome 1, ref 33).
  • data fusion and state estimation techniques may be implemented to determine states x that are not directly measured.
  • the orientation of a body segment 15 or piece of equipment 20 requires an attitude estimator which combines angular rate data from the gyroscopes, the accelerations from the accelerometer and the magnetic field strength from the magnetometer.
  • An example of data fusion and estimation is the use of a vision- based tracking algorithm, applied to video data from video cameras, combined with IMU data from a device on either the body segment or equipment, to extract body segment or equipment motion information.
  • a vision- based tracking algorithm applied to video data from video cameras, combined with IMU data from a device on either the body segment or equipment, to extract body segment or equipment motion information.
  • Such a data fusion system can be used to provide an accurate estimation of absolute pose of body segment or equipment.
  • the combination of motion processing such as based on IMU and computer vision enables the extraction of video frames associated with certain events in the agent-environment interactions.
  • a phase transition 27 such as the forward swing initiation 26
  • a specific event in the task or environment such as a ground impact of the ball 32
  • can be relevant in assessing the agent strategy taking into account visual attention (gaze 81), body location of the subject 10, footwork (shoes or feet) 60, and movement preparation, or particular phase segment initiations 26, 27.
  • HMM Hidden Markov Models
  • various wearable devices can be configured to generate a range of communication modalities such as audio, haptic, or visual. These devices can operate along different levels of the information processing hierarchy discussed earlier. Such cueing devices can be worn on the body, skin; integrated in the equipment such as in the racket grip 21, shoes 60; or even implanted in the skin or body such as muscles 15. They can be configured to provide different modalities of feedbacks such as audio, haptic stimuli, or visual cues.
  • Another class of output devices include an augmented reality (AR) system 80 that can be configured to provide visual cues superimposed on the natural environment. Speakers, or visual signaling devices such as cones, markers, etc.
  • AR augmented reality
  • Implantable devices can also be used as part of the augmented system and for example provide functional muscle stimulation 15.
  • Outputs can also be communicated via the typical wearable devices, mobile and portable devices and computers that are part of the augmented skill ecosystem, such as smart watches, phones, or tablets.
  • Typical human cyber-physical systems are described formally using hybrid system notation.
  • This notation system combines continuous and discrete quantities.
  • the movement of a user, subject, or other agent may be governed by physical laws that result in nonlinear continuous time differential equations.
  • Discrete variables may be used to evaluate conditions associated with specific events, such as counting strokes in a tennis game or scoring the game based on ball trajectory relative to the task environment and rules.
  • Categories of state variables include: controlled variables, specific behavioral variables such as the visual gaze vector, and features used as cues by the agent to make decisions.
  • Actions are typically taken by the user and represent the addition of force or energy to the system such as the racket ball impact 30. Actions are typically applied to specific locations such as the end effector or equipment. As already discussed, actions are often motivated by a deliberate desire to achieve particular outcomes 33-35. In tennis, for example, the player wants to impart a specific effect on the ball (velocity and spin) 33, with the ultimate goal of driving it to a specific location on the opponent's court side 35. Events can be defined by particular state conditions. For example, in tennis, a major event is the impact of the ball on the racket 30.
  • Events can be expressed formally by constraints on the system states, e.g., racket acceleration exceeding a threshold due to the impact, or alternatively, the impact can be detected when the ball and racket velocity are equal.
  • Other relevant events in tennis include contact of the ball with the ground and when the ball crosses the net (see FIGS. 7 and 9).
  • outcomes are defined as quantities that capture the relevant characteristics of the agent's behavior in the performance of task.
  • outcomes can be categorized hierarchically, e.g., primary outcomes, secondary outcomes, etc. (see FIGS. 7 and 10).
  • the definition of outcomes are a function of the scope and level of the analysis.
  • outcomes are a subsect of the system states (e.g., at specific times, defined by events) or a function of the states.
  • primary outcomes are the characteristics associated with the racket-ball impact 30, such as the spin of the ball when it leaves the racket or the ball's velocity 33.
  • Primary outcomes could also include the location of the ball on the racket's string bed 30.
  • more comprehensive outcomes include the location of the ball's net crossing 34 or impact on the court 35.
  • agent A The skill of an agent A is the effectiveness with which the agent is using its body and/or tool, equipment, etc., to achieve desired task outcomes TO and more generally interact with, and/or adapt to the environment elements EE and task elements TE.
  • Miscellaneous additional quantities that can be added to the description of the task or activity performance include task or game rules (e.g., rules of the tennis game), which provide the basis to determine task success or completion and various task performance characteristics, as well as various decision rules and control laws for other computer-controlled or autonomous agents, apparatus, or equipment or accessory.
  • task or game rules e.g., rules of the tennis game
  • various decision rules and control laws for other computer-controlled or autonomous agents, apparatus, or equipment or accessory.
  • control law, rules, and algorithms that specify the behavior and actions of the apparatus in the environment.
  • These systems can include a prosthetic limb, an apparatus that reacts to the environment or task interactions, or even the various components of a robotics system such as a surgical tele -robotic system.
  • FIG. 21 illustrates an overview of the system and is followed by a description of the "augmented human system,” and finally, the general motion model, skill model, and the different augmentation modalities illustrated in FIGS. 22, 23, and 24.
  • the iterative training process illustrated in FIG. 21 illustrates three primary feedback loops: 1) A skill assessment loop (AL) 200 that tracks the overall progress in movement performance in the task domain, updates information about the user's skills, including motion models and skill models, as well as diagnostic tools used to identify specific deficiencies in movement technique that provide the basis for the synthesis of training goals; 2) A training loop (TL) 208 that tracks the progress in specific areas of the skill captured by training goals and configures the augmentation system; 3) A feedback augmentation loop (FL) 202 that provides relevant information during the movement performance.
  • a skill assessment loop A 200 that tracks the overall progress in movement performance in the task domain, updates information about the user's skills, including motion models and skill models, as well as diagnostic tools used to identify specific deficiencies in movement technique that provide the basis for the synthesis of training goals
  • a training loop (TL) 208 that tracks the progress in specific areas of the skill captured by training goals and configures the augmentation system
  • a feedback augmentation loop (FL) 202 that provides relevant information during the movement performance.
  • the identified motion and skill models combined with the diagnostic assessment, provide the basis for generating a set of instructions, which can be used to organize the training process, and synthesize cueing laws used to drive the augmentation.
  • a user receives two primary forms of feedback: instructions and real-time cues.
  • the instructions are typically generated during a session at particular intervals, e.g., completion of a training set, or after a training session. Instructions are typically presented in visual form and emphasize more comprehensive aspects of performance and skill.
  • the augmentation loop can be used to exercise movement on movement characteristics that have been identified through the diagnostic tools.
  • the cueing process targets specific characteristics to directly impact movement outcome and performance.
  • the cueing system computes feedback signals using algorithms that are synthesized based on the motion and skill models derived during the assessment. These cues are communicated in real-time to the user.
  • the assessment and augmentation feedback are delivered following the hierarchical organization that takes into account the hierarchical structure of skill development and the temporal characteristics of the movement and skill attributes.
  • the training assessment loop is managed by a training agent.
  • the augmentation loop is managed by a cueing agent. These agents operationalize the two processes and are able to track progress at these two levels and provide user with the interactions to run this system (see FIG. 21).
  • the motion model captures the comprehensive movement performance through the movement repertoire which organizes the range of movements as classes of movement patterns and their associated outcomes.
  • the repertoire model provides the ability to identify gaps or weaknesses in patterns. Gaps in the repertoire, i.e., missing motion patterns, manifest as the inability to produce actions and outcomes in areas that are relevant to the task performance. Gaps can also manifest as the inability to deal successfully with the range of prevailing operating and task conditions that are required to enable high level of task performance or from contingencies or environmental disturbances. Movement patterns are represented to describe relevant functional characteristics, such as phases and their associated biomechanical constraints.
  • the primary functions needed to support data-driven augmented training include:
  • the system provides a range of feedback types that act as drivers to modify subjects' behavior toward improving their skills.
  • the feedbacks are based on information and knowledge extracted from the motion and skill models, as well as from the extended analysis based on performer population, which make it possible to account for broader factors.
  • the feedback operate at various levels of the human information processing systems. These encompass a broad range of neuro-cognitive mechanisms. For example, the highest-level feedbacks are based on drivers that are rooted in social aspects of performance. These include leaderboards with ranking, side- by-side comparisons between players (e.g., via the skill profile, see FIG. 17), or role models that can be selected from the population analysis.
  • TABLE 4 summarizes the drivers, derived from the data-driven modeling and assessment, according to their levels of operation in the hierarchy.
  • Top-level drivers Mid-level drivers (cues)
  • Low-level drivers (signals) (cognitive)
  • Patterns to develop/form Quality of outcomes e.g., Feedback augmentation
  • the population group capturing the skill development provides the direction for the orientation of the training, such as movement architecture
  • the system relies on a movement capture and measurement system (shown in FIG. 2 and FIG. 21).
  • This system collects data from relevant movement quantities, including movement of equipment and body segments; physiological quantities, including electrical muscle activity (e.g., via surface or implantable electrodes); and other relevant quantities from the recorded performance data.
  • Data also includes task relevant quantities, such as outcome of the action or movements, as well as its effect on the larger task outcomes.
  • the system can track multiple users and their interactions.
  • the three primary feedback loops 200, 202, 208 shown in FIG. 21 are closed around the augmented human system detailed in FIGS. 22 and 23.
  • the human movement activity is augmented at three primary feedback levels which are communicated to the user through different modalities.
  • the feedback forms are organized according to the primary levels of human information processing and include: instructions or
  • communication modalities include audio, visual or haptic stimuli (potentially also direct functional muscle stimulation or even stimulations of the subject's peripheral and central nervous system).
  • feedback augmentation also includes activity interactions provided by an apparatus.
  • Instructions provide information about the training elements and the associated training goal. This information contributes to the formation of mental representations. They are typically communicated verbally, symbolically, or graphically.
  • Notifications provide information about progress with respect to a training goal. These are considered at the knowledge level of human information processing and can be communicated verbally, symbolically, or graphically.
  • Cues provide information to highlight specific features about the performance or outcome. They are typically communicated through discrete audible, tactile (haptic), or visual signals and contribute to the formation of rules that allow efficient processing of information both for motor and perceptual functions.
  • Signals provide real-time information to guide movement and enhance relevant movement features. They are typically communicated through continuous, or piece-wise continuous audible, tactile (haptic), or visual signals (potentially also direct functional muscle stimulation or even stimulations of the subject's peripheral and central nervous system).
  • Apparatuses enable interactions at the activity level to emphasize particular task states or conditions, such as a ball machine that can throw balls with different trajectories (pace, spin, height, depth, etc.) to the player.
  • An apparatus can also be used to physically guide movements, such as with an assistive robotic device.
  • the system is configured to receive various inputs from the user, coach, or physical therapist.
  • the user interactions are enabled by a graphical user interface (GUI) and/or natural language interface (NLI).
  • GUI or NLI enable the user to browse or interrogate the skill assessment and configure the training process.
  • users can select which training elements to track and which feedback forms (notifications, cues, signals) are preferred.
  • the user can also provide inputs related to the outcomes or technique of the movement during performance. For example, they can tag a particular action or movement they believe is relevant for further analysis. Users can also rate individual sets in a session, for example, based on their perceived training effectiveness. These feedbacks about performance can be used to highlight particular qualities during the assessment and diagnosis process. For example, they can serve as additional assessment signals.
  • the training loop is managed by a training agent which provides various degrees of autonomy and provides functions to assess skills along specific skill elements that have been designated as training elements.
  • the TL helps structure practice by organizing training goals in schedules. It also manages the configuration of the different components of the augmentation system.
  • the feedback augmentation loop is managed by a cueing agent, which tracks effectiveness of the selected cueing profiles (cueing law, apparatus interactions).
  • the Assessment Loop describes feedback that takes place over longer periods, spanning one session to multiple sessions, associated with the skill acquisition process.
  • the unit of time for the AL is an epoch, which as already discussed is defined by the data set requirements for modeling and assessment, denoted by superscript k.
  • the primary functions of the AL are computing and updating the movement models (M k ) and skill model (skill status S k ).
  • the skill status S k is a collection of skill element extracted from the repertoire that are assessed with respect to the skill learning stage.
  • Information about movement and skills are used to plan training activities and synthesize the various forms of augmentations.
  • the training activities are codified by training elements and goals. These are represented as projected changes in skill elements. Overall changes in skill are measured as incremental changes in skill status AS k .
  • the unit of time for the TL is the set, denoted by the subscript n. Changes in training elements during a session are measured as changes toward training goals As n .
  • the Feedback Augmentation Loop encompasses the feedback during movement performance, including the various forms of cueing, and those mediated by the apparatus to support the active training goals.
  • the FL focuses on the interactions that take place during performance and directly impacts movement behavior such as provided by real-time cueing. Users dispose of a range of instructions and feedback modalities to augment their training or playing experience including instructions, feedback cues, and/or apparatus interactions.
  • Typical training or play sessions can be described as periods of performance interrupted by pauses (see FIG. 58). Pauses subdivide the session into sets. Usually, users start a session by planning their activity and setting active training goals. Not all sessions are explicitly structured or planned. Even if this is the case, users can improvise and at any time enable various augmentations and access skill analysis and training
  • the main user interaction application supports browsing functions to: review past and current data, view existing movement skill status, select active training elements, view details of training goals, and enable augmentation profiles.
  • a training epoch k is a time scale ranging from one to several sessions.
  • the set n associated with the training loop is a time scale ranging from a few to many occurrences of a motion pattern set, i.e., provide a unit of time to organize sessions.
  • a set n can have one or more active training elements.
  • the time t corresponds to the actual time and is typically associated with the augmentation loop (real-time feedback from the cueing system or the apparatus based on measurements y .
  • the motion measurement data y is processed to determine the movement state data x.
  • the data can also include other behavioral data (e.g., visual gaze) and task specific data (e.g., movement and location of task elements and objects, and various types of outcomes).
  • Raw measurements are often extended through an estimation process to determine relevant state information based on available measurement data y.
  • the motion data is processed to extract the primary movement units associated with the actions performed in a task or activity (described elsewhere).
  • This process can be formalized based on human movement system theory or principles.
  • the movement repertoire R k can be obtained through classification of the ensemble of movement units into a collection of movement patterns ⁇ Pi ⁇ , which can be divided into classes (FIG. 12).
  • the movement patterns result from sensory-motor schemas or programs (described elsewhere). Through motion modeling, these movement patterns can be described by a sequence of movement phases that are related to the functional characteristics, including muscle synergies, biomechanical constraints, perceptual mechanisms, and task constraints.
  • HMM hidden Markov model
  • the elements of the motion model M k provide the basis for skill assessment and diagnosis to extract skill attributes
  • the skill status S k contains a collection of sorted skill elements.
  • the skill elements are sorted based on their acquisition stage.
  • the collections correspond to the three acquisition stages: formation, consolidation, and optimization:
  • S k — S k form U S k con U S k opt, [4] where e.g. S k form is the subset of skill elements that contain motion patterns satisfying the criteria discussed earlier for the formation stage.
  • Skill profile p S kiii(S k ) describes how different skill elements combine to create the subject's overall performance. This information can for example be determined by adding up the composite scores for each skill element across the repertoire:
  • the primary purpose of the training list is to designate which skill elements to focus on, and also to configure the augmentation system.
  • the active training list describes an order of importance, with the top-listed training goal representing the most significant training goal. These elements have priority on using system resources such as notifications or real-time feedback.
  • skill elements are organized hierarchically to describe their acquisition stages, which reflects their relative importance to the activity performance.
  • the active training list can be generated automatically from the skill status taking into account the relevance of skill elements, or selected by the user taking into account information such as their preference, available time, and conditions.
  • Training goals can be explicitly pursued, e.g., during a dedicated training set.
  • performance related to the training goals can be tracked during the "free" performance of the activity.
  • Relevant information about these goals can be used to notify the subject. Such notifications can, for example, highlight when significant progress toward a goal has been achieved.
  • the longitudinal analysis coupled with the population data provides both the microscopic and macroscopic information to support training planning.
  • the population sub-group and its association with the subjects' individual characteristics provide the information needed to manage the skill development: at the microscopic level, by providing references of realistic and preferred skill and performance characteristics relative to the group at a given level; and at the macroscopic level, by providing directions on movement architecture, and other attributes such as movement functional characteristics, to adopt for efficient and safe performance.
  • Another population analysis is performer profiling. Specific characteristics of a subject groups can be captured by their skill profiles, which can be described by composite metrics that emphasize different attributes. These profiles make it possible for the assessment and diagnostics that drive a subject's behavior in the direction of that subgroups' style.
  • the motion model, skill status, skill elements, and training elements provide the quantities needed to implement training as a data-driven, iterative process.
  • the training goals in the active training list are tracked to provide progress reviews or notifications.
  • the motion model and skill status can be re-assessed, leading to an update in skill status.
  • the user may continue with the remaining elements in the training list or re-assess which aspects of skill to emphasize.
  • One disclosed capability is the management of comprehensive information relevant to a user's movement performance and its application to drive and manage training.
  • the disclosure also addresses the problem of how this information is communicated to the performer.
  • the system can support several modes of interactions. These modes distinguish themselves by the levels of augmentation (types of feedback) and how the training elements are used to direct training.
  • training agent selects the training elements and provides a training plan that specifies which training elements are exercised and when to switch training elements. This mode also includes drills.
  • Training agent selects the training elements and user determines the order of training elements to exercise and when to switch training elements.
  • Interactive, augmented play training elements are selected by user, user determines the order of training elements to exercise and when to switch training elements. In augmented play, training elements can be integrated within regular playing sessions. Tracking takes place in the background and the training agent provides notifications on various milestones for the selected elements.
  • Free augmented play users can take advantage of feedback
  • the technology can also be used by a coach as a tool or training assistant.
  • the coach will essentially become an element in the feedback training loop.
  • the "augmented" coach within this system can play several functions, including interpreting the results of the skill assessment, planning the session, and providing verbal and other instructions such as demonstrating the movement.
  • FIG. 49 shows the top- level logic diagram for the overall system and its primary processes, depicted in FIG. 21.
  • the main blocks in the diagram are as follows.
  • Data Acquisition 110 represents the process of capturing performance data, which includes movement measurements from the activity, and other relevant activity data.
  • the movement measurement covers the motion of the agent and his or her segments.
  • the activity data covers the quantities useful to assess outcomes related to the performance and goals of the task or activity.
  • Modeling 120 represents the processes of modeling the subject or agent's movement and the relevant interactions with the task and environment elements. It includes extracting and processing primary movement units (PMUs), which are subsequently decomposed into movement components (described earlier), such as movement phases and muscle synergies that are relevant to the functional understanding of the movement patterns.
  • PMUs primary movement units
  • movement components such as movement phases and muscle synergies that are relevant to the functional understanding of the movement patterns.
  • Skill Assessment and Diagnosis 130 represents the processes used to determine the parameters that characterize a subject's skill elements, which subsequently determine a subject's skill profile and skill status. These processes can also take population data as inputs, denoted here as reference data. This additional data enables the determination of the player' s profile.
  • Training Goal and Feedback Synthesis 140 represents the processes involved in designing the various feedback augmentations (instruction and notification, cueing laws for real-time feedback, interaction laws for the apparatus).
  • Planning 150 represents the process of selecting the training elements, as well as planning and possibly scheduling of the training goal sequence, that will be used to manage the training or activity session.
  • Activity Management 160 represents the actual performance of the activity, including processes through which the feedback acts on the users during the various interactions. It also includes the process of tracking the training, managing the session, and configuring the overall system. The configuration determines the feedback profiles and how the training goals are tracked during the performance.
  • Data Acquisition 110 is the collection of all relevant performance data from a given performance. It is achieved through a variety of motion capture technologies, including IMUs that are worn by the subject or embedded into equipment or clothing, and optical or vision-based motion tracking systems. Data Acquisition also encompasses measurement of other activity or task relevant quantities such as outcomes and performers' behavioral data (e.g., visual gaze data). In addition, estimation techniques can be applied to estimate unmeasured quantities from the available measurements.
  • the measurements y encompass all relevant data for the desired level of analysis. They can include other behavioral measurements, such as gaze or muscle activity, as well as contextual data (information about set, session, player/subject, task, and environment conditions, etc.). Some quantities can be estimated. In the following, the notation, y encompasses any type of data, measured or estimated.
  • Movement modeling 120 uses the captured performance data and possibly previous movement models 260 to form the subject's up-to-date movement model. Movement modeling is an ongoing process that evolves in parallel with the skill acquisition. Therefore, it typically accounts for previous model information, in an iterative process.
  • the modeling process described in FIG. 50 includes the following steps:
  • the first step in movement modeling is the identification and extraction 210 of the movement patterns associated with the primary movement units (PMU).
  • PMUs are described in terms of movement unit profiles, which can be represented as time histories or trajectories in the state space. These profiles are then classified 220 to determine the membership information needed to determine the movement repertoire.
  • the measured movement data is segmented to extract the set of primary Motion Units ⁇ s ⁇ for the activity and their associated actions, events, and outcomes.
  • the repertoire is obtained through classification of the set of primary movement units ⁇ over the period of activity, for example the set of movement units for an epoch k.
  • the classified movement units can then be further analyzed to determine additional information relevant to the selected level of analysis.
  • PMUs can be further segmented into movement phases 230 associated with the muscle synergies, or other forms of segments relevant to the execution and functional analysis of movement.
  • the logic to select the level of analysis is shown in the inset in FIG. 50.
  • the phase segmentation 230 generates finite-state motion models, used in the functional analyses of movement, as well as a finite-state estimator, which is used in the cueing system.
  • the synergy decomposition 240 generates muscle synergies that can be used for physical and musculoskeletal analysis.
  • the result of the motion modeling is a set of motion models
  • a motion model can be a finite- state statistical model (HMM, etc.) such as:
  • the synergy decomposition 240 uses the movement phase profiles to determine components of muscle activation patterns that are combined to produce the resulting movement throughout a phase.
  • adequate determination of movement synergies requires measurement or estimation of the individual body segments movement, and possibly other relevant quantities, including physiological quantities such as electro-myographic measurements of electrical muscle activation.
  • Modeling the relationship between movement components and the musculoskeletal system provides information that can be used to estimate the biomechanical load and in turn help prevent excessive wear and injury.
  • Skill assessment and diagnostics 130 builds on the elements of the motion model (repertoire, movement phases, etc.) and the skill and performance attributes that can be generated through various metrics.
  • Movement pattern classes Pi and associated motion models 5i provide the structure to perform skill modeling, various forms of assessments, and diagnostics. The assessment is primarily a descriptive process of various skill characteristics relevant to the movement activity. As shown in FIG. 52A, the overall movement skill assessment metrics encompasses several levels: physical performance 312, pattern performance 313, task performance 314, and competitive performance 315. Each level, if selected 311, will be assessed across several components described earlier: outcomes, functional, perceptual and memory and learning (see FIG. 10).
  • Skill modeling uses the attributes generated in the assessment process and integrates them 316 to enable movement diagnosis.
  • the assessment step includes determining relevant quantities from the movement data, elements of the movement model, and movement activity performance. Reference values 317 from population analysis or individuals can also be incorporated in the assessment of the skill elements.
  • the diagnostic step includes interpreting these quantities to identify which aspects of the movement technique or other physical attributes need to be changed to improve the outcomes and other behavioral characteristics critical for movement activity performance. This process is achieved by determining the relationship between outcomes and the various skill attributes.
  • the movement functional analysis plays a critical role in movement technique diagnostics since it describes the mechanics of how movement accomplishes its outcomes.
  • the assessment evaluates skill in terms of the physical effort required to achieve the outcomes and in terms of characteristics associated with the biomechanical constraints, such as the strain on the muscles or torques and forces on the skeleton, ligaments and joints.
  • the movement physical performance assessment is based on metrics such as energy or jerk. These quantities can then be related to the outcomes, or used to determine movement efficiency.
  • This assessment level also evaluates the relationship between movement patterns, specific movement phases, and wear and strain on the associated musculoskeletal structures. The features extracted from this assessment can then be used to generate feedback to help modify aspects of the associated movement execution and thereby help mitigate injury.
  • the assessment evaluates movement technique, as well as all other supporting functions, such as perception, that a subject uses to achieve outcomes under changing and uncertain conditions.
  • the pattern performance assessment provides critical information for the movement diagnostics.
  • Basic movement skill assessment includes the analysis of movement technique by extracting attributes of the movement trajectories within a given movement class.
  • Typical movement skill attributes include:
  • Consistency The movement profiles in a class represent general motor programs (described elsewhere). Therefore, proficient subjects are expected to display consistent trajectory characteristics within a class.
  • Timing The successful execution of movements and their associated outcomes depends on accurate spatial and temporal coordination. Critical timing characteristics can be evaluated and used as skill metrics.
  • More advanced movement skill assessment builds on the movement structure (e.g., phase decomposition) and is based on derivatives computed using sensitivity analysis.
  • the primary functional metrics are derivatives that capture how different features describing the movement technique, participate in outcomes and adaptation to task conditions.
  • assessments evaluate a subject's skills in terms of the range of movement patterns in the repertoire developed to tackle the movement activity requirements and handle the range of conditions prevailing during performance.
  • task and outcome metrics mi f(Pi , 5i) which typically represent descriptive quantities determined from the movement model and outcome measurements and/or estimations. They can include: success rate, movement outcome/result, variability, precision, as well as statistical characteristics, for a specific session and/or relative to historical data. COMPETITIVE PERFORMANCE ASSESSMENT
  • assessments evaluate a subject's skills in terms of overall strategies developed to tackle the task and handle the range of conditions prevailing during performance.
  • the motion patterns can be used as the state of the agent to describe the agent- environment interaction dynamics at the task and competitive level.
  • the function ⁇ can capture a player's strategy, as well as their ability to perceive the game status and opponent actions and intention. Therefore, the function ⁇ contains information that can be used to asses a player' s competitive performance.
  • Such an HMM model can be extended to include any relevant state information such as the position of the subjects or the position of the ball.
  • the Skill Attributes for a particular movement pattern provide information for the overall assessment of the movement skill, performance, as well as other relevant considerations such as injury risk and the learning process itself.
  • a Skill Element provides the formal definition of the concept of a unit of skills.
  • the cost function Q can be defined as the weighted sum of attributes, with the weights indicating the relative importance of each attribute:
  • a critical aspect of skill assessment is the acquisition stage (e.g., formation, consolidation, optimization). This information is described by the concept of skill status (FIG. 52B), which provides information about the acquisition stage of each skill element. This information is useful for the determination of training or rehabilitation intervention.
  • the skill status can be determined by applying criteria derived from a variety of skill attributes a; and their associated statistics 321. TABLES 1 and 2 describes examples of criteria that can be used to determine the acquisition stage of movement patterns from the repertoire.
  • the Skill Status S k for an epoch k can be represented as a partition on the set of skill elements that covers the movement repertoire:
  • the repertoire combines the collection of movement patterns that have been acquired by an individual to deal with the task requirements and environment conditions.
  • the motion model encompasses the movement repertoire, its movement classes, and movement phases and synergies.
  • the extracted attributes from the various skill metrics provides additional information to determine other quantities relevant to learning and training.
  • the skill elements e; combined with the skill status provide comprehensive information about the subject's movement technique and performance. This information can be used to generate a so-called skill profile 330, which describes the overall skill and performance of a subject.
  • the skill profile can be illustrated graphically for example by displaying the skill composite of each skill elements (see FIG. 17).
  • One potential output of this assessment process is to generate a list of the movement patterns sorted by skill level based on composite score and development stage 326. This list provides the basis to define the training elements.
  • Movement classes can be arranged relative to physical and biomechanical criteria. Typically, skill and physical attributes evolve in parallel during learning;
  • subjects can adopt techniques that may be effective in achieving outcomes but detrimental to their musculoskeletal health.
  • Possible physical development stages include, "physical build up,” i.e., patterns where the technique is affected primarily by a lack of adequate strength, "endurance,” i.e., patterns that exhibit premature wear, and "excessive load,” i.e., patterns that are executed with a level of force that produces excessive wear and strain on the body.
  • This information can for example be used to determine an injury index for each skill element. This index can then be added across the repertoire to determine the injury profile.
  • the skill profile can be tuned to particular task requirements or styles of performance. For example, some of the strokes and outcomes are more fundamental to player performance. These can be characterized as core strokes. Different tiers of skill elements or strokes can be defined, and the skill profile therefore can be decomposed into different profile components to capture different characteristics.
  • the relative weights assigned to the skill elements in the skill profile enable characterizations of performer or player types, which can be used to define a player profile 340. For example, in tennis, strokes that are used in defensive play as opposed to offensive play provide the information to characterize the player type. This information is further developed under population analysis.
  • Reference values can be used to provide absolute references, for example to measure how the various extracted attributes compare to a representative group of players. For example, in tennis, this allows a subject to understand if their topspin amount produced for a particular stroke class is high or low.
  • Reference values can be determined by extracting statistical distributions across attributes for a group of subjects with similar movement technique. The statistical data can then be used to generate percentile ranks for any relevant attribute, and using those for example to discretize the reference ranges into tiers (such as low, medium, high, very high).
  • Population information can be used to determine leaderboards that can be helpful for a coach or a physical therapist. It can also help motivate subjects to understand how they compare with other individuals, e.g., in an absolute ranking, as well as to understand the specific aspect of their skills or performance that is responsible for their ranking and which aspects of their movement technique or performance is the most actionable to help them progress within their group.
  • Training Goals and Feedback Synthesis 150 represents the determination and specification of training goals, discussed earlier, and associated augmentations that can be used to drive training. These are selected across the different feedback modalities.
  • the specification of a training goal 410 can be viewed as a dual problem to the determination of an augmentation that will help drive the training process toward the goal. Ideally, goal and feedback synthesis is performed simultaneously.
  • the synthesized feedback determines the "augmentation space" available to skill goals (FIG. 21). These augmentations define the scope of user interactions within which the user can then choose to operate.
  • FIG. 23 describes how the augmentation environment is enabled and operated during
  • the acquisition stage in the skill status computation provides information that allows to determine the appropriate feedback forms. For example, the formation of new patterns requires different augmentations than the refinement or optimization of an existing movement pattern. In general, several feedback modalities can be combined (e.g., instructions, feedback cues, and apparatus interactions). The feedback
  • configuration 426 describes how feedback modalities are combined to produce the user augmentation.
  • Training goals help make training actionable, and enable subjects to focus their attention during performance.
  • the quantitative specification of the training goal also means that it can be measured or estimated, which allows to objectively track the training progress for the particular skill element.
  • Training goals can be derived from the statistical analyses of a subject's skill at the various assessment levels such as task performance level, for example, based on the attributes within a skill composite score, taking into account population reference data (see FIG. 19). Or, for example at the physical performance level based on the functional analysis (see FIG. 20 and FIG. 37). The training goal can for example be derived based on an increment (or a fraction increment) in a percentile tier for a skill level or outcome level, respectively.
  • a training goal at the performance level can be determined from the global score ranking shown in FIG. 34. One can proceed, for example, by identifying the skill element in the repertoire that has the largest impact on ranking (critical skill element). And from there determine the skill attribute in the skill profile composite (FIG.
  • attribute ai Since skill deficiencies often manifest across multiple attributes, one or more component, or even some combination of components of attribute ai, can be selected as the critical attribute to drive a particular goal gi. Furthermore, the attributes can require targeted movement technique optimization. Therefore, the attribute goal can be combined with the functional analysis.
  • Dimensionality reduction or embedding techniques can be used to determine the functional relationship. This level of analysis is typically conducted during functional movement modeling.
  • Training Element (ei, gi,t > ) describes a Skill Element e; combined with a Training Goal gi,t>.
  • the specific goals include the spatial definition of the movement configuration. This corresponds to the cognitive stage of skill acquisition where the subject forms an understanding and representation of the movement primarily focusing on its spatial configuration.
  • the knowledge for example includes the movement phases, including the configuration of the body segments, and end effector and equipment (system state), at phase transitions. Also relevant at the formation stage is the understanding of the relationship between the movement phases and their synchronization with the environment and task elements and objects.
  • the specific goals include the refinement of the movement patterns and associated functions to achieve best outcomes within the subject's bio-mechanical constraints.
  • the quantities that are optimized include the functional characteristics (features associated with movement outcome) and physical performance characteristics (musculoskeletal loads).
  • the subject can form mental representations that enable them to focus on features in the technique that influence the outcome, or gain an understanding of which elements of the task convey information that helps with the movement modulation or timing.
  • the feedback laws are synthesized 420 using the training elements (combining skill elements and training goals), including information from skill profile and status (FIG. 54B).
  • the terminology of feedback is used in the larger sense, with the following two primary feedback types: instructions 421, and feedback cues 422.
  • an apparatus 423 see e.g., ball machine in FIG. 2 can be used to provide additional interactions for movement performance and training (see later discussion, see FIG. 23).
  • Instructions are synthesized from the skill model parameters and assessment 424, in particular the skill profile and player profiles.
  • the communication modalities include visual 431, verbal 432, and text 433.
  • Instructions 434 represent feedback that operate at the "knowledge" level. They include aspects such as descriptions of the training elements for the next training set, or details about the movement features that will be augmented through feedback cues. Instructions can also include visual descriptions or simulations of the spatial configuration of formation patterns.
  • Cueing mechanisms 439 are synthesized from the motion model and in particular the functional movement model 425.
  • the cueing mechanisms include validation cues 435, outcome optimization cues 436, alerts 437, and pattern formation cues 438. These cues are used as feedback augmentations.
  • the cueing laws for the real-time feedback cues are determined from the functional movement modeling and analysis.
  • the instructions, cues, and apparatus can be combined to create different augmentation profiles that lead to different interaction forms.
  • the synthesized instructions and cueing mechanisms are first integrated to determine best combinations. The goal is to combine these feedback modes to achieve synergy.
  • the settings and parameters define the available feedback configurations 426 (FIG. 54B). These combinations are then used to determine configuration parameters for the
  • Augmentations can operate at the symbolic/cognitive level, cue level, and signal level.
  • feedback is in the form of instructions prior to performance, reports following the performance, and notifications during the
  • Instructions can be used to help subjects form mental representations of the movement pattern focusing on aspects that are relevant to current training elements.
  • Notifications can be used to provide feedback on training progress, e.g., on a specific training goal. Reports provide a summative overview of the subject's skills and training activity. The generation of textual and other symbolic or graphical information is performed via a communication system with an instruction generator such as an expert system. Notification can be implemented in the form of a state machine, or even using a conversational agent which can output either text or natural language.
  • Feedback cues target the movement characteristics associated with the training goal (through outcome validation, feature validation, etc.), as well as associated sensory and perceptual processes. Cueing and signal level feedback can operate as reinforcement or deterrents.
  • the cueing system can also provide visual cues to help form visual attention needed to support a particular interaction for the task or activity.
  • the cueing system combines a cueing law Ki specific to the training element ⁇ that computes cueing signals and a cue generator that translates and encodes these signals into understandable signal forms (audible, visual, haptic, etc.).
  • the cueing laws are implemented for example by a state machine which uses the movement measurement data y t , states x t , and/or movement features fi to compute the cue signal.
  • the cue and signal level also encompass the interaction laws for a possible apparatus.
  • the primary role of the apparatus interactions is to expand the operating range, for example to help form new patterns.
  • the apparatus can also provide physical interactions such as those provided by an assistive robotic device or exoskeleton.
  • the apparatus action is driven, similarly to the feedback cueing, by a feedback law and/or program.
  • Training planning addresses the question of which aspects of movement performance are to be emphasized during training.
  • the plan or schedule describes the organization of a session in terms of the training elements and associated training goals.
  • the plan also provides the structure to schedule and manage the session during the performance of the activity 160 (see FIG. 56).
  • Planning typically takes into account the overall training goals, available time, and other resources.
  • the prioritization can be determined from the stage of skill acquisition of the skill elements and for the significance of the skill elements to the task objective.
  • the training elements can be prioritized based on the skill status (S k ) of each skill element 415.
  • training elements can be selected. These selected training elements produce a so-called training list.
  • the active training element it indicates to the augmentation and tracking system which aspects of movement performance have to be monitored and actively cued.
  • the training process is formalized as an iterative learning process.
  • This model makes it possible to determine how the data is managed.
  • epochs can be defined to coincide with major developmental changes during a training cycle over which a new skill level is reached with significant changes in attributes to result in a new profile.
  • This epoch has an associated data set with motion model, skill model and various skill attributes, and statistical descriptions.
  • For each epoch there are associated training elements and goals that when completed will lead to a new skill level.
  • the delineation between epochs can be arbitrary. More objective criteria can be used to determine training epochs, for example, the validity of the motion model used for motion pattern classification.
  • the training system can prompt the user and the assessment cycle can be re-initialized, which provides a new baseline for training.
  • the motion model enables the analysis of the skill acquisition process for an individual and also across the larger population. Therefore, patterns in skill acquisition can also be used to manage the training process and determine the larger-scale training goals.
  • the training list provides a way to emphasize a group of training elements.
  • the goals at the top of the list have the highest priority.
  • Training priority can be determined from the skill status parameters and criteria (see TABLE 2), the development stage, and information about the relevance of particular movement patterns (skill elements) and associated outcome for the task (see FIG. 13).
  • the designation of the priority of a training element in the training list can be performed either manually by the user, or automatically based on assignment of the skill elements (see primary, secondary, tertiary in FIG. 13).
  • Each training element may include a stopping criterion to signal when to transition to the next training goal. Stopping criteria could be the number of movements to repeat in that particular class, performance over a time duration, given progress toward the goal (given fraction), or the accomplishment of the goal, which can be determined statistically such as in clinical significance tests.
  • the ensemble of training elements and goals can be used to systematically plan and manage training or playing sessions. For example, a training schedule composed of sets can be generated before the session (see FIG. 45B). Each set can emphasize one or more training goals.
  • a training or play session is typically divided into time periods. These periods are designated as sets. Each set can have one or more training goals. These elements are arranged across several sets to form a training schedule. This structure makes it possible to decompose longer-term training goals into intermediate goals. [0810] The implementation of the training process takes place through the augmented human system (see FIGS. 22 and 23).
  • Different feedback modalities call for different frequencies of user interactions. Instructions for example are presented intermittently typically following the selection of a training element. Real-time feedback cues, on the other hand, are applied concurrently with the movement performance. Real-time feedback can also be communicated continuously or at discrete time periods during the execution, or just following the movement outcome.
  • an apparatus is used as part of the performance.
  • a typical apparatus in tennis is the ball machine.
  • the apparatus can be programmed to work conjointly with the feedback cues and instructions.
  • the main parameters for the augmentation system configuration 620 are designating the targets (e.g., subject, coach, etc.), and specifying the type of instructions (e.g., verbal, audio, etc.) and the type of feedback cues.
  • the primary systems that mediate interactions are shown in FIG. 22. They include the communication systems (e.g., tablet or smart watch), the cueing systems (e.g., wearable device), and the apparatus system (e.g., ball machine).
  • different targets can be selected based on the training format. For example, in one scenario, a coach interprets and communicates the instructions to the subject. In this case, the coach would receive information about the subject's performance during a particular set and use this information to coach the subject before the next set. In another scenario, the subject uses instructions to assess the progress on a given training element.
  • the feedback forms under instructions include visual, verbal, or text. These forms provide different modes of interactions. For example, they can invite the user to browse the movement repertoire. Or, they can invite the user to learn about technique for a particular movement pattern.
  • a typical scenario includes the refinement or optimization of a movement pattern.
  • the cue profile combines phase transition cues with outcome validation cues.
  • the subject could use cueing during the performance to assist the formation of a new movement pattern or to optimize an existing pattern.
  • the subject can start with the activity performance 630. During performance, movement and system behaviors are monitored 640 and data acquisition continues. However, the emphasis of the assessment is characterizing the movement skills with the augmentation and the performance relative to the training goals. The activity can be paused at any instant 690.
  • Planning may take place ahead of the session or proceed incrementally.
  • Initial training elements and schedule are defined based on current skill status.
  • the training goals and elements for the subsequent set are defined as a function of the subject's completion of the training goals and overall performance, as well as other factors such as wear, fatigue, or motivation.
  • the training system enables interactive management during the performance of the activity.
  • Managing the training activity is an interactive process.
  • the management of the session 610 includes specifying which training goals are pursued at a given time period in the activity and updating the configuration of the augmentation system (instructions, feedback cues, apparatus mode of action).
  • the training goals are typically provided as part of a training schedule specifying training elements and associated goals.
  • the goals are pursued through the interaction of the subjects with the augmentations.
  • training goals provide a quantitative description of the change in a training element and can take into account the augmentation profile available for the element.
  • the augmentation systems are configured based on the goals of the next period of activity.
  • the system configuration 620 (FIG. 57A) determines how the different feedback modalities 621, 622, 623 are combined to create performance interactions that are most effective for the pursuit of the training goals.
  • FIG. 58 describes a session temporal structure delineating the different periods of activity, shown as sets #1 to #4. A set can be followed by a pause in movement activity. During activity periods, the performer receives cues, and or notifications.
  • the performer can review the performance data, and if needed modify goals and system configurations.
  • the progress towards the training goals can be tracked during the training activity 640.
  • the change in training elements provides the basis to provide feedback on progress.
  • the monitoring system 640 (FIG. 57B) provides notifications 644 to the performer (or coach).
  • Notification criteria 643 can be used such as the number of repetitions of a training element 641, the achievement of a certain fraction of the training goal, or the time elapsed.
  • Notifications 644 indicate if a training goal has been achieved 642, which can be determined using a form of clinical significance test. The significance test determines when the subject's technique has progressed sufficiently for the skill attribute to have stabilized near the target level.
  • a subject's movement skill profile and skill status can be assessed at various time intervals to accommodate for the different rates at which various aspects of movement skills evolve. Therefore, the assessment loop is closed (updated) at different rates for different system configurations and different assessment levels.
  • the changes that result from these inputs first need to be assimilated.
  • the movement repertoire does not change rapidly since it requires consolidation of movement into procedural memory. Therefore, the assessment at the task performance (repertoire) level is typically made at the interval of the epoch, spanning sessions to days or months. Epochs may be linked to changes in a subject's movement repertoire, but as described earlier the associated time periods are defined based on the creation and maintenance of sets of movement data and models (see FIG. 25).
  • the notification 644 can be issued using a range of communication devices and signals.
  • the subject can be prompted 645 through an audible signal and a message can be displayed on a smart watch.
  • notifications can be translated by a natural language processor and via voice communication.
  • the message can indicate the progress toward a specific training goal, or attainment of a particular outcome threshold.
  • the system can also prompt the user for an input 645. For example, this allows the performer to make a note or comment 646, or to simply tag a particular movement pattern, for example to indicate an issue or an outstanding result.
  • the user can also prompt the system, e.g., via a smartwatch to tag an event.
  • ACTIVITY INTERRUPTIONS [0826] Depending on the notification and the status of the training or activity, performance may be paused 690. Interruption in the activity can be prompted by the subject, the systems, or the coach. Typical scenarios include the following:
  • the cueing system detects a decrease in
  • the cueing agent notices that the movement performance has achieved the target level of the training goal.
  • the user receives instruction to pause to select the next training goal(s).
  • the coach monitors the performance via the communication system and decides to interrupt the performance to change the configuration of the augmentation system.
  • the system detects changes in the outcomes or attributes that may be due to the onset of fatigue or wear, or even injury.
  • the user can receive instructions to pause, for example through a smartwatch, and subsequently pauses the session.
  • the activity can be resumed immediately 690, or suspended for a longer period to allow for data review and changes in plans and configuration.
  • the performance data can be reviewed in greater detail 650, and then depending on the required attention, the performance is resumed or the session can be suspended.
  • the augmentation profile 670 and training goals 680 can be updated.
  • the change in performance associated with an active training element may require updating the training priorities within the existing skills status and thus can prompt a review of the training goals 680 in the training list.
  • Large changes in skill status may require an update of the motion and skill models (iteration of the assessment loop leading to a new skill status S k+1 )-
  • the review is typically mediated by the communication system, i.e., tablet or smart phone.
  • the purpose of the review is for the user or coach to go over the progress for the current training or to address issues that have been brought up by the training agent.
  • the user has two options: to end the activity or to resume it 660. If the user decides to end the activity, it closes the session. If the user resumes the activity, it can be done under the same training list and augmentation profile 670, or a new augmentation profile can be selected which leads to the system re-configuration 620. Alternatively, a new activity or training plan can be selected before resuming performance 610.
  • the activity review provides details on the cause of the interruption. The user will then typically be prompted to return to the system configuration 620 or activity planning 610.
  • the systems may use movement skill assessment and diagnostics at distinct levels of the human movement system hierarchy to specify training goals.
  • the systems may provide different forms of augmentations synthesized to help pursue the training goals.
  • the system may also include a system to track and/or manage the learning process.

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Business, Economics & Management (AREA)
  • Physical Education & Sports Medicine (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Educational Technology (AREA)
  • Educational Administration (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • Medical Informatics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Epidemiology (AREA)
  • Public Health (AREA)
  • Primary Health Care (AREA)
  • Biophysics (AREA)
  • Databases & Information Systems (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • Electrically Operated Instructional Devices (AREA)

Abstract

L'invention concerne un système de formation d'habileté de mouvement pilotée par des données. Le système fait appel à une évaluation et à des diagnostics d'habileté de mouvement à des niveaux distincts de la hiérarchie de système de mouvement humain pour spécifier des objectifs de formation d'un utilisateur. Le système peut fournir diverses augmentations qui sont synthétisées pour aider l'utilisateur à suivre les objectifs de formation. Le système peut comprendre des caractéristiques pour suivre et/ou gérer des processus de formation ou d'apprentissage.
EP18829049.8A 2017-07-06 2018-07-06 Systèmes et procédés de formation d'habileté de mouvement pilotée par des données Pending EP3649633A4 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201762529412P 2017-07-06 2017-07-06
PCT/US2018/041118 WO2019010435A1 (fr) 2017-07-06 2018-07-06 Systèmes et procédés de formation d'habileté de mouvement pilotée par des données

Publications (2)

Publication Number Publication Date
EP3649633A1 true EP3649633A1 (fr) 2020-05-13
EP3649633A4 EP3649633A4 (fr) 2021-03-10

Family

ID=64904003

Family Applications (1)

Application Number Title Priority Date Filing Date
EP18829049.8A Pending EP3649633A4 (fr) 2017-07-06 2018-07-06 Systèmes et procédés de formation d'habileté de mouvement pilotée par des données

Country Status (4)

Country Link
US (1) US20190009133A1 (fr)
EP (1) EP3649633A4 (fr)
CN (1) CN110998696B (fr)
WO (1) WO2019010435A1 (fr)

Families Citing this family (73)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8944940B2 (en) 2011-08-29 2015-02-03 Icuemotion, Llc Racket sport inertial sensor motion tracking analysis
US11673042B2 (en) 2012-06-27 2023-06-13 Vincent John Macri Digital anatomical virtual extremities for pre-training physical movement
US10096265B2 (en) 2012-06-27 2018-10-09 Vincent Macri Methods and apparatuses for pre-action gaming
US11904101B2 (en) 2012-06-27 2024-02-20 Vincent John Macri Digital virtual limb and body interaction
EP2997511A1 (fr) 2013-05-17 2016-03-23 Vincent J. Macri Système et procédé pour l'exercice et le contrôle de la préparation des mouvements et des actions
US10111603B2 (en) 2014-01-13 2018-10-30 Vincent James Macri Apparatus, method and system for pre-action therapy
EP3194036A4 (fr) 2014-08-11 2017-12-27 Icuemotion LLC Système de codification et de repérage pour des interactions humaines au tennis et dans d'autres activités sportives et professionnelles
US10854104B2 (en) 2015-08-28 2020-12-01 Icuemotion Llc System for movement skill analysis and skill augmentation and cueing
US11030918B2 (en) * 2015-09-10 2021-06-08 Kinetic Telemetry, LLC Identification and analysis of movement using sensor devices
JP2017097214A (ja) * 2015-11-26 2017-06-01 ソニー株式会社 信号処理装置、信号処理方法及びコンピュータプログラム。
WO2018081795A1 (fr) 2016-10-31 2018-05-03 Zipline Medical, Inc. Systèmes et procédés de surveillance d'une thérapie physique du genou et d'autres articulations
CN109521927B (zh) 2017-09-20 2022-07-01 阿里巴巴集团控股有限公司 机器人互动方法和设备
US11288977B1 (en) * 2017-10-11 2022-03-29 Hrl Laboratories, Llc System and method for predicting performance to control interventions by assistive technologies
WO2019161857A1 (fr) * 2018-02-23 2019-08-29 Sens Innovation Aps Dispositif et procédé pour fournir un transfert de données sûr par l'intermédiaire d'un protocole de communication non sécurisé
US10963698B2 (en) * 2018-06-14 2021-03-30 Sony Corporation Tool handedness determination for surgical videos
GB2574074B (en) 2018-07-27 2020-05-20 Mclaren Applied Tech Ltd Time synchronisation
US11557215B2 (en) * 2018-08-07 2023-01-17 Physera, Inc. Classification of musculoskeletal form using machine learning model
US11227507B2 (en) * 2018-10-11 2022-01-18 International Business Machines Corporation Wearable technology employed in injury detection, prevention and skills training
US11164319B2 (en) 2018-12-20 2021-11-02 Smith & Nephew, Inc. Machine learning feature vector generator using depth image foreground attributes
RU193501U1 (ru) * 2019-02-05 2019-10-30 Раиса Юрьевна Будко Устройство замещения утраченных функций человека с использованием обратной связи
US20200302331A1 (en) * 2019-03-20 2020-09-24 International Business Machines Corporation Intelligent problem solving using visual input
US11482127B2 (en) * 2019-03-29 2022-10-25 Indiavidual Learning Pvt. Ltd. System and method for behavioral analysis and recommendations
JP7147696B2 (ja) * 2019-06-27 2022-10-05 トヨタ自動車株式会社 学習装置、リハビリ支援システム、方法、プログラム、及び学習済みモデル
US11314828B2 (en) * 2019-07-12 2022-04-26 Iq2 Holdings, Inc. Dynamic and personalized product placement in electronic files
CN110355761B (zh) * 2019-07-15 2022-06-14 武汉理工大学 一种基于关节刚度和肌肉疲劳的康复机器人控制方法
US20220184481A1 (en) * 2019-07-31 2022-06-16 Intel Corporation Game Status Detection and Trajectory Fusion
US20240206768A1 (en) * 2019-10-17 2024-06-27 Polyvalor, Limited Partnership Methods and systems for the acquisition of kinematic data for neuromotor assessment
GB2588236B (en) 2019-10-18 2024-03-20 Mclaren Applied Ltd Gyroscope bias estimation
KR102324690B1 (ko) * 2019-11-27 2021-11-11 숭실대학교 산학협력단 딥러닝을 적용한 탁구 로봇 시스템
EP4093522A4 (fr) * 2020-01-21 2024-05-22 Fine-Tuned Components Limited Procédé de réglage fin d'un putter et sa fabrication
US11182807B1 (en) * 2020-05-08 2021-11-23 International Business Machines Corporation Oligopoly detection
US11541265B2 (en) * 2020-05-13 2023-01-03 Samsung Electronics Co., Ltd. Wearable device and operation method thereof
US20210394020A1 (en) * 2020-06-17 2021-12-23 FitForm Technologies Inc. Tracking three-dimensional motion during an activity
US11343154B2 (en) 2020-06-30 2022-05-24 Eagle Technology, Llc Systems and method for providing an ontogenesis wisdom and action engine
US20220004957A1 (en) * 2020-07-01 2022-01-06 EDUCATION4SIGHT GmbH Systems and methods for providing knowledge bases of assessment items
CN111772640B (zh) * 2020-07-10 2023-09-29 深圳市丞辉威世智能科技有限公司 肢体运动训练指导方法、装置及存储介质
JP2022024766A (ja) * 2020-07-28 2022-02-09 トヨタ自動車株式会社 訓練システム、訓練方法およびプログラム
CN112229642B (zh) * 2020-08-06 2022-08-19 沈阳工业大学 一种基于工效学的乘用车驾驶动态舒适性测试分析方法
US11654566B2 (en) 2020-08-12 2023-05-23 General Electric Company Robotic activity decomposition
US11897134B2 (en) 2020-08-12 2024-02-13 General Electric Company Configuring a simulator for robotic machine learning
GB2597961A (en) * 2020-08-12 2022-02-16 Sony Europe Bv A method, device and computer software for determining a relative performance measure between at least two players in a sport
JP7435363B2 (ja) * 2020-08-26 2024-02-21 トヨタ自動車株式会社 訓練システム、訓練計画生成方法およびプログラム
CN114259720B (zh) * 2020-09-15 2022-10-18 荣耀终端有限公司 动作识别方法及装置、终端设备、运动监测系统
US11972330B2 (en) 2020-09-30 2024-04-30 International Business Machines Corporation Capturing and quantifying loop drive ball metrics
US11369844B2 (en) 2020-09-30 2022-06-28 International Business Machines Corporation Machine learning augmented loop drive training
CN214808291U (zh) * 2020-09-30 2021-11-23 深圳华锐互动科技有限公司 一种球类运动训练板组件
US11417136B2 (en) * 2020-09-30 2022-08-16 International Business Machines Corporation Capturing and quantifying body postures
US11301269B1 (en) 2020-10-14 2022-04-12 UiPath, Inc. Determining sequences of interactions, process extraction, and robot generation using artificial intelligence / machine learning models
CN116249574A (zh) * 2020-10-20 2023-06-09 株式会社爱世克私 运动分析装置、运动分析方法及运动分析程序
US11928607B2 (en) 2020-10-30 2024-03-12 AstrumU, Inc. Predictive learner recommendation platform
US12005573B2 (en) * 2020-12-06 2024-06-11 Cionic, Inc. Mobility based on machine-learned movement determination
US11957605B2 (en) 2020-12-06 2024-04-16 Cionic, Inc. Machine-learned movement determination based on intent identification
CN116830167A (zh) * 2020-12-21 2023-09-29 伊虎智动有限责任公司 用于开放式运动技能的评估和增强系统
WO2022145258A1 (fr) * 2020-12-28 2022-07-07 国立研究開発法人科学技術振興機構 Dispositif d'instruction de mouvement, système d'instruction de mouvement, procédé d'instruction de mouvement, et programme
CN112859598B (zh) * 2021-01-07 2022-08-19 河北工业大学 一种重组式经验变换型迭代学习控制方法
CN113189986B (zh) * 2021-04-16 2023-03-14 中国人民解放军国防科技大学 一种自主机器人的二阶段自适应行为规划方法及系统
CN113221708B (zh) * 2021-04-30 2023-11-10 北京科技大学 面部运动单元检测模型训练方法及装置
US20220374812A1 (en) * 2021-05-24 2022-11-24 Skillsacpe Analytics LLC Systems and methods for generation and traversal of a skill representation graph using machine learning
US11756687B2 (en) 2021-06-27 2023-09-12 Motionize Israel Ltd. Method and system for assessing performance
CN115731602A (zh) * 2021-08-24 2023-03-03 中国科学院深圳先进技术研究院 基于拓扑表征的人体活动识别方法、装置、设备、存储介质
EP4177868A1 (fr) * 2021-11-03 2023-05-10 Sony Group Corporation Rétroaction basée sur la performance pour une activité dans un environnement à faible gravité
WO2023112096A1 (fr) * 2021-12-13 2023-06-22 日本電信電話株式会社 Dispositif de soutien à l'entraînement, procédé de soutien à l'entraînement et programme
USD1006661S1 (en) 2021-12-22 2023-12-05 Icuemotion Llc Sensor cover
US20230218974A1 (en) * 2022-01-12 2023-07-13 Mangolytics, Inc. Portable drill instructor for a sport
US20230303084A1 (en) * 2022-03-23 2023-09-28 Robert Bosch Gmbh Systems and methods for multi-modal data augmentation for perception tasks in autonomous driving
WO2023183536A1 (fr) * 2022-03-23 2023-09-28 Xeed, Llc Communications en cascade de dispositifs de détection à une liaison
US20230401512A1 (en) * 2022-06-13 2023-12-14 At&T Intellectual Property I, L.P. Mitigating temporal generalization for a machine learning model
US11784833B1 (en) * 2022-07-25 2023-10-10 Gravystack, Inc. Apparatus and method for generating an endpoint path associated with a user
WO2024058848A1 (fr) 2022-09-15 2024-03-21 Arcutis Biotherapeutics, Inc. Compositions pharmaceutiques de roflumilast et de solvants capables de dissoudre des quantités élevées de médicament
CN115937895B (zh) * 2022-11-11 2023-09-19 南通大学 一种基于深度相机的速度与力量反馈系统
US11857862B1 (en) * 2023-01-11 2024-01-02 Lindsey Lombardi Thomas Method and system for assessing tennis stroke heaviness
CN116110584B (zh) * 2023-02-23 2023-09-22 江苏万顶惠康健康科技服务有限公司 一种人体健康风险评估预警系统
US11957969B1 (en) * 2023-08-02 2024-04-16 Gotta Sports Inc. System and method for match data analytics

Family Cites Families (27)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4508510A (en) * 1983-12-07 1985-04-02 Mona Clifford Method for psychomotor training of physical skills
EP1722872A1 (fr) * 2004-01-26 2006-11-22 Modelgolf Llc Systemes et procedes pour mesurer et evaluer les performances obtenues lors de l'accomplissement d'un exercice physique et celles de l'equipement utilise pour accomplir l'exercice
EP1846115A4 (fr) * 2005-01-26 2012-04-25 Bentley Kinetics Inc Procede et systeme destines a l'analyse et a l'enseignement de mouvement athletique
CN103028228B (zh) * 2006-09-07 2015-08-12 耐克创新有限合伙公司 运动表现感测系统和方法
ES2433437T3 (es) * 2006-12-19 2013-12-11 Deakin University Método y aparato para control háptico
US8358965B2 (en) * 2006-12-30 2013-01-22 Realtime Learning Systems, Llc Internet based learning systems
RU2364436C2 (ru) * 2007-06-18 2009-08-20 Максим Алексеевич Васин Способ тренировки и устройство для его осуществления
EP2260453A4 (fr) * 2008-02-14 2016-03-23 Infomotion Sports Technologies Inc Analyse électronique d'une performance athlétique
WO2010108128A2 (fr) * 2009-03-20 2010-09-23 The Johns Hopkins University Procédé et système de quantification de la compétence technique
US8540560B2 (en) * 2009-03-27 2013-09-24 Infomotion Sports Technologies, Inc. Monitoring of physical training events
US9626554B2 (en) * 2010-08-26 2017-04-18 Blast Motion Inc. Motion capture system that combines sensors with different measurement ranges
EP2817794A4 (fr) * 2012-02-22 2015-11-04 Jocelyn Faubert Procédé et système d'apprentissage perceptif, cognitif et moteur
US9082312B2 (en) * 2012-05-09 2015-07-14 Antennasys, Inc. Physical activity instructional apparatus
CA2880434A1 (fr) * 2012-07-30 2014-02-06 Treefrog Developments, Inc. Surveillance athletique
JP6439235B2 (ja) * 2012-08-10 2018-12-19 カシオ計算機株式会社 情報報知装置、情報報知方法及びプログラム
US9652992B2 (en) * 2012-10-09 2017-05-16 Kc Holdings I Personalized avatar responsive to user physical state and context
US11263919B2 (en) * 2013-03-15 2022-03-01 Nike, Inc. Feedback signals from image data of athletic performance
EP3003149A4 (fr) * 2013-06-03 2017-06-14 Kacyvenski, Isaiah Capteur de mouvement et analyse
US20150057111A1 (en) * 2013-08-20 2015-02-26 Quattriuum Inc. System, device and method for quantifying motion
US20150104768A1 (en) * 2013-10-10 2015-04-16 Modernstat Llc Trainable skill practice planning and reporting system
CN106102846A (zh) * 2014-03-19 2016-11-09 索尼公司 信息处理设备、信息处理方法及记录介质
US20160361593A1 (en) * 2015-06-10 2016-12-15 Wesley Charles Elliott System, apparatus and method for determining and analyzing sport related activities in conjunction with low latency transmission and processing
US20170225033A1 (en) * 2015-06-23 2017-08-10 Ipcomm Llc Method and Apparatus for Analysis of Gait and to Provide Haptic and Visual Corrective Feedback
US10854104B2 (en) * 2015-08-28 2020-12-01 Icuemotion Llc System for movement skill analysis and skill augmentation and cueing
AU2017230456B2 (en) * 2016-03-11 2019-09-26 Niklas KNAB Signaling device and apparatus
SK288743B6 (sk) * 2016-05-25 2020-04-02 Hdts A S Integrovaný viacúčelový hokejový trenažér a spôsob jeho ovládania/riadenia pre individuálny tréning a testovanie korčuliarskych a hokejových zručností
WO2017217050A1 (fr) * 2016-06-16 2017-12-21 ソニー株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations et support de stockage

Also Published As

Publication number Publication date
EP3649633A4 (fr) 2021-03-10
US20190009133A1 (en) 2019-01-10
CN110998696B (zh) 2023-01-10
CN110998696A (zh) 2020-04-10
WO2019010435A1 (fr) 2019-01-10

Similar Documents

Publication Publication Date Title
US20190009133A1 (en) Systems and methods for data-driven movement skill training
US11367364B2 (en) Systems and methods for movement skill analysis and skill augmentation
US20220198368A1 (en) Assessment and augmentation system for open motor skills
Land et al. From action representation to action execution: exploring the links between cognitive and biomechanical levels of motor control
Schack et al. Mental representation and the cognitive architecture of skilled action
Kimmel The micro-genesis of interpersonal synergy. Insights from improvised dance duets
Schack Mental representation in action: A cognitive architecture approach
Schack et al. Elements and construction of motor control
US20230285806A1 (en) Systems and methods for intelligent fitness solutions
Camarillo-Abad et al. An environment for motor skill transfer based on wearable haptic communication
Carson et al. Skill change in elite-level kickers: Interdisciplinary considerations of an applied framework
Glushkova et al. Interactive sensorimotor guidance for learning motor skills of a glass blower
Lewkowicz et al. Predictable real-time constraints reveal anticipatory strategies of coupled planning in a sequential pick and place task
Ramezanzade et al. The effect of sonification of visual modeling on relative timing: The role in motor learning
Tadayon A person-centric design framework for at-home motor learning in serious games
Lioulemes Adaptive user and haptic interfaces for smart assessment and training
Tits Expert gesture analysis through motion capture using statistical modeling and machine learning
Jensen Design Considerations and Application Examples for Embedded Classification Systems
Lin Changing Movement Patterns Using Reinforcement Learning
Carson et al. 15 Skill change in elite-level kickers
Spooner The science of volleyball practice development and drill design: from principles to application
Trajkova aiDance: A Non-Invasive Approach in Designing AI-Based Feedback for Ballet Assessment and Learning
Hu et al. Sensorimotor Control for Human Motion Generation—A Review
Verhoeven How individual differences in pre-dispositions of behavioral repertoires influence motor learning and performance
Gopher 17 Skill Training in Virtual Reality Environments

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20200123

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
A4 Supplementary search report drawn up and despatched

Effective date: 20210210

RIC1 Information provided on ipc code assigned before grant

Ipc: G09B 19/00 20060101AFI20210204BHEP

Ipc: A61B 5/11 20060101ALI20210204BHEP

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20220301

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230809