GB2608576A

GB2608576A - Exercise performance system

Info

Publication number: GB2608576A
Application number: GB2100202.7A
Authority: GB
Inventors: Emilian Teodorescu Viorel
Original assignee: Wizhero Ltd
Current assignee: Wizhero Ltd
Priority date: 2021-01-07
Filing date: 2021-01-07
Publication date: 2023-01-11
Also published as: GB202100202D0

Abstract

A method and system of monitoring exercise performance comprises monitoring a plurality of images 46 of a user performing a given exercise, generating a user model from the images 46 comprising nodes 50 representing skeletal joints of the user and angles subtended between the nodes. A predetermined model of performance of the given exercise comprising datum node positions and/or angles subtended there-between is accessed. The method determines whether the user model is within a given threshold of similarity to the predetermined model and records data for an image relating to the user performance if the user model for said image is within the given threshold of similarity of the predetermined model. One or more defect is assigned to the recorded data, the defect indicative of the sub-optimal performance of the given exercise. The defect may identify musculoskeletal defects and remedial exercises may be suggested to correct the defect. The exercise preferably includes a static pose.

Description

Exercise performance system The present disclosure relates to a system for monitoring and/or analysing the performance of a physical exercise.

Background of the invention

"Remote personal training" may be used to monitor the exercise of a user by an instructor using remote communication means. For example, a user may exercise in front of an internet connected webcam or the like. The instructor may view the user exercising on their own computing device in a remote location from the user. The instructor may provide feedback on the user's exercise technique in real time or at the end of a session.

The inventor has found numerous problems with the prior art arrangements.

Typically, such personal training is provided 1-on-1 (i.e. a single user per instructor). This is time consuming for the instructor, and thus is typically expensive for the user. The instructor must use their personal judgement to provide feedback. However, poor video quality or camera angles may render this process difficult as the instructor may not be able to accurately determine the user's body position etc. Alternatively, the instructor may view a number of different users simultaneously (i.e. in a class). Whilst this may be more cost effective for each user, it can be difficult for the instructor to provide adequate feedback, as they cannot watch each user simultaneously with sufficient attention.

Furthermore, it is time consuming and awkward to help correct individual users' technique over streamed sessions or video calls. Different aspects of the users' techniques may require different corrective steps and/or may have different underlying causes. The advice given to one user may be detrimental to another, if overheard.

It is an aim of the present invention to overcome or ameliorate one or more of the above problems.

Statement of Invention

According to a first aspect of the invention, there is provided: a method of monitoring exercise performance comprising: monitoring a plurality of images of a user performing a given exercise; generating a user model from one or more of the images comprising nodes representing skeletal joints of the user and angles subtended between the nodes; accessing a predetermined model of performance of the given exercise comprising datum node positions and/or angles subtended there-between; determining whether the user model is within a given threshold of similarity to the predetermined model; recording data for an image relating to the user performance if the user model for said image is within the given threshold of similarity of the predetermined model; and assigning one or more defect to the recorded data, the defect indicative of the sub-optimal performance of the given exercise.

The defect may relate to the position and/or orientation of the user's body or a portion thereof. The defect may be assigned automatically (e.g. by a computer).

The defect may be assigned according to a computational model (e.g. an Artificial Neural Network and/or Artificial Intelligence).

One or more root cause may be assigned to the defect, the root cause comprising one or more musculoskeletal defect at least partially contributing to the sub-optimal performance of the given exercise.

One or more root cause may be assigned a weighting indicative of the magnitude of the contribution to the defect.

One or more defect maybe assigned a weighting indicative of the magnitude of the contribution to the sub-optimal performance of the given exercise.

One or more remedial exercise may be associated with a root cause, the remedial exercise configured to at least partially remedy the root cause.

One or more remedial exercise may be assigned a weighting indicative of the magnitude of the contribution to the remediation of the root cause.

The predetermined model may be generated by captured data relating to a performance of the given exercise by the user and/or a further user (e.g. an instructor). The predetermined model may be generated by captured data relating to a performance of the given exercise comprising a defect. The defect may be performed intentionally. Performance of the given exercise may be pre-recorded (e.g. using previously captured images). The performance of the given exercise comprising a defect may be used to create a computational model (e.g. a training model).

The predetermined model may be variably determined according to one or more physical characteristic of the user.

The predetermined model may comprise a skeletal model of a datum or ideal performance of the given exercise, the skeletal model indictive of the position and/or orientation of the user's body or portions thereof.

The user model of the user performing the exercise may be constructed during the monitoring of the images, the recording data comprising automatically selecting one or more image closest to the predetermined model as being indicative of the user performance of the given exercise.

The predetermined model may be compared with the selected image of the user's exercise performance to determine the one or more defect.

Only select portion of the user's exercise performance may be stored and/or analysed when the performance is within the given threshold of similarity.

Only a most similar portion of the user's exercise performance may be stored/analysed when the performance is within the given threshold of similarity (i.e. only the user exercise performance closest to the predetermined model is assigned a defect).

A plurality of portions of the user's exercise performance may be captured and unselected images are discarded.

The method may comprise recording a plurality of respective defects/root causes for two or more exercises and aggregating the defects/root causes for a given user. The exercises may comprise different exercises. The exercises may comprise different sport or physical activity types. Aggregation of the defects/root causes may be provided automatically.

The method may comprise selecting one or more user performance according to whether the user performance is within a given threshold of similarity to the ideal performance and using the user performance to at least partially determine the ideal performance.

Assigning of the defect may be performed asynchronously from recording of the user performance.

Assigning of the defect may be performed manually or assigning of the defect is performed automatically but confirmed manually. Assignment of the defect may be provided during generation of the predetermined model. The defect may be assigned to predetermined model comprising an intentional defect.

Monitoring of the user performance may be performed in real time with the user performance.

The weightings of root causes or defects may be adjusted for a user based on combined weightings assigned for different exercises.

The given exercise comprises a static/still/fixed pose.

According to a second aspect of the invention, there is provided: a system for monitoring exercise performance comprising: a user device having a camera for capturing a plurality of images of a user performing a given exercise; a monitoring system comprising one or more processor arranged to generate a user model from one or more of the images comprising nodes representing skeletal joints of the user and angles subtended between the nodes; the monitoring system accessing from a data store a predetermined model of performance of the given exercise comprising datum node positions and/or angles subtended there-between; the monitoring system determining whether the user model is within a given threshold of similarity to the predetermined model; the monitoring system controlling recording data for an image relating to the user performance if the user model for said image is within the given threshold of similarity of the predetermined model; and the data store comprising defect data for the predetermined models and the monitoring system assigning one or more defect to the recorded data, the defect indicative of the sub-optimal performance of the given exercise.

According to a third aspect of the invention, there is provided: a computer processor comprising machine readable instructions for operating in accordance with the monitoring system.

According to a fourth aspect of the invention, there is provided: a data carrier or data storage medium comprising machine readable instructions for control of one or more processor to operate in accordance with the monitoring system.

According to a fifth aspect of the invention, there is provided: a method of monitoring exercise performance comprising: monitoring a one or more images of a user performing a given exercise; generating a user model from one or more of the images; accessing a predetermined model of performance of the given exercise comprising; determining whether the user model is within a given threshold of similarity to the predetermined model; recording data for an image relating to the user performance if the user model for said image is within the given threshold of similarity of the predetermined model; and assigning one or more defect to the recorded data, the defect indicative of the sub-optimal performance of the given exercise.

Figure 1 shows a schematic view of an exercise performance monitoring system; Figure 2 shows a schematic overview of an exercise performance monitoring process; Figure 3 shows a schematic view of initial training process of the system; Figure 4 shows a sport/exercise channel user interface; Figure 5 shows an exercise/pose channel user interface; Figure 6 shows a schematic view of an instructor recording system; Figure 7 shows a schematic view of an instructor analysis interface; Figure 8 shows a schematic view of a user exercise capture process; Figure 9 shows a schematic view user exercise analysis process; Figure 10 shows a first view of user exercise analysis interface; Figures lla and 11b show second and third views a user exercise analysis interface; Figure 12 shows a schematic view user exercise feedback process; Figure 13 shows a user exercise channel interface; Figures 14 and 15 show first and second views a user exercise feedback interface; Figure 16 shows an instructor feedback interface; Figure 17 shows a schematic view of An exercise monitoring and/or training system 2 is shown figure 1. The training system 2 comprises a camera 4 configured to capture image data of a user 6. The camera 4 is operatively connected to of a processing unit 8. The processing unit 8 is configured to receive and/or process data captured by the camera 4. The processing unit 8 may comprise any conventional computer, including, inter alia: a desktop computer; a laptop computer; mobile (cell) phone; tablet device; games console; "smart" TV; microcomputer; or any other suitable device. The camera 4 may comprise any conventional camera, including, inter alia: discrete digital camera; webcam; integrated camera (i.e. integrated into a mobile phone, tablet or laptop); or other suitable camera. In some embodiments, the camera 4 may comprise a stereoscopic camera (e.g. comprising two or more spaced image sensors). Typically, the camera 4 is digital, however, it can be appreciated, that a non-digital image can be captured subsequently digitised.

The processing unit 8 is operatively connected to a remote processing device 10. The remote processing device 10 comprises a storage device 12. The storage device 12 provides a database of the like. The remote processing device 10 comprises a processing unit configured to process data from the user processing unit 8. The remote processing device may comprise a system, e.g. a server system hosting a service for end users and/or instructirs.The remote processing device 10 may analyse and/or store data captured using the camera 4. The remote processing device 10 may comprise any conventional computer, including, inter alia: a server; a server cluster; a computer cluster; a computer grid; supercomputer; or any computer discussed above.

A third processing device 16 is operatively connected to the remote processing device 10. The third processing device 16 is configured to be operated by an instructor 18. The instructor processing device 16 may allow an instructor to view and/or analyse data captured by the camera 4 and/or access data made available by the remote processing device. The instructor processing device 16 may comprise any conventional computer as described above.

The remote processing device 10, user processing device 8 and the instructor processing device 16 are operatively connected, e.g. to a local or wide area network, by any suitable means; including, inter alia: LAN; WAN; the internet; "The Cloud" etc. In the present embodiment, the device 10 and user processing device 8 are connected via the internet 14.

In some embodiments, remote processing device 10 may not be provided. Analysis and/or storage of the capture camera data may therefore be performed by one or both of the user processing device 8 or the instructor processing device 16. In some embodiments, a single device may used to capture camera data and to provide instructor analysis. The data may be analysed/stored on the remote device 10 or using the single device.

It can be appreciated that the exact hardware configuration is not pertinent to the invention at hand, and any suitable configuration may be used. The following described will be described only with reference to a preferred embodiment of the invention.

The exercise training system 2 is shown schematically in figure 2. The exercise training system 2 is configured to analyse the performance of a user performing a particular exercise, and provide feedback on their technique or performance.

In a first step 20 an initial training model is created. The initial training model creates an ideal or near ideal model of a user performing a particular exercise. This provides a baseline for comparison with a further user performing the particular exercise. The initial training step 20 is shown in further details in figures 3-7.

In a first step 30, the instructor 18 selects the exercise in which to create an ideal model. The instructor 18 is provided with a GUI on the instructor device 16. As shown in the user interface of figure 4, the exercises are organised into a number of categories or classifications. In a top category tier, the instructor can select the type of sport 32 (e.g. "Yoga", "Gym", "Football" etc.). In the next tier, the instructor 18 can select an exercise type 34 within the sport category 32. In the example shown in figure 5, within the "Yoga" category, the instructor 18 can select "Bodyweight exercises", "Agachamentos (squats)" or "Warrior practice" etc. exercises.

In the next tier, the instructor 18 can select a pose 36 within the exercise category.

In the example shown in figure 5, within the "Warrior practice" category, the instructor 18 can select "Triangle", "Deep squat" or "Mountain" etc. poses.

The sport/exercise/pose type may be selected from a database of prepopulated sport/exercise/pose types. Where a specific sport/exercise/pose type is not provided in the database, the instructor 18 may create their own sport/exercise/pose records accordingly. Thus, the instructor 18 may help to populate the database (i.e. in an ad hoc fashion). A thumbnail or picture may be added to aid the user with selection of the sport/exercise/pose. Each tier may provide a channel or the like.

The instructor may be able to bookmark sport/exercise/pose type. The instructor can then easily retrieve the record at a later stage. This allows the instructor to build a customisable database of sport/exercise/pose types they are interested in.

It can be appreciated that the classification system described above is merely exemplary. Any suitable hierarchical structures may be used. The hierarchy may comprise any number of tiers. In some embodiments, no classification is provided, and the instructor may be able to view all or any poses 36 simultaneously. The instructor 18 may then search for a specific pose either manually or using a search function.

In the next stage 38, the instructor 18 performs an ideal version of the selected pose. As shown in figure 6, the performance is captured 40 using a camera 42. The camera 42 may comprise any suitable form. The captured pose is then transmitted to the remote processing device 10. In some embodiments, the ideal pose may be pre-recorded. The instructor 18 may merely upload the recording into the system.

In the next step 44, the captured image is analysed to quantify the user's ideal pose. This allows precise computational determination of the ideal pose (i.e. translates the captured image into a machine readable format). In the present embodiment, this is achieved by constructing a "skeleton" indicative of various portions of the user's body.

As shown in figure 7, a captured image 46 is translated into a skeletal model 48. The skeletal model 48 comprises a number of nodes 50 indicative of various portions of the user's body, e.g. skeletal joints. Typically, a portion of nodes are indicative of a user's joints, for example: * 50a: wrists; * 50b: elbow; * 50c: shoulder; * 50d: hips; * 50e: knee; * 50f: ankle.

Other nodes may be indicative of a fixed portion of the user's body, for example: * 50g: head * 50h: waist * 50i: groin * 50j: chest/neck.

It can be appreciated that any number of nodes 50 may be provided accordingly to the level of detail required. For example, nodes 50 may be provided to indicate fingers, toes, facial features, body outline etc. The exact process or algorithm for creating the skeletal model 48 from the captured image 46 will not be described in detail for the sake of brevity, and such a technique will be understood by the skilled person. The system may use an Artificial Neural Network (ANN) and/or Artificial Intelligence (Al) to process the recorded image to detect the features representative of the nodes 50 (e.g. using image recognition).

The nodes 50 are mapped to a three-dimensional (3D) virtual space 52. The instructor 18 may be able to view the model 48 in a 3D workspace (e.g. can perform 3D translation or rotation of the model 48). Co-ordinates 54 of the each of the nodes 50 are provided. The co-ordinates may be provided and/or recorded in any suitable co-ordinate system. In the present embodiment, the co-ordinates indicate the relative position of the nodes 50, however, in alternative embodiments, the co-ordinates may indicate relative angular orientations of the nodes 50.

The positions of the nodes 50 may be manually edited, for example, to correct a sub-optimal pose, or error. The instructor 18 may manually edit the co-ordinates and/or the nodes 50 in the 3D workspace, or by changing the co-ordinates. In some embodiments, gross errors may be corrected automatically during creation of the skeletal model 48. In some embodiments, the gross errors may be indicated to the instructor 18 to allow manual correction.

In other examples, instead of recording a performed ideal version of a pose, the skeletal model for the pose could be created from scratch, e.g. by defining angles between the lines joining the nodes. Distances between nodes may also be specified, although it will be appreciated that the size height of individual users/instructors will result in variation in such dimensions.

Once the instructor 18 is satisfied the ideal pose has been captured and recorded, the pose record is stored in a database 12 (i.e. on the storage medium) on the remote processing device 10. The pose record may then be retrieved and/or edited at a later time. The instructor 18 may repeat the process for any number of poses, exercises and/or sports as required. Thus each channel or tier may be populated with multiple different poses etc. The instructor may able to add and/or remove any poses from the channels.

The instructor may record one or more of their physical characteristics on the system. The instructor may provide user data indicative of one or more of their: * Age * Height * Weight * BMI and/or body fat levels * Sex * Fitness level * Skill/experience level * Health problems (e.g. musculoskeletal disorders) * Flexibility * Environmental factors (e.g. tobacco consumption, alcohol consumption, pollution levels). The user data is recorded with each pose. The user data may be anonymised or pseudo-anonymised to provide privacy for the instructor. The instructor may record the user data in a user profile. The system may therefore retrieve the user data from the instructor profile when a pose is recorded, and so the instructor is not required to enter the data for each pose. The instructor may be periodically prompted to update their data.

The capture process for a given pose may be repeated by the same instructor 18 and/or via a different instructor. Thus, a plurality of datasets are created for each pose. In the next step 58, the datasets are aggregated to determine an ideal pose. The ideal pose is determined by feeding the plurality of datasets into an Artificial Neural Network (ANN). The datasets thus initialise the training model. Again, this process will be understood by the person skilled in the art and will not be described in detail.

The user data provides a plurality of input variables. Thus, an ideal pose may be determined across the range of one or more of the user data. For example, an ideal pose for a range of ages, heights or weights etc. is determined. It can be appreciated that the ANN is configured to provide multivariate analysis, and therefore can determine an ideal pose using a plurality of combined variables. Where sufficient data is not provided to cover a range of a particular variable and/or combination of variables, the ANN may interpolated/extrapolate from known data to estimate a value within the range.

In the next step 22, the performance of the user 6 is monitored during their performance of the pose. This process is described in detail with reference to figure 8. Before the monitoring step 22, the user 6 may create a user profile 56.

The user may input one or more variable of the user data described above into their profile 56. Thus, the system 2 can provide an ideal pose customised for each user according to their physical characteristics (e.g. adjusted according to age, weight, height etc.). The user profile 56 is recorded in the database 12.

In some embodiments, the system 2 may use the ANN and/or an Al to estimate one or more of the variables. For example, this may be achieved using images captured by the user's camera 4 and image recognition techniques. This may be beneficial where the user wishes to retain data privacy. Additionally, this provide a "low barrier to entry" as the user is not required to provide lots of personal data.

In the next step 58, the user 6 selects a pose 36. The process is substantially the same as previously described with reference to figures 4 and 5. The user 6 may be able to bookmark any desired sport/exercise/pose types. Once the desired pose 36 is selected, the system 2 determines an ideal pose 60 according the user data provided/determined by the system 2.

The user 6 then performs the pose 36. The pose 36 is performed in front of the user's camera 4. Typically, the user 6 is positioned such that the camera 4 may captured the entire body of the user 6 (as shown in figure 9). However, where an exercise only requires a portion of the body (e.g. an upper body exercise), the user 6 may position themselves accordingly.

The system 2 monitors the images/video captured by the camera 4. The system 2 compares the pose performed by the user 6 with the predetermined ideal pose 60. This is achieved my constructing a skeletal model 48 of the user 6 from the captured images 46 (see figure 9). Typically, this is performed in real-time. An image of the skeletal model 48 may be provided on the user's device 8, providing real-time feedback to the user 6.

The skeletal model 48 of the user's pose is compared with the skeletal model 48 of the ideal pose 60. If the user's pose is within a predetermined similarity or "closeness" to the ideal pose, then the user's image data and/or skeletal model is recorded. The ideal pose 60 thus provides a datum or reference pose. The record is then recorded 66 in the database 12 for later retrieval. This ensures only relevant pose data is record in the database 12 (i.e. images of user not performing the pose 36 are immediately discarded).

The exact method of determining the similarity will not be described in detail.

However, it can be appreciated, such a method typically comprises comparing a reference/datum node 50 on the ideal skeletal model with that of the user skeletal model. If the position of the node 50 and/or the orientation of the node 50 with respective to an adjacent/further node is within a predetermined threshold or criterion, then the node 50 will be sufficiently "similar". The predetermined threshold may be dependent one or more of the user variables. For example, the threshold may be greater for users with a low skill or flexibility level. The system 2 thus provides some tolerance for poor performance of the pose 36.

This process may then be repeated for any or all of the other nodes 50. Again, there may be a threshold for the number of nodes 50 that be sufficiently similar for the complete pose to be deemed sufficiently similar. This threshold number may be dependent on one or more user variables. Some nodes 50 may be given a different similarity threshold dependent on their important for a given pose 36. For example, a head node 50g may be given a greater similarity threshold for a pose in which leg positioning is deemed important.

In the present embodiment, the system 2 is configured to record a number of attempts of the user performing the pose 36. The system 2 then determines a captured pose 64 closest to the ideal pose 60. The closest pose 64 is then recorded on the database 12 and any other poses are discarded.

In a first example, the user 6 performs a number of discrete/separate attempts at performing the pose. The system 2 may prompt the user to perform a predetermined number of attempts. For example, the user may be prompted to perform the pose for a fixed time, return to a relaxed position, and repeat for the predetermined number of attempts. This may be performed in timed sequence (i.e. all the attempts are captured sequentially in continual process). Alternatively, the user prompt the device 8 to begin the next attempt. The system 2 monitors each attempt and records the respective pose datasets. The datasets are analysed by the system 2 and the closest dataset to the ideal pose 60 is recorded. The other datasets may then be discarded.

In a second example, the system 2 may continually monitor the user performing the pose. The system 2 may periodically capture pose data. Pose data may be captured until a predetermined number of datasets are captured and/or predetermined time period elapsed. Alternatively, pose datasets may be captured until the user manually ends the session. This may allow the user to perform a more "freeform" session, for example, to rest or stretch between exercises. The system 2 may automatically determine when a pose is being performed, thus requiring no input from the user. The closest pose is then select as previously described.

It can be appreciated that the above regimes are merely exemplary, and any suitable regime may be used to capture a plurality of pose datasets.

In some embodiments, a plurality of closest poses are selected. For example, the best two or three closest poses may be selected. This may be beneficial where performance of the pose is poor in a plurality of different ways between the poses.

In the next step 26, the instructor 18 manually analyses the recorded user poses to assign the causes of poor performance of the poses. Typically, this will be performed asynchronously from user the performing the pose (i.e. so the instructor can perform analysis at a time suitable for them). The process is described with reference to figure 9-11. The instructor selects 68 one or more recorded user pose to analyse. The instructor 18 may select a specific user profile. The instructor 18 may then select a pose from the user profile. Additionally or alternatively, the instructor may select a specific sport/exercise type. The instructor 18 may then select from a number of recorded poses from different users.

The instructor 18 views 70 the pose record 72. The pose record 72 is shown in figures 10 and 11. The pose record 72 comprises an image 46 of the user pose. A skeletal model 48 is provided. The pose record 72 may be substantially the same as the ideal pose record shown in figure 7.

In the next step 74, the instructor 18 compares the user pose with an ideal pose. 5 The ideal pose may be a pose created by an instructor or an ideal pose customised according to the user. The system 2 may display the ideal pose, such that the instructor 18 can make a direct comparison. In some embodiments, the system 2 may compare the ideal pose with the captured pose and provide an indication to the instructor 18 of defects in the pose. For example, the magnitude 10 of the deviation of a user pose node 50 from an ideal pose node 50 may be indicated. This may provide a "heat map" of the location and/or magnitude of the defects. This allows quantitative and/or qualitative interpretation of the data by the instructor.

The instructor 18 records a defect 76 for the pose. The defect 76 typically indicates poor positioning and/or orientation of the user's body and/or a portion thereof. The defect 76 thus provides one or more factors indicative of the suboptimal performance of the pose (i.e. a mistake). In the example shown in figure 11a, "Knee too far forward" is recorded as the defect 76. The instructor 18 may further provide a description 78 to provide further information on the defect (i.e. a

more detailed description thereof).

The defect 76 may be selected from a predetermined list of defects. Each defect may be associated with a machine readable identifier. This may allow the same defect to be recorded across a number of different poses, thus providing easier aggregation of defect data. Where a defect is not provided in the predetermined list of defects, a new defect may be added by the instructor 18. The new defect is then added to the defect list (e.g. stored on the database 12).

The instructor 18 records a "root cause" 80 of the defect 76. Typically, the "root cause" 80 relates a musculoskeletal defect which may result in a defect 76 in the pose. For example, the root causes may comprise: * A weakness of a muscle, limb and/or joint; * Inflexibility; * Postural defects (e.g. rounded back or closed chest) In the example shown in figure 11a, a root cause of "weak calf muscles" is recorded as the root cause of "knee too far forward".

A number of root causes 80 may be selected from a predetermined list of root causes. Each root cause 80 may be associated with a machine readable identifier. A new root cause 80 may be added by the instructor 18. One or more root causes 80 may be permanently associated with each defect 76. The root causes 80 may therefore be prepopulated when a defect 76 is selected. The instructor 18 may then delete/add any root causes 80 in an ad hoc basis as they see fit.

A weighting 82 is assigned to each root cause 80. The weighting 82 is indicative of the probability of the root cause 80 resulting in the defect 76 and/or a relative contribution of an individual root cause 80 where multiple root causes 80 are present. The weighting 82 is normalised (i.e. comprises a standardised scale). In the present embodiment, the weighting may be assigned an integer value between 0 and 10. The weighting 82 may be selected using a slider 84 on the user interface. It can be appreciated that any suitable means may be used to input the weighting 82.

Any number of root causes 80 may assigned to each defect 76 as required. A respective weighting 82 is then applied to each defect 76. The weighting 82 may thus indicate the relative contributions of each root cause 80. Similarly, any number of defects 76 may be recorded for each pose. Each defect 76 comprises roots causes and weightings etc. An overall weighting 86 may be recorded on the pose. The overall weighting 86 may indicate the overall magnitude of the defects 76. For example, for a well performed pose, the defects 76 have a little effect on the performance, and so provide a small overall contribution. The defects are therefore normalised across a number of poses.

Once the defects 76 have been recorded, the pose record is updated 88 on the database. The process is repeated for any number of poses/exercises/sports. Typically, analysis of the user 4 performance is performed by an instructor 18, however, it can be appreciated that analysis may be performed by any party for any other party (e.g. the instructor 18 may analyse another instructor 18, or the user 4 may analyse another user 4 etc.). In some embodiments, analysis may be performed by a third party (e.g. via dedicated analysists or crowdsourcing etc.).

In the next stage, the user 4 is provided with feedback on their recorded poses.

The process is described with reference to figures 12-16. In a first step 90, the user 6 accesses their profile. Any recorded and/or analysed poses are aggregated for each respective user. The poses may then be classified as previously described. The user 4 may then select a pose 36. As shown in figure 13, the user 4 may then view all the defects 76 recorded for their pose. The user 4 is thus provided feedback for their pose.

The system may aggregate 94 the defects 76 for each respective user. The defects 76 may be aggregated for a given pose (e.g. where number of same poses are recorded). In some embodiments, the defects 76 are aggregated over a given exercise and/or sport type. In some embodiments, the defects 76 are aggregated across all poses in all exercise or sports types. The user 4 is therefore given a complete picture of their musculoskeletal performance. Similar data may be aggregated for each root cause 60.

As shown in figure 14, the user 4 is provided with a list of roots causes 80. The system therefore provides feedback 96 of root causes of sub-optimal performance. The user 4 may select each root causes and view any poses associated with the root causes. The user 4 may view the weighting 82 for each root causes 80, thus providing an indication of the root causes 80 to focus on for improvement.

The defects 76 and/or root causes 80 may be weighted according to their aggregated contribution to user performance. This provides the user 4 with on overview of their performance issues. The weighting is derived from each individual root cause weighting 82 and the overall pose weighting 86 recorded during the analysis stage. Typically, the weighting is derived as the sum of the individual weightings: Overall weighting = Iroot cause weighting x pose weighting The weighting indicates which root causes occur most often and provides the largest contribution. In some embodiment, the overall weighting may be normalised/averaged (e.g. by dividing the overall weighting by the number of poses the root cause appears in).

The instructor 18 may assign one or more sport/exercise/pose to each root cause 80. The more sport/exercise/pose may then be performed by the user 4 to improve their performance with respect to the root cause 80. This provides a remedial exercise 98 for each root cause 80. The remedial exercise may improve, alleviate or mitigate the root cause 80. For example, as shown in figure 15, a "Downward facing dog" exercise is assigned to a root cause of "weak calf muscle". The user may therefore perform the "Downward facing dog" exercise to strengthen their calf muscles.

As shown in figure 16, the instructor may assign a weighting 100 to each remedial sport/exercise/pose 98. The weighting 100 may determine its effectiveness/utility in improving the root cause 80. Thus, the user 4 may be provided with a number of ranked sport/exercise/poses to improve the root cause 80. The remedy and weighting 98 are stored on the database 12. The remedial exercise(s) 98 may be listed on the user's profile. The user may therefore perform the remedial exercise without the need for detailed review of each pose analysis.

The user poses and defect data are input into a training model for an ANN. The system is therefore trained to detect defects and/or root causes using an automated means (i.e. instructor analysis is not required). The system may operate in substantially the same way as the monitoring steps described with reference to figures 8 and 9. The user selects a pose; performs the pose; a closest pose is captured; and defects/root causes are assigned by the ANN. The user is then provided with automated feedback and remedial exercises assigned by the instructor at the ANN training stage.

In alternative embodiments, the instructor 18 intentionally performs a pose comprising a defect. Thus, instructor poses are used to train the ANN, rather than user poses. The process is described in further detail with reference to figure 17. The instructor 18 selects 102 a pose to perform. The instructor 18 intentionally performs 104 a pose with a particular defect. The pose is captured by the system.

The instructor manually assigns 106,108 a defect 76 and a respective weighting 82 to the captured pose as previously described. The database 12 is updated 110. User data (e.g. physical characteristics) may be recorded. The instructor 18 may use a pre-recorded image/video comprising a particular defect. For example, the instructor 18 may use stock/archival images comprising the defect. This reduces the need for the instructor 18 to perform each defect manually.

The defect pose may be recorded a number of times by the instructor and/or by a number of different instructors. This process may be repeated for a number of different defects for each pose. The data is input to an ANN, thus allowing automated detection of defects/root causes in user poses.

The user 6 may then monitor their performance as previously described with reference to figure 8. The user selects a pose; performs the pose; and closest pose is captured. The closest posed in analysed by the ANN and defects/root causes are assigned to the pose. Rather than comparing to an ideal pose, the user pose may be compared to a plurality of poses comprising defects. If the user pose is similar to a defect pose, then the defect contained within the defect pose is assigned to the user pose. The user is then provided with automated feedback on the defects/root causes present and remedial exercises assigned by the instructor at the ANN training stage. The defects/root causes may be aggregated across a number of different poses/exercises/sports as previously described.

It can be appreciated that a mixture of user poses and intentionally defect instructor poses may be used to train the ANN to detect defect/root cases in a user's performance. Manual analyses of the user's performance may be performed periodically to check for accuracy of the ANN and/or to ensure convergence thereof.

The system comprises a plurality of ANNs. The ANNs may be operate independently, or one or more of the ANNs may be interlinked. One or more ANN may be used to provide 2-dimensional analysis (e.g. analysing/processing the 2-dimensional captured images). One or more ANN may be used to provide 3-dimensional analysis (e.g. analysing/processing 3-dimensional nodes arrays or skeletal models).

The system may provide real time feedback to user as they are performing their pose. The system may suggest exercises to remedy the root causes. The user poses may also be used to improve the ideal pose 38 model. The contribution toward the ideal pose model for each user pose may be weighted according to the overall weighting 86. Thus, poorly performed poses will have a small or no contribution to the ideal pose model. This helps to build the ideal pose model with a large variety or user variables, building a more accurate model.

The present embodiment is described in terms of a static pose. Such static poses may be useful for analysing performance of yoga, Pilates, isometric exercises, weightlifting, stretches or other sports which require holding a specific position or pose. However, it can be appreciated that they system may be used to record and/or analyse a moving exercise. For example, the system may be used to analyse a golf/bat swing, swimming stroke, running gait, throwing technique etc. The system may record a predetermined segment of the moving exercise for analysis. A skeletal model may be constructed as previously described. The instructor may provide analysis of the moving exercise at a predetermined time step and/or during key frames.

In some embodiments, the system may monitor a specific point or part of the moving exercise. For example, the system may analyse the starting or finishing point of the exercise. Thus, the system may apply techniques for analysing static poses to moving exercises.

Any of the sports, exercises and/or poses discussed herein are merely exemplary. The system may be used with any suitable physical activity and/or in situations where the positioning and/or orientation of a user's body determines the performance of a specific activity. For example, the system may be used to analyse posture during working or sitting etc. The present arrangement allows simple and convenient analysis of a user's performance of an exercise by an automated system. The system captures quantitative data (i.e. the skeletal model) of the user's and/or instructor's performance allowing qualitative analysis by the instructor. The analysis process may be performed in several minutes. The analysis is used to provide data for training model to monitor the performance of the user's exercise, thus requiring little or no further input from the instructor once created. The training model may be created in a distributive manner, reducing the burden on any one instructor.

Only the closest pose to an ideal pose is captured, thus reducing the analysis required by the system, thereby reducing computational burden thereon. This allows the instructor/system to provide feedback to many users in a cost effective and distributive manner.

The system provides feedback to the user to identify the root cause of poor performance of their exercise. The system may then provide further exercise routines to improve their performance.

Claims

Claims: 1. A method of monitoring exercise performance comprising: monitoring a plurality of images of a user performing a given exercise; generating a user model from one or more of the images comprising nodes representing skeletal joints of the user and angles subtended between the nodes; accessing a predetermined model of performance of the given exercise comprising datum node positions and/or angles subtended there-between; determining whether the user model is within a given threshold of similarity to the predetermined model; recording data for an image relating to the user performance if the user model for said image is within the given threshold of similarity of the predetermined model; and assigning one or more defect to the recorded data, the defect indicative of the sub-optimal performance of the given exercise.
2. A method according to claim 1, where the defect relates to the position and/or orientation of the user's body or a portion thereof.
3. A method according to any of claims 1 and 2, where one or more root cause is assigned to the defect, the root cause comprising one or more musculoskeletal defect at least partially contributing to the sub-optimal performance of the given exercise.
4. A method according to claim 3, where the one or more root cause is assigned a weighting indicative of the magnitude of the contribution to the defect.
5. A method according to any preceding claim, where the one or more defect is assigned a weighting indicative of the magnitude of the contribution to the sub-optimal performance of the given exercise.
6. A method according to any of claims 3-5, where one or more remedial exercise is associated with a root cause, the remedial exercise configured to at least partially remedy the root cause.
7. A method according to claim 6, where the one or more remedial exercise is assigned a weighting indicative of the magnitude of the contribution to the remediation of the root cause.
8. A method according to any preceding claim, where the predetermined model is generated by captured data relating to a performance of the given exercise by the user and/or a further user.
9. A method according to any preceding claim, where the predetermined model is variably determined according to one or more physical characteristic of the user.
10. A method according to any preceding claim, where the predetermined model comprises a skeletal model of a datum or ideal performance of the given exercise, the skeletal model indictive of the position and/or orientation of the user's body or portions thereof.
11. A method according to any preceding claim, where the user model of the user performing the exercise is constructed during the monitoring of the images, the recording data comprising automatically selecting one or more image closest to the predetermined model as being indicative of the user performance of the given exercise.
12. A method according to claim 11, where the predetermined model is compared with the selected image of the user's exercise performance to determine the one or more defect.
13. A method according to any preceding claim, where only select portion of the user's exercise performance is stored when the performance is within the given threshold of similarity.
14. A method according to any preceding claim, where only a most similar portion of the user's exercise performance is stored when the performance is within the given threshold of similarity.
15. A method according to any of claims 13 or 14, where a plurality of portions of the user's exercise performance are captured and unselected images are discarded.
16. A method according to any preceding claim, comprising recording a plurality of respective defects/root causes for two or more exercises and aggregating the defects/root causes for a given user.
17. A method according to any preceding claim, comprising selecting one or more user performance according to whether the user performance is within a given threshold of similarity to the ideal performance and using the user performance to at least partially determine the ideal performance.
18. A method according to any preceding claim, where assigning of the defect is performed asynchronously from recording of the user performance.
19. A method according to any preceding claim, where assigning of the defect is performed manually or assigning of the defect is performed automatically but confirmed manually.
20. A method according to any preceding claim, where monitoring of the user performance is performed in real time with the user performance.
21. A method according to any of claims 4-20, where weightings of root causes or defects are adjusted for a user based on combined weightings assigned for different exercises.
22. A method according to any preceding claim, where the given exercise comprises a static pose.
23. A system for monitoring exercise performance comprising: a user device having a camera for capturing a plurality of images of a user performing a given exercise; a monitoring system comprising one or more processor arranged to generate a user model from one or more of the images comprising nodes representing skeletal joints of the user and angles subtended between the nodes; the monitoring system accessing from a data store a predetermined model of performance of the given exercise comprising datum node positions and/or angles subtended there-between; the monitoring system determining whether the user model is within a given threshold of similarity to the predetermined model; the monitoring system controlling recording data for an image relating to the user performance if the user model for said image is within the given threshold of similarity of the predetermined model; and the data store comprising defect data for the predetermined models and the monitoring system assigning one or more defect to the recorded data, the defect indicative of the sub-optimal performance of the given exercise.
24. A computer processor comprising machine readable instructions for operating in accordance with the monitoring system.
25. Data carrier or data storage medium comprising machine readable instructions for control of one or more processor to operate in accordance with the monitoring system.