WO2015162158A1

WO2015162158A1 - Human motion tracking

Info

Publication number: WO2015162158A1
Application number: PCT/EP2015/058672
Authority: WO
Inventors: Victor SHOLUKHA; Serge Van Sint Jan
Original assignee: Université Libre de Bruxelles
Priority date: 2014-04-22
Filing date: 2015-04-22
Publication date: 2015-10-29

Abstract

The invention relates to a computer-implemented method for 3-dimensional modelling of skeletal kinematics of a subject, wherein the method comprises a data enrichment phase, wherein the data enrichment phase comprises the steps of: a) obtaining a set of N skeletal points from a non-marker-based motion capture device; b) converting the set of N skeletal points to a set of M skeletal points, wherein M > N, by calibrating a constant limb length of at least one limb; and c) outputting the set of M skeletal points into a six degrees-of-freedom joint skeletal kinematics analysis system.

Description

HUMAN MOTION TRACKING

FIELD OF THE INVENTION

The present invention is in the field of 3-dimensional modelling of skeletal kinematics of a subject.

BACKGROUND TO THE INVENTION

Human motion tracking is widely used for movement analysis and biomechanical representation of the musculoskeletal system. Currently, most movement analysis laboratories use Marker Based Systems (MBS). Although precision of these kinds of devices is high, practical problems still occur in daily practice: such systems are cumbersome and expensive. Applying the markers to the subject is time consuming: at first, markers need to be placed carefully on the subject's skin overlying some anatomical reliefs located underneath the skin surface, for example some bony tuberosities. Furthermore, result validation is still an issue in the literature, due to reproducibility and accuracy issues. Errors during placement of the markers will induce errors during motion representation (i.e., based on the marker placement), and therefore the results may show relatively low reproducibility. Motion artefacts caused by skin deformations can also reduce the measurement precision.

MarkerLess Systems (MLS) do not show these typical disadvantages: subject preparation is fast or inexistent, since no markers need to be placed. Furthermore, the absence of markers results in a reduced reproducibility error. Gaming camera devices, such as the Microsoft Kinect™ sensor commercially developed by PrimeSense technology (Tel Aviv, Israel), herein simply referred to as Kinect, are markerless motion devices. Such devices provide a cheap and easy to use way to obtain motion information of a person. They typically use a stick model skeleton to model the person.

However, precision of markerless systems depends on the number of cameras used (single camera vs. multiple cameras systems), types of algorithms (such as annealed particle filtering, stochastic propagation, silhouette contour, silhouette based techniques), and whether the entire body is estimated or only a specific region.

Furthermore, the points used by a markerless system, for example the Kinect, do not correspond to the actual joints. Therefore, the positions and angles obtained by a Kinect are not anatomically relevant. For example, the pelvis area is inadequately described by the Kinect skeletal points, and is anatomically incorrect. The skeleton model may comprise 20 points, located only approximately at the joint level, which movements are being captured by a camera. These 20 points are gross estimations of the centre of the major joints of the human body (FIG. 1 ). This kind of model only allows for very simple motion assessment (e.g., vector angle between 3 points for knee or elbow flexion, simple geometric approach to estimate shoulder abduction between upper arm and thorax) with limited precision. This leads to a relatively crude assessment of motion.

This assessment is furthermore not compliant with standard clinical conventions because this limited skeleton is merely a simplified representation of the human anatomy, and therefore cannot adequately represent the human skeleton in 3 dimensions (3D). It must be stressed that in order to be used in clinics for the evaluation and the follow-up of patients, the standard provided skeleton must be improved to include anatomical knowledge to meet anatomical conventions.

The field of motion assessment, for example in a clinical motion analysis or biomechanics, requires a larger set of skeletal points. Furthermore, the skeletal points need to be anatomically accurate, and need to correspond to actual joints in the body. Clinical conventions using such points are typically based on anatomical planes.

Therefore, there is a need for cost-efficient and more effective motion assessment tools that still provide anatomically correct and accurate data, suitable for use in clinical motion analysis. SUMMARY OF SOME EMBODIMENTS OF THE INVENTION

The present invention overcomes one or more of the aforementioned disadvantages, based on advanced 3-dimensional modelling of skeletal kinematics of a subject. Preferred embodiments of the present invention overcome one or more of the aforementioned disadvantages.

According to a first aspect, the present invention provides a computer-implemented method for 3-dimensional modelling of skeletal kinematics of a subject, wherein the method comprises a data enrichment phase, wherein the data enrichment phase comprises the steps of:

a) obtaining a set of N skeletal points from a non-marker-based motion capture device; b) converting the set of N skeletal points to a set of M skeletal points, wherein M > N, by calibrating a constant limb length of at least one limb; preferably wherein external constraints are used based on a 3-point algorithm; and

c) outputting the set of M skeletal points into a six d eg rees-of -freedom joint skeletal kinematics analysis system.

According to a second aspect, the invention provides a method for 3-dimensional modelling of skeletal kinematics of a subject, comprising the steps of:

i) providing a non-marker-based motion capture device;

ii) providing a six degrees-of-freedom joint skeletal kinematics analysis system;

iii) obtaining a set of N skeletal points of the subject with the non-marker- based motion capture device;

iv) providing the set of N skeletal points of the subject for use in a data enrichment phase, wherein the data enrichment phase comprises the steps of:

a) obtaining the set of N skeletal points from the non-marker-based motion capture device;

b) converting the set of N skeletal points to a set of M skeletal points, wherein M > N, by calibrating a constant limb length of at least one limb; preferably wherein external constraints are used based on a 3-point algorithm; and

c) outputting the set of M skeletal points into the six degrees-of- freedom joint skeletal kinematics analysis system.

In some preferred embodiments, the invention provides a method according to the first or second aspect as described above, wherein the non-marker based motion capture device in step a) is a feature extraction-based motion capture device, preferably a markerless camera device, more preferably a gaming camera device.

In some preferred embodiments, the invention provides a method according to the first or second aspect as described above, wherein the set of N skeletal points comprises at most 28 points, preferably at most 26 points, preferably at most 24 points, preferably at most 22 points, preferably 20 points. In some preferred embodiments, the invention provides a method according to the first or second aspect as described above, wherein the set of M skeletal points comprises at least 30 points, preferably at least 31 points, preferably at least 32 points, preferably at least 33 points, preferably 34 points.

In some preferred embodiments, the invention provides a method according to the first or second aspect as described above, wherein step b) is performed by processing each link sequentially from the root, preferably the pelvis, to the end joint. In some preferred embodiments, the invention provides a method according to the first or second aspect as described above, wherein external constraints, for example floor constraints, are used in step b), preferably based on a 3-point algorithm.

In some preferred embodiments, the invention provides a method according to the first or second aspect as described above, wherein additional points in step b) are obtained based on the orientation of two neighbouring links.

In some preferred embodiments, the invention provides a method according to the first or second aspect as described above, further comprising the step of:

b') optimizing the set of M points based on anatomical knowledge.

In some preferred embodiments, the invention provides a method according to the first or second aspect as described above, wherein the set of M skeletal points in step c) corresponds to the standard output of a marker-based motion capture system, for example a marker-based stereophotogrammetry motion capture system.

In some preferred embodiments, the invention provides a method according to the first or second aspect as described above, wherein step c) comprises outputting the set of M skeletal points into a conventional gait model, preferably into a plug-in-gait motion representation model.

In some preferred embodiments, the invention provides a method according to the first or second aspect as described above, wherein steps a) to c) are performed in real-time. In some preferred embodiments, the invention provides a method according to the first or second aspect as described above, wherein the subject is a human subject. Preferably external constraints are used based on a 3-point algorithm. This means that a a 3-point algorithm is used when imposing the external constraints. In some preferred embodiments, the 3-point algorithm comprises a pose estimation based on the determination of a triangle representing the limb-of-interest from the original linear representation.

According to a third aspect, the present invention provides a computer-implemented method for performing a postural or gait analysis of a subject, comprising the method for modelling skeletal kinematics of the subject according to the first or second aspect of the invention and preferred embodiments thereof.

According to a fourth aspect, the present invention provides use of the method according to the first or second aspect of the invention and preferred embodiments thereof for biomechanics and/or clinical functional analysis.

FIGURE LEGENDS

FIG. 1 : Stick model skeleton obtained with the Kinect™ sensors and Kinect for

Windows SDK (source: http://i. msdn.microsoft.com/dynimg/IC584844.png)

FIG. 2: FIG. 2A illustrates the anterior view of a stick model diagram of a subject in the upright position, tracked with the Kinect™ sensor, showing that the joint centres (numbered according to Kinect for Windows SDK output) are well recognized. FIG. 2B and 2C illustrates the anterior and lateral view respectively of a stick model diagram of the subject performing a deep squat movement, tracked with the Kinect™ sensor. The arrow indicates that the left knee is wrongly tracked.

FIG. 3: FIG. 3A-C are similar to FIG. 2A-C. FIG. 3D and 3E show the same squat motion for the 347 skeletal points, the arrow indicating that the left knee is in a more natural position. FIG. 3F illustrates the optimized skeleton in the upright position.

FIG. 4: FIG. 4A illustrates the raw Kinect model (upper body) with 14 skeletal points, FIG. 4B illustrates the optimized model with 220 skeletal points, and FIG. 4C illustrates the optimized model fused with a generic skeleton.

FIG. 5: Right knee position correction taking into account floor constraints (right posterolateral view). FIG. 6A-B: Example of right knee and ankle joint angles reconstruction from extended 3D model (FIG. 6A) and comparison with original stick-based model (FIG. 6B).

FIG. 7: Flow chart illustrating a preferred embodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

Before the present methods of the invention are described, it is to be understood that this invention is not limited to particular methods or combinations described, since such methods and combinations may, of course, vary. It is also to be understood that the terminology used herein is not intended to be limiting, since the scope of the present invention will be limited only by the appended claims.

As used herein, the singular forms "a", "an", and "the" include both singular and plural referents unless the context clearly dictates otherwise.

The terms "comprising", "comprises" and "comprised of" as used herein are synonymous with "including", "includes" or "containing", "contains", and are inclusive or open-ended and do not exclude additional, non-recited members, elements or method steps. It will be appreciated that the terms "comprising", "comprises" and "comprised of" as used herein comprise the terms "consisting of", "consists" and "consists of".

The recitation of numerical ranges by endpoints includes all numbers and fractions subsumed within the respective ranges, as well as the recited endpoints. Whereas the terms "one or more" or "at least one", such as one or more or at least one member(s) of a group of members, is clear per se, by means of further exemplification, the term encompasses inter alia a reference to any one of said members, or to any two or more of said members, such as, e.g., any≥3,≥4,≥5,≥6 or≥7 etc. of said members, and up to all said members.

All references cited in the present specification are hereby incorporated by reference in their entirety. In particular, the teachings of all references herein specifically referred to are incorporated by reference. Unless otherwise defined, all terms used in disclosing the invention, including technical and scientific terms, have the meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. By means of further guidance, term definitions are included to better appreciate the teaching of the present invention.

In the following passages, different aspects of the invention are defined in more detail. Each aspect so defined may be combined with any other aspect or aspects unless clearly indicated to the contrary. In particular, any feature indicated as being preferred or advantageous may be combined with any other feature or features indicated as being preferred or advantageous. Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment, but may. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner, as would be apparent to a person skilled in the art from this disclosure, in one or more embodiments. Furthermore, while some embodiments described herein include some but not other features included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the invention, and form different embodiments, as would be understood by those in the art. For example, in the appended claims, any of the claimed embodiments can be used in any combination.

In the present description of the invention, reference is made to the accompanying drawings that form a part hereof, and in which are shown by way of illustration only of specific embodiments in which the invention may be practiced. It is to be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the present invention. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present invention is defined by the appended claims.

According to a first aspect, the present invention concerns a computer-implemented method for 3-dimensional modelling of skeletal kinematics of a subject, wherein the method comprises a data enrichment phase, wherein the data enrichment phase comprises the steps of: a) obtaining a set of N skeletal points from a non-marker-based motion capture device;

c) outputting the set of M skeletal points into a six degrees-of-freedom joint skeletal kinematics analysis system.

In some preferred embodiments, the invention provides a method for 3-dimensional modelling of skeletal kinematics of a subject, comprising the steps of:

i) providing a non-marker-based motion capture device;

iv) providing the set of N skeletal points of the subject for use in a data enrichment phase as described in the first aspect of the invention. According to a second aspect, the invention provides a method for 3-dimensional modelling of skeletal kinematics of a subject, comprising the steps of:

v) providing a non-marker-based motion capture device;

vi) providing a six degrees-of-freedom joint skeletal kinematics analysis system;

vii) obtaining a set of N skeletal points of the subject with the non-marker- based motion capture device;

viii) providing the set of N skeletal points of the subject for use in a data enrichment phase, wherein the data enrichment phase comprises the steps of:

b) converting the set of N skeletal points to a set of M skeletal points, wherein M > N, by calibrating a constant limb length of at least one limb; preferably wherein external constraints are used based on a 3-point algorithm; and c) outputting the set of M skeletal points into the six degrees-of- freedom joint skeletal kinematics analysis system.

The method according to the first or second aspect of the invention comprises the step of:

a) obtaining a set of N skeletal points from a non-marker-based motion capture device.

As used herein, the term "non-marker-based motion capture device" preferably refers to a markerless motion capture device, such as a markerless camera. The non-marker-based motion capture device may comprise a single camera, or multiple cameras. Preferably, the N skeletal points are obtained from single camera markerless data (e.g. the Kinect). Other 3D camera's that may be used can be commercially provided by Asus (such as the Xtion Pro Live), Mesa imaging, Intel (such as the Creative Interactive Gesture Camera Developer Kit, or the RealSense 3D Camera), Panasonic (such as the D-imager), Softkinetic (such as DepthSense), and Simi (such as the Simi Shape 3D).

Each frame of the original data can be collected from the original hardware and available as 2D or 3D coordinates of crude approximation of the main human joints (e.g., shoulder, elbow, hip, knee, etc.). By piecewise linear connection of those joints one can develop stick-based model (i.e., each adjacent set of point is linked together by a line representing human segments) for visualization and motion analysis. The major lack of this approach is the inability of allowing anatomically correct descriptions of the joint angular motion according to present clinical conventions.

Single camera markerless data are preferably captured and recognized in real time. In order to be used for scientific purposes, these data usually require filtering (e.g., point trajectories smoothing), reframing (i.e., to guaranty frame frequency stability) and calibration.

In some preferred embodiments, the non-marker based motion capture device in step a) is a feature extraction-based motion capture device, preferably a markerless camera device, more preferably a gaming camera device (for example a Microsoft Kinect™ device). Preferably, the set of N skeletal points was obtained by feature extraction of N joints. In some preferred embodiments, the set of N skeletal points is insufficient for full 3- dimensional skeletal kinematics analysis of the subject, but wherein the set of M skeletal points allows for full 3-dimensional skeletal kinematics analysis of the subject. For example, the set of N skeletal points may only provide a simplified representation.

In some preferred embodiments, the set of N skeletal points comprises at least 16 points and at most 28 points, preferably at least 17 points and at most 26 points, preferably at least 18 points and at most 24 points, preferably at least 19 points and at most 22 points, preferably 20 points. In some preferred embodiments, the set of N skeletal points comprises at most 28 points, preferably at most 26 points, preferably at most 24 points, preferably at most 22 points, preferably 20 points. For example, the set of N skeletal points may comprise 16, 17, 18, 19, 20, 21 , 22, 23, 24, 25, 26, 27, or 28 points. In another embodiment, only a subset of the set of N skeletal points is used in the present method, comprising fewer points that the set of N skeletal points as set out above.

b) converting the set of N skeletal points to a set of M skeletal points, wherein M > N, by calibrating a constant limb length of at least one limb preferably wherein external constraints are used based on a 3-point algorithm. According to the invention, one or more additional constraints are imposed on the set of N skeletal points to obtain the set of M skeletal points. Preferred constraints are a constant limb length for one or more limbs, preferably for all limbs (i.e., lower and upper limbs, and trunk). Preferably, the limb length is obtained from a starting pose, preferably a vertical upright static starting pose. Other preferred constraints can be a floor constraint, whereby the vertical position of a skeletal point of a lower limb, ankle, or foot is constrained by the vertical position of the starting pose, i.e. by the floor.

It is noted that the addition of points (M>N) is not an optimisation step as such. It is a step used to obtain a set of initial conditions, that may be used as a starting point for an additional optimisation step. However, the inventors have surprisingly found that the initial conditions obtained by the method of the invention are of such high quality, that a further optimisation step may not be necessary, but merely optional. Alternatively, when a further optimisation step is used, the initial conditions obtained by the method of the invention provide an improved starting point for such an optimisation step. According to the invention, a calibration procedure is adopted that is preferably based on static pose capturing. Scaling factors are estimated by comparison with external anthropometric body links measurements (for example, directly measured by the therapist in charge of the motion data capture). Calibration allows estimation of the link's constant size. It is important to underline that markerless cameras typically show an error in longitudinal direction one order higher compared to frontal plane directions, and such error leads to substantial inaccuracy in current link sizes (e.g., the distance between neighbour joints is not always constant). In a preferred embodiment, the link size is corrected based on the assumption that the set of N skeletal points (for example, a raw stick-based model) supplies proper line orientation. In some embodiments, the method starts from the native pelvis stick model. This has the advantage that it is usually the most reliable and stable data collected from a markerless camera. In some preferred embodiments, step b) is performed by processing each link sequentially from the root, preferably the pelvis, to the end joint. The spatial location of extremity joints, and therefore segment size, can be substituted by processing each link sequentially from the root (e.g., pelvis segment) to the end joint (e.g., hip joint, then knee joint, then ankle joint). FIG.5 shows an example of such an approach (for the right leg, HC-ιΑ-ι obtained from raw data HCA).

Constant limb length implementation may be based on analysis of the current frame joint 3D positions:

for example whereby N = 20, as presented in Figure 2A.

Joint connection may be defined by a topology table:

for example whereby N = 20, as presented in Figure 2A: T = [0, 1 , 2, 3, 3, 5, 6, 7, 3, 9, 10, 1 1 , 1 , 13, 14, 15, 1 , 17, 18, 19]. For each limb, a normalized orientation vector may be calculated as:

where is the Euclidian vector norm.

Suppose that during a calibration step limb length

was calculated, then unconstrained new joint positions

may be calculated as:

In some preferred embodiments, external constraints, for example external floor constraints, are furthermore used in step b), preferably based on a 3-point algorithm. When external constraints are available (e.g., by assuming that the feet are kept on the floor), the corresponding joint position can be adjusted in order to keep proper position and/or orientation of the contact links. Example of raw data showing the floor constraints are illustrated in FIG. 5 (HCA). In some embodiments, the constraints are applied based on a three-point algorithm for the pose estimation. The estimation is preferably based on the determination of a triangle representing the limb-of-interest from the original linear representation. Such estimation requires therefore the determination of a supplementary point, or vertex, in order to define the entire segment triangle. This is clarified below by taking an example of a foot on the floor, taken as external constraint, during squat motion, and as illustrated in FIG. 5 and FIG. 6, by keeping the lower limb length constant.

One of the advantages of the 3-point algorithm is that it is straight-forward as well as highly accurate from an anatomical viewpoint, allowing real-time calculations with realistic anatomical results. The calibration as such is not a static calibration based on a starting pose prior to the kinematic measurements, but instead it is a dynamic calibration obtained during kinematic measurements. FIG. 5 illustrates a right knee position correction according to a preferred embodiment of the invention, taking into account floor constraints (right posterolateral view). Points A, H, and C are points estimated from a markerless camera. Point K is a reconstructed new knee position taking into account constant size of the links. Point P is the projection of K on the HA side. Small balls near knee (K) and ankle (A) are newly-estimated points determined by the invention and available from the extended model; these points give an estimation of the frontal plane (for the knee, defined by medial and lateral epicondyle landmarks, for the ankle joint, defined by the medial and lateral malleolus landmarks). The skeleton HdA-i shows the right lower limb size estimated without external constraints, note the exaggerated lower limb length compared to the real skeleton. The skeleton HKA shows the same lower limb corrected with the external floor constraints. Without loss of generality, it may be assumed that the total leg length is elongated due to thigh and shank link sizes correction in order to keep a constant lower limb length. In order to keep the foot position on the floor after total leg elongation, one must keep both hip and ankle positions by altering the knee joint position. The new knee joint position can then be estimated by a square calculation of the AKH triangle with side lengths from the distance between hip and ankle and new thigh and shank sizes (e.g., using well-known available algorithms such as the Heron's formula). Then the projection (point P) of the new knee joint point towards the hip-ankle line can be estimated from the previously- determined AKH triangle. Note that the distance between hip and ankle joints, and the estimation of the new knee position lay in the same plane as the original one.

The last supposition does not allow satisfactory solution when the current knee angle is small (e.g., when lies within the expected error of measurement). In such cases, the normal (illustrated by the lateral arrow on the pelvis local coordinate system visible in FIG. 5) to the pelvis frontal plane and thigh (or shank) can be used instead for the new knee position evaluation.

Finally, the thigh link defined between the hip joint point and the new knee joint point can be supplied by two additional points within the frontal plane (e.g., small balls in FIG. 5) using thigh and shank orientations. In some preferred embodiments, additional points in step b) are obtained based on the orientation of two neighbouring links. This approach is preferably used for the lower limbs (knee and ankle joints) and the upper limb (elbow joint) extremities. It can be extended straightforwardly for any other joints if the original data comprise enough information about the linear segment architecture of the body segments to analyse.

When additional points in step b) are obtained based on the orientation of two neighbouring links, this allows for a more accurate representation of certain limbs, which is more realistic from an anatomical viewpoint. For example, two additional points may be obtained perpendicular to the plane comprising two neighbouring links, for example for a lower limb. Without these additional points, only the flexion/extension of the lower limb can be accurately estimated, but with the addition of these points, the 3D rotation of the thigh may be estimated as well. When applied to the elbow joint, this may also allow for the 3D movement of the shoulder (e.g. humerus). When applied to the wrist joint, this may also allow for the 3D movement of the forearm (e.g. ulna and radius). In some preferred embodiments, anatomical knowledge is additionally used to create new skeletal points. For example, some angles, or some directions of motions are anatomically impossible or improbable. The anatomical knowledge allows for additional constraints to be imposed on the motion of the skeletal points, from which new skeletal points may be obtained. In some preferred embodiments, the method according to the first or second aspect of the invention (and preferred embodiments thereof) further comprises the step of:

b') optimizing the set of M points based on anatomical knowledge.

This anatomical knowledge can be mostly based on range of joint motion limitation. For example, knee and elbow joints do not hyperextend in normal individuals. Range of joint motions and joint behaviour are also available from the literature (e.g., so-called coupled motions).

FIG. 6 presents an example of results of lower limbs performing deep squatting. Original raw data from markerless cameras are preferably first filtered and pre-processed. From these data, relative angular orientation of the segments can be estimated. In addition, the same data are processed by the method as described above and supplementary points are estimated. From all available points (raw and estimated), anatomical frames can be obtained (here for the pelvis, thigh and shank with the axes X, Y, and Z respecting clinical conventions). From the anatomical frames, segment orientation can be processed. This example was created from Kinect data (20 points per frame) and converted to PiG like (PiG refers to "Plug in Gait" which is used in the Vicon Clinical Manager software package) data (34 points per frame).

The results show acceptable agreement in the sagittal plane, i.e., around the Z-axis (right laterally oriented) in FIG. 6A, between the stick model and PiG model: here in this particular example, the maximal difference for the knee joint is about 8°, as shown in FIG. 6B. Values of flexion angles for the stick model (N skeletal points) are 60° and 92° for the hip and knee joint, respectively. Flexion angles values estimated from the extended model (M skeletal points) are 96°(knee) and 67°(ankle). Supplementary angles amplitude values are in the range [0,20]°, which is within normal anatomical range. Note that in FIG. 5, the knee angle (HCA, raw data) value was almost 25° smaller compare to the corrected one (HKA). Supplementary anatomical angles (abduction/adduction and internal/external rotation) can be evaluated for the PiG model only, which is valuable for scientific and clinical reporting.

The method according to the first or second aspect of the invention comprises the step of: c) outputting the set of M skeletal points into a six d eg rees-of -freedom joint skeletal kinematics analysis system.

The M points may be used to determine six degrees-of-freedom joint kinematic (i.e., angle, translation, velocity, and acceleration) directly without optimisation. Results are preferably further presented according to conventional clinical and anatomical motion representations. Representing six degrees-of-freedom joint kinematic is an important issue for further realistic representations of joint behaviour and joint kinematics. Such representation allows production of quantitative results that answer and respect current biomechanical and clinical standards (which is impossible when using the linear output of current markerless cameras). Next to producing research and clinical reports according to international standards, a six degrees-of-freedom joint description also improves joint kinematic representation for further modelling (e.g., processing muscle and ligament behaviour during musculoskeletal modelling based on the six degrees-of-freedom produced by the invention). For example, hip, knee and ankle joints crossed by so-called double joint muscles (e.g. quadriceps etc) and their moment arm evaluation is very sensitive to the amount of joint degrees of freedom, especially when knee and ankle joint translations are compared to simplified rotational degrees of freedom only. In some preferred embodiments, the set of M skeletal points comprises at least 30 points and at most 42 points, preferably at least 31 points and at most 40 points, preferably at least 32 points and at most 38 points, preferably at least 33 points and at most 36 points, preferably 34 points. In some preferred embodiments, the set of M skeletal points comprises at least 30 points, preferably at least 31 points, preferably at least 32 points, preferably at least 33 points, preferably 34 points. For example, the set of M skeletal points may comprise 30, 31 , 32, 33, 34, 35, 36, 37, 38, 39, or 40 points. In another embodiment, only a subset of the set of N skeletal points is used in the present method, resulting in only a subset of the set of M skeletal points, comprising less points that the set of M skeletal points as set out above. In some preferred embodiments, the set of M skeletal points comprises at least 40 points, preferably at least 50 points, preferably at least 75 points, preferably at least 125 points, preferably at least 150 points, preferably at least 175 points, preferably at least 200 points.

The new model architecture allows for proper anatomical and clinical motion representations, and further analysis for example to be included in a conventional patient file. The final enriched model is preferably similar to current standards in clinical motion analysis (such as the PiG model used by the VICON company, or any clinical conventions recommended by standardization committees, such as for example the International Society of Biomechanics, the Gait and Clinical Movement Analysis Society, the European Society for Movement Analysis in Adults and Children, etc.)- In some preferred embodiments, the set of M skeletal points corresponds to the standard output of a marker- based motion capture system, for example a marker-based stereophotogrammetry motion capture system (such as most available marker-based systems found on the market, e.g. VICON, MotionAnalysis, etc). In some preferred embodiments, step c) comprises outputting the set of M skeletal points into a conventional gait model, preferably into a plug-in-gait motion representation model.

In some preferred embodiments, steps a) to c) are performed in real-time. The subject can be a human, animal or robotic subject, preferably a human or animal subject. In some preferred embodiments, the subject is a human subject.

According to a third aspect, the present invention provides a computer-implemented method for performing a postural or gait analysis of a subject, preferably according to world-wide clinical conventions, comprising the method for modelling skeletal kinematics of the subject according to the first or second aspect of the invention and preferred embodiments thereof.

According to a fourth aspect, the present invention provides use of the method according to the first, second, or third aspect of the invention and preferred embodiments thereof for biomechanics and/or clinical functional analysis, preferably related to the assessment, diagnosis or follow-up of the musculoskeletal systems in normal subjects (e.g., body tracking, sport analysis, etc) or in patients (e.g., suffering from musculoskeletal or neurological disorders leading to locomotor disturbances).

According to a fifth aspect, the present invention provides a computer program, or a computer program product directly loadable into the internal memory of a computer, or a computer program product stored on a computer readable medium, or a combination of such computer programs or computer program products, for performing the method according to the first, second, or third aspect of the invention and preferred embodiments thereof. According to a sixth aspect, the invention also provides a computer or a computer- readable medium on which the computer program or computer program product according to the fifth aspect is stored. Preferably, a skeletal kinematics analysis system is also stored on the computer or computer-readable medium. Preferred embodiments for the skeletal kinematics analysis system are as described above for the first or second aspect of the invention.

The invention also provides a system suitable for performing the method according to the first, second, or third aspect of the invention. According to a seventh aspect, the invention also provides a system, comprising:

a non-marker-based motion capture device, preferably a feature extraction-based motion capture device, more preferably a markerless camera device, more preferably a gaming camera device; and

a computer or a computer-readable medium according to the sixth aspect of the invention.

Preferred embodiments for the non-marker-based motion capture device are as described above for the first or second aspect of the invention. Preferably, the system according to the seventh aspect of the invention also comprises a skeletal kinematics analysis system. Preferred embodiments for the skeletal kinematics analysis system are as described above for the first or second aspect of the invention.

FIG. 7 shows a flow chart illustrating a preferred embodiment of the method of the invention using a system of the invention. A non-marker based motion capture device captures images of a subject and provides N skeletal points of the subject for each image. Using constant limb length of at least one limb, the data enrichment phase converts these N skeletal points into M skeletal points. The M skeletal points are subsequently used as input for a 6 degrees of freedom (6DoF) skeletal kinematics system to provide a more detailed image. Based on the input of M skeletal points, the 6DoF skeletal kinematics system can be used for analysis of the subject, for example postural or gait analysis. EXAMPLES

Twenty healthy adults (24 ± 6 years old, 172 ± 8 cm height, 68 ± 10 kg weight, 23 ± 3 kg/m² BMI, 10 women) were recruited to participate to this study.

The MLS (Kinect™ sensor, frequency: 30 Hz) was used as a motion analysis device. Custom-made software integrating the native Microsoft Kinect SDK (version 1.6) was developed to collect the raw MLS data (Fig.lA). Note that these data were relatively crude (i.e., stick-like model without any real 3D representation of anatomical segments). The MLS sensor was placed on a tripod at 1.5 m above the floor. Subjects stood at 2.5 m from the camera. Subjects were in underwear to allow reliable placement of the markers for the MBS analysis taking place simultaneously. Prior to motion analysis, the subjects were asked to stand still in anatomical position facing the MLS camera. Subjects were then asked to maintain three different poses (3s for each of the poses) before recording the motion in order to calibrate the MLS data processing pipeline.

MBS data were simultaneously collected from a state-of-the-art stereophotogrammetric system (Vicon, 8 MXT40s cameras, Vicon Nexus software, frequency: 90Hz) that tracks the spatial trajectories of the reflective markers set on the subjects. A modified PiG model has been adopted. Next to the usual PiG markers, markers have been set on the medial epicondyle of the humeral and femoral bones. Thirty four markers were positioned by the same observer during the entire study.

After performing the calibration, the subjects were asked to perform three different motions that are used in daily clinical practice to evaluate the function of the upper limb: putting the hand to the top of the head ("do one's hair"), hand-to-mouth ("eat") and hand- to-back ("take your wallet in the pocket"). Five repetitions of each motion were performed with right and left arms successively. In order to assess repeatability of the measurements the same protocol was repeated one week later by the same operator.

For the MLS, three different approaches were used:

Raw results (Raw): Using the raw figure model provided by the MLS software, the different angles were directly obtained using Equations 1 to 3 (note that it is not possible to estimate shoulder orientation in the 3 clinically-relevant anatomical planes with this simple approach). No filtering method was applied on the native MLS data.

PiG like model (PiG), using a method according to the present invention: Following the method that introduced a model based approach for interpolating additional markers, the 20 anatomical landmarks (ALs) from the raw MLS model were converted to 34 ALs to obtained a "PiG-like" model. Then, for each limb-of-interest an anatomical local coordinate system (LCS) was created based on three ALs. For pelvis and thorax four ALs were used for the LCS definition. From these LCS, joint anatomical angles were computed using Euler's angles.

Optimization (Optimized), using a method according to the present invention: Based on the PiG like model data described above, an extended model was tuned (e.g. scaled) and fitted frame by frame to PiG data using constrained optimization. In summary, this procedure was applied starting from the generic pelvis model using affine registration (as most reliable raw data). Then, thorax and each extremity location were sequentially optimized by constrained joint degrees of freedom (DoF) variations. The initial pose for optimization was predicted from previous frames or generated directly from PiG data. Finally, each DoF was smoothed to obtain continuous curves. For each of the above specified three approaches, the mean of the five repetitions of the motions were calculated by selecting manually the start and the end of each. Ranges of Motion (RoM) for shoulder flexion, abduction and rotation and elbow flexion were finally computed as the difference between the maximal and the minimal values on the averaged motions.

Normality of the data was checked using the Shapiro-Wilk test. Mean values and standard deviations were calculated. Discrepancies between the MBS and the different versions of MLS were tested using Pearson's correlation coefficient (R). The test-retest reliability was investigated using Intra-class Coefficient Correlation (ICC) (two-way random average measures). Two other relevant clinical parameters were processed: the Standard Error of Measurement (SEM) is a reliability measure that assesses response stability and the Minimal Detectable Change (MDC) that estimates the smallest amount of change that can be detected by a measure that corresponds to a noticeable change in clinics, using equations 4 and 5.

Due to discrepancies between the three different approaches, SEM and MDC were also presented in percentage (NSEM and NMDC respectively) of the RoM to facilitate interpretation and later comparison (Normalized variable=Variable/RoM^*100). All statistics and data processing were performed in Matlab (MathWorks, Natick, Massachusetts, U.S.A.).

Mean RoM results for MBS and MLS (Raw, PiG and Optimized) and correlation are presented in Table 1. Concerning the reproducibility of measurement, results (ICC, SEM and MDC) are presented in Table 2.

Table 1

Table 2

Claims

1. Computer-implemented method for 3-dimensional modelling of skeletal kinematics of a subject, wherein the method comprises a data enrichment phase, wherein the data enrichment phase comprises the steps of:

a) obtaining a set of N skeletal points from a non-marker-based motion capture device;

b) converting the set of N skeletal points to a set of M skeletal points, wherein M > N, by calibrating a constant limb length of at least one limb; wherein external constraints are used based on a 3-point algorithm; and c) outputting the set of M skeletal points into a six degrees-of-freedom joint skeletal kinematics analysis system.

2. Method for 3-dimensional modelling of skeletal kinematics of a subject, comprising the steps of:

i) providing a non-marker-based motion capture device; ii) providing a six degrees-of-freedom joint skeletal kinematics analysis system;

iii) obtaining a set of N skeletal points of the subject with the non- marker-based motion capture device;

a) obtaining the set of N skeletal points from the non- marker-based motion capture device;

b) converting the set of N skeletal points to a set of M skeletal points, wherein M > N, by calibrating a constant limb length of at least one limb; wherein external constraints are used based on a 3-point algorithm; and

c) outputting the set of M skeletal points into the six degrees-of-freedom joint skeletal kinematics analysis system.

3. The method according to any one of claims 1 or 2, whereby additional points in step b) are obtained based on the orientation of two neighbouring links.

4. The method according to any one of claims 1 to 3, wherein the 3-point algorithm comprises a pose estimation based on the determination of a triangle representing the limb-of-interest from the original linear representation.

5. The method according to any one of claims 1 to 4, wherein the non-marker based motion capture device in step a) is a feature extraction-based motion capture device, preferably a markerless camera device, more preferably a gaming camera device.

6. The method according to any one of claims 1 to 5, wherein the set of N skeletal points comprises at least 16 points and at most 28 points, preferably at least 17 points and at most 26 points, preferably at least 18 points and at most 24 points, preferably at least 19 points and at most 22 points, preferably 20 points.

7. The method according to any one of claims 1 to 6, wherein the set of M skeletal points comprises at least 30 points and at most 42 points, preferably at least 31 points and at most 40 points, preferably at least 32 points and at most 38 points, preferably at least 33 points and at most 36 points, preferably 34 points.

8. The method according to any one of claims 1 to 7, wherein step b) is performed by processing each link sequentially from the root, preferably the pelvis, to the end joint.

9. The method according to any one of claims 1 to 8, further comprising the step of:

b') optimizing the set of M points based on anatomical knowledge.

10. The method according to any one of claims 1 to 9, wherein the set of M skeletal points in step b) corresponds to the standard output of a marker-based motion capture system, for example a marker-based stereophotogrammetry motion capture system.

1 1. The method according to any one of claims 1 to 10, step c) comprises outputting the set of M skeletal points into a conventional gait model, preferably into a plug- in-gait motion representation model.

12. The method according to any one of claims 1 to 1 1 , wherein steps a) to c) are performed in real-time.

13. The method according to any one of claims 1 to 12, wherein the subject is a human subject.

14. Computer-implemented method for performing a postural or gait analysis of a subject, comprising the method for modelling skeletal kinematics of the subject according to any one of claims 1 to 13.

15. Use of the method according to any one of claims 1 to 14 for biomechanics and/or clinical functional analysis.