LU100034B1

LU100034B1 - Physical activity feedback

Info

Publication number: LU100034B1
Application number: LU100034A
Authority: LU
Inventors: Michel Antunes; Girum Demisse; Djamila Aouada
Original assignee: Univ Luxembourg
Priority date: 2017-01-31
Filing date: 2017-01-31
Publication date: 2018-07-31

Abstract

The invention is directed to a method for analysing the position of a human body using a computing device, comprising the following steps: (a) capturing at least one frame; (b) detecting on the at least one captured image frame the position of joints of the at least one portion of the body; (c) registering the detected positions and predefined reference positions of the joints in a common reference system; (d) computing a displacement of at least one of the joints that minimizes the error between the detected position and the corresponding predefined reference position of said joints; (e) using output means of said computing device, providing an indication of said displacement as an output. In step (d), the computed displacement is at least one rotation of one of the at least one body part.

Description

PHYSICAL ACTIVITY FEEDBACK

Technical field [0001] The invention is directed to the field of physical training activity, more particularly to automated physical activity training and feedback.

Background art [0002] Physical training activity is vital for the general population for maintaining a healthy lifestyle. It is crucial for elderly people in the prevention of diseases, maintenance of independence and improvement of quality of life. For stroke survivors it is critical and essential for recovering some autonomy in daily life activities. Despite the benefits of physical activity, many stroke survivors do not exercise regularly due to many reasons, such as lack of motivation, confidence, and skill levels. Traditionally, the post-stroke patients are initially subject to physical therapy under the supervision of a health professional aimed at restoring and maintaining activities of daily living in rehabilitation centres. The physiotherapist explains the movement to be performed by the patient, and continuously advises her/him how to improve the motion as well as interrupts the exercise in case of health related risk issues. Unfortunately, and due to the high economic burden, the on-site rehabilitation is usually of a short period of time and prescribed treatments and activities for home based rehabilitation are usually suggested. Unfortunately, stroke patients, and more frequently older adults, do not appropriately adhere to the recommended treatments, because, among other factors, they do not always understand or remember well enough what and how they are supposed to do the physical treatment.

[0003] In order to support the rehabilitation of stroke patients at home, human tracking and gesture therapy systems are being investigated for monitoring and assistance purposes. These home rehabilitation systems are advantageous not only because they are less costly for the patients and for the health care systems, but also because having it at home and regularly available, the users tend to do more exercise. A well accepted sensing technology for these purposes are RGB-D sensors (e.g. Kinect™ of

Microsoft®) that are affordable and versatile, allowing to capture in real-time colour and depth information.

[0004] Existing systems and research either (1) combine exercises with video games as a means to educate and train people, while keeping a high level of motivation; or (2) try to emulate a physical therapy session. These works usually involve the detection, recognition and analysis of specific motions and actions performed. Very recent works tackle the problem of assessing how well the people perform certain actions, which can be used in rehabilitation e.g. to evaluate mobility and measure the risk of relapse.

[0005] The scientific publication of Pirsiavash, H., Vondrick, C., Torralba, A.: “Assessing the quality of actions” in Computer Vision-ECCV 2014, pp. 556-571, Springer (2014), uses computer vision, i.e. videos, for assessing the quality of movements of persons practising sport. The approach followed in that document is based on a regression model that correlates spatiotemporal pose features of the body with scores obtained from expert judges. More specifically, the pose of the person is obtained in every frame. The body joint positions are computed as vectors. In the regression model, the observed videos are compared with reference videos showing perfect execution. In addition to provide a quality assessment, a feedback is also provided to the performer by differentiating a scoring function with joint location. More specifically, the gradient of the scoring function with respect to the location of each joint is computed in order to compute the movements of the joints that maximizes the score. Feedback vectors are also computed and superposed on specific joint(s) on the video for providing an improvement incentive to the performer. The corrective feedback is also analysed per joint, which involves a complex set of instructions for suggesting a particular body-part motion.

[0006] In the medical community, the publication of Ofli, F., Kurillo, G., Obdrz'alek, S., Bajcsy, R., Jimison, H.B., Pavel, M.: “Design and evaluation of an interactive exercise coaching system for older adults: Lessons learned”, IEEE J. Biomedical and Health Informatics (2016), provides assistive feedback during the performance of exercises. For each particular movement, they define constraints such as keeping hands close to each other or maintaining the torso in an upright position. These constraints are constantly measured during the exercise for assessing if the movement is performed correctly and in case pre-defined values for metrics on these constraints are violated, then corrective feedback is provided. The motion constraints are however action specific and manually defined.

Summary of invention

Technical Problem [0007] The invention has for technical problem to alleviate at least one drawback of the above mentioned prior art. More specifically, the invention has for technical problem to provide an improved feedback to persons practising physical exercise.

Technical solution [0008] The invention is directed to a method for analysing the position of a human body using a computing device, comprising the following steps: (a) capturing, using image capturing means of said device, at least one frame comprising at least a portion of a human body, said portion comprising joints of articulation of said body; (b) detecting on the at least one captured image frame the position of the joints of the at least one portion of the body; (c) registering the detected positions and predefined reference positions of the joints, said predefined reference positions being pre-stored in a memory element, in a common reference system; (d) computing a displacement of at least one of the joints that minimizes an error between the detected position of the joints, and the corresponding predefined reference position of said joints; (e) using output means of said computing device, providing an indication of said displacement as an output; wherein in step (d) the computed displacement is at least one rotation of one of the at least one body part of the at least one portion of the human body, each of said at least one body part comprising at least two of the joints.

[0009] According to a preferred embodiment, in step (d) the at least one portion of the human body is split into a set of N body-parts B = {b1,..., bk,..., bN} where each body-part bk is composed by nk joints where bk = [bï.....(¾.

[0010] According to a preferred embodiment, in step (d) the at least one body part whose rotation is computed is/are the one among the body parts whose rotation causes the smallest error for said body part.

[0011] According to a preferred embodiment, in step (d) the computed displacement is a sequence of rotations of the at least one rotation, said rotations being sorted digressively according to the impact of each rotation on the error for the corresponding body part, the rotation with the highest error reduction being the first one of said sequence.

[0012] According to a preferred embodiment, the error for a body part is based on the Euclidian distance between the joints of said body part and the predefined reference positions of said joints.

[0013] According to a preferred embodiment, the Euclidian distance mk for a body part bk is computed as follows mk = Σ”=ι||^/ - bk\\2 where the predefined reference positions are B = {B\..., Bk, for the set of N body-parts B.

[0014] According to a preferred embodiment, the sequence of rotations is computed as follows R = [Rv..., fy,..., RN} where is a rotation Rk for each body part bk that minimizes the error ek(Rk) = %jti\\Rkbk - bf ||2.

[0015] According to a preferred embodiment, the sorting of the rotations is based on iteratively selecting the body part bk that maximizes the cost ck = mk-ek(Rk) where in each iteration i the body part bk selected in the previous i - 1 iterations are not taken into account.

[0016] According to a preferred embodiment, in step (e) the indication of the displacement comprises at least one feedback vector illustrating the at least one rotation of the body-part(s).

[0017] According to a preferred embodiment, each of the at least one feedback vector is anchored to the corresponding body-part and/or to a spatial centroid of the corresponding body-part.

[0018] According to a preferred embodiment, each of the at least one feedback vector fk is calculated as follows fk = Rkck — ck where ck is a vector of a centroid of the body part bk.

[0019] According to a preferred embodiment, in step (e) the indication of the displacement comprises visual and/or oral feedback information describing the displacement and identifying the corresponding body parts.

[0020] According to a preferred embodiment, the feedback information indicating the displacement is based on the at least one feedback vector.

[0021] According to a preferred embodiment, in step (e) the Cartesian coordinates of the at least one feedback vector are analysed to identify the coordinate showing the highest magnitude, the feedback information indicating the displacement being determined by the direction and/or the sign of said coordinate.

[0022] According to a preferred embodiment, the feedback information indicating the displacement comprises expressions that correspond to at least one of a right, left, forward, backward, downward and/or upward displacement.

[0023] According to a preferred embodiment, steps (a) to (e) are executed in an iterative manner for different successive frames.

[0024] The invention is also directed to a computer program comprising instructions that are executable by a computer, wherein the instructions are configured for executing the steps of the method according to the invention when running on said computer.

[0025] The invention is also directed to a computing device with a storage medium for a computer program, wherein the storage medium comprises a computer program according to the invention.

[0026] According to a preferred embodiment, the computing device further comprises: the image capturing means; the output means; a computer connected with the image capture means and the output means for computing the feedback to the user.

[0027] According to a preferred embodiment, the output means comprises a visual display device and/or a sound output device.

Advantages of the invention [0028] The invention is particularly interesting in that it does not compute feedback for single joints, but rather rotations for body-parts, defined as configurations of skeleton joints that may or may not move rigidly. Such a computation is less complex and requires less computer resources than some solutions of the prior art.

[0029] Also, feedback proposals are automatically computed by comparing the movement being performed with a template action, without specifying pose constraints of joint configurations.

[0030] In addition, feedback instructions are not only presented visually, but also human interpretable feedback can be proposed from discretized spatial transformations that can be suggested to the user using, for example, audio messages.

Brief description of the drawings [0031 ] Figure 1 illustrates a schematic representation of the skeleton of a performer body composed of a series of joints.

[0032] Figures 2 and 3 illustrates different body-parts of the skeleton of figure 1.

[0033] Figure 4 illustrates a first example of a feedback proposal according to the invention.

[0034] Figure 5 illustrates a second example of a feedback proposal according to the invention.

[0035] Figure 6 illustrates two successive feedback proposals for the movement illustrated in figure 4.

[0036] Figure 7 illustrates a template pose, the effective poses of two different subjects, and the best poses of these subjects that minimize that relative error.

Description of an embodiment [0037] Figure 1 illustrates in a schematic way the skeleton of a subject that intends to perform a physical movement to achieve a given body pose. The skeleton comprises a series of joints, for instance 21 joints, interconnected by skeleton sections that are considered rigid.

[0038] Let 5 = {j\,... ,jn,... ,jN] denotes a skeleton with N joints, where each joint is given by its 3D coordinates j = \jxJy,jz]T. We define an action or movement as being a skeleton sequence M = [S1(...,Sr, ...,SF], where F is the number of frames of the sequence. Given a template skeleton sequence U and a subject performing a movement M, it will be provided, at each time instant, feedback proposals such that the movement can be iteratively improved to better match M.

[0039] As a first step, pre-processing on the input skeleton data is achieved. Existent approaches were previously introduced in the literature (e.g. Vemulapalli, R., Arrate, F., Chellappa, R.: Human action recognition by representing 3d skeletons as points in a lie group. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2014)), and are adapted for the present invention.

[0040] A first requirement for comparing two skeletal sequences is that they need to be spatially registered. This is achieved by transforming the joints of each skeleton 5 such that the world coordinate system is placed at the hip center, and the projection of the vector from the left hip to the right hip onto the x-y plan is parallel to the x-axis. Then, for achieving invariance to absolute locations, the skeletons in M are normalized such that the body-part lengths match the corresponding part lengths of the skeletons in A?. This is performed without modifying the joint angles.

[0041] Different subjects, or the same subject at different times, perform a particular action or movement at different rates. In order to handle rate variations and mitigate the temporal misalignment of time series, Dynamic Time Warping (DTW) can be usually employed (e.g. Rabiner, L, Juang, B.H.: Fundamentals of speech recognition. Prentice hall (1993)). In the present case, it is sought to align a given sequence M with a template sequence M. The template sequence M can be aligned with respect to M, or vice-versa. It is assumed that the subject is trying to replicate the same action as M, and given M, it is sought to provide feedback proposals. Since a feedback proposal is to be computed for each temporal instant of M, it is reasonable to compute the temporal correspondences of M with respect to M.

[0042] After the spatial and temporal alignment processing described in the previous section, the skeleton instance Sf in A? will be in correspondence with Sf in M. This section explains how to compute the body motion required to align corresponding body-parts of aligned skeletons 5 and 5, and discloses a method for extracting human-interpretable feedback from these transformations.

[0043] With reference to figure 2, the human motion is analysed using a body-part based representation of the subject. The skeleton 5 is represented by a set of body-parts B = {b1,...,bk,..., bN). Each body part bk is composed by nk joints bk = {bk, -,bkk} and has a local reference system defined by the joint bk. In figure 2, the skeleton comprises 12 body-parts blt...,b12 being the right forearm, the left forearm, the back, the right arm, the left arm, the right leg, the left leg, the torso, the upper body, the lower body, the full upper body and the full body. It is however understood that other definitions of the body-parts can be considered.

[0044] Given the aligned skeletons S and 5, the objective is to compute the motion that each body-part of 5 needs to undergo to better match the template skeleton. This analysis is performed for each body-part using the corresponding local coordinate system. As a metric for measuring how similar is the pose of corresponding body-parts, we use the Euclidean distance as the scoring function. Following this, the error between bk and bk is given by:

[0045] It is to be noticed that \\bk -bk|| = 0, because the previous computation is performed using the local coordinate systems that are assumed to be in correspondence.

[0046] For providing feedback to the performer of skeleton 5 on how the movement can be improved to better match 5, we compute the transformation that each body-part bk needs to undergo for decreasing the scoring function mk. We anchor the reference joints bk and bk of the corresponding body-parts. The aim is then to compute the rotation Rk e 50(3) that minimizes the following error:

which can be computed in closed form.

It is important to refer that since the human motion is articulated, depending on the movement being performed, a given body-part bk may or may not move rigidly. This is not a critical issue because body-parts that do not moving rigidly have high joint matching error and will be considered not relevant by the method described next. Note that different body-parts bk can contain subsets of the same joints, which implies that the transformation Rk will also have impact on the location of the other body-parts bl*k. Taking this into account, we want to compute a sequence of transformations R = [Rlt...,Ri,...,Rn}, one rotation Rt = Rk for each body-part bk, such that the first rotation R± has the highest decrease in the joint location error until RN, which has the lowest impact in the human pose matching. This sorting is performed maximizing the following cost:

where in iteration i, the body-partsôfe selected in the previous i - 1 iterations are not taken into account.

[0047] Here is the pseudo code for computing the above:

[0048] The rotations = Rk correspond to the motion required for the best alignment of bk and bk. However, it might be difficult to present this rigid-body transformation as feedback proposals on, for example, a screen. For overcoming this, we can compute feedback vectors for suggesting improvements on the motion. For each body-part, we can pre-calculate the spatial centroid cfc(note that in case of single limbs, this point is located on the body-part itself). Then, the feedback vector anchored to ck is defined as

[0049] Figure 4 illustrates a feedback proposal for a reaching a waving target pose S. The left images are two views of the target pose 5 that is for instance a waving pose. The central images are two corresponding views of the actual pose S of the subject, for instance a clapping pose. The right images are the superposition of the left and central views and illustrate the feedback vector fk anchored to the body-part being the right arm (corresponding to the body-part ó4 as illustrated in figure 2). The vector suggest the subject to move the right arm upwardly in order to reach the clapping target pose S. Here only the first feedback proposal is shown.

[0050] Figure 5 illustrates another feedback proposal for a reaching another target pose 5. Similarly to figure 4, the left images are two views of the target pose 5 that is for instance a standing pose. The central images are two corresponding views of the actual pose 5 of the subject, for instance a bending pose. The right images are the superposition of the left and central views and illustrate the feedback vector fk anchored to the body-part being the torso (corresponding to the body-part be as illustrated in figure 2). The vector suggest the subject to move the right arm upwardly in order to reach the clapping target pose S. The vector suggest the subject to move the torso backwards and upwards in order to reach the standing target pose S. Similarly to figure 4, here only the first feedback proposal R1 is shown.

[0051] Not all the persons have the same spatial awareness to realize how to perform the motion suggested by the feedback vector fk as discussed above and illustrated in figures 4 and 5. This difficulty is even more evident in cognitive impaired individuals. In order to support the patient in improving their movements, simple human-interpretable feedback messages can be shown or/and spoken to the patient by the computer system.

[0052] Let us analyse the case of the body-part bk that needs to undergo the largest motion ^ = Rk. Initially, to each bk was assigned a body-part name BN, e.g. ^ is the right forearm and b8 is the torso (see Fig. 2). These labels are used directly for informing the user which body-parts should be moved. Then, the feedback vector fk = [fk,fk,fz] is discretized by selecting the dimension c/with highest magnitude |/d|. The messages regarding the direction of the motion BD are then defined as:

The feedback proposal messages are represented as the concatenation of strings: Feedback message := ’’Move" + BN + BD.

[0053] Figure 6 illustrate feedback proposals comprising a vector representation and a message according to the above. Here first and second feedback proposals Rx and R2 are shown. The two images on the left illustrate a waving target pose S and an actual clapping pose S of the subject. The two images in the rights illustrate a superposition of the target and actual poses with a feedback 1 and a feedback 2, respectively. The feedback 1 comprises a vector anchored on the right arm and oriented upwards (similarly to figure 4) and a message reading “Move Right Arm up”. The feedback 2 comprises a vector anchored to the left arm and oriented also generally upwards and a message reading “Move Left Arm up”. A colour coding can be used for identifying the directions BD.

[0054] In this section, we experimentally evaluate the proposed system using data captured using the Kinect™ version 2. The idea is to simulate a person who suffered a stroke: the bad arm issue due to the paralysis of an upper limb is simulated by lifting a kettle-bell using one of the arms, and the balance problem is replicated using a balance ball. The objective in this section is to simulate a simple physiotherapy session at home, and test if the feedback proposals are able to guide the user. We assume that a person needs to perform a template human pose S. The subject puts himself above the balance ball and lifts the kettle-bell. Giving only the guidance of the feedback vectors, body-part motion intensity and feedback messages, the objective is to converge to the template pose without actually seeing it. The exercise lasts for 20 seconds and feedback proposals are shown at each time instant.

[0055] The experimental results are shown in Figure 7. Part (a) show two views of the template pose 5. Parts (b) and (c) show a first pose St and the best pose SBest for two subjects. The best pose SBest is the one that minimizes the error m12 for the body-part b12 (i.e. the whole body, see figure 2). Parts (d) and (e) show the relative error (difference between initial and current error divided by the initial error) in % for the body-part b12.

Claims

A method of analyzing the position of a human body using a computing device, comprising the steps of: (a) capturing, using image capturing means of said device, at least one frame comprising at least a portion of a human body, said portion comprising joints of said body; (b) detecting on the at least one captured picture frame the position of the joints of the at least a portion of the body; (c) registering the detected positions and predefined reference positions of the joints, said predefined reference positions being pre-stored in a memory element, in a common reference system; (d) calculating a displacement of at least one of the joints which minimizes the error between the detected position of the joints and the corresponding predefined reference position of said joints; (e) using output means of said computing device to provide an indication of said displacement as an output; characterized in that in step (d), the calculated displacement is at least one rotation of one of the at least one body portion of the at least one portion of the human body, each of said at least one body portion comprising at least two of the joints.

2. The method according to claim 1, wherein in step (d), the at least one portion of the human body is divided into a set of N body parts B = {b1, ..., bN} where each part of body bk is composed of nk articulations where bk = [bf, ..., bkk}.

3. Method according to one of claims 1 and 2, wherein in step (d) the at least one body part whose rotation is calculated is / are one of the body parts whose rotation causes the smallest error for said body part.

4. Method according to any one of claims 1 to 3, wherein in step (d), the calculated displacement is a sequence of rotations of the at least one rotation, said rotations being sorted degressively as a function of I the impact of each rotation on Terror for the corresponding body part, the rotation with the greatest error reduction being the first of said sequence.

The method of claim 2 and one of claims 3 and 4, wherein Terror for a body portion is based on the Euclidean distance between the joints of said body portion and the predefined reference positions of said joints.

The method according to claim 5, wherein the Euclidean distance mk for a body part bk is computed as follows mk = Σ "= ι || & / - bf || 2 ού the predefined reference positions are B = [ b1, ..., bk, ..., bN} for all N body parts B.

The method of claim 4 and one of claims 5 and 6, wherein the sequence of rotations is calculated as follows R = R1, R1,..., RN, Rt is a rotation Rk for each part. of body bk which minimizes Terror ek (Rk) = 2f = 1 \\ Rkbk-bk \\ 2.

The method of claim 7, wherein the sorting of the rotations is based on an iterative selection of the body part bk which maximizes the cost cf = mk-ek (Rk) or at each iteration i the selected body portions bk in the previous iterations i - 1 are not taken into account.

9. A method as claimed in any one of claims 1 to 8, wherein in step (e) the displacement representation comprises at least one feedback vector illustrating the at least one rotation of the at least one part (s). of body.

The method of claim 9, wherein each of the at least one feedback vector is anchored to the corresponding body part and / or to a spatial centroid of the corresponding body part.

11. Method according to one of claims 7 and 8 and according to one of claims 9 and 10, wherein each of the at least one feedback vector fk is calculated as follows fk = Rkck - ck ού ck is a vector of a centroid of the body part bk.

12. A method according to any one of claims 1 to 11, wherein in step (e) the displacement indication comprises visual and / or verbal feedback information describing the displacement and identifying the portions of the corresponding bodies.

13. The method of any one of claims 9 to 11, and claim 12, wherein the feedback information describing the displacement is based on the at least one feedback vector.

14. Method according to claim 13, characterized in that, in step (e), the Cartesian coordinates of the at least one feedback vector are analyzed to identify the coordinate showing the greatest amplitude, the feedback information. describing the displacement being determined by the direction and / or sign of said coordinate.

15. The method of claim 14, wherein the feedback information describing the motion includes expressions that correspond to at least one of right, left, forward, backward, downward and / or upward motion.

16. A method according to any one of claims 1 to 15, wherein steps (a) and (e) are performed iteratively for different successive frames.

A computer program comprising computer executable instructions, characterized in that the instructions are configured to perform the steps of the method according to any one of claims 1 to 16 when performed on said computer.

Computer device with a storage medium for a computer program, characterized in that the storage medium comprises a computer program according to claim 17.

The computing device of claim 18, wherein said device further comprises: the image capturing means; the output means; a computer connected to the image capturing means and the output means for calculating the feedback to the user.

20. The computing device according to claim 19, characterized in that the output means comprises a visual display device and / or a sound output device.