WO2017158569A1

WO2017158569A1 - Kinect based balance analysis using single leg stance (sls) exercise

Info

Publication number: WO2017158569A1
Application number: PCT/IB2017/051567
Authority: WO
Inventors: Kingshuk CHAKRAVARTY; Brojeshwar Bhowmick; Aniruddha Sinha; Suraj SUMAN; Abhijit Das
Original assignee: Tata Consultancy Services Limited
Priority date: 2016-03-18
Filing date: 2017-03-17
Publication date: 2017-09-21

Abstract

Synchronized and coordinated activation of the postural muscles of the trunk and lower limbs is required for maintaining equilibrium and balance in human body. Poor postural balance control causes injury or falls in huge population and is supposed to be a critical factor of common motor skills. Single Leg Stance (SLS) is a good option for measuring postural control in any stance, which not only assesses postural steadiness in a static position by a temporal measurement (SLS-duration) but also analyses the role of body skeleton joints in postural stability and correction. This method provides a quick, reliable and easy way to screen their patients for fall risks and is easily incorporated into a comprehensive functional evaluation for older adults. An automatic unobtrusive system is proposed here to measure SLS duration and body balance. For this purpose, vibration-jitter analysis is performed which gives a clear view of relative variation of frequency of different skeleton joints over time. The whole processing is done on the dataset of skeleton joints obtained from Kinect. The dataset of skeleton joints through the Kinect setup, is used instead of video, which addresses user privacy concerns.

Description

KINECT BASED BALANCE ANALYSIS USING SINGLE LEG STANCE (SLS)

EXERCISE

CROSS REFERENCE TO RELATED APPLICATIONS AND PRIORITY

[0001] The present invention claims priority to Indian Provisional specification (Title: System and method for retail pricing within product linkages) No. 201621009600 filed in India on March 18, 2016.

TECHNICAL FIELD

[0002] The present application generally refers to an unobtrusive system to compute for a human subject, Single Leg Stance (SLS) duration and analyze body vibration in terms of 3D movements of skeleton joints using skeleton joint data obtained from Kinect, which is a motion sensing input device generally used for game consoles. This computation and analysis will aid medical practitioners to analyze crucial factors for fall risk minimization, injury prevention, and prescribe fitness and rehabilitation programs for the human subject.

BACKGROUND

[0003] There are a few related prior arts that suggest similar form of patent literature. The most relevant references were reviewed carefully to determine the Novelty, Non obviousness and Industrial Applicability of the present invention.

[0004] Balance in Single Leg Stance (SLS) method has so far been assessed in terms of both SLS duration and body sway which can be measured by center of pressure (COP) movement registered using stabilometry with force platforms. The COP reflects both the horizontal location of the center of gravity and the reaction forces due to muscular activity but does not inform about how postural perturbation creates instability and oscillation in body parts and skeleton joints. Some works are reported where only amplitude of center of pressure (COP) movement is recorded but frequency associated with each joint vibration is omitted where a clear relationship exists between the oscillation of COP and COM (center of mass). Certain marker-based motion capture and analysis systems have been used for body sway measurement which is expensive, complex and the test can only be performed in the lab or clinic environment.

[0005] In one prior art a method and syste™ i¾ inscribed for analyzing movements of main body parts of a moving person, the metfr ising the steps of attaching one or more sensors to selected main body parts, each sensor comprising means for wireless communication of data, calibrating data from the one or more sensors, and mapping the calibrated sensor data to a virtual 3D avatar. This method is, however obtrusive in nature as it is based on wearable sensor based technology, whereas the proposed solution is completely unobtrusive.

[0006] In another prior art, a kinetic rehabilitation system is presented, comprising a kinetic sensor comprising a motion- sensing camera; and a computing device comprising: (a) a non- transient memory comprising a stored set of values of rehabilitative gestures each defined by a time series of spatial relations between a plurality of theoretical skeleton joints, and wherein each time series comprises: initial spatial relations, mid-gesture spatial relations and final spatial relations, and (b) a hardware processor configured to continuously receive a recorded time series of frames from said motion-sensing camera, wherein each frame comprises a three-dimensional position of each of a plurality of skeleton joints of a patient, wherein said hardware processor is further configured to compare, in real time, at least a portion of the recorded time series of frames with the time series of spatial relations, to detect a rehabilitative gesture performed by the patient. This invention is about Kinect based posture/gesture recognition. However in our proposed system, data obtained for a plurality of skeleton joints from Kinect is used to implement Single Leg Stance (SLS) exercise where SLS is monitored to analyze human body balance.

[0007] In another prior art a kinetic rehabilitation system is presented, comprising a kinetic sensor comprising a motion- sensing camera; and a computing device comprising: (a) a communication module; (b) a non-transient memory comprising a stored set of values of rehabilitative gestures each defined by a time series of spatial relations between a plurality of theoretical skeleton joints, and wherein each time series comprises: initial spatial relations, mid-gesture spatial relations and final spatial relations, and (c) a hardware processor configured to: (i) continuously receive a recorded time series of frames from said motion- sensing camera, wherein each frame comprises a three-dimensional position of each of a plurality of skeleton joints of a patient, (ii) compare, in real time, at least a portion of the recorded time series of frames with the time series of spatial relations, to detect a rehabilitative gesture performed by the patient, (iii) detect a discrepancy between the rehabilitative gesture performed by the patient and a corresponding one of said stored set of values of rehabilitative gestures, (iv) log data pertaining to said discrepancy and to said rehabilitative gesture performed by said patient, (v) send said data to a therapist via said communication module and provide a report to said therapist. Though it is related to Kinect sensor based rehab system, but still it is not related to human static balance analysis using skeleton joint data obtained from Kinect.

SUMMARY

[0008] The following presents a simplified summary of some embodiments of the disclosure in order to provide a basic understanding of the embodiments. This summary is not an extensive overview of the embodiments. It is not intended to identify key/critical elements of the embodiments or to delineate the scope of the embodiments. Its sole purpose is to present some embodiments in a simplified form as a prelude to the more detailed description that is presented below.

[0009] The proposed system involves an unobtrusive system to compute the SLS duration and body vibration in terms of 3D movements of a plurality of skeleton joints using only skeleton joint data obtained from Kinect. This system is focused on balance analysis during SLS exercise. The SLS duration, which is time span during which a human subject stands on single leg needs to be evaluated. For this a new eigenvector based curvature analysis method is proposed which performs better than previous state-of-art reported method. This fact is verified by Bland-Altman plot. This process divides the whole signal into three segments i.e. before, during and after SLS.

[0010] In one embodiment, each time frame segment is further analyzed to quantify the postural balance. The skeleton joint data obtained from Kinect is filtered using bandpass filter in order to overcome the effect of noise. In this embodiment the velocity profile of each joint in each time frame segment is subdivided into 50 samples window and Fourier transform of each window is evaluated. The frequency corresponding to maximum amplitude in each window is used to obtain the mean frequency of each time frame segment separately. From here the vibration jitter i.e. relative variation of frequency for each joint are evaluated which indicate that the jitter during SLS is maximum than the other two.

[0011] In order to quantify the force imposed on each joint, Force per unit mass (FPUM) parameter is evaluated which provides a measure of force during each section in different joint, thus enabling a quantification of the vibration and force imposed on each joint.

BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The embodiments herein will be better understood from the following detailed description with reference to the drawings, in which:

[0013] FIG 1 is a block diagram of the system for analyzing balance in Single Leg Stance using Kinect according to an embodiment of the present disclosure;

[0014] FIG 2 is a representative illustration of the proposed methodology of analyzing balance in Single Leg Stance using Kinect according to an embodiment of the present disclosure;

[0015] FIG 3 depicts the skeleton joint data capture setup showing the relative placement of the Kinect apparatus with relation to a human subject, and also illustrating a plurality of skeleton joints identified, whose coordinates are tracked by the system, according to an embodiment of the present disclosure;

[0016] FIG 4 depicts a timeframe graph tracking the Y coordinate of the left ankle of the human subject, according to an embodiment of the present disclosure;

[0017] FIG 5 depicts the curvature points identification method using variations in Y coordinates of the left ankle of the human subject, according to an embodiment of the present disclosure;

[0018] FIG 6 is a graphical representation of body vibration analysis for the human subject; FIG 6(a) being Jitter for 3 skeleton joints for 3 timeframe segments; and FIG 6(b) being FPUM for the same 3 skeleton joints for the similar 3 timeframe segments according to an embodiment of the present disclosure;

DETAILED DESCRIPTION OF EMBODIMENTS

[0019] The embodiments herein and the various features and advantageous details thereof are explained more fully with reference to the non-limiting embodiments that are illustrated in the accompanying drawings and detailed in the following description. The examples used herein are intended merely to facilitate an understanding of ways in which the embodiments herein may be practiced and to further enable those of skill in the art to practice the embodiments herein. Accordingly, the examples should not be construed as limiting the scope of the embodiments herein.

[0020] The words "comprising," "having," "containing," and "including," and other forms thereof, are intended to be equivalent in meaning and be open ended in that an item or items following any one of these words is not meant to be an exhaustive listing of such item or items, or meant to be limited to only the listed item or items.

[0021] It must also be noted that as used herein and in the appended claims, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. Although any systems and methods similar or equivalent to those described herein can be used in the practice or testing of embodiments of the present disclosure, the preferred, systems and methods are now described.

[0022] Some embodiments of this disclosure, illustrating all its features, will now be discussed in detail. The disclosed embodiments are merely exemplary of the disclosure, which may be embodied in various forms.

[0023] Before setting forth the detailed explanation, it is noted that all of the discussion below, regardless of the particular implementation being described, is exemplary in nature, rather than limiting. For example, although selected aspects, features, or components of the implementations are depicted as being stored in memories, all or part of the systems and methods consistent with the Kinect based balance analysis system and method may be stored on, distributed across, or read from other machine-readable media.

[0024] FIG. 1 is a block diagram of a Kinect based balance analysis system 100 according to an embodiment of the present disclosure. The Kinect based balance analysis system 100 comprises a memory 102, a hardware processor 104, and an input/output (I/O) interface 106. The memory 102 further includes one or more modules 108 (or modules 108). The memory 102, the hardware processor 104, the input/output (I/O) interface 106, and/or the modules 108 may be coupled by a system bus or a similar mechanism.

[0025] The memory 102, may store instructions, any number of pieces of information, and data, used by a computer system, for example Kinect based balance analysis system 100 to implement the functions (or embodiments) of the present disclosure. The memory 102 may include for example, volatile memory and/or non-volatile memory. Examples of volatile memory may include, but are not limited to volatile random access memory (RAM). The non-volatile memory may additionally or alternatively comprise an electrically erasable programmable read only memory (EEPROM), flash memory, hard drive, or the like. Some examples of the volatile memory includes, but are not limited to, random access memory, dynamic random access memory, static random access memory, and the like. Some example of the non-volatile memory includes, but are not limited to, hard disks, magnetic tapes, optical disks, programmable read only memory, erasable programmable read only memory, electrically erasable programmable read only memory, flash memory, and the like. The memory 102 may be configured to store information, data, applications, instructions or the like for enabling the Kinect based balance analysis system and to carry out various functions in accordance with various example embodiments.

[0026] The hardware processor 104 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and/or any devices that manipulate signals based on operational instructions. Further, the hardware processor 104 may comprise a multi-core architecture. Among other capabilities, the hardware processor 104 is configured to fetch and execute computer-readable instructions or modules stored in the memory 102.

[0027] FIG 2 is a representative system diagram 100 for the Kinect based balance analysis system using Single Leg Stance (SLS) Exercise according to an embodiment of the present disclosure wherein skeleton joint data captured by Kinect in the form of (x, y, z) coordinates of the 20 skeleton joints of the human body are received by a data capturing module (200) of the system and directed toward the SLS duration measurement module (260) and vibration analysis module (270). The SLS Duration Measurement module (260) comprises of two sub modules. One sub-module (210) uses the said input skeleton joint data from Kinect and tracks the variation in coordinates of a lifted leg's ankle in the y-plane which is the vertical plane. This coordinate variation data produced by this submodule 210 is received by the 2^nd sub module 220 which performs an eigenvector based curvature analysis to compute SLS duration. The SLS duration data is utilized in performing the vibration analysis of each joint.

[0028] Balance is considered to be a crucial factor for fall risk minimization, injury prevention, fitness and rehabilitation programs. Poor postural or balance control causes injury or falls in huge population and supposed to be critical factor of common motor skills.

[0029] The Single Leg Stance (SLS) is a means of measuring postural control, which assesses postural steadiness in a static position and also analyses the role of the plurality of skeleton joints in postural corrections.

[0030] In an embodiment of the method, as illustrated in the setup in FIG 3, SLS exercise is performed by a human subject (310) with eyes open and arms on the hips while the human subject (310) must stand unassisted on one leg 7 to 8 feet distance away from Kinect (300) placed 4.5 feet high from the floor level, and is timed in seconds from the time one foot is flexed off the floor to the time when the foot touches the floor level or the standing leg or an arm leaves the hips. SLS test measures postural stability (i.e., balance) as it is more difficult to stand on single leg due to the narrow base of support required to do the test. For clinicians, it provides a quick, reliable and easy way to screen their patients and clients for fall risks and is easily incorporated into a comprehensive functional evaluation for older adults. For patients suffering with Parkinson's disease, a SLS cut-off time of 10 seconds provides the highest sensitivity and specificity for history of one or more falls.

[0031] Skeleton joint data obtained from Kinect is noisy even when the human subject stands static. Among the plurality of skeleton joints, certain skeleton joints are always moving in small amounts when the human subject is static and this contributes to the noise in skeleton joint data obtained from Kinect. Additionally, room lighting, infra-red interference and quantization noise add to the noise of skeleton joint data obtained from Kinect. The noisy skeleton joint data of the plurality of skeleton joints is filtered and cleaned using Euclidean distance based outlier detection algorithm. It uses the fact that mutual Euclidean distance between any plurality of physical skeleton joints should not vary with time i.e. static features should be constant over all frames. A clustering approach is used to detect the outlier frames, where the static features of the frames are partitioned into two clusters - one containing the good frames and the other containing the outlier frames.

[0032] An algorithm to calculate the SLS duration, which is the duration during which the human subject stands on one leg, using eigenvector based curvature analysis method, is proposed. The vibration for the plurality of skeleton joints are measured in terms of frequency variation i.e. vibration-jitter and force per unit mass (FPUM).

[0033] For SLS duration measurement, in an embodiment, the variation in coordinates of the ankle of the lifted leg of the human subject, during SLS is measured. The 3D spatio-temporal information of a plurality of skeleton joints obtained from Kinect are 3D coordinates (x,y,z) where x represents left and right variation, y represents up and down variation in relation to the floor level and z represents to and from variation of the plurality of skeleton joints of the human subject with respect to Kinect. Hence in this embodiment, changes in the lifted leg's ankle coordinate can give meaningful information about the precise timing when the human subject lifts the leg above the floor level. In an embodiment of this method the left leg is lifted, although the method is valid for both the legs. K-means clustering algorithm is used to capture the variation in left-ankles displacement in Y-plane with time. It will help differentiate one leg stance portion. In FIG 5, the three segments i.e. S-to-A (time frame segment 1) (510), A-to-B (time frame segment 2) (520) and B-to-E (time frame segment 3) (530) are obtained using the proposed method.

Where

- i if is a Euclidean distance between a data point ¹ and the cluster center ,. Frames belonging to R-to-F will form one cluster, whereas rest will group into another one, as O is the indicator of the distance of the N data points from their respective cluster centers. FIG 5 shows output of k-means algorithm i.e. frame A and B which are far away from our desired frames R and F. Consider data points X in region S-A. Curvature point will lie in the direction of minimum variance of data. Covariance matrix XX of mean subtracted data X and eigenvalue decomposition of the matrix is computed. This is the principle behind Principle Component Analysis (PC A) [18] to find the direction of maximum variance. The eigenvector (say, E_min) corresponding to least eigenvalue provides the direction of minimum variance of the skeleton joint data and so reveals the direction towards curvature points. The curvature points R and F are obtained through minimum projection error of the eigenvector corresponding to smallest eigenvalue using the following equation:

argmin_r [P_r — (P_r. u)u\

where P_r is the original signal value at frame r and ύ is the unit vector along E_min. Finally SLS duration is measured by finding difference between timestamps corresponding R and F frames.

[0034] During SLS, exercise while standing on single leg, the human subject swings/oscillates in order to maintain/preserve the balance. For any posture of the human body, the human subject cannot move with ease and flexibility certain skeleton joints for example HipCenter, ShoulderCenter. Hence, the plurality of skeleton joints (320) have different degrees of freedom, for example it is high for hand but low for HipCenter.

[0035] To measure the oscillation/swing quantitatively, velocity profile of each skeleton joint of the plurality of skeleton joints is used for vibration analysis. Velocity is the rate change of displacement in respective directions x, y and z. Vibration is composed of frequency and amplitude. Higher frequency indicates more vibration and less balance. Velocity is obtained from the filtered skeleton joint data as follows:

Vel = [v_x ^J, v_y ^], v_z ^J] = ^ [x^j, y^j, z>]

Where x , y, i is the displacement in (x, y, z) direction respectively for j^th skeleton joint. It is observed from the AnkleLeft's velocity profile that velocity is maximum near R and minimum (considering sign) near F. Also the mean velocity of AnkleLeft in first time frame segment S-to-R (FIG 4) is almost similar to the third one, whereas the velocity in the second is much higher than the other two. Start (R) and end (F) frames/time of one leg stance posture have already been identified in the subsection 2.3. Hence, three different segments namely S- to-R (time frame segment- 1), R-to-F (time frame segment-2) and F-to-E (time frame segment-3) are analyzed separately. This fact is also true for all the 20 skeleton joints in this embodiment. To get the information about frequency, every skeleton joint data in each time frame segment is partitioned into a window of 50 samples in an embodiment, and Fourier transform of each time frame segment is evaluated as follows:

where ν^ (ω) is the frequency response of the i^th window for the j^th joint velocity v_k ^J . In this embodiment the above Fourier transform is carried out for all the 20 skeleton joints and in all three directions (x, y, z). Frequency (f^ ) corresponding to the maximum amplitude (A^J _k) in each window is selected and the mean frequency of each time frame segment is evaluated as follows: fj _ ∑ ^jfk where, is the frequency corresponding to maximum amplitude in the k^th window for j^th joint.

[0036] Using the above equation, mean frequencies f^ l -to-F, fm \R-to-F, fm \F-to-E in each time frame segment are computed. These calculated mean frequencies allow analysis of relative frequency variation (vibration) in corresponding segments i.e. before, during and after SLS. Jitter in each time frame segment is the difference of frequencies and the mean frequency given as following,

where A_;2,3 is vibration jitter and is mean frequency in each time frame segment whereas f _k is the frequency for all windows in each time frame segment. This jitter vector A_;2,3 quantifies vibration in terms of frequency for 3 segments, as more the variation in the values of jitter, more the vibration and worse the balance. The variation of jitter is shown in FIG 6(a).

[0037] The dominant component of velocity for each window can be written as v_k ^J [n] =

A^J _k cos(2nf_k ri) where f_k is the frequency corresponding to the maximum amplitude A^J _k in k^th window for j^th joint of each time frame segment.

[0038] As per Biomechanics principles it is quite evident that, the force imposed on each joint during SLS is due to body weight, abductor muscles force and joint reaction force which is required to restore the equilibrium state of the skeleton joints. This force can be a good measure for joint balance estimation. Thus the reaction force per unit mass for each joint is measured as the rate of change of velocity for that joint. This can be best explained using Newton's law of motion i.e,

F

F = ma → a = —

m where, F is the force, m is the mass and a is the acceleration. The variation of force per unit mass (FPUM) is shown in FIG 6(b).

[0039] Therefore two parameters for each joint i.e. jitter and force per unit mass (i.e. FPUM) is derived to study joint- wise body vibration in (x, y, z) plane for SLS.

Claims

WE CLAIM:

1) A hardware processor implemented method for Single Leg Stance balance analysis, the method comprising of: creating, by a data capturing module (200), dataset of skeleton joints from inputs received from a Kinect setup; measuring, by an SLS duration measurement module (260), SLS (Single Leg Stance) duration measurement from said dataset of skeleton joints; computing, by a vibration analysis module (270), velocity profile of SLS vibration; jitter of SLS vibration; force-per unit mass (FPUM) of SLS vibration; from said dataset of skeleton joints.

2) The method of claim 1, wherein the Kinect setup comprises of placing the Kinect in front of a human subject, with the human subject having bare feet, eyes open, arms on the hips looking straight ahead.

3) The method of claim 2, wherein 3D spatio-temporal information of skeleton joints of the human subject are obtained from Kinect in the form of 3D coordinates (x, y, z).

4) The method of claim 1, wherein dataset of skeleton joints obtained from Kinect, is filtered of noise using Euclidean distance based outlier detection method.

5) The method of claim 1 wherein measuring SLS duration is achieved by observing variation of coordinates in the 3 dimensions (x, y, z) of a plurality of skeleton joints namely sideways, up-down with relation to the floor level, and to and fro with relation to Kinect.

6) The method of claim 5 wherein the variation of y-component of a lifted leg joint is selected for time frame segmenting before, during and after SLS. 7) The method of claim 5 wherein eigenvector based curvature analysis is used to calculate the SLS duration from skeleton joint data obtained from Kinect.

8) The method of claim 1 wherein the velocity profile and frequency variations of joint movement is determined for each joint in the dataset of skeleton joints.

9) The method of claim 8 wherein the frequency corresponding to the maximum amplitude is considered from each time segment of before, during and after single leg stance, and the mean frequency corresponding to each time frame segment is evaluated to obtain relative frequency variations (vibration) before, during and after SLS.

10) The method of claim 9 wherein the variation in relative frequency is calculated as vibration jitter.

11) A system for Single Leg Stance balance analysis, the system comprising of: a memory storing instructions; a processor communicatively coupled to said memory, wherein said processor is configured by said instructions to: create, by a data capturing module (200), dataset of skeleton joints from inputs received from a Kinect setup; measure, by a duration measurement module (260), SLS (Single Leg Stance) duration measurement from said dataset of skeleton joints; compute, by a vibration analysis module (270), velocity profile of SLS vibration from said dataset of skeleton joints; jitter of SLS vibration from said dataset of skeleton joints; force-per unit mass (FPUM) of SLS vibration; from said dataset of skeleton joints.