WO2024020838A1

WO2024020838A1 - Apparatus, method, device and medium for dynamic balance ability evaluation

Info

Publication number: WO2024020838A1
Application number: PCT/CN2022/108182
Authority: WO
Inventors: Zhongxuan Liu; Ming Lu; Dongqi CAI; Zhigang Wang; Hao ZHAO; Xu Zhang; Yurong Chen
Original assignee: Intel Corporation
Priority date: 2022-07-27
Filing date: 2022-07-27
Publication date: 2024-02-01

Abstract

The disclosure provides an apparatus, method, device, and medium for dynamic balance ability evaluation. The apparatus includes interface circuitry configured to receive a sequence of video frames including pose information of a person; and processor circuitry coupled to the interface circuitry and configured to extract the pose information from the sequence of video frames; determine one or more pose features for each video frame based on the pose information; identify different phases of motion of the person based on the one or more pose features for each video frame; obtain a plurality of indices for the dynamic balance ability evaluation based on the one or more pose features for each video frame or the different phases of motion of the person; and evaluate the dynamic balance ability of the person based on each index and a corresponding threshold.

Description

APPARATUS, METHOD, DEVICE AND MEDIUM FOR DYNAMIC BALANCE ABILITY EVALUATION

Technical Field

Embodiments of the present disclosure generally relate to image processing, and in particular to an apparatus, method, device, and medium for dynamic balance ability evaluation.

Background Art

With ageing, physical ability declines. People’s mobility may be affected and they may experience difficulties in maintaining their independence. A large category of difficulties concern falls, which may have catastrophic outcomes to the health state of the falling people. Falls affect millions of people each year and result in significant injuries, particularly among the elderly. In fact, it has been estimated that falls are one of the top three causes of death in elderly people. A fall is defined as a sudden, uncontrolled and unintentional downward displacement of the body to the ground, followed by an impact, after which the body stays down on the ground.

Evaluating risks of falls for elderly people is important for selection of right caring and habitation methods and levels. One of the most important parts of fall risk evaluation is to evaluate people’s balance abilities. Among the balance abilities, the dynamic balance ability is more complex and is difficult to be evaluated. In order to evaluate the dynamic balance ability, a plurality of indices related to abilities of keeping balance when walking should be taken into consideration.

Summary

According to an aspect of the disclosure, an apparatus for dynamic balance ability evaluation is provided. The apparatus includes interface circuitry configured to receive a sequence of video frames including pose information of a person; and processor circuitry coupled to the interface circuitry and configured to extract the pose information of the person from the sequence of video frames; determine one or more pose features for each of the video frames based on the pose information of the person; identify different phases of motion of the person based on the one or more pose features for each of the video frames; obtain a plurality of indices for the dynamic balance ability evaluation based on the one or more pose features for each of the video frames or the different phases of motion of the person; and evaluate the dynamic balance ability of the person based on each of the plurality of indices and a corresponding threshold.

According to another aspect of the disclosure, a computer-implemented method for dynamic balance ability evaluation is provided. The method includes obtaining a sequence of video frames including pose information of a person; extracting the pose information of the person from the sequence of video frames; determining one or more pose features for each of the video frames based on the pose information of the person; identifying different phases of motion of the person based on the one or more pose features for each of the video frames; obtaining a plurality of indices for the dynamic balance ability evaluation based on the one or more pose features for each of the video frames or the different phases of motion of the person; and evaluating the dynamic balance ability of the person based on each of the plurality of indices and a corresponding threshold.

Another aspect of the disclosure provides a machine readable storage medium having instructions stored thereon, which when executed by a machine cause the machine to perform the above method for dynamic balance ability evaluation.

Another aspect of the disclosure provides a computing device including means for implementing the above method for dynamic balance ability evaluation.

Brief Description of the Drawings

In the drawings, which are not necessarily drawn to scale, like numerals may describe similar components in different views. Like numerals having different letter suffixes may represent different instances of similar components. The drawings illustrate generally, by way of example, but not by way of limitation, various embodiments discussed in the document.

Fig. 1 shows a schematic scene of capturing motion of a person for dynamic balance ability evaluation of the person, in accordance with an embodiment of the disclosure.

Fig. 2 shows a flowchart of a process for dynamic balance ability evaluation, in accordance with some embodiments of the disclosure.

Fig. 3 shows a flowchart of a process of identifying different phases of motion of the person, in accordance with some embodiments of the disclosure.

Fig. 4 shows a flowchart of a process of obtaining a plurality of indices for dynamic balance ability evaluation, in accordance with some embodiments of the disclosure.

Fig. 5 shows evaluation results of eight indices for dynamic balance ability evaluation by applying the approach for dynamic balance ability evaluation to a video of a normal gait.

Fig. 6 shows evaluation results of eight indices for dynamic balance ability evaluation by applying the approach for dynamic balance ability evaluation to a video of an abnormal gait.

Fig. 7 is a block diagram illustrating components, according to some example embodiments, able to read instructions from a machine-readable or computer-readable medium and perform any one or more of the methodologies discussed herein.

Fig. 8 is a block diagram of an example processor platform in accordance with some embodiments of the disclosure.

Detailed Description of Embodiments

Various aspects of the illustrative embodiments will be described using terms commonly employed by those skilled in the art to convey the substance of the disclosure to others skilled in the art. However, it will be apparent to those skilled in the art that many alternate embodiments may be practiced using portions of the described aspects. For purposes of explanation, specific numbers, materials, and configurations are set forth in order to provide a thorough understanding of the illustrative embodiments. However, it will be apparent to those skilled in the art that alternate embodiments may be practiced without the specific details. In other instances, well known features may have been omitted or simplified in order to avoid obscuring the illustrative embodiments.

Further, various operations will be described as multiple discrete operations, in turn, in a manner that is most helpful in understanding the illustrative embodiments; however, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations need not be performed in the order of presentation.

The phrases “in an embodiment” “in one embodiment” and “in some embodiments” are used repeatedly herein. The phrase generally does not refer to the same embodiment; however, it may. The terms “comprising, ” “having, ” and “including” are synonymous, unless the context dictates otherwise. The phrases “A or B” and “A/B” mean “ (A) , (B) , or (A and B) . ”

As mentioned, in order to evaluate the dynamic balance ability, a plurality of indices related to abilities of keeping balance when walking should be taken into consideration. For standardization, the Ministry of Civil Affairs of the People’s Republic of China has issued the “Basic Specification for falls prevention of the elderly in senior care organization” on December 10, 2021. The Specification provides a test sheet for the elderly’s balance abilities, in which the dynamic balance ability are embodied by eight indices, i.e., Starting, Step height, Step length, Step uniformity, Step continuity, Step alignment, Torso stability, and Turnaround ability.

A traditional method for dynamic balance ability evaluation is to track three-dimension (3D) positions of markers fixed on people’s joints by multiple high frame-rate cameras and make analysis. The shortcomings of this method may include restrictions on people’s moving freedom; complexity of adding and removing the markers and configuration for cameras (calibration should be performed by tools when any camera is changed) ; a high price (normally tens of thousand dollars) ; limited space and environment (comparatively large space and indoor environment with controlled lightness) . There have been tries to use multiple depth cameras to evaluate the balance ability in recent years, but it meets difficulties of limitations of environment lightness, complex configuration and moving distance, and thus no dynamic balance results are provided, as described by an article “Predicting Fall Probability Based on a Validated Balance Scale” (published on Institute of Electrical and Electronic Engineers (IEEE) Conference on Computer Vision and Pattern Recognition (CVPR) 2020, pages 302-303) of Alaa Masalha et al. When red-green-blue (RGB) cameras are used for the balance evaluation, only the Center of Mass (CoM) and the Center of Pressure (CoP) are extracted, while the required indices by the above Specification are not extracted and cannot be directly obtained by CoM and CoP, as described by an article “Towards balance assessment using Openpose” proposed by Brighton Li et al. on the 43 ^rd Annual International Conference of the IEEE Engineering in Medicine &Biology Society (EMBC) that was held during October 31 to November 4, 2021.

Recently, the progress of deep learning based pose estimation for videos opens the door to solve this challenge by an approach called Pose Estimation based Balance Ability Evaluation (PEBA) . Firstly, the PEBA does not need markers and high frame rate cameras. Secondly, the PEBA can process videos captured for a person in a long distance. Thirdly, the PEBA has no requirement to limit environment lightness and can even be used outdoor. Fourthly, the PEBA does not need the comparatively expensive cameras with hard configuration such as fixed high frame rate for markers capturing and the depth cameras, and also does not need parameters difficult to get (such as a person’s height and weight) . However, this approach meets following difficulties or defects: the accuracy of joint 3D localization is far less than the marker based method and the data of pose estimation cannot be used for dynamic balance ability evaluation directly because of uncertain deviations of deep learning, coverage and the person’s angle.

The present application provides computer-implemented approaches for dynamic balance ability evaluation. The approaches of the present application can automatically extract and evaluate a plurality of indices related to abilities of keeping balance when walking, including the eight indices as described by the above-mentioned Specification, and therefore can overcome some or all of the difficulties or defects of the PEBA.

In the approaches of the present application, only pose features confident enough are used, such as a locally maximum distance between a person’s feet and a distance between the person’s shoulders when the person is forward/backward; important phases are extracted firstly, such as a starting phase, a turnaround phase and one or more maximum step phases; indices for dynamic balance ability evaluation are extracted based only on the confident pose features in confident phases; and a ratio between a distance between two feature points corresponding to the person and a feature distance (which will be defined below) is used to get certain indices utilizing a principle that a corresponding index is proportional to a size (such as the height and weight) of the person.

Fig. 1 shows a schematic scene 100 of capturing motion of a person for dynamic balance ability evaluation of the person, in accordance with an embodiment of the disclosure. As shown in Fig. 1, in order to evaluate the dynamic balance ability of the person (such as an elderly person) , the person is required to walk from a starting point to an end point and turn back to the starting point. Alternatively, the person can simply walk from a starting point to an end point without turning back to the starting point. The motion of the person is captured by a camera (such as a camera with a wide-angle lens) from a direction perpendicular to a moving direction of the person to obtain a video. The video can be sampled to obtain a sequence of video frames with an appropriate frame rate. The sequence of video frames include pose information of the person.

In another embodiment, in order to prevent the person from being too nervous to walk normally, the required motion of the person can be captured without notifying the person, by setting a control line to let the person do the motion unconsciously.

Fig. 2 shows a flowchart of a process 200 for dynamic balance ability evaluation, in accordance with some embodiments of the disclosure. The process 200 may be implemented, for example, by one or more processors of a computing device. An example of the processors is to be shown in Fig. 8.

The process 200 includes, at block 210, obtaining a sequence of video frames (such as the sequence of video frames obtained in Fig. 1) including pose information of a person. As mentioned, the sequence of video frames may be obtained by sampling a captured video of a motion of the person. Alternatively, the sequence of video frames may be received or retrieved from a database storing video frames.

The process 200 includes, at block 220, extracting the pose information of the person from the sequence of video frames. The pose information of the person may include horizontal coordinates and vertical coordinates of a set of feature points in each of the video frames, for example. The set of feature points are used to describe particular features and actions of the person. In an embodiment, generally, eighteen feature points corresponding to the person’s head, chest, shoulder/elbow/hand of each arm, leg/knee/foot of each leg, two ears, and two eyes, respectively may be used. In another embodiment, more or less feature points may be used, which is not limited herein. For example, a feature point corresponding to the person’s waist may also be taken into consideration. For example, the feature points corresponding to the person’s two ears, and two eyes may be ignored.

The process 200 includes, at block 230, determining one or more pose features for each of the video frames based on the pose information of the person. As an example, generally, the pose features may include one or more of a distance between the person’s two shoulders, a distance between the person’s two feet, a distance between the person’s head and chest, a distance between the person’s chest and waist, and the like. In an embodiment, the one or more pose features may be determined by a combination of one or more feature points. For example, for each video frame, the one or more pose features of the person may be determined based on horizontal coordinates and vertical coordinates of one or more feature points from the set of feature points in the video frame. The pose features of the person should be robust during the motion of the person, i.e., in the sequence of video frames. When determining the one or more pose features, the horizontal coordinates and vertical coordinates of one or more feature points can be weighted before being combined, in order to improve the robustness.

The process 200 includes, at block 240, identifying different phases of motion of the person based on the one or more pose features for each of the video frames. Taking the schematic scene 100 of Fig. 1 as an example, the motion of the person includes walking from the starting point to the end point and turning back to the starting point, and in this case, the different phases of the motion of the person may include a starting phase, a turnaround phase and one or more maximum step phase. As used herein, the term “turnaround” refers to a turn of 180 degrees, i.e., a “U” turn. A particular process of identifying different phases of motion of the person will be described below with reference to Fig. 3.

The process 200 includes, at block 250, obtaining a plurality of indices for the dynamic balance ability evaluation based on the one or more pose features for each of the video frames or the different phases of motion of the person. For example, in order to be consistent with the Basic Specification for falls prevention of the elderly in senior care organization, the plurality of indices may include Starting, Step height, Step length, Step uniformity, Step continuity, Step alignment, Torso stability, and Turnaround ability. However, the plurality of indices may include more or less indices, which is not limited herein. Some of the pose features are related to a single video frame, such as Torso stability, and can be determined directly from the one or more pose features for the video frame, while other pose features are related to a series of video frames, even the whole sequence of the video frames, such as Step length and Turnaround ability etc., and should be determined based on both the one or more pose features for each of the video frames and the different phases of motion of the person. A particular process of obtaining the plurality of indices for the dynamic balance ability evaluation will be described below with reference to Fig. 4.

In some embodiments, for each of the plurality of indices for the dynamic balance ability evaluation, a normalized value may be obtained. As such, the value of the index may be a value in the range of 0～1.

The process 200 includes, at block 260, evaluating the dynamic balance ability of the person based on each of the plurality of indices and a corresponding threshold. For example, the dynamic balance ability of the person may be evaluated by comparing a score of each index with the corresponding threshold to distinguish whether the score of the index is acceptable. The threshold may be predetermined by professional grading. For example, one or more doctors can give their evaluations on the indices for a number of selected videos, and the threshold may be determined by analysis of the doctors’ evaluations. For example, the threshold may be a critical value to distinguish “Good” and “Bad” . Alternatively, the threshold may be an acceptable range, a value lower than the lower limit value of the acceptable range is evaluated as “Bad” , a value in the acceptable range is evaluated as “Fine” , and a value upper than the upper limit value of the acceptable range is evaluated as “Good” . There may be other types of thresholds, which will not be enumerated herein.

Optionally, a comprehensive evaluation on the dynamic balance ability of the person can be obtained by a combination of the scores of the plurality of indices, such as a weighted sum of the scores of the plurality of indices.

For the elderly, caring and habitation methods and levels can be selected based on evaluations of their balance abilities, including dynamic balance abilities.

Fig. 3 shows a flowchart of a process 300 of identifying different phases of motion of the person, in accordance with some embodiments of the disclosure. The process 300 may be implemented, for example, by one or more processors of a computing device. An example of the processors is to be shown in Fig. 8.

As an example, the person is required to walk from the starting point to the end point and turn back to the starting point. In this case, the different phases of the motion of the person may include a starting phase, a turnaround phase and one or more maximum step phase. In other cases, the motion of the person may include other phases. For example, when the person is required to walk from the starting point to the end point without turning back, the different phases of the motion of the person may simply include the starting phase and the maximum step phase.

The process 300 is described with regard to the case that the motion of the person includes walking from the starting point to the end point and turning back to the starting point.

The one or more pose features for each of the video frames are determined at block 230 may include a barycenter of the person, a distance between two shoulders of the person ( “shoulders distance” for short) , a distance between two feet of the person ( “feet distance” for short) , a distance between two knees of the person, a distance between the person’s head and chest, a distance between the person’s chest and waist, a vertical distance from the person’s knee to foot, and the like. The pose features listed herein are for illustrating only and are not exhausting.

The process 300 may include, at block 310, identifying a feature distance for the sequence of video frames from the one or more pose features for each of the video frames. As used herein, the “feature distance” is considered to be a constant across all the video frames. For example, the feature distance can be identified based on horizontal coordinates and vertical coordinates of the set of feature points in the sequence of video frames.

For different videos, there may be one or more feature distances. For the robustness of calculation, a most “constant” pose feature can be identified for the sequence of video frames. In an embodiment, the feature distance is selected to be the distance (e.g., the vertical distance) between the person’s head and chest, or the distance (e.g., the vertical distance) between the person’s chest and waist.

The process 300 may include, at block 320, determining a moving direction of the person based on horizontal coordinates of the feature distance in the sequence of video frames. For example, when the feature distance is selected to be the distance between the person’s head and chest, or the distance between the person’s chest and waist, the horizontal coordinates of the feature distance can reflect the moving direction of the person.

The process 300 may include, at block 330, identifying a turnaround based on a ratio of the distance between the two shoulders of the person to the feature distance in each of the video frames, and a threshold. The threshold may be predefined.

As mentioned previously, the video is obtained by capturing the motion of the person from the direction perpendicular to the moving direction of the person, and thus when the person is walking without turning, the distance between two shoulders of the person is shortest. When the person begins to turn, the distance between two shoulders of the person becomes longer, and the ratio of the distance between the two shoulders of the person to the feature distance becomes greater. When the person turns 90 degrees, the distance between the two shoulders of the person to the feature distance is longest, and the ratio of the distance between the two shoulders of the person to the feature distance has the greatest value. During this turning process, the moving direction of the person keeps forward.

After the 90 degrees turn, the moving direction of the person changes to backward. As the person further turns, the distance between two shoulders of the person becomes shorter, and the ratio of the distance between the two shoulders of the person to the feature distance becomes smaller. When the person turns 180 degrees (i.e., a turnaround is finished) , the distance between two shoulders of the person is shortest again. During this turning process.

The threshold is used to exclude normal swings of the shoulders. Only when the ratio of the distance between the two shoulders of the person to the feature distance exceeds the threshold, a turn is identified.

The process 300 may include, at block 340, determine, with reference to the moving direction of the person, a time duration from a beginning of the turnaround to an end of the turnaround as the turnaround phase.

In an embodiment, whether the turnaround is a left side turnaround or a right side turnaround can be determined based on horizontal coordinates of the two shoulders of the person in the sequence of video frames during the turnaround phase

The process 300 may include, at block 350, determining video frames each with a local maximum value of the distance between the two feet of the person as candidate frames, and at block 360, filtering the candidate frames based on predefined rules. The predefined rules may include one or more of:

● filtering the candidate frames by non-maximum suppression;

● removing video frames corresponding to the turnaround phase from the candidate frames; and

● filtering the candidate frames by an average value of the distance between the two feet of the person in each of the candidate frames.

The process 300 may include, at block 370, determining a time duration corresponding to remaining candidate frames after the filtering as the maximum step phase. It should be noted that there may be one or more maximum step phases, that is, the video frames corresponding to the maximum step phases may not be continuous.

The process 300 may include, at block 380, detecting a static standing state of the person based on a position of a barycenter of the person in each of the video frames, and at block 390, determining a time duration from beginning time of the motion of the person (i.e., the time when the static standing state of the person changes) to a first maximum step phase as the starting phase.

It should be noted that some of the blocks of the process 300 can be performed in parallel, such as

blocks

320, 330, 350, 380, or blocks 310, 380 and 350, and therefore the execution sequence of the blocks is not limited by the describing order as presented herein.

Fig. 4 shows a flowchart of a process 400 of obtaining the plurality of indices for the dynamic balance ability evaluation, in accordance with some embodiments of the disclosure. The process 400 may be implemented, for example, by one or more processors of a computing device. An example of the processors is to be shown in Fig. 8.

After the one or more pose features for each of the video frames are determined and the different phases of motion of the person are identified, the plurality of indices for the dynamic balance ability evaluation can be obtained based on one or both of them. In order to be consistent with the Basic Specification for falls prevention of the elderly in senior care organization, the plurality of indices may include Starting, Step height, Step length, Step uniformity, Step continuity, Step alignment, Torso stability, and Turnaround ability. The process 400 describes how to obtain each of these indices and/or their meanings.

The index “Starting” , which may also be referred to as “Hesitation for Starting” , can be represented by the beginning time of the motion of the person. At block 410, Starting can be determined based on the starting phase that has been determined at block 390. For convenience of evaluation, the index “Starting” may have a value greater than or equal to 0 and smaller than or equal to 1 by normalization. Just as an example, as a rule of thumb, the time from preparing to starting may have a lower limit of 1 second and a upper limit of 10 seconds, and if it is observed that, for the person, the time is t second (s) , the value of Starting may be calculated by (10-t) / (10-1) . In this case, the greater the value of Starting, the better the ability of starting.

The index “Step height” can be represented by an angle of the knee of the person’s back leg, since the foot cannot be high enough, if the angle of the knee is not large enough. The person’s back leg can be identified according to the moving direction of the person. At block 420, Step height can be determined based on the moving direction of the person and the maximum step phases that have been determined at block 370. For convenience of evaluation, the index “Step height” may also have a value greater than or equal to 0 and smaller than or equal to 1 by normalization. Similarly as Starting, Step height can also be set to have greater value when the angle of the knee of the person’s back leg is greater.

The index “Step length” can be represented by represented by a ratio of an average value of distances between two feet of the person in the maximum step phases to the feature distance. At block 430, Step length can be determined based on the maximum step phases. For convenience of evaluation, the index “Step length” may also have a value greater than or equal to 0 and smaller than or equal to 1 by normalization. Similarly as Starting, Step length can also be set to have greater value when the angle of the ratio of the average value of distances between two feet of the person in the maximum step phases to the feature distance is greater.

The index “Step uniformity” can be represented by a variance of Step lengths. At block 440, Step uniformity can be determined based on Step lengths as determined at block 430. For convenience of evaluation, the index “Step uniformity” may also have a value greater than or equal to 0 and smaller than or equal to 1 by normalization. Step uniformity can be set to have greater value when the Step lengths is smaller.

The index “Step continuity” can be represented by a variance of time differences between two neighbor maximum step phases. At block 450, Step continuity can be determined based on the maximum step phases. For convenience of evaluation, the index “Step continuity” may also have a value greater than or equal to 0 and smaller than or equal to 1 by normalization. Step continuity can be set to have greater value when the variance of time differences between two neighbor maximum step phases is smaller.

The index “Step alignment” can be represented by a variance of vertical coordinates of the person’s feet in the maximum step phases. At block 460, Step alignment can be determined based on the maximum step phases. For convenience of evaluation, the index “Step alignment” may also have a value greater than or equal to 0 and smaller than or equal to 1 by normalization. Step uniformity can be set to have greater value when the vertical coordinates of the person’s feet in the maximum step phases is smaller.

The index “Torso stability” can be represented by a variance of angels between the feature distance and a horizontal axis of each of the video frames. At block 470, Torso stability can be determined based on the one or more pose features for each of the video frames as determined at block 230. For convenience of evaluation, the index “Torso stability” may also have a value greater than or equal to 0 and smaller than or equal to 1 by normalization. Torso stability can be set to have greater value when the variance of the angels between the feature distance and the horizontal axis of each of the video frames is smaller.

The index “Turnaround ability” can be represented by a time duration of the turnaround phase. At block 480, Turnaround ability can be determined based on the turnaround phase that has been determined at block 340. For convenience of evaluation, the index “Turnaround ability” may have a value greater than or equal to 0 and smaller than or equal to 1 by normalization. Just as an example, as a rule of thumb, the time duration of the turnaround phase may have a lower limit of 0.5 second and a upper limit of 2.5 seconds, and if it is observed that, for the person, the time duration is t second (s) , the value of Turnaround ability may be calculated by (2.5-t) / (2.5-0.5) . In this case, the greater the value of Turnaround ability, the better the turnaround ability.

It should be noted that some of the blocks of the process 400 can be performed in parallel, such as

blocks

410, 420, 430, 450, 460, 470 and 480, and therefore the execution sequence of the blocks is not limited by the describing order as presented herein.

After the values of the indices are obtained, the dynamic balance ability of the person can be evaluated based on each of the plurality of indices and the corresponding threshold, as described at block 260.

As can be seen, by the settings of the present application, for each of the indices, the greater value means the better ability corresponding to the index. Therefore, a comprehensive evaluation on the dynamic balance ability of the person can be obtained by a combination of the scores of the plurality of indices, such as a weighted sum of the scores of the plurality of indices. For the elderly, caring and habitation methods and levels can be selected based on evaluations of their balance abilities, including dynamic balance abilities.

More particularly, the process 200 of Fig. 2, the process 300 of Fig. 3 and the process 400 of Fig. 4 may be implemented in one or more modules as a set of logic instructions stored in a machine-readable or computer-readable storage medium such as random access memory (RAM) , read only memory (ROM) , programmable ROM (PROM) , firmware, flash memory, etc., in configurable logic such as, for example, programmable logic arrays (PLAs) , field programmable gate arrays (FPGAs) , complex programmable logic devices (CPLDs) , in fixed-functionality logic hardware using circuit technology such as, for example, application specific integrated circuit (ASIC) , complementary metal oxide semiconductor (CMOS) or transistor-transistor logic (TTL) technology, or any combination thereof.

For example, computer program code to carry out operations shown in the process 200 of Fig. 2, the process 300 of Fig. 3 and the process 400 of Fig. 4 may be written in any combination of one or more programming languages, including an object oriented programming language such as JAVA, SMALLTALK, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. Additionally, logic instructions might include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, state-setting data, configuration data for integrated circuitry, state information that personalizes electronic circuitry and/or other structural components that are native to hardware (e.g., host processor, central processing unit/CPU, microcontroller, etc. ) .

Fig. 5 shows evaluation results of the eight indices for dynamic balance ability evaluation by applying the approach for dynamic balance ability evaluation to a video of a normal gait. Fig. 6 shows evaluation results of the eight indices for dynamic balance ability evaluation by applying the approach for dynamic balance ability evaluation to a video of an abnormal gait (e.g., a simulated gait of a hemiplegia patient) .

As mentioned previously, by the settings of the present application, for each of the indices, the greater value means the better ability corresponding to the index. Fig. 5 illustrates that each of the eight indices has an acceptable value as compared with the corresponding threshold, i.e., a reference value, and is thus evaluated as “Good” or “Fine” . Fig. 6 illustrates that some of the eight indices, i.e., Starting, Step uniformity, Step alignment, Torso stability and Turnaround ability, have acceptable values as compared with the corresponding thresholds, and are thus evaluated as “Good” or “Fine” , but Step height, Step length, Step continuity have unacceptable values as compared with corresponding thresholds, and are thus evaluated as “Bad” .

It is demonstrated that the approach for dynamic balance ability evaluation of the present application can obviously differentiate the two gaits and point out that the abnormal gait (i.e., the hemiplegia gait) has big problems with the indices, i.e., Step height, Step length and Step continuity.

Fig. 7 is a block diagram illustrating components, according to some example embodiments, able to read instructions from a machine-readable or computer-readable medium (e.g., a non-transitory machine-readable storage medium) and perform any one or more of the methodologies discussed herein. Specifically, Fig. 7 shows a diagrammatic representation of hardware resources 700 including one or more processors (or processor cores) 710, one or more memory/storage devices 720, and one or more communication resources 730, each of which may be communicatively coupled via a bus 740. For embodiments where node virtualization (e.g., NFV) is utilized, a hypervisor 702 may be executed to provide an execution environment for one or more network slices/sub-slices to utilize the hardware resources 700.

The processors 710 may include, for example, a processor 712 and a processor 714 which may be, e.g., a central processing unit (CPU) , a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU) , a digital signal processor (DSP) such as a baseband processor, an application specific integrated circuit (ASIC) , a radio-frequency integrated circuit (RFIC) , another processor, or any suitable combination thereof.

The memory/storage devices 720 may include main memory, disk storage, or any suitable combination thereof. The memory/storage devices 720 may include, but are not limited to any type of volatile or non-volatile memory such as dynamic random access memory (DRAM) , static random-access memory (SRAM) , erasable programmable read-only memory (EPROM) , electrically erasable programmable read-only memory (EEPROM) , Flash memory, solid-state storage, etc.

The communication resources 730 may include interconnection or network interface components or other suitable devices to communicate with one or more peripheral devices 704 or one or more databases 706 via a network 708. For example, the communication resources 730 may include wired communication components (e.g., for coupling via a Universal Serial Bus (USB) ) , cellular communication components, NFC components,

components (e.g.,

Low Energy) ,

components, and other communication components.

Instructions 750 may comprise software, a program, an application, an applet, an app, or other executable code for causing at least any of the processors 710 to perform any one or more of the methodologies discussed herein. The instructions 750 may reside, completely or partially, within at least one of the processors 710 (e.g., within the processor’s cache memory) , the memory/storage devices 720, or any suitable combination thereof. Furthermore, any portion of the instructions 750 may be transferred to the hardware resources 700 from any combination of the peripheral devices 704 or the databases 706. Accordingly, the memory of processors 710, the memory/storage devices 720, the peripheral devices 704, and the databases 706 are examples of computer-readable and machine-readable media.

Fig. 8 is a block diagram of an example processor platform in accordance with some embodiments of the disclosure. The processor platform 800 can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network) , a mobile device (e.g., a cell phone, a smart phone, a tablet such as an iPad ^TM) , a personal digital assistant (PDA) , an Internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a gaming console, a personal video recorder, a set top box, a headset or other wearable device, or any other type of computing device.

The processor platform 800 of the illustrated example includes a processor 812. The processor 812 of the illustrated example is hardware. For example, the processor 812 can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. The hardware processor may be a semiconductor based (e.g., silicon based) device. In some embodiments, the processor implements one or more of the methods or processes described above.

The processor 812 of the illustrated example includes a local memory 813 (e.g., a cache) . The processor 812 of the illustrated example is in communication with a main memory including a volatile memory 814 and a non-volatile memory 816 via a bus 818. The volatile memory 814 may be implemented by Synchronous Dynamic Random Access Memory (SDRAM) , Dynamic Random Access Memory (DRAM) ,

Dynamic Random Access Memory

and/or any other type of random access memory device. The non-volatile memory 816 may be implemented by flash memory and/or any other desired type of memory device. Access to the

main memory

814, 816 is controlled by a memory controller.

The processor platform 800 of the illustrated example also includes interface circuitry 820. The interface circuitry 820 may be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB) , a

interface, a near field communication (NFC) interface, and/or a PCI express interface.

In the illustrated example, one or more input devices 822 are connected to the interface circuitry 820. The input device (s) 822 permit (s) a user to enter data and/or commands into the processor 812. The input device (s) can be implemented by, for example, an audio sensor, a microphone, a camera (still or video) , a keyboard, a button, a mouse, a touchscreen, a track-pad, a trackball, and/or a voice recognition system.

One or more output devices 824 are also connected to the interface circuitry 820 of the illustrated example. The output devices 824 can be implemented, for example, by display devices (e.g., a light emitting diode (LED) , an organic light emitting diode (OLED) , a liquid crystal display (LCD) , a cathode ray tube display (CRT) , an in-place switching (IPS) display, a touchscreen, etc. ) , a tactile output device, a printer and/or speaker. The interface circuitry 820 of the illustrated example, thus, typically includes a graphics driver card, a graphics driver chip and/or a graphics driver processor.

The interface circuitry 820 of the illustrated example also includes a communication device such as a transmitter, a receiver, a transceiver, a modem, a residential gateway, a wireless access point, and/or a network interface to facilitate exchange of data with external machines (e.g., computing devices of any kind) via a network 826. The communication can be via, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-site wireless system, a cellular telephone system, etc.

For example, the interface circuitry 820 may include a training dataset inputted through the input device (s) 822 or retrieved from the network 826.

The processor platform 800 of the illustrated example also includes one or more mass storage devices 828 for storing software and/or data. Examples of such mass storage devices 828 include floppy disk drives, hard drive disks, compact disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.

Machine executable instructions 832 may be stored in the mass storage device 828, in the volatile memory 814, in the non-volatile memory 816, and/or on a removable non-transitory computer readable storage medium such as a CD or DVD.

The following paragraphs describe examples of various embodiments.

Example 1 includes an apparatus for dynamic balance ability evaluation comprising: interface circuitry configured to receive a sequence of video frames including pose information of a person; and processor circuitry coupled to the interface circuitry and configured to extract the pose information of the person from the sequence of video frames; determine one or more pose features for each of the video frames based on the pose information of the person; identify different phases of motion of the person based on the one or more pose features for each of the video frames; obtain a plurality of indices for the dynamic balance ability evaluation based on the one or more pose features for each of the video frames or the different phases of motion of the person; and evaluate the dynamic balance ability of the person based on each of the plurality of indices and a corresponding threshold.

Example 2 includes the apparatus of Example 1, wherein the sequence of video frames is obtained by capturing a video of the motion of the person from a direction perpendicular to a moving direction of the person and sampling the video.

Example 3 includes the apparatus of Example 1 or Example 2, wherein the pose information of the person comprising horizontal coordinates and vertical coordinates of a set of feature points in each of the video frames.

Example 4 includes the apparatus of Example 3, wherein the set of feature points comprises feature points selected from points representing the person’s head, chest, shoulder/elbow/hand of each arm, and leg/knee/foot of each leg.

Example 5 includes the apparatus of Example 3 or Example 4, wherein the processor circuitry is configured to determine each of the one or more pose features for each of the video frames based on horizontal coordinates and vertical coordinates of one or more feature points from the set of feature points in the video frame.

Example 6 includes the apparatus of Example 3 or Example 4, wherein the motion of the person comprising walking from a starting point to an end point and turning back to the starting point, and the different phases of the motion of the person comprises a starting phase, a turnaround phase and one or more maximum step phases.

Example 7 includes the apparatus of Example 6, wherein the processor circuitry is configured to: identify a feature distance for the sequence of video frames from the one or more pose features for each of the video frames; determine a moving direction of the person based on horizontal coordinates of the feature distance in the sequence of video frames; identify a turnaround based on a ratio of a distance between two shoulders of the person to the feature distance in each of the video frames, and a threshold; and determine, with reference to the moving direction of the person, a time duration from a beginning of the turnaround to an end of the turnaround as the turnaround phase.

Example 8 includes the apparatus of Example 7, wherein the processor circuitry is configured to: determine whether the turnaround is a left side turnaround or a right side turnaround based on horizontal coordinates of two shoulders of the person in the sequence of video frames during the turnaround phase.

Example 9 includes the apparatus of Example 7 or Example 8, wherein the processor circuitry is further configured to: determine a distance between two feet of the person in each of the video frames; determine video frames each with a local maximum value of the distance between two feet of the person as candidate frames; and determine one or more time durations corresponding to remaining candidate frames after any one or more of following operations as the one or more maximum step phases: filtering the candidate frames by non-maximum suppression; removing video frames corresponding to the turnaround phase from the candidate frames; and filtering the candidate frames by an average value of the distance between two feet of the person in each of the candidate frames.

Example 10 includes the apparatus of Example 9, wherein the processor circuitry is further configured to: determine a time duration from beginning time of the motion of the person to a first maximum step phase as the starting phase, and wherein the beginning of the motion of the person is determined based on a position of a barycenter of the person in each of the video frames.

Example 11 includes the apparatus of Example 10, wherein the plurality of indices for the dynamic balance ability evaluation comprise one or more of: Starting, represented by the beginning time of the motion of the person; Step height, represented by an angle of the knee of the person’s back leg; Step length, represented by a ratio of an average value of distances between two feet of the person in the maximum step phases to the feature distance; Step uniformity, represented by a variance of Step lengths; Step continuity, represented by a variance of time differences between two neighbor maximum step phases; Step alignment, represented by a variance of vertical coordinates of the person’s feet in the maximum step phases; Torso stability, represented by a variance of angels between a distance between the person’s head and chest and a horizontal axis of each of the video frames; and Turnaround ability, represented by a time duration of the turnaround phase.

Example 12 includes the apparatus of any of Examples 1-11, wherein obtaining the plurality of indices for the dynamic balance ability evaluation comprises obtaining a normalized value of each of the plurality of indices.

Example 13 includes the apparatus of Example 12, wherein the processor circuitry is configured to evaluate the dynamic balance ability of the person by comparing the normalized value of each of the plurality of indices with the corresponding threshold, the corresponding threshold being predetermined according to professional grading.

Example 14 includes a computer-implemented method for dynamic balance ability evaluation, comprising: obtaining a sequence of video frames comprising pose information of a person; extracting the pose information of the person from the sequence of video frames; determining one or more pose features for each of the video frames based on the pose information of the person; identifying different phases of motion of the person based on the one or more pose features for each of the video frames; obtaining a plurality of indices for the dynamic balance ability evaluation based on the one or more pose features for each of the video frames or the different phases of motion of the person; and evaluating the dynamic balance ability of the person based on each of the plurality of indices and a corresponding threshold.

Example 15 includes the method of Example 14, further comprising: capturing a video of the motion of the person from a direction perpendicular to a moving direction of the person; and sampling the video to obtain the sequence of video frames.

Example 16 includes the method of Example 14 or Example 15, wherein the pose information of the person comprising horizontal coordinates and vertical coordinates of a set of feature points in each of the video frames.

Example 17 includes the method of Example 16, wherein the set of feature points comprises feature points selected from points representing the person’s head, chest, shoulder/elbow/hand of each arm, and leg/knee/foot of each leg.

Example 18 includes the method of Example 16 or Example 17, further comprising: determining each of the one or more pose features for each of the video frames based on horizontal coordinates and vertical coordinates of one or more feature points from the set of feature points in the video frame.

Example 19 includes the method of Example 16 or Example 17, wherein the motion of the person comprising walking from a starting point to an end point and turning back to the starting point, and the different phases of the motion of the person comprises a starting phase, a turnaround phase and one or more maximum step phases.

Example 20 includes the method of Example 19, further comprising: identifying a feature distance for the sequence of video frames from the one or more pose features for each of the video frames; determining a moving direction of the person based on horizontal coordinates of the feature distance in the sequence of video frames; identifying a turnaround based on a ratio of a distance between two shoulders of the person to the feature distance in each of the video frames, and a threshold; and determining, with reference to the moving direction of the person, a time duration from a beginning of the turnaround to an end of the turnaround as the turnaround phase.

Example 21 includes the method of Example 20, further comprising: determining whether the turnaround is a left side turnaround or a right side turnaround based on horizontal coordinates of two shoulders of the person in the sequence of video frames during the turnaround phase.

Example 22 includes the method of Example 20, further comprising: determining a distance between two feet of the person in each of the video frames; determining video frames each with a local maximum value of the distance between two feet of the person as candidate frames; and determining one or more time durations corresponding to remaining candidate frames after any one or more of following operations as the one or more maximum step phases: filtering the candidate frames by non-maximum suppression; removing video frames corresponding to the turnaround phase from the candidate frames; and filtering the candidate frames by an average value of the distance between two feet of the person in each of the candidate frames.

Example 23 includes the method of Example 22, further comprising: determining a time duration from beginning time of the motion of the person to a first maximum step phase as the starting phase, wherein the beginning of the motion of the person is determined based on a position of a barycenter of the person in each of the video frames.

Example 24 includes the method of Example 23, wherein the plurality of indices for the dynamic balance ability evaluation comprise one or more of: Starting, represented by the beginning time of the motion of the person; Step height, represented by an angle of the knee of the person’s back leg; Step length, represented by a ratio of an average value of distances between two feet of the person in the maximum step phases to the feature distance; Step uniformity, represented by a variance of Step lengths; Step continuity, represented by a variance of time differences between two neighbor maximum step phases; Step alignment, represented by a variance of vertical coordinates of the person’s feet in the maximum step phases; Torso stability, represented by a variance of angels between a distance between the person’s head and chest and a horizontal axis of each of the video frames; and Turnaround ability, represented by a time duration of the turnaround phase.

Example 25 includes the method of any of Examples 14-24, further comprising obtaining a normalized value of each of the plurality of indices.

Example 26 includes the method of Example 25, further comprising evaluating the dynamic balance ability of the person by comparing the normalized value of each of the plurality of indices with the corresponding threshold, the corresponding threshold being predetermined according to professional grading.

Example 27 includes machine readable storage medium having instructions stored thereon, the instructions when executed by a machine, causing the machine to: obtain a sequence of video frames comprising pose information of a person; extract the pose information of the person from the sequence of video frames; determine one or more pose features for each of the video frames based on the pose information of the person; identify different phases of motion of the person based on the one or more pose features for each of the video frames; obtaining a plurality of indices for the dynamic balance ability evaluation based on the one or more pose features for each of the video frames or the different phases of motion of the person; and evaluating the dynamic balance ability of the person based on each of the plurality of indices and a corresponding threshold.

Example 28 includes the machine readable storage medium of Example 27, wherein the instructions when executed by a machine, cause the machine to capture a video of the motion of the person from a direction perpendicular to a moving direction of the person; and sample the video to obtain the sequence of video frames.

Example 29 includes the machine readable storage medium of Example 27 or Example 28, wherein the pose information of the person comprising horizontal coordinates and vertical coordinates of a set of feature points in each of the video frames.

Example 30 includes the machine readable storage medium of Example 29, wherein the set of feature points comprises feature points selected from points representing the person’s head, chest, shoulder/elbow/hand of each arm, and leg/knee/foot of each leg.

Example 31 includes the machine readable storage medium of Example 29 or Example 30, wherein the instructions, when executed by a machine, cause the machine to determine each of the one or more pose features for each of the video frames based on horizontal coordinates and vertical coordinates of one or more feature points from the set of feature points in the video frame.

Example 32 includes the machine readable storage medium of Example 29 or Example 30, wherein the motion of the person comprising walking from a starting point to an end point and turning back to the starting point, and the different phases of the motion of the person comprises a starting phase, a turnaround phase and one or more maximum step phases.

Example 33 includes the machine readable storage medium of Example 32, wherein the instructions when executed by a machine, cause the machine to: identify a feature distance for the sequence of video frames from the one or more pose features for each of the video frames; determine a moving direction of the person based on horizontal coordinates of the feature distance in the sequence of video frames; identify a turnaround based on a ratio of a distance between two shoulders of the person to the feature distance in each of the video frames, and a threshold; and determine, with reference to the moving direction of the person, a time duration from a beginning of the turnaround to an end of the turnaround as the turnaround phase.

Example 34 includes the machine readable storage medium of Example 33, wherein the instructions when executed by a machine, further cause the machine to determine whether the turnaround is a left side turnaround or a right side turnaround based on horizontal coordinates of two shoulders of the person in the sequence of video frames during the turnaround phase.

Example 35 includes the machine readable storage medium of Example 33, wherein the instructions when executed by a machine, further cause the machine to: determine a distance between two feet of the person in each of the video frames; determine video frames each with a local maximum value of the distance between two feet of the person as candidate frames; and determine one or more time durations corresponding to remaining candidate frames after any one or more of following operations as the one or more maximum step phases: filtering the candidate frames by non-maximum suppression; removing video frames corresponding to the turnaround phase from the candidate frames; and filtering the candidate frames by an average value of the distance between two feet of the person in each of the candidate frames.

Example 36 includes the machine readable storage medium of Example 35, wherein the instructions when executed by a machine, further cause the machine to determine a time duration from beginning time of the motion of the person to a first maximum step phase as the starting phase, wherein the beginning of the motion of the person is determined based on a position of a barycenter of the person in each of the video frames.

Example 37 includes the machine readable storage medium of Example 36, wherein the plurality of indices for the dynamic balance ability evaluation comprise one or more of: Starting, represented by the beginning time of the motion of the person; Step height, represented by an angle of the knee of the person’s back leg; Step length, represented by a ratio of an average value of distances between two feet of the person in the maximum step phases to the feature distance; Step uniformity, represented by a variance of Step lengths; Step continuity, represented by a variance of time differences between two neighbor maximum step phases; Step alignment, represented by a variance of vertical coordinates of the person’s feet in the maximum step phases; Torso stability, represented by a variance of angels between a distance between the person’s head and chest and a horizontal axis of each of the video frames; and Turnaround ability, represented by a time duration of the turnaround phase.

Example 38 includes the machine readable storage medium of any of Examples 27-37, wherein the instructions when executed by a machine, further cause the machine to obtain a normalized value of each of the plurality of indices.

Example 39 includes the machine readable storage medium of Example 38, wherein the instructions when executed by a machine, further cause the machine to evaluate the dynamic balance ability of the person by comparing the normalized value of each of the plurality of indices with the corresponding threshold, the corresponding threshold being predetermined according to professional grading.

Example 40 includes a computing device, comprising: means for obtaining a sequence of video frames comprising pose information of a person; means for extracting the pose information of the person from the sequence of video frames; means for determining one or more pose features for each of the video frames based on the pose information of the person; identifying different phases of motion of the person based on the one or more pose features for each of the video frames; means for obtaining a plurality of indices for the dynamic balance ability evaluation based on the one or more pose features for each of the video frames or the different phases of motion of the person; and means for evaluating the dynamic balance ability of the person based on each of the plurality of indices and a corresponding threshold.

Example 41 includes the computing device of Example 40, further comprising: means for capturing a video of the motion of the person from a direction perpendicular to a moving direction of the person; and means for sampling the video to obtain the sequence of video frames.

Example 42 includes the computing device of Example 40 or Example 41, wherein the pose information of the person comprising horizontal coordinates and vertical coordinates of a set of feature points in each of the video frames.

Example 43 includes the computing device of Example 42, wherein the set of feature points comprises feature points selected from points representing the person’s head, chest, shoulder/elbow/hand of each arm, and leg/knee/foot of each leg.

Example 44 includes the computing device of Example 42 or Example 43, further comprising: means for determining each of the one or more pose features for each of the video frames based on horizontal coordinates and vertical coordinates of one or more feature points from the set of feature points in the video frame.

Example 45 includes the computing device of Example 42 or Example 43, wherein the motion of the person comprising walking from a starting point to an end point and turning back to the starting point, and the different phases of the motion of the person comprises a starting phase, a turnaround phase and one or more maximum step phases.

Example 46 includes the computing device of Example 45, further comprising: means for identifying a feature distance for the sequence of video frames from the one or more pose features for each of the video frames; determining a moving direction of the person based on horizontal coordinates of the feature distance in the sequence of video frames; means for identifying a turnaround based on a ratio of a distance between two shoulders of the person to the feature distance in each of the video frames, and a threshold; and means for determining, with reference to the moving direction of the person, a time duration from a beginning of the turnaround to an end of the turnaround as the turnaround phase.

Example 47 includes the computing device of Example 46, further comprising: means for determining whether the turnaround is a left side turnaround or a right side turnaround based on horizontal coordinates of two shoulders of the person in the sequence of video frames during the turnaround phase.

Example 48 includes the computing device of Example 46, further comprising: means for determining a distance between two feet of the person in each of the video frames; determining video frames each with a local maximum value of the distance between two feet of the person as candidate frames; and means for determining one or more time durations corresponding to remaining candidate frames after any one or more of following operations as the one or more maximum step phases: filtering the candidate frames by non-maximum suppression; removing video frames corresponding to the turnaround phase from the candidate frames; and filtering the candidate frames by an average value of the distance between two feet of the person in each of the candidate frames.

Example 49 includes the computing device of Example 47, further comprising: means for determining a time duration from beginning time of the motion of the person to a first maximum step phase as the starting phase, wherein the beginning of the motion of the person is determined based on a position of a barycenter of the person in each of the video frames.

Example 50 includes the computing device of Example 49, wherein the plurality of indices for the dynamic balance ability evaluation comprise one or more of: Starting, represented by the beginning time of the motion of the person; Step height, represented by an angle of the knee of the person’s back leg; Step length, represented by a ratio of an average value of distances between two feet of the person in the maximum step phases to the feature distance; Step uniformity, represented by a variance of Step lengths; Step continuity, represented by a variance of time differences between two neighbor maximum step phases; Step alignment, represented by a variance of vertical coordinates of the person’s feet in the maximum step phases; Torso stability, represented by a variance of angels between a distance between the person’s head and chest and a horizontal axis of each of the video frames; and Turnaround ability, represented by a time duration of the turnaround phase.

Example 51 includes the computing device of any of Examples 40-50, further means for comprising obtaining a normalized value of each of the plurality of indices.

Example 52 includes the computing device of Example 51, further comprising means for evaluating the dynamic balance ability of the person by comparing the normalized value of each of the plurality of indices with the corresponding threshold, the corresponding threshold being predetermined according to professional grading.

Example 53 includes an apparatus comprising one or more processors to implement the one or more of the processes as shown and described in the description.

Example 54 includes a method comprising one or more of processes as shown and described in the description.

Example 55 includes an apparatus comprising one or more memories to store computer-readable instructions for implementing one or more of the processes as shown and described in the description.

The above description is intended to be illustrative, and not restrictive. For example, the above-described examples (or one or more aspects thereof) may be used in combination with each other. Other embodiments may be used, such as by one of ordinary skill in the art upon reviewing the above description. The Abstract is to allow the reader to quickly ascertain the nature of the technical disclosure and is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Also, in the above Detailed Description, various features may be grouped together to streamline the disclosure. This should not be interpreted as intending that an unclaimed disclosed feature is essential to any claim. Rather, inventive subject matter may lie in less than all features of a particular disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separate embodiment. The scope of the embodiments should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Although certain embodiments have been illustrated and described herein for purposes of description, a wide variety of alternate and/or equivalent embodiments or implementations calculated to achieve the same purposes may be substituted for the embodiments shown and described without departing from the scope of the present disclosure. The disclosure is intended to cover any adaptations or variations of the embodiments discussed herein. Therefore, it is manifestly intended that embodiments described herein be limited only by the appended claims and the equivalents thereof.

Claims

An apparatus for dynamic balance ability evaluation, comprising:

interface circuitry configured to receive a sequence of video frames comprising pose information of a person; and

processor circuitry coupled to the interface circuitry and configured to:

extract the pose information of the person from the sequence of video frames;

determine one or more pose features for each of the video frames based on the pose information of the person;

identify different phases of motion of the person based on the one or more pose features for each of the video frames;

obtain a plurality of indices for the dynamic balance ability evaluation based on the one or more pose features for each of the video frames or the different phases of motion of the person; and

evaluate the dynamic balance ability of the person based on each of the plurality of indices and a corresponding threshold.
The apparatus of claim 1, wherein the sequence of video frames is obtained by capturing a video of the motion of the person from a direction perpendicular to a moving direction of the person and sampling the video.
The apparatus of claim 1, wherein the pose information of the person comprising horizontal coordinates and vertical coordinates of a set of feature points in each of the video frames.
The apparatus of claim 3, wherein the set of feature points comprises feature points selected from points representing the person’s head, chest, shoulder/elbow/hand of each arm, and leg/knee/foot of each leg.
The apparatus of claim 3, wherein the processor circuitry is configured to determine each of the one or more pose features for each of the video frames based on horizontal coordinates and vertical coordinates of one or more feature points from the set of feature points in the video frame.
The apparatus of claim 3, wherein the motion of the person comprising walking from a starting point to an end point and turning back to the starting point, and the different phases of the motion of the person comprises a starting phase, a turnaround phase and one or more maximum step phases.
The apparatus of claim 6, wherein the processor circuitry is configured to:

identify a feature distance for the sequence of video frames from the one or more pose features for each of the video frames;

determine a moving direction of the person based on horizontal coordinates of the feature distance in the sequence of video frames;

identify a turnaround based on a ratio of a distance between two shoulders of the person to the feature distance in each of the video frames, and a threshold; and

determine, with reference to the moving direction of the person, a time duration from a beginning of the turnaround to an end of the turnaround as the turnaround phase.
The apparatus of claim 7, wherein the processor circuitry is configured to:

determine whether the turnaround is a left side turnaround or a right side turnaround based on horizontal coordinates of two shoulders of the person in the sequence of video frames during the turnaround phase.
The apparatus of claim 7, wherein the processor circuitry is further configured to:

determine a distance between two feet of the person in each of the video frames;

determine video frames each with a local maximum value of the distance between two feet of the person as candidate frames; and

determine one or more time durations corresponding to remaining candidate frames after any one or more of following operations as the one or more maximum step phases:

filtering the candidate frames by non-maximum suppression;

removing video frames corresponding to the turnaround phase from the candidate frames; and

filtering the candidate frames by an average value of the distance between two feet of the person in each of the candidate frames.
The apparatus of claim 9, wherein the processor circuitry is further configured to:

determine a time duration from beginning time of the motion of the person to a first maximum step phase as the starting phase, and wherein the beginning of the motion of the person is determined based on a position of a barycenter of the person in each of the video frames.
The apparatus of claim 10, wherein the plurality of indices for the dynamic balance ability evaluation comprise one or more of:

Starting, represented by the beginning time of the motion of the person;

Step height, represented by an angle of the knee of the person’s back leg;

Step length, represented by a ratio of an average value of distances between two feet of the person in the maximum step phases to the feature distance;

Step uniformity, represented by a variance of Step lengths;

Step continuity, represented by a variance of time differences between two neighbor maximum step phases;

Step alignment, represented by a variance of vertical coordinates of the person’s feet in the maximum step phases;

Torso stability, represented by a variance of angels between a distance between the person’s head and chest and a horizontal axis of each of the video frames; and

Turnaround ability, represented by a time duration of the turnaround phase.
The apparatus of claim 1, wherein obtaining the plurality of indices for the dynamic balance ability evaluation comprises obtaining a normalized value of each of the plurality of indices.
The apparatus of claim 12, wherein the processor circuitry is configured to evaluate the dynamic balance ability of the person by comparing the normalized value of each of the plurality of indices with the corresponding threshold, the corresponding threshold being predetermined according to professional grading.
A computer-implemented method for dynamic balance ability evaluation, comprising:

obtaining a sequence of video frames comprising pose information of a person;

extracting the pose information of the person from the sequence of video frames;

determining one or more pose features for each of the video frames based on the pose information of the person;

identifying different phases of motion of the person based on the one or more pose features for each of the video frames;

obtaining a plurality of indices for the dynamic balance ability evaluation based on the one or more pose features for each of the video frames or the different phases of motion of the person; and

evaluating the dynamic balance ability of the person based on each of the plurality of indices and a corresponding threshold.
The method of claim 14, further comprising:

capturing a video of the motion of the person from a direction perpendicular to a moving direction of the person; and

sampling the video to obtain the sequence of video frames.
The method of claim 14, wherein the pose information of the person comprising horizontal coordinates and vertical coordinates of a set of feature points in each of the video frames.
The method of claim 16, further comprising:

determining each of the one or more pose features for each of the video frames based on horizontal coordinates and vertical coordinates of one or more feature points from the set of feature points in the video frame.
The method of claim 16, wherein the motion of the person comprising walking from a starting point to an end point and turning back to the starting point, and the different phases of the motion of the person comprises a starting phase, a turnaround phase and one or more maximum step phases.
The method of claim 18, further comprising:

identifying a feature distance for the sequence of video frames from the one or more pose features for each of the video frames;

determining a moving direction of the person based on horizontal coordinates of the feature distance in the sequence of video frames;

identifying a turnaround based on a ratio of a distance between two shoulders of the person to the feature distance in each of the video frames, and a threshold; and

determining, with reference to the moving direction of the person, a time duration from a beginning of the turnaround to an end of the turnaround as the turnaround phase.
The method of claim 19, further comprising:

determining whether the turnaround is a left side turnaround or a right side turnaround based on horizontal coordinates of two shoulders of the person in the sequence of video frames during the turnaround phase.
The method of claim 19, further comprising:

determining a distance between two feet of the person in each of the video frames;

determining video frames each with a local maximum value of the distance between two feet of the person as candidate frames; and

determining one or more time durations corresponding to remaining candidate frames after any one or more of following operations as the one or more maximum step phases:

filtering the candidate frames by non-maximum suppression;

removing video frames corresponding to the turnaround phase from the candidate frames; and

filtering the candidate frames by an average value of the distance between two feet of the person in each of the candidate frames.
The method of claim 21, further comprising:

determining a time duration from beginning time of the motion of the person to a first maximum step phase as the starting phase, wherein the beginning of the motion of the person is determined based on a position of a barycenter of the person in each of the video frames.
The method of claim 22, wherein the plurality of indices for the dynamic balance ability evaluation comprise one or more of:

Starting, represented by the beginning time of the motion of the person;

Step height, represented by an angle of the knee of the person’s back leg;

Step length, represented by a ratio of an average value of distances between two feet of the person in the maximum step phases to the feature distance;

Step uniformity, represented by a variance of Step lengths;

Step continuity, represented by a variance of time differences between two neighbor maximum step phases;

Step alignment, represented by a variance of vertical coordinates of the person’s feet in the maximum step phases;

Torso stability, represented by a variance of angels between a distance between the person’s head and chest and a horizontal axis of each of the video frames; and

Turnaround ability, represented by a time duration of the turnaround phase.
A machine readable storage medium having instructions stored thereon, the instructions when executed by a machine, causing the machine to perform the computer-implemented method for dynamic balance ability evaluation of any one of claims 14 to 23.
A computing device, comprising means for performing the method for dynamic balance ability evaluation of any one of claims 14 to 23.