GB2605623A - Methods and tools for remote frailty and gait analysis - Google Patents

Abstract

Using a motion capture system with a camera and computer arrangement, performing gait analysis of a person using machine learning (artificial intelligence) to automatically diagnose frailty. Video is taken of patients and used as input and a human pose estimation software retrieves coordinates of body parts which are then transformed into signals and movement histograms to be used as feature descriptors. The camera and software work in a high frames per second rate and thus lead to greater accuracy. No marker-based (i.e. wearable markers/devices) motion capture is needed, meaning the method of diagnosing frailty is less invasive.

Description

Methods and Tools for remote Frailty and Gait Analysis This invention relates to a software incorporated solution and workflow methodology for health monitoring and intervention in frailty by applying machine vision and indoor positioning technologies.

Figure 1 shows the complete frailty and gait analysis system tools.

Figure 2 shows the exemplary components of the frailty and gait analysis method structure.

Gait can be defined as a manner or style of walking. There are studies asserting that every individual has aunique gait pattern, what has led gait to be considered as a new biometric feature. Gait analysis is the systematic study of human walking for recognizing of gait pattern abnormalities, postulating its causes, and proposing suitable treatments. Gait analysis is commonly used in clinical applications for recognition of a health problem or monitoring patient's recovery status. The traditional clinical gait analysis is performed by clinicians who observe the patients' gait characteristics while he/she is walking.However, this method is subjective and depends on the experience and judgment of the clinician. As a result, it can lead to a confusion and has a negative effect on the diagnosis and treatment decision making of pathologies.

The process of clinical gait analysis can be facilitated through the use of new technologies, which allow an objective measurement and reduces the confusion and error margin of the subjective methods.

There has been extensive research in the recent past to develop a methodology using different techniques for gait analysis. Previously, methods based on silhouettes, like Gait Energy Image (GEO/were employed to solve the gait analysis problem. Although these methods have shown some advancement in the process of gait analysis, there is the unavailability of a proper gait analysis mechanism for the subjects where problematic constraints such as the view angle, walking speed, clothing, surface, carrying status, shoe and elapsed time are considered and minimalized in an efficacious way.

To address these needs, the present invention proposes an automatic system for analysing key clinically validated aspects of physical frailty including gait speed, gait instability, waddling gait, using props to stabilize gait, decreasing stride length, widened gait, reduced or abnormal cadence, decreased foot ground clearance and the time taken to rise out of a chair. The invention analyzes and classifies gait patterns using digital cameras as the only required equipment and provides a tool hr constant and ubiquitous gait monitoring of patients and elderly people while living in their homes. Such a tool can be additionally used on patients with orthopedic problems, Parkinson's disease as well as on post stroke patients to evaluate their gait and allow medical professionals to offer personalized treatment to individuals suffering from varying degrees of gait degeneration.

Human pose estimation is combined with a rule-based method for gait analysis and recognition. This information is processed generating signals and movement histograms, which are used as features, robust to the aforementioned aspects. The reason to use human pose estimation for gait analysis, is that in pose estimation, deep learning-based Convolutional Neural Network (CNN) is used to detect the body points of a person without the worry of any problematic constraint. The algorithm used in this invention takes an RGB video as input and produces the two-dimensional (2D) locations of anatomical key points, such as knees, elbows, hips, for each person in every frame of the video. This provides skeletal images where body points are joined to form a skeleton of a person. Only these obtained skeletal images are further used for classification and gait analysis.

In Figure 1: The Patient Frailty Analysis System may be configured to analyse input data (e.g., image data, pose metrics) from the Edge MV device in order to identify and quantify arange of motion of a body part or limb and to assist a clinician with the diagnosis. [2]

The Patient Frailty Analysis System may include a display or display device. The display may be a screen. The display screen may or may not be a touchscreen. The display may be configured to show a user interface (UI), or a graphical user interface (GUI) rendered through an application (e.g.) via an application programming interface (API) executed on the patient frailty analysis system). The GUI may show images, charts and analytics relating to the gait. The Ul may be configured for representing and delivering analytics, sensor data (e.g.) video), and processed data to a user (e.g., clinician). A user may navigate within the GUI through the application. For example, the user may select a link by directly touching the screen (e.g., touchscreen). Alternatively, the user may select a portion of an image with aid of a user interactive device (e.g., mouse, joystick) keyboard, trackball, touchpad, or any other device). The patient frailty analysis system would provide a front end to the API to review the patients results in detail, compare to previous tests,track improvements, etc. Also in Figure 1: The Edge MV (Machine Vision) Device includes both hardware and software elements. The Edge MV Device may contain a camera or imaging sensor operable coupled to an embedded computing board. The Edge MV Device can be controlled by an application/software configured to take image(s) or video(s) of the patient. The camera is able to capture dynamic image data (e.g., video). The camera may comprise optical elements (e.g., lens,filters). The camera may capture colour images (RGB images). The camera may be a monocular camera and images of the patient may be taken from a single view/angle.

The Edge MV Device is configured to obtain image data to track motion and posture of a patient. As described herein, computer vision techniques and deep learning techniques may be used to reconstruct the pose using 2D imaging data. The Edge MV Device may include a display. The display may be a screen. The display screen may or may not be a touchscreen. The display may be configured to show a user interface (UI), or a graphical user interface (GUI) rendered through an application (e.g., via an application programming interface (API) executed on the Edge MV Device). The GUI may assist the clinician with the patient's gait test management. The Edge MV Device may employ machine learning and computer vision-based pose estimation algorithm to provide accurate quantification of range of motion based on image data. The pose estimation algorithm may be capable of quantifying range of motion using 2D image data with improved precision and accuracy. The pose estimation algorithm can be any type of machine learning network such as aneural network. Examples of neural networks include a deep neural network, convolutional neural network (CNN), and recurrent neural network (RNN). The machine learning algorithm may comprise one or more of the following: a support vector machine (SVM), a naive Bayes classification, a linear regression, a quantile regression, a logistic regression, a random forest, a neural network, CNN, RNN, a gradient-boosted classifier or repressor, or another supervised or unsupervised machine learning algorithm (e.g., generative adversarial network (GAN), Cycle-GAN, etc.) As illustrated in Figure 1: The Network itself may be a network that is configured to provide communication between the various components illustrated in Figure 1 which includes the illustrated Cloud API and DB. In this context, Cloud APIs are application programming interfaces used to build applications in the cloud computing market. Cloud APIs allow software to request data and computations from one or more services through a direct or indirect interface. The DB element illustrated in Figure 1 is a generic term for any interfacing Data Base.

In Figure 1: The Cloud API & DB: includes a server in a data network (e.g.) a cloud computing network,) and can be computer programmed to accept requests (e.g.) HTTP, or other protocols that can be initiate data transmission) from a computing device (e.g.) Edge MV Device and/or Patient Frailty Analysis System) and to serve the computing device with requested data.

The Cloud API & DB may include a data management system that may construct the database for fast and [3] efficient data retrieval, query, and delivery. The databases may store, for example, raw data collected by the camera located on the Edge MV Device. The databases may also store userinformation, historical data patterns, data relating to a testing, medical records, analytics, user input (e.g., statements or comments indicative of how the user is feeling at different points in time, etc.), and so forth.

The Cloud API & DB is essentially handling all of the data, accepting new data, and allows it to be retrieved at a subsequently required point in time.

The Network as illustrated in Figure 1 may be implemented as one or more networks that connect devices and/or components in the network layout for allowing communication between them.

For example, the Patient Frailty Analysis System, Edge MV Device, and Cloud API & DB may be in operable communication with one another over network. Direct communications may be provided between two or more of the above components. The direct communications may occur without requiring any intermediary device or network. Indirect communications may be provided between two or more of the above components. The indirectcommunications may occur with the aid of one or more intermediary devices or network. For instance, indirect communications may utilize a telecommunications network. Indirect communications may be performed with the aid of one or more routers, communication towers, satellites, or any other intermediary device or network. Examples of types of communications may include, (but are not limited to,) communications via the Internet, Local Area Networks (LANs), Wide Area Networks (WANs), Bluetooth, Near Field Communication (NFC) technologies,networks based on mobile data protocols -such as General Packet Radio Services (GPRS), GSM, Enhanced Data GSM Environment (EDGE), 3G, 4G, SG or Long Term Evolution (LTE) protocols, Infra-Red (IR) communication technologies, and/or Wi-Fi, and may be wireless, wired, or a combination thereof.

In essence, the Network itself may be wireless, wired, and/or a combination thereof.

Figure 2 shows the exemplary components and methodology of the Patient Frailty Analysis System.

The system may comprise of a VioCapture module, Pose Estimation module, Pose Post-Processing module, Activity Recognition module, Gait Analysis module and an API Upload module.

The Video Capture module provides the video of the patient's gait during a test captured by the camera of the Edge MV Device.

The Pose Estimation module is configured to analyse the video stream from the Video Capture module to quantify a range ofmotion in real-time using algorithms described above.

The Pose Post-Processing module filters the output of the pose estimation module. For example, a Kalman filter is used to track the joint key pointsestimated by the previous module, and a simple filter is used to remove bad poses. Moreover, person tracking is applied to differentiate between people in a frame of the video stream.

The Activity Recognition module is used for the gait recognition and provides information about the gait activity of the patient (e.g., walking, standing, sitting).

The Gait Analysis module takes the pose data produced in the previous modules and turns them into useful metrics that describe a patient's gait.

The API Upload module takes the patient data from a test (e.g., the calculated metrics) and sends them to the API that would store them in a database. [4]

Claims (6)

1. Claims 1. Human gait analysis software that can automatically diagnose the onset of physical frailty in clinical patients using machine learning based human pose estimation and biometricanalysis.
2. 2. The frailty analysis software works in a high frames per second rate and thus, issues like frame skipping, that lead to reduced accuracy of the pose estimation and tracking, are avoided.
3. 3. The frailty analysis software does not comprise a marker-based motion capture method where a multitude of markers need to be carefully positioned on an individual's body. These marker-based methods are very time consuming and fatiguing processes and are also not suitable for real-life noninvasive applications and the equipment needed is quite expensive.
4. 4. The frailty analysis software is not based on human silhouette as this method suffers from manyfactors such as movement on scene, clothing and carrying conditions.
5. 5. The human pose estimation algorithm used in the frailty analysis software does not result inlocal maxima as in the case of the Iterative Closet Point (ICP) algorithm where initial configuration is critical.
6. 6. The frailty analysis software does not make use of any Inertial Measurement Unit (IMU) sensorsand wearable devices to measure patient motion which increases the complexity of the processing and disturbs the in-clinic patient's experience.