US20190029569A1 - Activity analysis, fall detection and risk assessment systems and methods - Google Patents
Activity analysis, fall detection and risk assessment systems and methods Download PDFInfo
- Publication number
- US20190029569A1 US20190029569A1 US16/108,432 US201816108432A US2019029569A1 US 20190029569 A1 US20190029569 A1 US 20190029569A1 US 201816108432 A US201816108432 A US 201816108432A US 2019029569 A1 US2019029569 A1 US 2019029569A1
- Authority
- US
- United States
- Prior art keywords
- processor
- walk
- dimensional data
- person
- walking
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
- A61B5/1126—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb using a particular sensing technique
- A61B5/1128—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb using a particular sensing technique using image analysis
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0002—Remote monitoring of patients using telemetry, e.g. transmission of vital signals via a communication network
- A61B5/0004—Remote monitoring of patients using telemetry, e.g. transmission of vital signals via a communication network characterised by the type of physiological signal transmitted
- A61B5/0013—Medical image data
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0002—Remote monitoring of patients using telemetry, e.g. transmission of vital signals via a communication network
- A61B5/0015—Remote monitoring of patients using telemetry, e.g. transmission of vital signals via a communication network characterised by features of the telemetry system
- A61B5/0022—Monitoring a patient using a global network, e.g. telephone networks, internet
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0033—Features or image-related aspects of imaging apparatus classified in A61B5/00, e.g. for MRI, optical tomography or impedance tomography apparatus; arrangements of imaging apparatus in a room
- A61B5/004—Features or image-related aspects of imaging apparatus classified in A61B5/00, e.g. for MRI, optical tomography or impedance tomography apparatus; arrangements of imaging apparatus in a room adapted for image acquisition of a particular organ or body part
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
- A61B5/0077—Devices for viewing the surface of the body, e.g. camera, magnifying lens
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
- A61B5/1113—Local tracking of patients, e.g. in a hospital or private home
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
- A61B5/1116—Determining posture transitions
- A61B5/1117—Fall detection
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
- A61B5/112—Gait analysis
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/74—Details of notification to user or communication with user or patient ; user input means
- A61B5/742—Details of notification to user or communication with user or patient ; user input means using visual displays
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/74—Details of notification to user or communication with user or patient ; user input means
- A61B5/746—Alarms related to a physiological condition, e.g. details of setting alarm thresholds or avoiding false alarms
-
- G06F19/00—
-
- G06K9/00342—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/215—Motion-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/285—Analysis of motion using a sequence of stereo image pairs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/23—Recognition of whole body movements, e.g. for sport training
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
- G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
- G16H40/67—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for remote operation
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/30—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for calculating health indices; for individual health risk assessment
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16Z—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS, NOT OTHERWISE PROVIDED FOR
- G16Z99/00—Subject matter not provided for in other main groups of this subclass
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30241—Trajectory
Definitions
- the present invention relates to methods and systems for activity monitoring of a patient, and more specifically, to methods and systems for obtaining measurements of temporal and spatial gait parameters of the patient for use in health risk assessment.
- Human activity analysis from video is an open problem that has been studied within the areas of video surveillance, homeland security, and eldercare.
- the monitoring of human activity is often employed in the medical industry to detect any abnormal or dangerous events, such as falls and/or the risk of falls for a patient.
- Various parameters such as gait parameters and/or other locomotive measurements corresponding to a medical patient, are often monitored and considered indispensable in the diagnosis of frailty and fall risk, and in particular, when providing medical care for the elderly.
- Falls are a significant issue among the elderly. For example, it is estimated that between 25-35% of people 65 years and older fall each year, and many of such falls result in serious injuries, such as hip fractures, head traumas, and the like. Moreover, the medical costs associated with such falls are astronomical. In the year 2000, it is estimated that over $19 billion dollars were spent treating fall-related injuries for the elderly. Such costs do not account for the decreased quality of life and other long term effects often experienced by many elderly patients after suffering a fall.
- a low-cost monitoring system that would allow for continuous, standardized assessment of fall risk can help address falls and the risk of falls among older adults. Moreover, to enable older adults to continue living longer, in particular, in an independent setting, and thus reduce the need for expensive care facilities, low-cost systems are needed that detect both adverse events such as falls, and the risk of such events.
- the present disclosure provides a method for determining the risk of a person falling.
- the method includes acquiring depth image data that comprises a plurality of frames that depict a person walking through a home, and extracting a foreground object from the depth image data.
- the method additionally includes generating a three-dimensional data object based on the foreground object, and identifying a walking sequence from the three-dimensional data object.
- the method further includes generating one or more gait parameters from the identified walking sequence, and comparing the one or more gait parameters against a standard clinical measure of the one or more gait parameters to determine a level of risk at which the person is of falling.
- FIG. 1 is a block diagram illustrating a computing environment for obtaining one or more parameters to perform health risk assessments, according to various embodiments of the present disclosure.
- FIG. 2 is a block diagram illustrating an example living unit, according to various embodiments of the present disclosure.
- FIG. 3 is a block diagram illustrating a remote device, according to various embodiments of the present disclosure.
- FIG. 4 is a flowchart illustrating an example for obtaining temporal and spatial gait parameters for performing health risk assessments, according to various embodiments of the present disclosure.
- FIG. 4A is a flowchart illustrating an example for obtaining temporal and spatial gait parameters, performing health risk assessments, and sending a an alert if a fall is detected, according to various embodiments of the present disclosure.
- FIG. 5 is a flowchart illustrating walk sequences, according to various embodiments of the present disclosure.
- Various embodiments of the present disclosure include methods and corresponding systems for performing health risk assessments for a patient in the home environment.
- depth image data for a medical patient is obtained and subsequently used to generate one or more parameters, such as temporal and spatial gait parameters.
- the generated parameters can be used with other medical information related to the patient, such as electronic health records, to perform various health risk assessments, such as for example, alerting health care professionals of alarming trends or other health risks associated with the patient.
- Falls represent a substantial health risk among the elderly, as the risk of falls generally increases with age. It is estimated that one out of every three older adults (age 65 and over) falls each year, many of which suffer serious injuries, such as hip fractures, head traumas, etc. Typically, the result of such falls is a reduction in a person's gait ability, such as a reduction in mobility and independence, all of which can ultimately increase the risk of early death.
- the causes of such falls are known as “risk” factors. Although, generally, no single risk factor can be considered the single cause of a given fall, the greater the number of risk factors to which an individual is exposed, the greater the probability of a fall and the more likely the results of the fall will threaten the person's independence.
- gait parameters which describe the pattern of movement in animals and humans, are indispensable in assessing risk factors, making fall risk assessments, the diagnosis of fall risk, or the like. For example, studies have indicated that gait parameters can be predictive of future falls and adverse events in older adults and, further, that scores on certain mobility tests are good indicators of fall risk. Despite these findings, gait parameters and mobility tests are generally assessed infrequently, if at all, and are typically monitored through observation by a clinician with a stop watch or a clinician using equipment in a physical performance lab, both of which are expensive and labor-intensive. Such sparse, infrequent evaluations can not be representative of a person's true functional ability.
- Various embodiments of the present disclosure involve methods and systems for monitoring patient gait parameters continuously, during everyday activity, in a cost-effective, efficient manner. Monitoring such parameters and/or activities can offer significant benefits for fall risk and mobility assessment.
- FIG. 1 illustrates an example system 100 for obtaining depth image data and subsequently processing the depth image data to generate gait parameters (both temporal and spatial) for use in health risk assessment, in accordance with various embodiments of the present disclosure.
- the system 100 is an example platform in which one or more embodiments of the methods can be used. However, it is contemplated that such methods and/or processes can also be performed on other conventional computing platforms, as are generally known in the art.
- a user such as an administrator, clinician, researcher, family member, etc., can use a remote device 102 to receive and/or otherwise obtain depth image data from one or more depth camera(s) 108 .
- Depth image data can include any type of data captured from a camera capable of being processed to generate a representation of an object, and in particular, a patient in a given location, such as a three-dimensional point cloud representation, of that person or patient.
- the depth image data can include audio and can be captured in an audio format, such as by one or more microphones associated with the depth cameras 108 , or other type of recording device.
- the remote device 102 can be located in a living unit, outside a living unit but in a living community, or in a location outside the living community such as a hospital setting, and can include various hardware and accompanying software computing components that can be configured to receive and/or otherwise capture and process the depth image data.
- the remote device 102 can execute an image analysis application 109 that receives depth image data associated with a particular patient. Subsequently, the image analysis application 109 can process the depth image data to extract, generate and/or otherwise compute temporal and/or spatial gait parameters of a patient for use in various health risk assessments.
- the image analysis application 109 can provide the temporal and/or spatial gait parameters and corresponding risk assessments for display, such as for example, as part of a graphical user interface.
- a user can use the remote device 102 as a stand-alone device to compute temporal and spatial gait parameters for use in health risk assessment, or can use the remote device 102 in combination with a central computing device 106 available over a network 104 .
- the central computing device 106 can also be under the control of the same user but at a remote location, such as a location outside of the living community.
- the remote device 102 can be in a client-server relationship with the central computing device 106 , a peer-to-peer relationship with the central computing device 106 , or in a different type of relationship with the central computing device 106 .
- the client-server relationship can include a thin client on the remote device 102 .
- the client-server relationship can include a thick client on the remote device 102 .
- the remote device 102 can communicate with the central processing device 106 over a network 104 , which can be the Internet, an intranet, a local area network, a wireless local network, a wide area network, or another communication network, as well as combinations of networks.
- the network 104 can be a Mobile Communications (GSM) network, a code division multiple access (CDMA) network, an Internet Protocol (IP) network, a Wireless Application Protocol (WAP) network, a WiFi network, or an IEEE 802.11 standards network, as well as various combinations thereof.
- GSM Mobile Communications
- CDMA code division multiple access
- IP Internet Protocol
- WAP Wireless Application Protocol
- WiFi Wireless Fidelity
- IEEE 802.11 IEEE 802.11
- the central computing device 106 can include various hardware and accompanying software computing components to operate in substantially the same manner as the remote device 102 to receive depth image data.
- the central computing device 106 can be a single device.
- the central computing device 106 can include multiple computer systems.
- the multiple computer systems can be in a cloud computing configuration.
- One or more depth cameras 108 and/or sets of depth cameras 108 can be included in the system 100 to generate video signals of the objects (e.g., persons) residing in the living unit.
- the depth cameras 108 and/or sets of depth cameras 108 can include various computing and camera/lense components such as an RGB camera, infrared sensitive camera, from which a depth image and/or depth image data can be obtained.
- Other computing and/or camera components can also be included, as are generally known in the art.
- An example configuration of one or more depth cameras 108 in various living areas is described in greater detail below.
- the remote device 102 , the central computing device 106 , or both can communicate with a database 110 .
- the database 110 can include depth image data 112 and parameters 114 .
- the depth image data 112 can be stored based on the video signals generated by the depth cameras 108 .
- the depth image data 112 can include depth data, such as a pattern of projected light, from which a depth image can be produced.
- the video signals generated by the depth cameras 108 prior to converting the images to depth images are not stored in the database 110 or elsewhere in the system 100 .
- the processing performed on the depth image data 112 can be stored as the parameters 114 in the database 110 .
- the depth image data 112 can be used to track the person's activity as described in greater detail below.
- While various embodiments of the present disclosure have been described as being performed using multiple devices within a computing environment, such as computing environment 100 shown in FIG. 1 , it is contemplated that such various embodiments can be performed locally, using only a single device, such as the central processing device 106 , and in such cases the remote device 102 is integrated into or otherwise in direct connection with the central processing device 106 . In such an arrangement, the central processing device 106 can be in direct communication with the depth cameras 108 and the database 110 .
- FIG. 2 illustrates an example living unit 200 , according to an example embodiment.
- the living unit 200 is shown to have a person 202 (e.g., a medical patient being monitored) in an area 204 of the living unit 200 .
- the depth cameras 108 of FIG. 1 are shown as two depth cameras 206 , 208 , although the system can be implemented using a single depth camera 208 or more than two depth cameras. These depth cameras 206 , 208 can be deployed in the living unit 200 to generate video signals depicting the person 202 from different views in the area 204 .
- the depth cameras 206 , 208 can be Microsoft KinectTM cameras that are placed at various locations within the area 204 , capable of performing 3D motion tracking using a skeletal model, gesture recognition, facial recognition, and/or voice recognition.
- Each Microsoft KinectTM camera can include one or more sensors, an IR sensitive camera, or the like, that use a pattern of actively emitted infrared light in combination with a complementary metal-oxide-semiconductor (“CMOS”) image sensor and/or an IR-pass filter to obtain depth image data, such as a depth image, that is generally invariant to ambient lighting.
- CMOS complementary metal-oxide-semiconductor
- Each Microsoft KinectTM camera can also include a standard RGB camera and/or other camera components as are generally known in the art.
- the depth cameras 206 , 208 can capture image depth data of the person 202 , such as 3D motion tracking data at 30 frames per second, all of which can be invariant to changes in visible light.
- the depth cameras 206 , 208 are static in the area 204 . As such, the depth cameras 206 , 208 cannot physically move locations within the living unit 200 , change focus, or otherwise alter their view of the area 204 .
- the depth cameras 206 , 208 can be deployed to generate additional video signals of the person 202 . The depth cameras 206 , 208 can then be appropriately deployed in the area 204 or elsewhere in the living unit 200 to generate video signals of the person 202 .
- the video signals generated by the depth cameras 206 , 208 can be provided to the remote device 102 shown in the form of a computing system 210 .
- the computing system 210 is deployed in the living unit 200 .
- the computing system 210 can be elsewhere.
- Any depth image data 112 captured from the depth cameras 206 , 208 e.g., a Microsoft KinectTM camera
- Entropy is used as a measure of regularity in gait. A comprehensive explanation of Entropy is described in an Appendix entitled: “In-Home Measurement Of The Effect Of Strategically Weighted Vests On Ambulation,” which is incorporated by reference in its entirety herein.
- FIG. 3 is an example block diagram illustrating the various hardware and/or software components of the remote device 102 according to one exemplary embodiment of the present disclosure.
- the remote device 102 can include a processing system 302 that can be used to execute the image analysis application 109 that receives depth image data (i.e. depth image data 112 ) and generates one or more temporal and/or spatial gait parameters for health risk assessment.
- the processing system 302 can include memory and/or be in communication with a memory 322 , which can include volatile and/or non-volatile memory.
- the processing system 302 can also include various other computing components.
- the remote device 102 can include a computer readable media (“CRM”) 304 , which can include computer storage media, communication media, and/or another available computer readable media medium that can be accessed by the processing system 302 .
- CRM 304 can include non-transient computer storage media and communication media.
- computer storage media includes memory, volatile media, non-volatile media, removable media, and/or non-removable media implemented in a method or technology for storage of information, such as machine/computer readable/executable instructions, data structures, program modules, or other data.
- Communication media includes machine/computer readable/executable instructions, data structures, program modules, or other data.
- the CRM 304 is configured with the image analysis application 109 .
- the image analysis application 109 includes program instructions and/or modules that are executable by the processing system 302 .
- program modules include routines, programs, instructions, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types.
- the image analysis application 109 can include a receiving module 306 that receives depth image data 112 from one or more depth cameras 108 .
- the receiving module 306 can receive a pattern of projected infrared light from a Microsoft KinectTM camera.
- the depth image data 112 received from the Microsoft KinectTM camera (at 30 frames per second) can be an 11-bit 640 ⁇ 480 image which is invariant to visible lighting.
- the precision of the distance measurement for each pixel is dependent on the distance from the KinectTM, with the precision decreasing from approximately one centimeter at two meters to approximately ten centimeters at six meters.
- the depth image data 112 can be stored in a database or other type of data store, where each data entry in the database corresponds to a walk and/or walk sequence identified in a particular space, such as an apartment corresponding to the patient.
- a calibration module 308 can estimate calibration parameters for the depth cameras 108 .
- the depth cameras 108 are Microsoft KinectTM cameras
- intrinsic, distortion, and stereo parameters for the IR and the RGB cameras of the KinectTM can be estimated according to a calibration pattern, such as a checkerboard calibration pattern and/or the like.
- calibration of any depth image data 112 returned from the Microsoft KinectTM can be performed, as the depth image data 112 returned from the KinectTM can require some form of transformation to obtain usable and accurate distances.
- the following equations can be used to transform a raw KinectTM depth image data depth value, D, an integer value typically in the range [660, 1065], for a given pixel, (x, y), to a distance, d:
- x′ and y′ are the normalized pixel coordinates computed using the intrinsic and distortion parameters of the IR camera.
- the parameters b, f, k 1 , k 2 , k 3 , and k 4 are optimized over a large ( ⁇ 3,000) set of training points and the equation attempts to adjust for distortion effects.
- the training points are obtained by placing a large checkerboard calibration pattern in the environment, while moving the KinectTM over a large range of distances and viewing angles with respect to the pattern.
- the position of the calibration pattern with respect to the camera in each frame can be estimated. Simultaneously, the values associated with the pattern in the depth image data can be recorded.
- a global optimization is performed using, for example, the CMA-ES algorithm although other optimization algorithms can be used.
- the CMA-ES algorithm is an optimization algorithm used to find a solution that minimizes an objective function.
- Example values for the parameters ⁇ b, f, k 1 , k 2 , k 3 , k 4 ⁇ used to transform the raw depth values to inches are ⁇ 14145.6, 1100.1, 0.027, ⁇ 0.014, 1.161, 3.719 ⁇ .
- the receiving module 306 can automatically initiate a computation module 310 that analyzes, parses, and/or otherwise processes the depth image data to generate one or more parameters by executing one or more algorithms and/or equations.
- the computation module 310 can extract gait parameters, such as walking speed, stride time, and stride length from the depth image data (e.g., a 3-dimensional representation of objects within a space, such as a room within a home or apartment).
- foreground objects represented as a set of 3D points
- a tracking algorithm can be used to track any extracted 3D objects and/or points. Walks can then be identified from the path histories of the tracked objects. More particularly, a set of criteria including path straightness, speed, duration, and distance can be used to identify suitable walks from the path histories.
- the computation module 310 can initiate a background model module 312 , which executes a background subtraction algorithm, optionally, in conjunction with a background model initialization algorithm and/or a background model updating algorithm, to generate a background model.
- the background module 312 can generate the background model from the depth image data 112 captured by the depth cameras 108 .
- the background modeling algorithm can use a mixture of distributions approach typically run at 15 frames per second. The distributions are simple ranges defined by a minimum and maximum value.
- the background model consists of K b background and K f foreground distributions for each pixel in the disparity image. Each distribution, D k (x,y), is defined by three floating point values, an upper bound, a lower bound, and a weight:
- the background modeling algorithm can be initialized over a set of training frames using the procedure defined in “Algorithm 1—Background Model Initialization” as described below:
- Dj(x,y) refers to distribution j for pixel x,y and contains and W j (x,y) refers to the weight of distribution j for pixel x,y.
- the variable I j (x,y) refers to the lower bound of distribution j for pixel x,y and u j (x,y) refers to the lower bound of distribution j for pixel x,y.
- p i (x,y) refers to the value of pixel x,y in image i.
- ⁇ W is a value that, in addition to being added to a distribution's weight if a distribution is matched, can be subtracted from a distribution's weight if the distribution is not matched given the pixel has a valid depth value.
- W adapt represents the threshold at which a foreground distribution will be converted to a background distribution.
- ⁇ R represents a value that is used to keep the upper and lower bounds of a distribution from simply growing apart over time.
- a foreground/segmentation module 314 can process any background models (i.e., the frames) generated by the background model module 312 to extract, segment, classify and/or otherwise identify a foreground and/or foreground pixels. Thus, the foreground/segmentation module 314 can classify a pixel as foreground and/or as background.
- a 3D Segmentation module 316 can generate three-dimensional (“3D”) models for tracking from any extracted foreground. Specifically, given the extracted foreground for a frame, 3D objects are formed and evaluated for tracking.
- the intrinsic and extrinsic calibration parameters generated by the calibration module 308 can be processed by the computation module 310 to convert the 3D foreground pixels into a set of 3D points.
- object segmentation by the 3D segmentation module 316 is performed. More particularly, the set of the 3D points can be projected onto a discretized (1 ⁇ 1 inch) ground plane and single-linkage clustering is used to group the points into objects.
- the ground plane is discretized to limit the number of points considered by the clustering algorithm, and a distance of six inches is used for the single-linkage clustering threshold.
- various parameters can be extracted from each 3D object (a cloud of 3D points), at each frame: avg x/y/z, max x/y/z, min x/y/z, covariance matrix, time stamp, ground plane projection of points below 22 inches and a correlation coefficient based on such a projection.
- V a threshold
- the 3D objects obtained from the current frame are compared against a set of currently tracked objects. All those new objects which match an existing object based on location and volume are used to update the existing tracked object, while all those that do not match an existing object are used to create new entries in the tracked object list. Each tracked object maintains a history of up to 30 seconds. Tracked objects are discarded if not updated for 20 seconds.
- a sequence identification module 318 can be used to automatically identify walking sequences from the 3D objects (e.g. the tracked objects). Subsequently, the analyzed walking sequences can be processed by the computation module 310 to generate various gait parameters such as, in one embodiment, a walking speed, average speed, peek speed, stride time (e.g. individual stride time), and/or stride length (e.g. individual stride length), average stride length, height of the person walking, among others.
- the identification of walk sequences can be determined using the histories of the tracked 3D objects. After each new frame, the history of each tracked object is evaluated to determine if a walk has just started, ended, or is currently in progress. For example, upon initialization of a 3D object in the tracked object set, the object is assumed to “not be in a walk” (operation 502 ). The object stays in the “not in walk” state until the speed of the object goes above a threshold, T. Thus, the state of the object changes from “not in walk” to “in walk” (operation 504 ).
- the object remains in such a state until one of two conditions is met: 1) the object's velocity drops below the threshold; or 2) the current walk does not meet a straightness requirement (operation 506 ). Upon one of these two conditions being met, the length and duration of the walk are assessed to determine if the walk should be analyzed for stride parameters and saved (operation 510 ). If the walk is saved, the state of the object returns to “not in walk” (operation 512 ). However, if the walk is not saved and the straightness requirement was the reason for termination, then the oldest points in the walk are iteratively discarded until the remaining points meet the straightness requirement. The state of the object is then returned to the in walk state (operation 514 ).
- the straightness requirement consists of two measures: a global measure focused on the straightness of the entire path, and a local measure focused on abrupt changes.
- the first measure represents the average squared distance of each point in the sequence to a best fit line.
- the second measure represents the maximum deviation in walking direction computed over a small sliding window vs. that of the best fit line for the entire walk. Thresholds for both measures control the degree of straightness required for a walking sequence to be saved. In order to diminish the potential impact of capturing the beginning or the end of a walk on the computed average speed, only the middle 50 percent (based on time) of each walk is used to compute average speed.
- the resulting output can be a dataset in which each entry corresponds to a walk identified in a given area.
- Each entry can be associated with the following features: height of the person, walking speeds, and, if possible, average stride time and average stride length, in addition to the time the walk occurred.
- each walk, x i is initially associated with either two or four features:
- x i ⁇ ⁇ h , s ⁇ if ⁇ ⁇ no ⁇ ⁇ stride ⁇ ⁇ data ⁇ h , s , st , sl ⁇ else
- h, s, st, and sl are height, walking speed, stride time, and stride length, respectively.
- stride time and stride length values are estimated for the walks lacking them using the mean of the three nearest neighbors with stride information.
- the dataset can include walks from all the persons (e.g. person 202 ) of the area 204 (e.g. an apartment), as well as any visitors. As such, before any gait measurement estimates can be performed, a procedure for identifying walks from the specific person(s) is necessary.
- GMM Gaussian Mixture Model
- the Gaussian distribution representing of each person is used to identify walks from that particular person. Any walk whose likelihood given a distribution is greater than a threshold is assumed to be from the person that the distribution represents, and is used in computing gait parameter estimates for that person.
- the classification can be performed independently for each distribution. Thus, a walk could be included in the estimates of more than one person, if the distributions overlap.
- the steps of model initialization and updating are described below and illustrated in FIG. 2 .
- An output module 320 processes the depth image data 112 and/or the one or more generated parameters 114 to perform one or more health risk assessments.
- the parameters 114 and depth image data 112 can be used to assess a patient's risk of falling and/or the onset of illness.
- the actual assessment of fall/health risk can be based on mapping the various gait parameters to standard clinical measures such as a Timed-up-and-Go (TUG) test, and the Habitual Gait Speed (HGS) test.
- TUG Timed-up-and-Go
- HGS Habitual Gait Speed
- a simple neural network model that can “predict” TUG time based on an individual person's average gait speed. It is contemplated that any gait parameter can be mapped to standard measures. For example, a TUG time above 16 or 20 seconds indicates a high risk of falling in the next year.
- the gait parameter data and/or gait parameter estimates can be used to predict a score that a person, such as a patient, would receive on various clinical measures, tests, and the like, such as the TUG, HGS, Berg Balance-Short Form, Short Physical Performance Battery, and Multi-Directional Reach Test data, etc.
- FIG. 4 depicts an example method and/or process 400 for obtaining depth image data and subsequently processing the depth image data to generate temporal and spatial gait parameters for use in health risk assessment.
- Process 400 can be executed by at least one processor encoded with, or executing instructions of, an image analysis application 109 .
- process 400 includes receiving depth image data for a particular patient from one or more depth cameras 108 .
- depth image data can be received from a Microsoft KinectTM camera device located in the home of an elderly woman.
- the depth image data can be analyzed to generate at least one three-dimensional object. For example, a three-dimensional representation of the elderly gentlemen patient can be generated.
- a walking sequence can be identified based on the at least one three-dimensional object. For example, a walking sequence corresponding to the elderly woman patient can be identified.
- one or more parameters can be generated from the walking sequence. For example, one or more temporal and spatial gait parameters can be generated corresponding to the elderly woman.
- the generated parameters are used to perform various health risk assessments for a particular patient (e.g. the elderly gentlemen) and results of the health risk assessments can be provided for display at 412 .
- the image analysis application 109 can additionally include an alert module 321 (shown in FIG. 3 ).
- an alert module 321 shown in FIG. 3 .
- operation of the system 100 and execution of the image analysis application 109 as described above, generates a health risk assessment indicating that a high risk of falling is present or detects that an actual fall has occurred, in addition to providing the health risk assessment results for display, as indicated at 412 , execution of the alert module 321 will send and alert message to a programmable list of caregivers, e.g., doctors and/or family members, as indicated at 414 .
- the alert message can be in any form and/or format suitable for alerting the list of caregivers.
- the alert can be a visual message (e.g., a flashing light and/or an email and/or a text message), and/or an audible message (e.g., a beep or selected ringtone) and/or a tactile message (e.g., vibration of smartphone) sent to the caregivers(s) via a smartphone, desktop or laptop computer, computer tablet, television or any other suitable personal data/communication device connected to the central processing device 106 via the Internet 104 or any of the network systems described above.
- a visual message e.g., a flashing light and/or an email and/or a text message
- an audible message e.g., a beep or selected ringtone
- a tactile message e.g., vibration of smartphone
- the alert message can contain information about the fall or the detection of a high risk of falling, such as confidence of detection, time of occurrence of the incident (i.e., the recorded depth image data) that evoked the alert message, location of the incident that evoked the alert message, presence of another person besides the patient in the room at the time of the incident that evoked the alert message.
- the alert message can include a hyperlink to video data, stored on the database 110 , containing the depth imagery of the detected incident.
- the video data can be any suitable type of video data, such as digital video data, analog video data, voxel (volume element) image video data, or some combination thereof.
- video data can be playable and viewed using any suitable media player software, e.g., a media player program or computer app, whereby the video can be rewound, replayed and fast forwarded to allow the caregiver(s) to review the depth imagery in detail over a specified time period.
- a video hyperlink feature with the rewind, replay and fast forward capability can aid the caregiver(s), e.g., doctors, in determining whether emergency assistance is required and/or whether additional diagnostic tests are warranted, e.g., testing for stroke, testing for heart attack, X-rays, etc.
- Voxel data represents data values on a regular grid in three dimensional space.
- Voxel is a combination of “volumetric” and “pixel” where pixel is a combination of “picture” and “element”.
- voxels themselves do not typically have their position (i.e., their coordinates) explicitly encoded along with their values. Instead, the position of a voxel is inferred based upon its position relative to other voxels (i.e., its position in the data structure that makes up a single volumetric image). Voxels are effectively utilized to represent regularly sampled spaces that are non-homogeneously filled.
- the described disclosure can be provided as a computer program product, or software, that can include a machine-readable medium having stored thereon instructions which can be used to program a computer system (or other electronic devices) to perform a process according to the present disclosure.
- a machine-readable medium includes any mechanism for storing information in a form (e.g., software, processing application) readable by a machine (e.g., a computer).
- the machine-readable medium can include, but is not limited to, magnetic storage medium (e.g., floppy diskette); optical storage medium (e.g., CD-ROM); magneto-optical storage medium; read only memory (ROM); random access memory (RAM); erasable programmable memory (e.g., EPROM and EEPROM); flash memory; or other types of medium suitable for storing electronic instructions.
- magnetic storage medium e.g., floppy diskette
- optical storage medium e.g., CD-ROM
- magneto-optical storage medium e.g., magneto-optical storage medium
- ROM read only memory
- RAM random access memory
- EPROM and EEPROM erasable programmable memory
- flash memory or other types of medium suitable for storing electronic instructions.
Abstract
A method for determining the risk of a person falling is provided. The method includes acquiring depth image data that comprises a plurality of frames that depict a person walking through a home, and extracting a foreground object from the depth image data. The method additionally includes generating a three-dimensional data object based on the foreground object, and identifying a walking sequence from the three-dimensional data object. The method further includes generating one or more gait parameters from the identified walking sequence, and comparing the one or more gait parameters against a standard clinical measure of the one or more gait parameters to determine a level of risk at which the person is of falling.
Description
- This application is a continuation of Ser. No. 15/424,375, which is a continuation of U.S. patent application Ser. No. 14/169,508 filed on Jan. 31, 2014, which is a continuation-in-part of U.S. patent application Ser. No. 13/871,816 filed on Apr. 26, 2013, which claims priority under 35 U.S.C. § 119(e) to co-pending provisional applications, including Application No. 61/788,748 entitled “Activity Analysis, Fall Detection And Risk Assessment Systems And Methods” filed on Mar. 15, 2013; Application No. 61/649,770 entitled “Activity Analysis, Fall Detection And Risk Assessment Systems And Methods” filed on May 21, 2012; and Application No. 61/687,608 entitled “Activity Analysis, Fall Detection, and Risk Assessment Using Depth Camera for Eldercare and Other Monitoring Applications” filed on Apr. 27, 2012. The disclosure of the above applications are incorporated herein by reference in their entirety.
- The present invention relates to methods and systems for activity monitoring of a patient, and more specifically, to methods and systems for obtaining measurements of temporal and spatial gait parameters of the patient for use in health risk assessment.
- The statements in this section merely provide background information related to the present disclosure and cannot constitute prior art.
- Human activity analysis from video is an open problem that has been studied within the areas of video surveillance, homeland security, and eldercare. For example, the monitoring of human activity is often employed in the medical industry to detect any abnormal or dangerous events, such as falls and/or the risk of falls for a patient. Various parameters, such as gait parameters and/or other locomotive measurements corresponding to a medical patient, are often monitored and considered indispensable in the diagnosis of frailty and fall risk, and in particular, when providing medical care for the elderly.
- Falls are a significant issue among the elderly. For example, it is estimated that between 25-35% of people 65 years and older fall each year, and many of such falls result in serious injuries, such as hip fractures, head traumas, and the like. Moreover, the medical costs associated with such falls are astronomical. In the year 2000, it is estimated that over $19 billion dollars were spent treating fall-related injuries for the elderly. Such costs do not account for the decreased quality of life and other long term effects often experienced by many elderly patients after suffering a fall.
- Thus, a low-cost monitoring system that would allow for continuous, standardized assessment of fall risk can help address falls and the risk of falls among older adults. Moreover, to enable older adults to continue living longer, in particular, in an independent setting, and thus reduce the need for expensive care facilities, low-cost systems are needed that detect both adverse events such as falls, and the risk of such events.
- It is with these concepts in mind, among others, that various embodiments of the present disclosure were conceived.
- The present disclosure provides a method for determining the risk of a person falling is provided. In various embodiments, the method includes acquiring depth image data that comprises a plurality of frames that depict a person walking through a home, and extracting a foreground object from the depth image data. In various embodiments, the method additionally includes generating a three-dimensional data object based on the foreground object, and identifying a walking sequence from the three-dimensional data object. In various embodiments, the method further includes generating one or more gait parameters from the identified walking sequence, and comparing the one or more gait parameters against a standard clinical measure of the one or more gait parameters to determine a level of risk at which the person is of falling.
- Further areas of applicability of the present teachings will become apparent from the description provided herein. It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present teachings.
- The foregoing and other objects, features, and advantages of the present disclosure set forth herein will be apparent from the following description of exemplary embodiments of those inventive concepts, as illustrated in the accompanying drawings. It should be noted that the drawings are not necessarily to scale; however, the emphasis instead is being placed on illustrating the principles of the inventive concepts. Also in the drawings, the like reference characters refer to the same parts throughout the different views. The drawings depict only exemplary embodiments of the present disclosure and, therefore, are not to be considered limiting in scope.
-
FIG. 1 is a block diagram illustrating a computing environment for obtaining one or more parameters to perform health risk assessments, according to various embodiments of the present disclosure. -
FIG. 2 is a block diagram illustrating an example living unit, according to various embodiments of the present disclosure. -
FIG. 3 is a block diagram illustrating a remote device, according to various embodiments of the present disclosure. -
FIG. 4 is a flowchart illustrating an example for obtaining temporal and spatial gait parameters for performing health risk assessments, according to various embodiments of the present disclosure. -
FIG. 4A is a flowchart illustrating an example for obtaining temporal and spatial gait parameters, performing health risk assessments, and sending a an alert if a fall is detected, according to various embodiments of the present disclosure. -
FIG. 5 is a flowchart illustrating walk sequences, according to various embodiments of the present disclosure. - The following description is merely exemplary in nature and is in no way intended to limit the present teachings, application, or uses. Throughout this specification, like reference numerals will be used to refer to like elements.
- Various embodiments of the present disclosure include methods and corresponding systems for performing health risk assessments for a patient in the home environment. In various embodiments, depth image data for a medical patient is obtained and subsequently used to generate one or more parameters, such as temporal and spatial gait parameters. Subsequently, the generated parameters can be used with other medical information related to the patient, such as electronic health records, to perform various health risk assessments, such as for example, alerting health care professionals of alarming trends or other health risks associated with the patient.
- Falls represent a substantial health risk among the elderly, as the risk of falls generally increases with age. It is estimated that one out of every three older adults (age 65 and over) falls each year, many of which suffer serious injuries, such as hip fractures, head traumas, etc. Typically, the result of such falls is a reduction in a person's gait ability, such as a reduction in mobility and independence, all of which can ultimately increase the risk of early death. The causes of such falls are known as “risk” factors. Although, generally, no single risk factor can be considered the single cause of a given fall, the greater the number of risk factors to which an individual is exposed, the greater the probability of a fall and the more likely the results of the fall will threaten the person's independence.
- Research has shown that gait parameters, which describe the pattern of movement in animals and humans, are indispensable in assessing risk factors, making fall risk assessments, the diagnosis of fall risk, or the like. For example, studies have indicated that gait parameters can be predictive of future falls and adverse events in older adults and, further, that scores on certain mobility tests are good indicators of fall risk. Despite these findings, gait parameters and mobility tests are generally assessed infrequently, if at all, and are typically monitored through observation by a clinician with a stop watch or a clinician using equipment in a physical performance lab, both of which are expensive and labor-intensive. Such sparse, infrequent evaluations can not be representative of a person's true functional ability. Various embodiments of the present disclosure involve methods and systems for monitoring patient gait parameters continuously, during everyday activity, in a cost-effective, efficient manner. Monitoring such parameters and/or activities can offer significant benefits for fall risk and mobility assessment.
-
FIG. 1 illustrates anexample system 100 for obtaining depth image data and subsequently processing the depth image data to generate gait parameters (both temporal and spatial) for use in health risk assessment, in accordance with various embodiments of the present disclosure. Thesystem 100 is an example platform in which one or more embodiments of the methods can be used. However, it is contemplated that such methods and/or processes can also be performed on other conventional computing platforms, as are generally known in the art. - Referring now to
FIG. 1 , a user, such as an administrator, clinician, researcher, family member, etc., can use aremote device 102 to receive and/or otherwise obtain depth image data from one or more depth camera(s) 108. Depth image data can include any type of data captured from a camera capable of being processed to generate a representation of an object, and in particular, a patient in a given location, such as a three-dimensional point cloud representation, of that person or patient. In one embodiment, the depth image data can include audio and can be captured in an audio format, such as by one or more microphones associated with thedepth cameras 108, or other type of recording device. Theremote device 102 can be located in a living unit, outside a living unit but in a living community, or in a location outside the living community such as a hospital setting, and can include various hardware and accompanying software computing components that can be configured to receive and/or otherwise capture and process the depth image data. For example, as illustrated, theremote device 102 can execute animage analysis application 109 that receives depth image data associated with a particular patient. Subsequently, theimage analysis application 109 can process the depth image data to extract, generate and/or otherwise compute temporal and/or spatial gait parameters of a patient for use in various health risk assessments. Theimage analysis application 109 can provide the temporal and/or spatial gait parameters and corresponding risk assessments for display, such as for example, as part of a graphical user interface. - A user can use the
remote device 102 as a stand-alone device to compute temporal and spatial gait parameters for use in health risk assessment, or can use theremote device 102 in combination with acentral computing device 106 available over anetwork 104. In some embodiments, thecentral computing device 106 can also be under the control of the same user but at a remote location, such as a location outside of the living community. For example, according to various embodiments, theremote device 102 can be in a client-server relationship with thecentral computing device 106, a peer-to-peer relationship with thecentral computing device 106, or in a different type of relationship with thecentral computing device 106. In one embodiment, the client-server relationship can include a thin client on theremote device 102. In another embodiment, the client-server relationship can include a thick client on theremote device 102. - The
remote device 102 can communicate with thecentral processing device 106 over anetwork 104, which can be the Internet, an intranet, a local area network, a wireless local network, a wide area network, or another communication network, as well as combinations of networks. For example, thenetwork 104 can be a Mobile Communications (GSM) network, a code division multiple access (CDMA) network, an Internet Protocol (IP) network, a Wireless Application Protocol (WAP) network, a WiFi network, or an IEEE 802.11 standards network, as well as various combinations thereof. Other conventional and/or later-developed wired and wireless networks can also be used. - The
central computing device 106 can include various hardware and accompanying software computing components to operate in substantially the same manner as theremote device 102 to receive depth image data. In one embodiment, thecentral computing device 106 can be a single device. Alternatively, in another embodiment, thecentral computing device 106 can include multiple computer systems. For example, the multiple computer systems can be in a cloud computing configuration. - One or
more depth cameras 108 and/or sets ofdepth cameras 108 can be included in thesystem 100 to generate video signals of the objects (e.g., persons) residing in the living unit. Thedepth cameras 108 and/or sets ofdepth cameras 108 can include various computing and camera/lense components such as an RGB camera, infrared sensitive camera, from which a depth image and/or depth image data can be obtained. Other computing and/or camera components can also be included, as are generally known in the art. An example configuration of one ormore depth cameras 108 in various living areas is described in greater detail below. - The
remote device 102, thecentral computing device 106, or both can communicate with adatabase 110. Thedatabase 110 can includedepth image data 112 andparameters 114. Thedepth image data 112 can be stored based on the video signals generated by thedepth cameras 108. In general, thedepth image data 112 can include depth data, such as a pattern of projected light, from which a depth image can be produced. In some embodiments, the video signals generated by thedepth cameras 108 prior to converting the images to depth images are not stored in thedatabase 110 or elsewhere in thesystem 100. The processing performed on thedepth image data 112 can be stored as theparameters 114 in thedatabase 110. Thedepth image data 112 can be used to track the person's activity as described in greater detail below. - While various embodiments of the present disclosure have been described as being performed using multiple devices within a computing environment, such as
computing environment 100 shown inFIG. 1 , it is contemplated that such various embodiments can be performed locally, using only a single device, such as thecentral processing device 106, and in such cases theremote device 102 is integrated into or otherwise in direct connection with thecentral processing device 106. In such an arrangement, thecentral processing device 106 can be in direct communication with thedepth cameras 108 and thedatabase 110. -
FIG. 2 illustrates anexample living unit 200, according to an example embodiment. Theliving unit 200 is shown to have a person 202 (e.g., a medical patient being monitored) in anarea 204 of theliving unit 200. Thedepth cameras 108 ofFIG. 1 are shown as twodepth cameras single depth camera 208 or more than two depth cameras. Thesedepth cameras living unit 200 to generate video signals depicting theperson 202 from different views in thearea 204. - According to one embodiment, the
depth cameras area 204, capable of performing 3D motion tracking using a skeletal model, gesture recognition, facial recognition, and/or voice recognition. Each Microsoft Kinect™ camera can include one or more sensors, an IR sensitive camera, or the like, that use a pattern of actively emitted infrared light in combination with a complementary metal-oxide-semiconductor (“CMOS”) image sensor and/or an IR-pass filter to obtain depth image data, such as a depth image, that is generally invariant to ambient lighting. Each Microsoft Kinect™ camera can also include a standard RGB camera and/or other camera components as are generally known in the art. - For example, in one particular embodiment, the
depth cameras person 202, such as 3D motion tracking data at 30 frames per second, all of which can be invariant to changes in visible light. In some embodiments, thedepth cameras area 204. As such, thedepth cameras living unit 200, change focus, or otherwise alter their view of thearea 204. Alternatively, in other embodiments, thedepth cameras person 202. Thedepth cameras area 204 or elsewhere in theliving unit 200 to generate video signals of theperson 202. The video signals generated by thedepth cameras remote device 102 shown in the form of acomputing system 210. As shown, thecomputing system 210 is deployed in theliving unit 200. However, thecomputing system 210 can be elsewhere. Anydepth image data 112 captured from thedepth cameras 206, 208 (e.g., a Microsoft Kinect™ camera) can be used to extract or otherwise generate gait parameters of walking speed, right/left stride time and/or right/left stride length, stride to stride variability, trunk sway, gait asymmetry, entropy, and the like. Entropy is used as a measure of regularity in gait. A comprehensive explanation of Entropy is described in an Appendix entitled: “In-Home Measurement Of The Effect Of Strategically Weighted Vests On Ambulation,” which is incorporated by reference in its entirety herein. -
FIG. 3 is an example block diagram illustrating the various hardware and/or software components of theremote device 102 according to one exemplary embodiment of the present disclosure. Theremote device 102 can include aprocessing system 302 that can be used to execute theimage analysis application 109 that receives depth image data (i.e. depth image data 112) and generates one or more temporal and/or spatial gait parameters for health risk assessment. Theprocessing system 302 can include memory and/or be in communication with amemory 322, which can include volatile and/or non-volatile memory. Theprocessing system 302 can also include various other computing components. - The
remote device 102 can include a computer readable media (“CRM”) 304, which can include computer storage media, communication media, and/or another available computer readable media medium that can be accessed by theprocessing system 302. For example,CRM 304 can include non-transient computer storage media and communication media. By way of example and not limitation, computer storage media includes memory, volatile media, non-volatile media, removable media, and/or non-removable media implemented in a method or technology for storage of information, such as machine/computer readable/executable instructions, data structures, program modules, or other data. Communication media includes machine/computer readable/executable instructions, data structures, program modules, or other data. TheCRM 304 is configured with theimage analysis application 109. Theimage analysis application 109 includes program instructions and/or modules that are executable by theprocessing system 302. Generally, program modules include routines, programs, instructions, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. - According to various embodiments, the
image analysis application 109 can include areceiving module 306 that receivesdepth image data 112 from one ormore depth cameras 108. For example, the receivingmodule 306 can receive a pattern of projected infrared light from a Microsoft Kinect™ camera. More particularly, thedepth image data 112 received from the Microsoft Kinect™ camera (at 30 frames per second) can be an 11-bit 640×480 image which is invariant to visible lighting. The precision of the distance measurement for each pixel is dependent on the distance from the Kinect™, with the precision decreasing from approximately one centimeter at two meters to approximately ten centimeters at six meters. Thedepth image data 112 can be stored in a database or other type of data store, where each data entry in the database corresponds to a walk and/or walk sequence identified in a particular space, such as an apartment corresponding to the patient. - Optionally, before the receiving
module 306 receives anydepth image data 112 from thedepth cameras 108, acalibration module 308 can estimate calibration parameters for thedepth cameras 108. For example, in the embodiment in which thedepth cameras 108 are Microsoft Kinect™ cameras, intrinsic, distortion, and stereo parameters for the IR and the RGB cameras of the Kinect™ can be estimated according to a calibration pattern, such as a checkerboard calibration pattern and/or the like. Subsequently, calibration of anydepth image data 112 returned from the Microsoft Kinect™ can be performed, as thedepth image data 112 returned from the Kinect™ can require some form of transformation to obtain usable and accurate distances. For example, the following equations can be used to transform a raw Kinect™ depth image data depth value, D, an integer value typically in the range [660, 1065], for a given pixel, (x, y), to a distance, d: -
- where x′ and y′ are the normalized pixel coordinates computed using the intrinsic and distortion parameters of the IR camera. The parameters b, f, k1, k2, k3, and k4 are optimized over a large (˜3,000) set of training points and the equation attempts to adjust for distortion effects. The training points are obtained by placing a large checkerboard calibration pattern in the environment, while moving the Kinect™ over a large range of distances and viewing angles with respect to the pattern. Using the known intrinsic parameters of the IR camera of the Kinect™ the position of the calibration pattern with respect to the camera in each frame can be estimated. Simultaneously, the values associated with the pattern in the depth image data can be recorded. Following collection of the training data, a global optimization is performed using, for example, the CMA-ES algorithm although other optimization algorithms can be used. The CMA-ES algorithm is an optimization algorithm used to find a solution that minimizes an objective function. Example values for the parameters {b, f, k1, k2, k3, k4} used to transform the raw depth values to inches are {14145.6, 1100.1, 0.027, −0.014, 1.161, 3.719}.
- After receiving, and optionally calibrating, depth image data, the receiving
module 306 can automatically initiate acomputation module 310 that analyzes, parses, and/or otherwise processes the depth image data to generate one or more parameters by executing one or more algorithms and/or equations. For example, thecomputation module 310 can extract gait parameters, such as walking speed, stride time, and stride length from the depth image data (e.g., a 3-dimensional representation of objects within a space, such as a room within a home or apartment). - A brief description of the various computations that can be performed by the
computation module 310 will now be provided. Initially, foreground objects, represented as a set of 3D points, can be identified from depth image data using a dynamic background subtraction technique. Subsequently, a tracking algorithm can be used to track any extracted 3D objects and/or points. Walks can then be identified from the path histories of the tracked objects. More particularly, a set of criteria including path straightness, speed, duration, and distance can be used to identify suitable walks from the path histories. - Accordingly, initially, in one embodiment, the
computation module 310 can initiate abackground model module 312, which executes a background subtraction algorithm, optionally, in conjunction with a background model initialization algorithm and/or a background model updating algorithm, to generate a background model. Specifically, thebackground module 312 can generate the background model from thedepth image data 112 captured by thedepth cameras 108. In one embodiment, the background modeling algorithm can use a mixture of distributions approach typically run at 15 frames per second. The distributions are simple ranges defined by a minimum and maximum value. The background model consists of Kb background and Kf foreground distributions for each pixel in the disparity image. Each distribution, Dk(x,y), is defined by three floating point values, an upper bound, a lower bound, and a weight: -
D k(x,y)={[I k(x,y),u k(x,y)],W k(x,y)} - The background modeling algorithm can be initialized over a set of training frames using the procedure defined in “Algorithm 1—Background Model Initialization” as described below:
-
Algorithm 1—Background Model Initialization CONSTANTS: Wmax , Winit , ΔW INPUT: set of training disparity images, I SET: Wj(x,y) = 0, j=1:Kb+Kf , x=1:width, y=1:height for each image iϵ I for each pixel pi(x,y) , x=1:width, y=1:height if pi(x,y) = valid disparity value for each distribution Dj(x,y), j=1:Kb if Wj(x,y) > 0 and pi(x,y) matches [Ij(x,y), uj(x,y)] //Update the distribution and weight Wj(x,y) = min( Wj(x,y) + ΔW, Wmax) Ij(x,y) = min( pi(x,y) −1, Ij(x,y) ) uj(x,y) = max( pi(x,y) +1, uj(x,y)) if no distributions were matched //Replace least weight BACKGROUND distribution j = arg min Wk(x,y) k=1:Kb Wj(x,y) = Winit Ij(x,y) = pi(x,y) − 1 uj(x,y) = pi(x,y) + 1
where Wmax represents the maximum allowed weight a distribution can have; Winit represents the initial weight given to a new distribution; ΔW is the increment added to a distribution's weight if the distribution is matched. Further, Dj(x,y) refers to distribution j for pixel x,y and contains and Wj(x,y) refers to the weight of distribution j for pixel x,y. The variable Ij(x,y) refers to the lower bound of distribution j for pixel x,y and uj(x,y) refers to the lower bound of distribution j for pixel x,y. Finally, pi(x,y) refers to the value of pixel x,y in image i. - It should be noted that only background distributions are initialized over the training frames. The foreground distributions are left uninitialized with Wk(x,y)=0. Once initialized, the model is updated at each new frame using the procedure defined in “Algorithm 2—Background Model Updating” as described below:
-
Algorithm 2 - Background Model Updating CONSTANTS: Wmax , Winit , ΔW , Wadapt , ΔR INPUT: new disparity image, i for each pixel pi(x,y) , x=1:width, y=1:height if pi(x,y) = valid disparity value for each distribution Dj(x,y), j=1:Kb+Kf if Wj(x,y) > 0 and pi(x,y) matches [Ij(x,y), uj(x,y)] //Update the distribution range and weight Wj(x,y) = min( Wj(x,y) + ΔW, Wmax) Ij(x,y) = min( pi(x,y) -1, Ij(x,y) +ΔR) uj(x,y) = max( pi(x,y) + 1, uj(x,y) - ΔR) else //Decay distribution weight Wj(x,y) = max( Wj(x,y) - ΔW, 0) if no distributions were matched //Replace least weight FOREGROUND distribution Wj(x,y) = Winit Ij(x,y) = pi(x,y) - 1 uj(x,y) = pi(x,y) + 1 for each distribution Dj(x,y), j= Kb+1:Kb+Kf //Adapt FOREGROUND to BACKGROUND if Wj(x,y) > Wadapt Wk(x,y) = Wj(x,y) Ik(x,y) = Ij(x,y) uk(x,y) = uj(x,y) Wj(x,y) = 0
In Algorithm 2, Wmax and Wmin are the same as described for Algorithm 1. ΔW is a value that, in addition to being added to a distribution's weight if a distribution is matched, can be subtracted from a distribution's weight if the distribution is not matched given the pixel has a valid depth value. Wadapt represents the threshold at which a foreground distribution will be converted to a background distribution. ΔR represents a value that is used to keep the upper and lower bounds of a distribution from simply growing apart over time. - A foreground/
segmentation module 314 can process any background models (i.e., the frames) generated by thebackground model module 312 to extract, segment, classify and/or otherwise identify a foreground and/or foreground pixels. Thus, the foreground/segmentation module 314 can classify a pixel as foreground and/or as background. - More particularly, given the background model, for each pixel, the first step of foreground segmentation is to compare the disparity value of each pixel from the current frame against its background model. If the disparity value of a pixel is found to match one of its active (Wk(x,y)>0) background distributions, then the pixel is classified as background; otherwise the pixel is classified as foreground. All pixels for which a valid disparity value is not returned are assumed to be background. A pixel is found to match a distribution if it lies within the range defined by the distribution, or its distance from the range is less than a threshold T (for this work T=0.25). Following such an initial classification, a block-based filtering algorithm can be applied to eliminate noise. Finally, morphological smoothing and hole-filling is used to further clean the image.
- A
3D Segmentation module 316 can generate three-dimensional (“3D”) models for tracking from any extracted foreground. Specifically, given the extracted foreground for a frame, 3D objects are formed and evaluated for tracking. In one embodiment, the intrinsic and extrinsic calibration parameters generated by thecalibration module 308 can be processed by thecomputation module 310 to convert the 3D foreground pixels into a set of 3D points. - Following conversion of the foreground pixels to a set of 3D points, object segmentation by the
3D segmentation module 316 is performed. More particularly, the set of the 3D points can be projected onto a discretized (1×1 inch) ground plane and single-linkage clustering is used to group the points into objects. The ground plane is discretized to limit the number of points considered by the clustering algorithm, and a distance of six inches is used for the single-linkage clustering threshold. In one embodiment, various parameters can be extracted from each 3D object (a cloud of 3D points), at each frame: avg x/y/z, max x/y/z, min x/y/z, covariance matrix, time stamp, ground plane projection of points below 22 inches and a correlation coefficient based on such a projection. - An estimate of volume can be obtained for each 3D object by summing the range of Z values for each location in the discretized ground plane that are part of the object. Any objects with a volume estimate greater than or equal to a threshold, V, are considered valid and retained for tracking, while any objects with volume estimates less than V are discarded. (For this work, V=725.) The 3D objects obtained from the current frame are compared against a set of currently tracked objects. All those new objects which match an existing object based on location and volume are used to update the existing tracked object, while all those that do not match an existing object are used to create new entries in the tracked object list. Each tracked object maintains a history of up to 30 seconds. Tracked objects are discarded if not updated for 20 seconds.
- A
sequence identification module 318 can be used to automatically identify walking sequences from the 3D objects (e.g. the tracked objects). Subsequently, the analyzed walking sequences can be processed by thecomputation module 310 to generate various gait parameters such as, in one embodiment, a walking speed, average speed, peek speed, stride time (e.g. individual stride time), and/or stride length (e.g. individual stride length), average stride length, height of the person walking, among others. - In one embodiment, as illustrated in
FIG. 5 , the identification of walk sequences can be determined using the histories of the tracked 3D objects. After each new frame, the history of each tracked object is evaluated to determine if a walk has just started, ended, or is currently in progress. For example, upon initialization of a 3D object in the tracked object set, the object is assumed to “not be in a walk” (operation 502). The object stays in the “not in walk” state until the speed of the object goes above a threshold, T. Thus, the state of the object changes from “not in walk” to “in walk” (operation 504). The object remains in such a state until one of two conditions is met: 1) the object's velocity drops below the threshold; or 2) the current walk does not meet a straightness requirement (operation 506). Upon one of these two conditions being met, the length and duration of the walk are assessed to determine if the walk should be analyzed for stride parameters and saved (operation 510). If the walk is saved, the state of the object returns to “not in walk” (operation 512). However, if the walk is not saved and the straightness requirement was the reason for termination, then the oldest points in the walk are iteratively discarded until the remaining points meet the straightness requirement. The state of the object is then returned to the in walk state (operation 514). - The straightness requirement consists of two measures: a global measure focused on the straightness of the entire path, and a local measure focused on abrupt changes. The first measure represents the average squared distance of each point in the sequence to a best fit line. The second measure represents the maximum deviation in walking direction computed over a small sliding window vs. that of the best fit line for the entire walk. Thresholds for both measures control the degree of straightness required for a walking sequence to be saved. In order to diminish the potential impact of capturing the beginning or the end of a walk on the computed average speed, only the middle 50 percent (based on time) of each walk is used to compute average speed.
- The resulting output can be a dataset in which each entry corresponds to a walk identified in a given area. Each entry can be associated with the following features: height of the person, walking speeds, and, if possible, average stride time and average stride length, in addition to the time the walk occurred. Thus, each walk, xi, is initially associated with either two or four features:
-
- where h, s, st, and sl, are height, walking speed, stride time, and stride length, respectively. In order to include the information from walks without stride parameters in the computations, which due to furniture placement, etc., can make up the majority of walks in some areas (e.g. area 204), stride time and stride length values are estimated for the walks lacking them using the mean of the three nearest neighbors with stride information.
- In one particular embodiment, the dataset can include walks from all the persons (e.g. person 202) of the area 204 (e.g. an apartment), as well as any visitors. As such, before any gait measurement estimates can be performed, a procedure for identifying walks from the specific person(s) is necessary.
- One approach makes the assumption that each person will create a cluster, or mode, in the dataset, representing their typical, in-home, habitual gait. These clusters are modeled as Gaussian distributions in the 4D feature space. The basic procedure is to fit a Gaussian Mixture Model (GMM), λ={ρr, μr, Σr}, r=1, . . . , K, with the number of distributions, K, equal to the number of
persons 202 in thearea 204 to the dataset, X={x1, . . . , xN}: -
- where g(x|μr,Σr), r=1, . . . , K, are the multivariate Gaussian distributions, and ρr, r=1, . . . , K, are the mixture weights.
- The Gaussian distribution representing of each person (
e.g. person 202 such as a person) is used to identify walks from that particular person. Any walk whose likelihood given a distribution is greater than a threshold is assumed to be from the person that the distribution represents, and is used in computing gait parameter estimates for that person. The classification can be performed independently for each distribution. Thus, a walk could be included in the estimates of more than one person, if the distributions overlap. The steps of model initialization and updating are described below and illustrated inFIG. 2 . - An
output module 320 processes thedepth image data 112 and/or the one or more generatedparameters 114 to perform one or more health risk assessments. For example, theparameters 114 anddepth image data 112 can be used to assess a patient's risk of falling and/or the onset of illness. - In one embodiment, the actual assessment of fall/health risk can be based on mapping the various gait parameters to standard clinical measures such as a Timed-up-and-Go (TUG) test, and the Habitual Gait Speed (HGS) test. For example, in one embodiment, a simple neural network model that can “predict” TUG time based on an individual person's average gait speed. It is contemplated that any gait parameter can be mapped to standard measures. For example, a TUG time above 16 or 20 seconds indicates a high risk of falling in the next year. Accordingly, the gait parameter data and/or gait parameter estimates can be used to predict a score that a person, such as a patient, would receive on various clinical measures, tests, and the like, such as the TUG, HGS, Berg Balance-Short Form, Short Physical Performance Battery, and Multi-Directional Reach Test data, etc.
-
FIG. 4 depicts an example method and/orprocess 400 for obtaining depth image data and subsequently processing the depth image data to generate temporal and spatial gait parameters for use in health risk assessment.Process 400 can be executed by at least one processor encoded with, or executing instructions of, animage analysis application 109. Initially, at 402,process 400 includes receiving depth image data for a particular patient from one ormore depth cameras 108. For example, depth image data can be received from a Microsoft Kinect™ camera device located in the home of an elderly gentleman. At 404, the depth image data can be analyzed to generate at least one three-dimensional object. For example, a three-dimensional representation of the elderly gentlemen patient can be generated. At 406, a walking sequence can be identified based on the at least one three-dimensional object. For example, a walking sequence corresponding to the elderly gentleman patient can be identified. At 408, one or more parameters can be generated from the walking sequence. For example, one or more temporal and spatial gait parameters can be generated corresponding to the elderly gentleman. At 410, the generated parameters are used to perform various health risk assessments for a particular patient (e.g. the elderly gentlemen) and results of the health risk assessments can be provided for display at 412. - Referring now to
FIG. 4A , in various embodiments, theimage analysis application 109 can additionally include an alert module 321 (shown inFIG. 3 ). In such embodiments, if operation of thesystem 100 and execution of theimage analysis application 109, as described above, generates a health risk assessment indicating that a high risk of falling is present or detects that an actual fall has occurred, in addition to providing the health risk assessment results for display, as indicated at 412, execution of thealert module 321 will send and alert message to a programmable list of caregivers, e.g., doctors and/or family members, as indicated at 414. The alert message can be in any form and/or format suitable for alerting the list of caregivers. For example, the alert can be a visual message (e.g., a flashing light and/or an email and/or a text message), and/or an audible message (e.g., a beep or selected ringtone) and/or a tactile message (e.g., vibration of smartphone) sent to the caregivers(s) via a smartphone, desktop or laptop computer, computer tablet, television or any other suitable personal data/communication device connected to thecentral processing device 106 via theInternet 104 or any of the network systems described above. - Additionally, in various embodiments, the alert message can contain information about the fall or the detection of a high risk of falling, such as confidence of detection, time of occurrence of the incident (i.e., the recorded depth image data) that evoked the alert message, location of the incident that evoked the alert message, presence of another person besides the patient in the room at the time of the incident that evoked the alert message. Furthermore, in various embodiments, the alert message can include a hyperlink to video data, stored on the
database 110, containing the depth imagery of the detected incident. In such embodiments, the video data can be any suitable type of video data, such as digital video data, analog video data, voxel (volume element) image video data, or some combination thereof. Still further, such video data can be playable and viewed using any suitable media player software, e.g., a media player program or computer app, whereby the video can be rewound, replayed and fast forwarded to allow the caregiver(s) to review the depth imagery in detail over a specified time period. Such a video hyperlink feature with the rewind, replay and fast forward capability can aid the caregiver(s), e.g., doctors, in determining whether emergency assistance is required and/or whether additional diagnostic tests are warranted, e.g., testing for stroke, testing for heart attack, X-rays, etc. - As used herein voxel data represents data values on a regular grid in three dimensional space. Voxel is a combination of “volumetric” and “pixel” where pixel is a combination of “picture” and “element”. As with pixels in a bitmap, voxels themselves do not typically have their position (i.e., their coordinates) explicitly encoded along with their values. Instead, the position of a voxel is inferred based upon its position relative to other voxels (i.e., its position in the data structure that makes up a single volumetric image). Voxels are effectively utilized to represent regularly sampled spaces that are non-homogeneously filled.
- The description above includes example systems, methods, techniques, instruction sequences, and/or computer program products that embody techniques of the present disclosure. However, it is understood that the described disclosure can be practiced without these specific details. In the present disclosure, the methods disclosed can be implemented as sets of instructions or software readable by a device. Further, it is understood that the specific order or hierarchy of steps in the methods disclosed are instances of example approaches. Based upon design preferences, it is understood that the specific order or hierarchy of steps in the method can be rearranged while remaining within the disclosed subject matter. The accompanying method claims present elements of the various steps in a sample order, and are not necessarily meant to be limited to the specific order or hierarchy presented.
- The described disclosure can be provided as a computer program product, or software, that can include a machine-readable medium having stored thereon instructions which can be used to program a computer system (or other electronic devices) to perform a process according to the present disclosure. A machine-readable medium includes any mechanism for storing information in a form (e.g., software, processing application) readable by a machine (e.g., a computer). The machine-readable medium can include, but is not limited to, magnetic storage medium (e.g., floppy diskette); optical storage medium (e.g., CD-ROM); magneto-optical storage medium; read only memory (ROM); random access memory (RAM); erasable programmable memory (e.g., EPROM and EEPROM); flash memory; or other types of medium suitable for storing electronic instructions.
- It is believed that the present disclosure and many of its attendant advantages will be understood by the foregoing description, and it will be apparent that various changes can be made in the form, construction and arrangement of the components without departing from the disclosed subject matter or without sacrificing all of its material advantages. The form described is merely explanatory, and it is the intention of the following claims to encompass and include such changes.
- While the present disclosure has been described with reference to various exemplary embodiments, it will be understood that these embodiments are illustrative and that the scope of the disclosure is not limited to them. Many variations, modifications, additions, and improvements are possible. More generally, embodiments in accordance with the present disclosure have been described in the context of exemplary implementations. Functionality can be separated or combined in blocks differently in various embodiments of the disclosure or described with different terminology. These and other variations, modifications, additions, and improvements can fall within the scope of the disclosure as defined in the claims that follow.
Claims (5)
1. A method for determining the risk of a person to falling, the method comprising:
acquiring by at least one processor of a computer-based remote device, depth image data from at least one depth camera, wherein the depth image data comprises a plurality of frames that depict the person walking through a home environment over time, the frames comprising a plurality of pixels, the remote device located remotely from the at least one depth camera, the remote device comprising electronic memory on which an image analysis application is electronically stored, and the at least one processor structured and operable to execute the image analysis application;
extracting, by the at least one processor, a foreground object from the depth image data;
segmenting, by the at least one processor, the pixels of the frames of the depth image data corresponding to the foreground object;
generating, by the at least one processor, a three-dimensional data object based on the foreground object;
tracking by the at least one processor, the three-dimensional data object over a plurality of frames of the depth images data;
identifying, by the at least one processor, a walking sequence from the tracked three-dimensional data object, wherein the identifying comprises:
the at least one processor determining a speed for the tracked three-dimensional data object over a time frame;
the at least one processor comparing the determined speed with a speed threshold;
in response to the comparison indicating that the determined speed is greater than the speed threshold, the at least one processor assigning a state indicative of walking to the tracked three-dimensional data object;
while the tracked three-dimensional data object is in the assigned walking state:
the at least one processor determining a walk straightness for the tracked three-dimensional data object;
the at least one processor determining a walk length for the tracked three-dimensional data object;
the at least one processor determining a walk duration for the tracked three-dimensional data object;
the at least one processor saving the tracked three-dimensional data object in memory as the identified walking sequence if the determined walk straightness exceeds a straightness threshold, the determined walk length exceeds a walk length threshold, and the determined walk duration exceeds a walk duration threshold;
generating, by the at least one processor, one or more gait parameters from the identified walking sequence;
comparing, by the at least one processor, the one or more gait parameters against a standard clinical measure of the one or more gait parameters to determine a level of risk at which the person is of falling.
2. The method of claim 1 , wherein the identified walking sequence is compared against a previously saved walking sequence of the person to confirm that the identified walking sequence is correctly associated with the person.
3. The method of claim 2 , wherein the comparison utilizes a Gaussian distribution.
4. The method of claim 1 , wherein the one or more gait parameters includes at least one of: walking speed, stride time, or stride length.
5. The method of claim 1 , wherein the standard clinical measure is selected from the group consisting of: Timed-Up-and-Go (TUG) and Habitual Gait Speed (HGS).
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/108,432 US20190029569A1 (en) | 2012-04-27 | 2018-08-22 | Activity analysis, fall detection and risk assessment systems and methods |
Applications Claiming Priority (7)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201261687608P | 2012-04-27 | 2012-04-27 | |
US201261649770P | 2012-05-21 | 2012-05-21 | |
US201361788748P | 2013-03-15 | 2013-03-15 | |
US13/871,816 US9408561B2 (en) | 2012-04-27 | 2013-04-26 | Activity analysis, fall detection and risk assessment systems and methods |
US14/169,508 US9597016B2 (en) | 2012-04-27 | 2014-01-31 | Activity analysis, fall detection and risk assessment systems and methods |
US15/424,375 US10080513B2 (en) | 2012-04-27 | 2017-02-03 | Activity analysis, fall detection and risk assessment systems and methods |
US16/108,432 US20190029569A1 (en) | 2012-04-27 | 2018-08-22 | Activity analysis, fall detection and risk assessment systems and methods |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/424,375 Continuation US10080513B2 (en) | 2012-04-27 | 2017-02-03 | Activity analysis, fall detection and risk assessment systems and methods |
Publications (1)
Publication Number | Publication Date |
---|---|
US20190029569A1 true US20190029569A1 (en) | 2019-01-31 |
Family
ID=50773876
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/169,508 Active 2033-06-20 US9597016B2 (en) | 2012-04-27 | 2014-01-31 | Activity analysis, fall detection and risk assessment systems and methods |
US15/424,375 Active US10080513B2 (en) | 2012-04-27 | 2017-02-03 | Activity analysis, fall detection and risk assessment systems and methods |
US16/108,432 Abandoned US20190029569A1 (en) | 2012-04-27 | 2018-08-22 | Activity analysis, fall detection and risk assessment systems and methods |
Family Applications Before (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/169,508 Active 2033-06-20 US9597016B2 (en) | 2012-04-27 | 2014-01-31 | Activity analysis, fall detection and risk assessment systems and methods |
US15/424,375 Active US10080513B2 (en) | 2012-04-27 | 2017-02-03 | Activity analysis, fall detection and risk assessment systems and methods |
Country Status (1)
Country | Link |
---|---|
US (3) | US9597016B2 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10485452B2 (en) * | 2015-02-25 | 2019-11-26 | Leonardo Y. Orellano | Fall detection systems and methods |
CN111027417A (en) * | 2019-11-21 | 2020-04-17 | 复旦大学 | Gait recognition method and gait evaluation system based on human body key point detection algorithm |
Families Citing this family (65)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130127620A1 (en) | 2011-06-20 | 2013-05-23 | Cerner Innovation, Inc. | Management of patient fall risk |
US10546481B2 (en) | 2011-07-12 | 2020-01-28 | Cerner Innovation, Inc. | Method for determining whether an individual leaves a prescribed virtual perimeter |
US9741227B1 (en) | 2011-07-12 | 2017-08-22 | Cerner Innovation, Inc. | Method and process for determining whether an individual suffers a fall requiring assistance |
WO2013100790A1 (en) * | 2011-12-30 | 2013-07-04 | Intel Corporation | Coarse-to-fine multple disparity candidate stereo matching |
US10096223B1 (en) | 2013-12-18 | 2018-10-09 | Cerner Innovication, Inc. | Method and process for determining whether an individual suffers a fall requiring assistance |
US9729833B1 (en) | 2014-01-17 | 2017-08-08 | Cerner Innovation, Inc. | Method and system for determining whether an individual takes appropriate measures to prevent the spread of healthcare-associated infections along with centralized monitoring |
US10225522B1 (en) | 2014-01-17 | 2019-03-05 | Cerner Innovation, Inc. | Method and system for determining whether an individual takes appropriate measures to prevent the spread of healthcare-associated infections |
US10078956B1 (en) | 2014-01-17 | 2018-09-18 | Cerner Innovation, Inc. | Method and system for determining whether an individual takes appropriate measures to prevent the spread of healthcare-associated infections |
US10514669B1 (en) | 2014-04-25 | 2019-12-24 | State Farm Mutual Automobile Insurance Company | Systems and methods for managing the operation of devices within a property |
CA2961139A1 (en) * | 2014-09-16 | 2016-03-24 | Hip Hope Technologies Ltd. | Fall detection device and method |
US10388135B1 (en) | 2014-10-07 | 2019-08-20 | State Farm Mutual Automobile Insurance Company | Systems and methods for analyzing sensor data to detect property intrusion events |
CN105654023B (en) * | 2014-11-12 | 2019-05-03 | 株式会社理光 | The method and apparatus for identifying object risk |
WO2016081994A1 (en) * | 2014-11-24 | 2016-06-02 | Quanticare Technologies Pty Ltd | Gait monitoring system, method and device |
US10090068B2 (en) | 2014-12-23 | 2018-10-02 | Cerner Innovation, Inc. | Method and system for determining whether a monitored individual's hand(s) have entered a virtual safety zone |
US10524722B2 (en) | 2014-12-26 | 2020-01-07 | Cerner Innovation, Inc. | Method and system for determining whether a caregiver takes appropriate measures to prevent patient bedsores |
US10091463B1 (en) | 2015-02-16 | 2018-10-02 | Cerner Innovation, Inc. | Method for determining whether an individual enters a prescribed virtual zone using 3D blob detection |
CN106033601B (en) * | 2015-03-09 | 2019-01-18 | 株式会社理光 | The method and apparatus for detecting abnormal case |
US10025989B2 (en) * | 2015-05-05 | 2018-07-17 | Dean Drako | 3D event capture and image transform apparatus and method for operation |
US10342478B2 (en) | 2015-05-07 | 2019-07-09 | Cerner Innovation, Inc. | Method and system for determining whether a caretaker takes appropriate measures to prevent patient bedsores |
CN106296628B (en) * | 2015-05-11 | 2019-03-05 | 株式会社理光 | The method and apparatus for detecting abnormal case |
US9892611B1 (en) | 2015-06-01 | 2018-02-13 | Cerner Innovation, Inc. | Method for determining whether an individual enters a prescribed virtual zone using skeletal tracking and 3D blob detection |
CN105488955A (en) * | 2015-12-11 | 2016-04-13 | 哈尔滨墨医生物技术有限公司 | A security alarm system for wards |
US10813572B2 (en) * | 2015-12-11 | 2020-10-27 | Electronic Caregiver, Inc. | Intelligent system for multi-function electronic caregiving to facilitate advanced health diagnosis, health monitoring, fall and injury prediction, health maintenance and support, and emergency response |
WO2017106485A1 (en) * | 2015-12-16 | 2017-06-22 | Hologic, Inc. | Systems and methods for presenting complex medical condition diagnoses |
US10614288B2 (en) | 2015-12-31 | 2020-04-07 | Cerner Innovation, Inc. | Methods and systems for detecting stroke symptoms |
US11283873B2 (en) | 2016-04-01 | 2022-03-22 | Intel Corporation | IoT sensor fusion |
US10522251B2 (en) | 2016-07-08 | 2019-12-31 | International Business Machines Corporation | Infrared detectors and thermal tags for real-time activity monitoring |
US10376186B2 (en) * | 2016-10-18 | 2019-08-13 | International Business Machines Corporation | Thermal tags for real-time activity monitoring and methods for fabricating the same |
CN108024088B (en) * | 2016-10-31 | 2020-07-03 | 杭州海康威视系统技术有限公司 | Video polling method and device |
CN108158587B (en) * | 2016-12-05 | 2020-12-29 | 中国移动通信有限公司研究院 | Method and device for measuring indoor human body exercise amount |
US10147184B2 (en) | 2016-12-30 | 2018-12-04 | Cerner Innovation, Inc. | Seizure detection |
US10325471B1 (en) | 2017-04-28 | 2019-06-18 | BlueOwl, LLC | Systems and methods for detecting a medical emergency event |
CN107194967B (en) * | 2017-06-09 | 2021-04-06 | 南昌大学 | Human body tumbling detection method and device based on Kinect depth image |
US10055961B1 (en) * | 2017-07-10 | 2018-08-21 | Careview Communications, Inc. | Surveillance system and method for predicting patient falls using motion feature patterns |
EP3673650A4 (en) | 2017-08-23 | 2020-07-15 | Siemens Healthcare Diagnostics Inc. | Vision system for laboratory workflows |
CN107578036A (en) * | 2017-09-28 | 2018-01-12 | 南通大学 | A kind of depth image tumble recognizer based on wavelet moment |
EP3729242A4 (en) * | 2017-12-22 | 2021-01-20 | Magic Leap, Inc. | Methods and system for generating and displaying 3d videos in a virtual, augmented, or mixed reality environment |
US10643446B2 (en) | 2017-12-28 | 2020-05-05 | Cerner Innovation, Inc. | Utilizing artificial intelligence to detect objects or patient safety events in a patient room |
US10482321B2 (en) | 2017-12-29 | 2019-11-19 | Cerner Innovation, Inc. | Methods and systems for identifying the crossing of a virtual barrier |
US20190259475A1 (en) * | 2018-02-20 | 2019-08-22 | SameDay Security, Inc. | Connected Kiosk for the Real-Time Assessment of Falls Risk |
US11062469B2 (en) | 2018-03-09 | 2021-07-13 | Microsoft Technology Licensing, Llc | 4D tracking utilizing depth data from multiple 3D cameras |
US11213224B2 (en) | 2018-03-19 | 2022-01-04 | Electronic Caregiver, Inc. | Consumer application for mobile assessment of functional capacity and falls risk |
US11094180B1 (en) | 2018-04-09 | 2021-08-17 | State Farm Mutual Automobile Insurance Company | Sensing peripheral heuristic evidence, reinforcement, and engagement system |
CN108520237B (en) * | 2018-04-10 | 2020-09-22 | 武汉斑马快跑科技有限公司 | Risk behavior identification method |
US11923058B2 (en) | 2018-04-10 | 2024-03-05 | Electronic Caregiver, Inc. | Mobile system for the assessment of consumer medication compliance and provision of mobile caregiving |
US10827951B2 (en) | 2018-04-19 | 2020-11-10 | Careview Communications, Inc. | Fall detection using sensors in a smart monitoring safety system |
US11488724B2 (en) | 2018-06-18 | 2022-11-01 | Electronic Caregiver, Inc. | Systems and methods for a virtual, intelligent and customizable personal medical assistant |
US10932970B2 (en) | 2018-08-27 | 2021-03-02 | Careview Communications, Inc. | Systems and methods for monitoring and controlling bed functions |
US10922936B2 (en) | 2018-11-06 | 2021-02-16 | Cerner Innovation, Inc. | Methods and systems for detecting prohibited objects |
EP3895141B1 (en) | 2018-12-15 | 2024-01-24 | Starkey Laboratories, Inc. | Hearing assistance system with enhanced fall detection features |
US11638563B2 (en) | 2018-12-27 | 2023-05-02 | Starkey Laboratories, Inc. | Predictive fall event management system and method of using same |
US11179064B2 (en) * | 2018-12-30 | 2021-11-23 | Altum View Systems Inc. | Method and system for privacy-preserving fall detection |
US11199561B2 (en) * | 2018-12-31 | 2021-12-14 | Robert Bosch Gmbh | System and method for standardized evaluation of activity sequences |
CN109886102B (en) * | 2019-01-14 | 2020-11-17 | 华中科技大学 | Fall-down behavior time-space domain detection method based on depth image |
JP2022519283A (en) | 2019-02-05 | 2022-03-22 | エレクトロニック ケアギヴァー,インコーポレイテッド | 3D environmental risk identification using reinforcement learning |
CA3128910A1 (en) | 2019-03-28 | 2020-10-01 | Huawei Technologies Co., Ltd. | System and method for signal detection at asynchronous devices and devices without a time frame structure |
US11113943B2 (en) | 2019-05-07 | 2021-09-07 | Electronic Caregiver, Inc. | Systems and methods for predictive environmental fall risk identification |
WO2020240525A1 (en) * | 2019-05-31 | 2020-12-03 | Georgetown University | Assessing diseases by analyzing gait measurements |
CN112120703A (en) * | 2019-06-25 | 2020-12-25 | 株式会社日立制作所 | Fall risk assessment method and device |
US11894129B1 (en) | 2019-07-03 | 2024-02-06 | State Farm Mutual Automobile Insurance Company | Senior living care coordination platforms |
US11367527B1 (en) | 2019-08-19 | 2022-06-21 | State Farm Mutual Automobile Insurance Company | Senior living engagement and care support platforms |
CN110547955A (en) * | 2019-09-06 | 2019-12-10 | 薛丁维 | blind guiding robot |
US20220020481A1 (en) | 2020-07-20 | 2022-01-20 | Abbott Laboratories | Digital pass verification systems and methods |
US11688516B2 (en) | 2021-01-19 | 2023-06-27 | State Farm Mutual Automobile Insurance Company | Alert systems for senior living engagement and care support platforms |
CN113496216B (en) * | 2021-08-31 | 2023-05-05 | 四川大学华西医院 | Multi-angle falling high-risk identification method and system based on skeleton key points |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5231483A (en) * | 1990-09-05 | 1993-07-27 | Visionary Products, Inc. | Smart tracking system |
US20100328436A1 (en) * | 2009-06-01 | 2010-12-30 | The Curators Of The University Of Missouri | Anonymized video analysis methods and systems |
US20120025320A1 (en) * | 2010-07-30 | 2012-02-02 | Au Optronics Corporation | Complementary metal oxide semiconductor transistor and fabricating method thereof |
US20130131985A1 (en) * | 2011-04-11 | 2013-05-23 | James D. Weiland | Wearable electronic image acquisition and enhancement system and method for image acquisition and visual enhancement |
US20130163879A1 (en) * | 2010-08-30 | 2013-06-27 | Bk-Imaging Ltd. | Method and system for extracting three-dimensional information |
Family Cites Families (29)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5309921A (en) | 1992-02-11 | 1994-05-10 | Spectrum Medical Technologies | Apparatus and method for respiratory monitoring |
US6002994A (en) | 1994-09-09 | 1999-12-14 | Lane; Stephen S. | Method of user monitoring of physiological and non-physiological measurements |
US7522186B2 (en) | 2000-03-07 | 2009-04-21 | L-3 Communications Corporation | Method and apparatus for providing immersive surveillance |
US6915008B2 (en) | 2001-03-08 | 2005-07-05 | Point Grey Research Inc. | Method and apparatus for multi-nodal, three-dimensional imaging |
US20030058111A1 (en) | 2001-09-27 | 2003-03-27 | Koninklijke Philips Electronics N.V. | Computer vision based elderly care monitoring system |
US7202791B2 (en) | 2001-09-27 | 2007-04-10 | Koninklijke Philips N.V. | Method and apparatus for modeling behavior using a probability distrubution function |
US20040030531A1 (en) | 2002-03-28 | 2004-02-12 | Honeywell International Inc. | System and method for automated monitoring, recognizing, supporting, and responding to the behavior of an actor |
KR100507780B1 (en) | 2002-12-20 | 2005-08-17 | 한국전자통신연구원 | Apparatus and method for high-speed marker-free motion capture |
US7330566B2 (en) | 2003-05-15 | 2008-02-12 | Microsoft Corporation | Video-based gait recognition |
US20050088515A1 (en) | 2003-10-23 | 2005-04-28 | Geng Z. J. | Camera ring for three-dimensional (3D) surface imaging |
US7831087B2 (en) | 2003-10-31 | 2010-11-09 | Hewlett-Packard Development Company, L.P. | Method for visual-based recognition of an object |
US8589315B2 (en) | 2004-08-14 | 2013-11-19 | Hrl Laboratories, Llc | Behavior recognition using cognitive swarms and fuzzy graphs |
US7843351B2 (en) | 2004-09-01 | 2010-11-30 | Robert Bourne | Back training device |
US7502498B2 (en) | 2004-09-10 | 2009-03-10 | Available For Licensing | Patient monitoring apparatus |
US20060055543A1 (en) | 2004-09-10 | 2006-03-16 | Meena Ganesh | System and method for detecting unusual inactivity of a resident |
US8022987B2 (en) | 2005-06-30 | 2011-09-20 | Sandia Corporation | Information-based self-organization of sensor nodes of a sensor network |
US7420472B2 (en) * | 2005-10-16 | 2008-09-02 | Bao Tran | Patient monitoring apparatus |
US7589637B2 (en) | 2005-12-30 | 2009-09-15 | Healthsense, Inc. | Monitoring activity of an individual |
US7586418B2 (en) | 2006-11-17 | 2009-09-08 | General Electric Company | Multifunctional personal emergency response system |
US8120498B2 (en) | 2007-09-24 | 2012-02-21 | Intel-Ge Care Innovations Llc | Capturing body movement related to a fixed coordinate system |
CN101874404B (en) | 2007-09-24 | 2013-09-18 | 高通股份有限公司 | Enhanced interface for voice and video communications |
KR20090032339A (en) | 2007-09-27 | 2009-04-01 | 한국전자통신연구원 | Method and apparatus for service to inquire after old people's health |
EP2445405B1 (en) * | 2009-06-24 | 2018-06-13 | The Medical Research, Infrastructure, And Health Services Fund Of The Tel Aviv Medical Center | Automated near-fall detector |
US20120023201A1 (en) | 2010-07-26 | 2012-01-26 | Atlas Advisory Partners, Llc | Unified Content Delivery Platform |
CN102551685B (en) * | 2010-12-30 | 2015-04-01 | 世意法(北京)半导体研发有限责任公司 | Object monitor |
US8718748B2 (en) * | 2011-03-29 | 2014-05-06 | Kaliber Imaging Inc. | System and methods for monitoring and assessing mobility |
WO2013058985A1 (en) | 2011-10-17 | 2013-04-25 | Kimmel Zebadiah M | Method and apparatus for detecting deterioration of health status |
US9817017B2 (en) | 2011-10-17 | 2017-11-14 | Atlas5D, Inc. | Method and apparatus for monitoring individuals while protecting their privacy |
EP2775904B1 (en) * | 2011-11-11 | 2020-07-22 | National University of Ireland, Galway | A system for the management and prevention of venous pooling |
-
2014
- 2014-01-31 US US14/169,508 patent/US9597016B2/en active Active
-
2017
- 2017-02-03 US US15/424,375 patent/US10080513B2/en active Active
-
2018
- 2018-08-22 US US16/108,432 patent/US20190029569A1/en not_active Abandoned
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5231483A (en) * | 1990-09-05 | 1993-07-27 | Visionary Products, Inc. | Smart tracking system |
US20100328436A1 (en) * | 2009-06-01 | 2010-12-30 | The Curators Of The University Of Missouri | Anonymized video analysis methods and systems |
US20120025320A1 (en) * | 2010-07-30 | 2012-02-02 | Au Optronics Corporation | Complementary metal oxide semiconductor transistor and fabricating method thereof |
US20130163879A1 (en) * | 2010-08-30 | 2013-06-27 | Bk-Imaging Ltd. | Method and system for extracting three-dimensional information |
US20130131985A1 (en) * | 2011-04-11 | 2013-05-23 | James D. Weiland | Wearable electronic image acquisition and enhancement system and method for image acquisition and visual enhancement |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10485452B2 (en) * | 2015-02-25 | 2019-11-26 | Leonardo Y. Orellano | Fall detection systems and methods |
CN111027417A (en) * | 2019-11-21 | 2020-04-17 | 复旦大学 | Gait recognition method and gait evaluation system based on human body key point detection algorithm |
Also Published As
Publication number | Publication date |
---|---|
US20170143240A1 (en) | 2017-05-25 |
US20140148733A1 (en) | 2014-05-29 |
US9597016B2 (en) | 2017-03-21 |
US10080513B2 (en) | 2018-09-25 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10080513B2 (en) | Activity analysis, fall detection and risk assessment systems and methods | |
US9408561B2 (en) | Activity analysis, fall detection and risk assessment systems and methods | |
US20210183516A1 (en) | Systems and methods to identify persons and/or identify and quantify pain, fatigue, mood, and intent with protection of privacy | |
Stone et al. | Unobtrusive, continuous, in-home gait measurement using the Microsoft Kinect | |
US20200205697A1 (en) | Video-based fall risk assessment system | |
Kwolek et al. | Human fall detection on embedded platform using depth maps and wireless accelerometer | |
Zhang et al. | A survey on vision-based fall detection | |
Chen et al. | A fall detection system based on infrared array sensors with tracking capability for the elderly at home | |
US11282367B1 (en) | System and methods for safety, security, and well-being of individuals | |
Stone et al. | Capturing habitual, in-home gait parameter trends using an inexpensive depth camera | |
Seredin et al. | A skeleton features-based fall detection using Microsoft Kinect v2 with one class-classifier outlier removal | |
Abobakr et al. | Rgb-d fall detection via deep residual convolutional lstm networks | |
Shoaib et al. | View-invariant fall detection for elderly in real home environment | |
JP7185805B2 (en) | Fall risk assessment system | |
Stone et al. | Mapping Kinect-based in-home gait speed to TUG time: a methodology to facilitate clinical interpretation | |
CN113397520A (en) | Information detection method and device for indoor object, storage medium and processor | |
Lo et al. | From imaging networks to behavior profiling: Ubiquitous sensing for managed homecare of the elderly | |
Xiang et al. | Remote safety monitoring for elderly persons based on omni-vision analysis | |
Boudouane et al. | Wearable camera for fall detection embedded system | |
Bansal et al. | Elderly people fall detection system using skeleton tracking and recognition | |
JP2010250775A (en) | Crime prevention device and program | |
CN114127816A (en) | System and method for on-floor detection without the need for wearable items | |
Agarwal et al. | Method to improve gait speed assessment for low frame rate AI enabled visual sensor | |
JP2021033359A (en) | Emotion estimation device, emotion estimation method, program, information presentation device, information presentation method and emotion estimation system | |
Medina-Quero et al. | Computer vision-based gait velocity from non-obtrusive thermal vision sensors |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: THE CURATORS OF THE UNIVERSITY OF MISSOURI, MISSOU Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SKUBIC, MARJORIE;STONE, ERIK;RANTZ, MARILYN;AND OTHERS;SIGNING DATES FROM 20150210 TO 20150707;REEL/FRAME:046750/0559 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |