CN110633736A - Human body falling detection method based on multi-source heterogeneous data fusion - Google Patents

Human body falling detection method based on multi-source heterogeneous data fusion Download PDF

Info

Publication number
CN110633736A
CN110633736A CN201910795220.0A CN201910795220A CN110633736A CN 110633736 A CN110633736 A CN 110633736A CN 201910795220 A CN201910795220 A CN 201910795220A CN 110633736 A CN110633736 A CN 110633736A
Authority
CN
China
Prior art keywords
data
frame
human body
depth image
skeleton
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201910795220.0A
Other languages
Chinese (zh)
Inventor
李巧勤
刘勇国
杨尚明
姜珊
王志华
陶文元
傅翀
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
University of Electronic Science and Technology of China
Original Assignee
University of Electronic Science and Technology of China
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by University of Electronic Science and Technology of China filed Critical University of Electronic Science and Technology of China
Priority to CN201910795220.0A priority Critical patent/CN110633736A/en
Publication of CN110633736A publication Critical patent/CN110633736A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
    • A61B5/1116—Determining posture transitions
    • A61B5/1117—Fall detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Veterinary Medicine (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • Surgery (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Image Analysis (AREA)
  • Public Health (AREA)
  • Pathology (AREA)
  • Dentistry (AREA)
  • Physiology (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Biomedical Technology (AREA)

Abstract

The invention belongs to the field of human body fall detection, and provides a human body fall detection method based on multi-source heterogeneous data fusion; acquiring a behavior depth image and skeleton information of a human body through a Kinect, and firstly, getting rid of the constraint of a wearable sensor in the selection of the sensor; secondly, the problems that wearable sensors cannot be used in specific scenes such as bathrooms and toilets, and meanwhile, the privacy of human bodies is invaded by monitoring through a common camera are solved; meanwhile, features are extracted from multi-source heterogeneous data through a deep learning model, keyless attention fusion is introduced in a data fusion mode, and data redundancy and computational complexity caused by data level fusion are avoided; compared with the prior art, the method obviously improves the accuracy of fall detection.

Description

Human body falling detection method based on multi-source heterogeneous data fusion
Technical Field
The invention belongs to the field of human body fall detection, and particularly relates to a human body fall detection method based on multi-source heterogeneous data fusion.
Background
With the development of social aging, falling becomes a common health problem among the elderly, and falling detection becomes an important research direction. Falls are generally defined as "involuntary movements to the ground or lower", and fall detection is mainly performed by monitoring body signals through various types of sensors for acquisition, processing and determination. The sensors are roughly divided into wearable sensors and environment perception sensors; among them, in wearable sensor systems, an accelerometer is usually used to sense the orientation change of a tester, a gyroscope is used to detect the angular momentum, or other types of sensors such as barometer, magnetometer, etc., and these wearable sensors are generally placed on the chest, waist, wrist, thigh, etc. of a wearer; in the environment sensing system, an infrared sensor, a vibration sensor, a microphone array, a radar sensor, a camera, and the like are generally used. However, the wearable sensor requires the old to wear the device all the time and charge the device instantly, which is inconvenient; the accuracy of the camera sensor adopted in the environment perception sensor is higher, but the camera is used for monitoring and invading the privacy of the old.
In the current fall detection, Microsoft Kinect is used for obtaining three-dimensional skeleton data, acceleration and skeleton data are calculated through the mass center of a human joint and serve as main biomechanical characteristics, and an LSTM model is adopted for fall detection; for example, the method used in the document ' Elders ' fall detection based on biological features using depth camera ' can dynamically capture the motion of a human body in a three-dimensional space, so that researchers can more accurately analyze the posture of the human body, and the capability of human behavior analysis is greatly improved; because three-dimensional skeleton data rather than original image data are used, the privacy of the user can be effectively protected while analyzing the behavior. In behavior Recognition, three modes of three-dimensional bones, Human Body partial images and moving image histories (MHIs) are integrated into a mixed deep learning framework for Human Body behavior detection, so that a good effect is obtained, for example, in the literature, "Human action on Recognition Based on integration Body position, part shape, and Motion", RGB is used for capturing Human Body posture, partial shape and Body Motion, and a model combines a Convolutional Neural Network (CNNs), long-time memory (LST M) and a fine-tuned pre-training system structure into a mixed system called MCLP: the multimodal CNN + LSTM + VGG16 is pre-trained in the ImageNet public dataset; the three submodels are subjected to decision-level fusion. Decision-level fusion can be regarded as a form of aggregation, so that each mode is completely independent of other modes, and results obtained independently are fused in the last step; because different modalities are only fused in the last step of the output of the independent model, the model cannot obtain the interaction between the modalities, and the falling recognition rate is reduced.
Disclosure of Invention
The invention aims to provide a human body falling detection method based on multi-source heterogeneous data fusion, aiming at the problems that signals are unstable, data are possibly lost and real-time monitoring cannot be achieved in signals transmitted by using a wearable sensor in the current falling detection. The method utilizes Microsoft Kinect equipment to obtain the depth image and the skeleton data for fall detection, and firstly, the restriction of a wearable sensor is eliminated in the selection of the sensor; secondly, the problems that wearable sensors cannot be used in specific scenes such as bathrooms and toilets, and meanwhile, the privacy of human bodies is invaded by monitoring through a common camera are solved; and finally, algorithm improvement is carried out on multi-source heterogeneous data in a fusion mode, and the accuracy of fall detection is improved.
In order to achieve the purpose, the invention adopts the technical scheme that:
a human body falling detection method based on multi-source heterogeneous data fusion comprises the following steps:
step 1, collecting human body skeleton node data and depth image data based on a Kinect v2 sensor system;
step 2, synchronous frame sampling is carried out on the collected human skeleton node data and the collected depth image data: setting sequence length, dividing an input sequence into N equal-length sections, and randomly selecting a frame from each equal-length section to obtain time sequence data with the length of N frames;
step 3, carrying out data processing on the sampled human skeleton node data and the depth image data;
step 3-1, aiming at human skeleton node data:
step 3-1-1, node processing: performing node processing on each frame of skeleton node data, selecting trunk skeleton nodes from 25 skeleton nodes provided by a Kinect v2 sensor, and arranging joints in a tree traversal sequence;
step 3-1-2. coordinate normalization: carrying out coordinate normalization on the three-dimensional coordinate points of each frame of skeleton nodes;
step 3-1-3, performing feature extraction on the preprocessed bone node data through a CNN-LSTM model to obtain bone data features ht；
Step 3-2. for depth image data:
step 3-2-1, generating a Motion History Image (MHI) according to the motion dynamics:
step 3-2-2, normalizing the motion history image;
3-2-3, performing image enhancement on the normalized motion history image to obtain a training data set;
step 3-2-4, performing feature extraction on the training data set by adopting a VGG16 model to obtain a one-dimensional feature vector h degrees;
step 4, carrying out keyless attention data fusion on the bone node data and the depth image data after feature extraction;
step 4-1, characterizing the bone data htAnd connecting depth image features h degrees in series to obtain:
Figure BDA0002180735640000021
wherein the content of the first and second substances,
Figure BDA0002180735640000031
human skeleton node characteristics, N, which represent the extraction of the nth frame of human skeleton node data, are 1, 2.., N;
step 4-2. mixingBAs an annotation vector, the attention mechanism gives the output vector c by the desired calculation of the annotation vector:
Figure BDA0002180735640000032
wherein λ isnTo represent
Figure BDA0002180735640000033
The weight of (c):
Figure BDA0002180735640000034
Figure BDA0002180735640000035
where ω is a learnable parameter of the same dimension as the annotation vector;
and 5, inputting the vector subjected to the data fusion processing into a classifier to obtain a classification result.
Further, the specific process of the step 3-2-1 is as follows: firstly, carrying out gray level conversion on each frame of depth image; then, taking the first frame as a background, calculating an absolute difference value between pixels corresponding to the second frame and the first frame, if the difference value of any pixel point is greater than a preset threshold value, determining the pixel point as a dynamic pixel and specifying the pixel point as white, otherwise, determining the pixel point as a static pixel and specifying the pixel point as black, and taking the synthesized graph as a new background; and by analogy, synthesizing the motion history image.
The invention has the beneficial effects that:
the invention provides a human body falling detection method based on multi-source heterogeneous data fusion, which is characterized in that a behavior depth image and skeleton information of a human body are obtained through a Kinect, so that the constraint of a wearable sensor is eliminated, and the problems that a common camera monitors and invades the privacy of the human body and the like are effectively avoided; meanwhile, features are extracted from multi-source heterogeneous data through a deep learning model, keyless attention fusion is introduced in a data fusion mode, and data redundancy and computational complexity caused by data level fusion are avoided; compared with the prior art, the method obviously improves the accuracy of fall detection.
Drawings
Fig. 1 is a flow chart of a human body fall detection method based on multi-source heterogeneous data fusion according to the present invention.
Fig. 2 is a specific flowchart of the human body fall detection method based on multi-source heterogeneous data fusion according to the present invention.
FIG. 3 is a human bone node traversal graph according to an embodiment of the present invention.
FIG. 4 is a flow chart of CNN-Maxpooling in an embodiment of the present invention.
FIG. 5 is a flow chart of an LSTM in an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples.
The embodiment provides a human body fall detection method based on multi-source heterogeneous data fusion, the flow of which is shown in fig. 1 and 2, and the method specifically comprises the following steps:
s1, collecting human body skeleton node data and depth image data based on a Kinect v2 sensor system;
kinect v2 is a 3D motion-sensing Camera available from microsoft, which has three cameras, RGB Camera, Depth Camera, IR Emitters, and a Microphon e Array; the Kinect sensor is connected with a PC end through a data transmission line, Kinect S DK provided by Microsoft is called to obtain skeleton data and depth image data of a human body, in the embodiment, the Kinect sensor locks a target position, the placing height is 1.2m, and the front-back distance is 1.5 m-1.8 m;
description of action types: the training set actions are divided into fall actions and daily behaviors, wherein the simulated fall actions are set as: the front fall and the back fall finally lie on the ground, the back fall finally sits on the ground, and the left fall and the right fall take five falling actions; the daily behavior is set as: walking, sitting, squatting, lying, sitting to standing, sitting to lying, lying to sitting, lying to standing, and lying to standing;
s2, carrying out frame sampling on the collected human body bone node data and the collected depth image data:
carrying out synchronous frame sampling on human skeleton node data and depth image data; the purpose of frame sampling is to fix the length of the input sequence, so that the input sequence is adaptive to a training network; and making the model more robust by introducing randomness; in this embodiment, the sampling frequency is set to 30Hz, and since the fall action time does not exceed 2 seconds generally, the sequence length is set to 2 seconds, the input sequence is divided into N equal-length segments, and then one frame is randomly selected from each equal-length segment to obtain time series data with the length of N frames;
s3, carrying out data processing on the sampled human skeleton node data and the depth image data;
s3-1, aiming at human skeleton node data:
s3-1-1, node processing: performing node processing on each frame of bone node data, wherein the purpose of the node processing is to enable a system to detect a spatial mode between related joints to express bone coordinates; selecting 16 bone nodes from 25 bone nodes provided by a Kinect v2 sensor, wherein the bone nodes are respectively the middle vertebra, the middle shoulder, the head, the left shoulder, the left elbow, the left hand, the right shoulder, the right elbow, the right hand, the base vertebra, the left thigh, the left knee, the left ankle, the right thigh, the right knee and the right ankle in sequence as shown in FIG. 3; wherein the joints are arranged in a tree traversal order while allowing for joint repetition during traversal, in this embodiment, the total number of joints becomes 31; stacking the joint coordinates such that joints in the same traversal path are placed adjacent to each other, making it easier for the network to learn the correlation pattern between adjacent joint coordinates;
s3-1-2 coordinate normalization: performing coordinate normalization on the three-dimensional coordinate points of each frame of skeleton nodes, wherein the aim is to prevent the joint coordinates from being changed by the position and the direction of the shooting object and the size of the skeleton of the shooting object; the normalization formula is as follows:
wherein max is the maximum value of the sample data, min is the minimum value of the sample data, x is the current sensor data, and x is the result after normalization;
s3-1-3, carrying out feature extraction through a CNN-LSTM model after preprocessing bone node data;
in this embodiment, since 31 skeleton nodes are processed by the node processing, the skeleton node data is composed of 93 values (31 joints × 3 dimensions), and when N is set to 8 in the input layer, 8 × 93 joint coordinate values are used as input;
in the feature extraction stage, in this embodiment, the CNN-LSTM model employs 3 continuous convolutional layers and 2 cascaded LSTM, where the 3 continuous convolutional layers (sequentially including 32, 48, and 64 cores) form the CNN model and are used to implement feature extraction, as shown in fig. 4; the output of each convolutional layer is halved by the maximum pooling layer; inputting the features extracted by the CNN model into a first LSTM, and inputting the output of the first LSTM into a second LSTM after the feature quantity is halved by using a filter layer; 2 LSTMs use the same architecture, the number of neurons is 128; each LSTM updates their input gates following the following formula (i)t) Forget gate (f)t) And an output gate (o)t) And cell state (C)t) And the output vector (h)t) (ii) a The update equation for LSTM is:
it＝sigmoid(WuiUt+Whiht-1+WciCt-1+bi) (1)
ft＝sigmoid(WufUt+Whfht-1+WcfCt-1+bf) (2)
ot＝sigmoid(WuoUt+Whoht-1+WcoCt-1+bo) (3)
Ct＝ftct-1+ittanh(WucUt+Whcht-1+bc) (4)
ht＝ottanh(Ct) (5)
wherein, UtAnd CtFor the input vector and cell state at time t, htAnd ht-1Output vectors (hidden states) for time t and time t-1, bcIs offset, and in addition, bi、bfAnd boBias of the input, forgetting and output gates, respectively, and Wui、Whi、Wci、Wuf、Whf、Wcf、Wuo、Who、Wco、Wuc、WhcIs a weight matrix; h istThe extracted human skeleton node characteristics are obtained; as shown in fig. 5.
S3-2, for depth image data:
s3-2-1, generating a Motion History Image (MHI) according to the motion dynamics:
firstly, carrying out gray level conversion on each frame of depth image; then, taking the first frame as a background, calculating an absolute difference value between pixels corresponding to the second frame and the first frame, if the difference value of any pixel point is greater than a preset threshold value, determining the pixel point as a dynamic pixel and specifying the pixel point as white, otherwise, determining the pixel point as a static pixel and specifying the pixel point as black, and taking the synthesized graph as a new background; and so on, synthesizing the motion history image;
s3-2-2, normalizing the motion history image, wherein the normalization formula is as follows:
Figure BDA0002180735640000061
wherein x isiRepresenting image pixel point values, min (x), max (x) representing maximum and minimum values of image pixels, respectively; the normalization does not change the information storage of the image, the value range of the processed pixel value is converted from 0-255 to 0-1, and the method has great benefit for the subsequent convolutional neural network processing, so that the image can resist the attack of geometric transformation;
s3-2-3, performing image enhancement on the normalized motion history image to obtain a training data set; for enhancing the generalization ability of the model; taking the image center as a rotation center, setting the maximum rotation angle to be 45 degrees, setting the up-down translation scale and the left-right translation scale to be 10, and zooming to be 0.2 to enhance the training data set;
s3-2-4, extracting features of the training data set by adopting a VGG16 model to obtain a one-dimensional feature vector h degrees;
the generated MHI image is resized to 224 x 224, and the image is three color value channels, so that the image is input to a VGG16 model to be trained (224 x 3) and serves as a feature extractor of the MHI; the extracted features (7 × 512) are then adjusted into a vector of size 25008;
the VGG convolutional neural network is a model proposed by Oxford university in 2014, and has very good results in image classification and target detection tasks; VGG16 is divided into 16 layers, and is a complex CNN network model, a picture of 224 × 3 is input, the size of a passed convolution kernel is 3 × 3, the step size stride of the convolution kernel movement is 1, and the dimension after the convolution operation becomes small, so that before each convolution operation, newly added positions are first rows, all first column elements are 0, and padding is used for representing, and padding is set to be 1; pooling (pooling) is a 2 x 2 maximal pooling (max pooling) mode; the whole framework is formed by laminating 13 convolution layers and 3 full-connection layers;
s4, carrying out keyless attention data fusion on the skeleton node data and the depth image data after feature extraction;
obtaining a bone data characteristic hiding state h in the three-dimensional bone node data processingt(ii) a One-dimensional feature vectors are obtained in the depth image data processing process for h degrees;
s4-1, characterizing the bone data by htAnd connecting depth image features h degrees in series to obtain:
wherein the content of the first and second substances,
Figure BDA0002180735640000063
representing the n-th frame of the human boneHuman skeleton node characteristics extracted from skeleton node data, N is 1, 2. [. the]Representing a concatenation of state vectors;
s4-2, performing data fusion through a keyless attention mechanism, and enabling h to be a keyBAs a sequence of input vectors, called annotation vectors, where the attention mechanism gives the output vector c by the desired computation of the annotation vector:
Figure BDA0002180735640000071
wherein each one
Figure BDA0002180735640000072
The weight calculation method comprises the following steps:
Figure BDA0002180735640000073
Figure BDA0002180735640000074
where ω is a learnable parameter of the same dimension as the annotation vector;
and S5, inputting the vector c subjected to data fusion processing into a classifier, wherein the classifier is an artificial neural network formed by sequentially connecting a full connection layer (dense), an intermediate filter layer (dropout) and a classification layer (softmax activation function), and obtaining a classification result.
In this embodiment, the fall detection task generally adopts accuracy (accuracy), specificity (specificity) and sensitivity (sensitivity) as evaluation indexes, that is, the fall detection task uses accuracy (accuracy), specificity (specificity) and sensitivity (sensitivity) as evaluation indexes
Accuracy ═ number of correct classes/total number of classes
Specificity＝TN/(TN+FP)
Sensitivity＝TP/(TP+FN)
Where TP is a true positive value, indicating that a fall event was evoked and the algorithm detected it; FP is false positive, indicating that no fall event occurred, but the algorithm detected it; TN being true negative means that no fall event has occurred, but the algorithm does not detect it; FN is false negative, indicating that a fall event has occurred, but the algorithm does not detect it. At present, Ghojogh.B et al (Fisherposesfor Human Action Recognition Using Kinect Sensor Data) have achieved 89% accuracy in the UTkinect (http:// cvrc. ec. utexa. edu/kinectDatases/HOJ3D. html) Data set. By adopting the falling detection method, experiments are carried out on the Kinect public data sets UR and UTKinect, and the recognition accuracy rate of 95 percent is obtained.
While the invention has been described with reference to specific embodiments, any feature disclosed in this specification may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise; all of the disclosed features, or all of the method or process steps, may be combined in any combination, except mutually exclusive features and/or steps.

Claims (2)

1. A human body falling detection method based on multi-source heterogeneous data fusion comprises the following steps:
step 1, collecting human body skeleton node data and depth image data based on a Kinect v2 sensor system;
step 2, synchronous frame sampling is carried out on the collected human skeleton node data and the collected depth image data: setting sequence length, dividing an input sequence into N equal-length sections, and randomly selecting a frame from each equal-length section to obtain time sequence data with the length of N frames;
step 3, carrying out data processing on the sampled human skeleton node data and the depth image data;
step 3-1, aiming at human skeleton node data:
step 3-1-1, node processing: performing node processing on each frame of skeleton node data, selecting trunk skeleton nodes from 25 skeleton nodes provided by a Kinect v2 sensor, and arranging joints in a tree traversal sequence;
step 3-1-2. coordinate normalization: carrying out coordinate normalization on the three-dimensional coordinate points of each frame of skeleton nodes;
step 3-1-3, performing feature extraction on the preprocessed bone node data through a CNN-LSTM model to obtain bone data features ht；
Step 3-2. for depth image data:
step 3-2-1, generating a Motion History Image (MHI) according to the motion dynamics:
step 3-2-2, normalizing the motion history image;
3-2-3, performing image enhancement on the normalized motion history image to obtain a training data set;
step 3-2-4, performing feature extraction on the training data set by adopting a VGG16 model to obtain a one-dimensional feature vector ho；
Step 4, carrying out keyless attention data fusion on the bone node data and the depth image data after feature extraction;
step 4-1, characterizing the bone data htAnd depth image feature hoCarrying out series connection to obtain:
Figure FDA0002180735630000011
wherein the content of the first and second substances,
Figure FDA0002180735630000012
human skeleton node characteristics, N, which represent the extraction of the nth frame of human skeleton node data, are 1, 2.., N;
step 4-2. mixingBCalled annotation vector, the attention mechanism gives the output vector c by the desired computation of the annotation vector:
Figure FDA0002180735630000013
wherein λ isnTo represent
Figure FDA0002180735630000014
The weight of (c):
Figure FDA0002180735630000015
Figure FDA0002180735630000016
where ω is a learnable parameter of the same dimension as the annotation vector;
and 5, inputting the vector subjected to the data fusion processing into a classifier to obtain a classification result.
2. The human body fall detection method based on multi-source heterogeneous data fusion according to claim 1, wherein the specific process of the step 3-2-1 is as follows: firstly, carrying out gray level conversion on each frame of depth image; then, taking the first frame as a background, calculating an absolute difference value between pixels corresponding to the second frame and the first frame, if the difference value of any pixel point is greater than a preset threshold value, determining the pixel point as a dynamic pixel and specifying the pixel point as white, otherwise, determining the pixel point as a static pixel and specifying the pixel point as black, and taking the synthesized graph as a new background; and by analogy, synthesizing the motion history image.
CN201910795220.0A 2019-08-27 2019-08-27 Human body falling detection method based on multi-source heterogeneous data fusion Pending CN110633736A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910795220.0A CN110633736A (en) 2019-08-27 2019-08-27 Human body falling detection method based on multi-source heterogeneous data fusion

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910795220.0A CN110633736A (en) 2019-08-27 2019-08-27 Human body falling detection method based on multi-source heterogeneous data fusion

Publications (1)

Publication Number Publication Date
CN110633736A true CN110633736A (en) 2019-12-31

Family

ID=68969142

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910795220.0A Pending CN110633736A (en) 2019-08-27 2019-08-27 Human body falling detection method based on multi-source heterogeneous data fusion

Country Status (1)

Country Link
CN (1) CN110633736A (en)

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111209848A (en) * 2020-01-03 2020-05-29 北京工业大学 Real-time fall detection method based on deep learning
CN111436944A (en) * 2020-04-20 2020-07-24 电子科技大学 Falling detection method based on intelligent mobile terminal
CN111488850A (en) * 2020-04-17 2020-08-04 电子科技大学 Neural network-based old people falling detection method
CN111507252A (en) * 2020-04-16 2020-08-07 上海眼控科技股份有限公司 Human body falling detection device and method, electronic terminal and storage medium
CN111681389A (en) * 2020-06-12 2020-09-18 电子科技大学 Old people falling behavior detection method based on blind source separation
CN112084899A (en) * 2020-08-25 2020-12-15 广东工业大学 Fall event detection method and system based on deep learning
CN112101235A (en) * 2020-09-16 2020-12-18 济南大学 Old people behavior identification and detection method based on old people behavior characteristics
CN112101201A (en) * 2020-09-14 2020-12-18 北京数衍科技有限公司 Pedestrian state detection method and device and electronic equipment
CN112270807A (en) * 2020-10-29 2021-01-26 怀化学院 Old man early warning system that tumbles
CN112613388A (en) * 2020-12-18 2021-04-06 燕山大学 Personnel falling detection method based on multi-dimensional feature fusion
CN112617813A (en) * 2020-12-15 2021-04-09 南京邮电大学 Multi-sensor-based non-invasive fall detection method and system
CN112801061A (en) * 2021-04-07 2021-05-14 南京百伦斯智能科技有限公司 Posture recognition method and system
CN112998697A (en) * 2021-02-22 2021-06-22 电子科技大学 Tumble injury degree prediction method and system based on skeleton data and terminal
CN113196289A (en) * 2020-07-02 2021-07-30 浙江大学 Human body action recognition method, human body action recognition system and device
CN114419842A (en) * 2021-12-31 2022-04-29 浙江大学台州研究院 Artificial intelligence-based falling alarm method and device for assisting user in moving to intelligent closestool

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108776775A (en) * 2018-05-24 2018-11-09 常州大学 Fall detection method in a kind of the elderly room based on weight fusion depth and skeleton character
CN109492612A (en) * 2018-11-28 2019-03-19 平安科技（深圳）有限公司 Fall detection method and its falling detection device based on skeleton point
CN109875565A (en) * 2019-01-25 2019-06-14 电子科技大学 A kind of cerebral apoplexy upper extremity exercise function method for automatically evaluating based on deep learning
CN109948459A (en) * 2019-02-25 2019-06-28 广东工业大学 A kind of football movement appraisal procedure and system based on deep learning

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108776775A (en) * 2018-05-24 2018-11-09 常州大学 Fall detection method in a kind of the elderly room based on weight fusion depth and skeleton character
CN109492612A (en) * 2018-11-28 2019-03-19 平安科技（深圳）有限公司 Fall detection method and its falling detection device based on skeleton point
CN109875565A (en) * 2019-01-25 2019-06-14 电子科技大学 A kind of cerebral apoplexy upper extremity exercise function method for automatically evaluating based on deep learning
CN109948459A (en) * 2019-02-25 2019-06-28 广东工业大学 A kind of football movement appraisal procedure and system based on deep learning

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
HANY EL-GHAISH ET AL.: "Human Action Recognition Based on Integrating Body Pose, Part Shape, and Motion", 《IEEE ACCESS》 *
XIANG LONG ET AL.: "Multimodal Keyless Attention Fusion for Video Classification", 《THE THIRTY-SECOND AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE》 *

Cited By (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111209848A (en) * 2020-01-03 2020-05-29 北京工业大学 Real-time fall detection method based on deep learning
CN111507252A (en) * 2020-04-16 2020-08-07 上海眼控科技股份有限公司 Human body falling detection device and method, electronic terminal and storage medium
CN111488850A (en) * 2020-04-17 2020-08-04 电子科技大学 Neural network-based old people falling detection method
CN111488850B (en) * 2020-04-17 2022-07-12 电子科技大学 Neural network-based old people falling detection method
CN111436944A (en) * 2020-04-20 2020-07-24 电子科技大学 Falling detection method based on intelligent mobile terminal
CN111681389B (en) * 2020-06-12 2021-02-26 电子科技大学 Old people falling behavior detection method based on blind source separation
CN111681389A (en) * 2020-06-12 2020-09-18 电子科技大学 Old people falling behavior detection method based on blind source separation
WO2022000420A1 (en) * 2020-07-02 2022-01-06 浙江大学 Human body action recognition method, human body action recognition system, and device
CN113196289A (en) * 2020-07-02 2021-07-30 浙江大学 Human body action recognition method, human body action recognition system and device
CN112084899A (en) * 2020-08-25 2020-12-15 广东工业大学 Fall event detection method and system based on deep learning
CN112101201B (en) * 2020-09-14 2024-05-24 北京数衍科技有限公司 Pedestrian state detection method and device and electronic equipment
CN112101201A (en) * 2020-09-14 2020-12-18 北京数衍科技有限公司 Pedestrian state detection method and device and electronic equipment
CN112101235A (en) * 2020-09-16 2020-12-18 济南大学 Old people behavior identification and detection method based on old people behavior characteristics
CN112270807A (en) * 2020-10-29 2021-01-26 怀化学院 Old man early warning system that tumbles
CN112617813A (en) * 2020-12-15 2021-04-09 南京邮电大学 Multi-sensor-based non-invasive fall detection method and system
CN112617813B (en) * 2020-12-15 2023-02-14 南京邮电大学 Multi-sensor-based non-invasive fall detection method and system
CN112613388A (en) * 2020-12-18 2021-04-06 燕山大学 Personnel falling detection method based on multi-dimensional feature fusion
CN112613388B (en) * 2020-12-18 2022-08-30 燕山大学 Personnel falling detection method based on multi-dimensional feature fusion
CN112998697A (en) * 2021-02-22 2021-06-22 电子科技大学 Tumble injury degree prediction method and system based on skeleton data and terminal
CN112998697B (en) * 2021-02-22 2022-06-14 电子科技大学 Tumble injury degree prediction method and system based on skeleton data and terminal
CN112801061A (en) * 2021-04-07 2021-05-14 南京百伦斯智能科技有限公司 Posture recognition method and system
CN114419842A (en) * 2021-12-31 2022-04-29 浙江大学台州研究院 Artificial intelligence-based falling alarm method and device for assisting user in moving to intelligent closestool
CN114419842B (en) * 2021-12-31 2024-05-10 浙江大学台州研究院 Fall alarm method and device for assisting user to fall to closestool based on artificial intelligence

Similar Documents

Publication Publication Date Title
CN110633736A (en) Human body falling detection method based on multi-source heterogeneous data fusion
Dorschky et al. CNN-based estimation of sagittal plane walking and running biomechanics from measured and simulated inertial sensor data
CN106650687B (en) Posture correction method based on depth information and skeleton information
CN110490109B (en) Monocular vision-based online human body rehabilitation action recognition method
Zhou et al. Learning to estimate 3d human pose from point cloud
Amsaprabhaa Multimodal spatiotemporal skeletal kinematic gait feature fusion for vision-based fall detection
Chang et al. A pose estimation-based fall detection methodology using artificial intelligence edge computing
CN113111767A (en) Fall detection method based on deep learning 3D posture assessment
Pasinetti et al. Assisted gait phase estimation through an embedded depth camera using modified random forest algorithm classification
Xu et al. Elders’ fall detection based on biomechanical features using depth camera
Ahmad et al. Human action recognition using convolutional neural network and depth sensor data
CN112101235B (en) Old people behavior identification and detection method based on old people behavior characteristics
CN113239892A (en) Monocular human body three-dimensional attitude estimation method based on data enhancement architecture
JP2019003565A (en) Image processing apparatus, image processing method and image processing program
Yan et al. Human-object interaction recognition using multitask neural network
Docto et al. Third eye hand glove object detection for visually impaired using You Only Look Once (YOLO) v4-tiny algorithm
CN116895098A (en) Video human body action recognition system and method based on deep learning and privacy protection
US20230103112A1 (en) System and method for monitoring activity performed by subject
Zhen et al. Hybrid Deep‐Learning Framework Based on Gaussian Fusion of Multiple Spatiotemporal Networks for Walking Gait Phase Recognition
Lin et al. Abnormal event detection using microsoft kinect in a smart home
Shah et al. An efficient and lightweight multiperson activity recognition framework for robot-assisted healthcare applications
Upadhyay et al. Body Posture Detection Using Computer Vision
CN116030533A (en) High-speed motion capturing and identifying method and system for motion scene
Nevalainen et al. Image analysis and development of graphical user interface for pole vault action
CN112861699A (en) Method for estimating height of human body in any posture based on single depth image and multi-stage neural network

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20191231

RJ01 Rejection of invention patent application after publication