CN110659677A - Human body falling detection method based on movable sensor combination equipment - Google Patents

Human body falling detection method based on movable sensor combination equipment Download PDF

Info

Publication number
CN110659677A
CN110659677A CN201910853810.4A CN201910853810A CN110659677A CN 110659677 A CN110659677 A CN 110659677A CN 201910853810 A CN201910853810 A CN 201910853810A CN 110659677 A CN110659677 A CN 110659677A
Authority
CN
China
Prior art keywords
data
sensor
input
modal
wrist
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201910853810.4A
Other languages
Chinese (zh)
Inventor
李巧勤
杨尚明
刘勇国
陶文元
杨晓帅
刘晞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
University of Electronic Science and Technology of China
Original Assignee
University of Electronic Science and Technology of China
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by University of Electronic Science and Technology of China filed Critical University of Electronic Science and Technology of China
Priority to CN201910853810.4A priority Critical patent/CN110659677A/en
Publication of CN110659677A publication Critical patent/CN110659677A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
    • A61B5/1116—Determining posture transitions
    • A61B5/1117—Fall detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Artificial Intelligence (AREA)
  • Pathology (AREA)
  • Medical Informatics (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Animal Behavior & Ethology (AREA)
  • Surgery (AREA)
  • Molecular Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Dentistry (AREA)
  • Physiology (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)

Abstract

The invention belongs to the technical field of electronic information detection, and discloses a human body falling detection method based on movable sensor combination equipment, which comprises the following steps: human user sensor data is collected based on a wearable sensor system. And carrying out numerical value normalization processing on the acquired sensor data. The collected sensor data is classified using a CorrRNN model based on time series multimodal learning. And respectively constructing classifiers by collecting data through the waist sensor and the wrist sensor, and performing weighted combination on classification results to obtain a falling type judgment result. The method proposed by the present invention combines two wearable sensors for fall detection, employs a CorrRNN model that is trained in an unsupervised manner, eliminating the need for labeling data, and combines GRUs to capture long-term dependencies and time-series input structures. The falling detection precision can be greatly improved.

Description

Human body falling detection method based on movable sensor combination equipment
Technical Field
The invention belongs to the technical field of electronic information detection, and particularly relates to a human body falling detection method based on movable sensor combination equipment.
Background
Currently, the closest prior art:
for elderly people who are solitary and without any supervision, it is important to be able to detect abnormal activity patterns early. In the context of such intelligent assistance applications and environments, the identification of physical activity and the detection of falls are considered essential functions. Due to the high impact of falls on health and healthcare costs, people are increasingly paying attention to automatic fall detection methods. The advent of wearable sensors, and in particular MEMS-based miniature inertial sensors (e.g., accelerometers and gyroscopes), has fueled this rapid growth. Their size and weight rapidly shrink to the point where they can be unobtrusively attached to the body.
The chinese patent CN108549900A human body fall detection method based on the wearing position of the mobile device provides a human body fall detection method based on the wearing position of the mobile device, which includes: firstly, a feature extraction method of fusion of rotation mode components and attitude angles is adopted, the data of an accelerometer and a gyroscope are utilized to calculate rotation radius, angular velocity amplitude and attitude angles and extract features, and then the rotation radius, the angular velocity amplitude and the attitude angles are classified to obtain the wearing position of the mobile equipment; a fall detection algorithm based on temporal analysis is then adaptively adjusted according to location. According to the technical scheme, under the condition that the wearing positions are different, the wearing positions of the sensors are automatically adjusted, the detection algorithm is determined according to the different wearing positions, but the falling activities of the human body are simple forward, backward and lateral falling, and the like.
At present, various wearable fall detection alarm devices exist, but the fall detection alarm devices have the problem that the detection precision can be reduced in real use due to the difference between human bodies under the detection precision of the test.
In summary, the problems of the prior art are as follows:
(1) the prior art has poor detection precision under the condition of complicated human body activity and cannot adapt to different environments.
(2) The current fall detection alarm device has the defect that the detection precision is reduced when the device is actually used under the detection precision of the test due to the difference between human bodies.
The difficulty of solving the technical problems is as follows:
the real use condition is too complex, the detection algorithm suitable for various wearing positions and various falling types is adjusted by enumerating the falling condition, the judgment is complex, and all scenes are difficult to be included. The real falling is different from the experiment environment, simple classification cannot be performed, a user can grasp a support with hands or support the ground through elbows when falling down, the difference exists between the user and the experiment data, and the detection precision is reduced.
The significance of solving the technical problems is as follows:
wear through wrist and waist sensor, to falling that various activities of user lead to in real activity and detecting the error and rectify, directly detect through the algorithm and fall simultaneously, be favorable to improving fall and detect the precision, detect more accurately.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a human body falling detection method based on movable sensor combination equipment.
The invention is realized in such a way, and provides a human body falling detection method based on movable sensor combination equipment. The human body falling detection method based on the movable sensor combination equipment comprises the following steps:
acquiring sensor data of a human body user based on a wearable sensor system;
secondly, carrying out numerical value normalization processing on the acquired sensor data;
classifying the acquired sensor data by adopting a CorrRNN model based on time sequence multi-modal learning;
and step four, respectively constructing classifiers through data collected by the waist sensor and the wrist sensor, and performing weighted combination on classification results to obtain falling type judgment results.
Further, the acquiring of the human body user sensor data based on the wearable sensor system in the step one specifically includes:
(1) a wearable sensor system for collecting user data is an Inertial Measurement Unit (IMU), the IMU can measure three-axis acceleration and three-axis angular velocity in daily actions, the IMU is connected with a Personal Computer (PC) through Bluetooth, wireless real-time transmission of data is achieved, and sampling frequency is set to be 20 Hz.
(2) The two sensors are respectively arranged on the wrist and the waist, the wearing position of the wrist is the position where the radius of the right hand is close to the carpal bone, and the wearing position of the waist is the position in the middle of the front side of the hip bone.
(3) An action category. The actions are divided into falling and daily actions, and the daily actions include basic actions such as sitting, standing, lying, jogging, running, going up and down stairs, and daily activities such as answering and making calls, squatting and the like.
Further, the performing of the numerical normalization processing on the acquired sensor data in the second step specifically includes:
and normalizing the signal value, and mapping all data to a range of 0-1. By using a linear function normalization method, the conversion function is as follows:
Figure BDA0002197714580000031
where max is the maximum value of the sample data and min is the minimum value of the sample data. x is the raw sensor signal data and x is the normalized result.
Further, the step three of classifying the collected sensor data by using the corrnn model based on time-series multimodal learning specifically includes:
(1) inputting data, collecting data of falling and daily actions of N testers as training samples, and recording a sample set as T ((X)1，Y1，X2，Y2，c)1,…,(X1，Y1，X2，Y2，c)N) Wherein X is1，Y1，X2，Y2Respectively representing data collected by two wearable sensors, X1Representing acceleration data collected by the lumbar sensor, Y1Representing the spirometer data collected by the lumbar sensor; x2Representing acceleration data collected by wrist sensors, Y2Representing the gyroscope data collected by the wrist sensor and c representing the motion class.
The data collected by the waist and the wrist are respectively input into the model for training, and the last two classifiers are combined for final classification. In the subsequent steps, the waist data is used as input, and the methods adopted by the wrist and waist sensor data are the same. Whereby the subsequent step will be X1，Y1Abbreviated x, y.
Both data (acceleration data, gyroscope data) are represented as a sequence of length T.
Figure BDA0002197714580000041
And
Figure BDA0002197714580000043
representing the m-dimensional characteristics of the X mode at time sequence t. For simplicity, the superscripts m and n are omitted subsequently.
(2) Time-series multi-modal learning
The two modes are fused at t, and the extension is expressed asAnd
Figure BDA0002197714580000045
l is the historical timing range. Given a multi-modal data sequence pair
Figure BDA0002197714580000046
Training a feature learning modelType M, learning multidimensional joint representation
Figure BDA0002197714580000047
(3) CorrRNN model structure
The model enables encoder-decoder operation, sequence-to-sequence learning, and learning of the sequence representation in an unsupervised manner. The model includes two regenerated neural networks: a multi-modal encoder and a multi-modal decoder. The multi-modal encoder maps the two input sequences to a common space. The multi-modal decoder reconstructs the two input sequences from the joint representation obtained by the encoder. Both the encoder and decoder are two-layer networks. The multi-modal input is first mapped to a separate hidden layer before being passed to a common layer called the fusion layer. The joint representation is first decoded to separate the hidden layers before reconstructing the multimodal input.
Using a pair of multimodal inputs to train the model, the activation of the fusion layer in the last step encoder is output as a sequence feature representation. Two types of feature representations can be obtained from the model input: if both input modalities exist, obtaining a joint representation of them; if only one form exists, an "enhanced" single-modality representation will be obtained.
(4) Multi-modal encoder
The multimodal encoder fuses the input modality sequences into a common representation, using three main modules at each time.
Dynamic weighting module (DW): the signals of the two modalities are dynamically weighted by evaluating the input signal's conformity with the recent history.
GRU module (GRU): the input modalities are fused to generate a fused representation. The module also uses the timing structure of the reset and update gate capture sequences.
Correlation module (Corr): intermediate states generated by the GRU module are used as inputs to compute the correlation-based penalty.
Wherein the dynamic weighting module assigns a weight to each modal input at a given timing step based on its consistency assessment over time. Is distributed to the inputThe dynamic weights of the inbound modality represent the consistency between the current input and the fused data of the previous timing step. Using bilinear function to evaluate consistency scores of two modalitiesAnd
Figure BDA0002197714580000052
namely:
Figure BDA0002197714580000053
here, the
Figure BDA0002197714580000054
Are parameters learned during the model training process. The weights for these two modes are obtained by normalizing the scores using Laplace smoothing:
Figure BDA0002197714580000055
the GRU module contains various gate units for modulating the flow of information within the module. The GRU module will x at time ttAnd ytAs input and tracking three quantities, i.e. fused representations
Figure BDA0002197714580000056
And modality specific representation
Figure BDA0002197714580000057
Fused representationA single representation of the historical multimodal input is constructed. Modality specific representation
Figure BDA0002197714580000059
Considered as projections that maintain modal input, a measure of their correlation is computed. The calculations in this module are expressed as follows:
Figure BDA00021977145800000511
Figure BDA00021977145800000512
Figure BDA00021977145800000513
Figure BDA00021977145800000514
Figure BDA00021977145800000515
Figure BDA00021977145800000516
Figure BDA00021977145800000517
where sigma is a logical sigmoid function,
Figure BDA00021977145800000519
is a hyperbolic tangent function, r and z are the inputs to the reset and update gates, h andrepresents activation and candidate activation of a standard GRU, respectively, b is the bias of the neuron; u shaperIs the input to the r-th neuron of the output layer.
The model uses separate weights for different inputs X and Y so that the model can capture modality relationships and specific aspects of each modality.
Correlathe motion module calculates modal input obtained from the GRU module
Figure BDA0002197714580000061
Correlation between the projections of (a). Formally, two modalities are given at time t
Figure BDA0002197714580000062
And
Figure BDA0002197714580000063
the correlation is calculated as follows:
Figure BDA0002197714580000064
wherein the content of the first and second substances,
Figure BDA0002197714580000065
expressing a correlation-based loss function as
Figure BDA0002197714580000066
The correlation between the two modalities is maximized by this function.
(5) Multi-modal decoder
Joint representation computed by multi-modal decoder from multi-modal encoder
Figure BDA0002197714580000067
The respective modality input sequences x and y are reconstructed simultaneously. By minimizing the reconstruction loss during training, the resulting joint representation retains as much information as possible from both modalities. In order to share information in the modalities, two additional reconstruction loss terms are introduced in the multi-modal decoder: cross reconstruction and self reconstruction. Finally, the multimode decoder includes three reconstruction losses:
fusion reconstruction loss from joint representation
Figure BDA0002197714580000068
Error of reconstruction:
loss of self-reconstruction fromMiddle reconstruction
Figure BDA00021977145800000611
From
Figure BDA00021977145800000612
Middle reconstruction
Figure BDA00021977145800000613
Error:
Figure BDA00021977145800000614
cross reconstruction loss fromMiddle reconstruction
Figure BDA00021977145800000616
From
Figure BDA00021977145800000617
Middle reconstruction
Figure BDA00021977145800000618
Error:
Figure BDA00021977145800000619
where β is a hyper-parameter that balances the relative proportions of the loss function values of the two input modalities, and f, g represent the feature maps implemented by the multi-modal encoder and decoder, respectively. The objective function used to train the model is represented as:
Figure BDA00021977145800000620
where λ is the hyper-parameter used to scale the contribution of the associated loss term and N is the size of the small batch of data used in the training phase. The objective function incorporates different forms of reconstruction loss computed by the decoder, with the associated loss computed as part of the encoding process. The objective function is optimized using a stochastic gradient descent algorithm with an adaptive learning rate.
(6) Classifying collected sensor data
Sensor features are obtained by computing the output of the last convolutional layer (layer 5) of the lstm (dcl) network, and features are extracted from the acquired normalized data. Sampling is performed from the persistent timing using a sliding window based on the extracted features. These motion sequences are used to train the CorrRNN model, and the fall detection model is trained using stochastic gradient descent.
Further, in the fourth step, classifiers are respectively constructed by collecting data through waist and wrist sensors, and the classification results are weighted and combined to obtain a fall type judgment result, which specifically comprises:
the classifiers for the waist and the wrist are respectively constructed, the two classifiers are combined in a weighting way, the single classification precision of the two classifiers is obtained, and the waist classification precision is O1Wrist classification accuracy of O2The weight of the waist classifier is O1/(O1+O2) The weight of the wrist classifier is O2/(O1+O2) And the two classifiers are combined in a weighting mode to form a final classifier for fall detection.
Another object of the present invention is to provide a wearable sensor applying the human fall detection method based on the movable sensor combination device.
Another object of the present invention is to provide an electronic information detecting system applying the human body falling detection method based on the movable sensor combination device.
In summary, the advantages and positive effects of the invention are:
compared with the prior art, the method provided by the invention has the advantages that the fall detection is carried out by combining two wearable sensors, and a CorrRNN model is adopted. The model is based on an encoder-decoder framework, learning a joint representation of a multivariate input by exploiting cross-modal correlations. The model is trained in an unsupervised manner (i.e., by minimizing the input-output reconstruction penalty term and maximizing the cross-modality based correlation term), which eliminates the need for labeling data, and incorporates GRUs to capture long-term dependencies and timing input structures. The falling detection precision can be greatly improved.
Because the acceleration data and the data of the screw instrument have different numerical ranges, the precision can be improved through normalization processing. The invention provides a human body falling detection method by wearing a waist and wrist sensor combination, which is characterized in that the detection precision and fault tolerance of falling detection are improved by wearing wearable sensors on the waist and the wrist to acquire data and fusing the data for falling detection, so that the falling detection precision is greatly improved.
Drawings
Fig. 1 is a general flow chart of a fall detection method provided by an embodiment of the invention.
Fig. 2 is a flowchart of a human fall detection method based on a movable sensor combination device according to an embodiment of the present invention.
Fig. 3 is a flow chart of the corrnn model according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The invention aims to provide a falling detection method based on a recurrent neural network, which uses a wearable sensor to acquire data. The method has the advantages of robustness, effectiveness and advanced performance. The invention uses two sets of sensors which are respectively worn at the right wrist part and the middle position of the front side of the hip bone at the waist part, and the falling detection method adopts a CorrRNN model. The model learns the timing dependencies between the joint representation and the modalities simultaneously, uses multiple loss terms in the objective function, including a maximum correlation loss term to enhance cross-learning modality information, and uses an attention model to dynamically adjust the contributions of different input modalities to the joint representation to achieve a desired effect. The accuracy of detection is further improved by constructing the CorrRNN models of the wrist and the waist respectively and combining the two CorrRNN models through weighting. The method has high detection precision and high effectiveness for fall detection of old people in different scenes.
The following detailed description of the principles of the invention is provided in connection with the accompanying drawings.
The invention uses two wearable sensor devices to detect human body falling. Two sensors are placed at the waist and wrist, respectively, and the collected data includes acceleration data and gyroscope data. Sensor data of two different positions are respectively input into a model to be trained to obtain two classifiers, and finally the two classifiers are combined to be classified, wherein the classifiers are constructed by adopting a CorrRNN model.
As shown in fig. 1, a method for detecting a human fall based on a movable sensor combination device provided by an embodiment of the present invention includes:
s101: collecting human user sensor data based on a wearable sensor system;
s102: carrying out numerical value normalization processing on the acquired sensor data;
s103: classifying the collected sensor data by adopting a CorrRNN model based on time sequence multi-modal learning;
s104: and respectively constructing classifiers by collecting data through the waist sensor and the wrist sensor, and performing weighted combination on classification results to obtain a falling type judgment result.
Further, the acquiring of the human body user sensor data based on the wearable sensor system in the step one specifically includes:
(1) a wearable sensor system for collecting user data is an Inertial Measurement Unit (IMU), the IMU can measure three-axis acceleration and three-axis angular velocity in daily actions, the IMU is connected with a Personal Computer (PC) through Bluetooth, wireless real-time transmission of data is achieved, and sampling frequency is set to be 20 Hz.
(2) The two sensors are respectively arranged on the wrist and the waist, the wearing position of the wrist is the position where the radius of the right hand is close to the carpal bone, and the wearing position of the waist is the position in the middle of the front side of the hip bone.
(3) An action category. The actions are divided into falling and daily actions, and the daily actions include basic actions such as sitting, standing, lying, jogging, running, going up and down stairs, and daily activities such as answering and making calls, squatting and the like.
Further, the performing of the numerical normalization processing on the acquired sensor data in the second step specifically includes:
and normalizing the signal value, and mapping all data to a range of 0-1. By using a linear function normalization method, the conversion function is as follows:
where max is the maximum value of the sample data and min is the minimum value of the sample data. x is the raw sensor signal data and x is the normalized result.
Further, the step three of classifying the collected sensor data by using the corrnn model based on time-series multimodal learning specifically includes:
(1) inputting data, collecting data of falling and daily actions of N testers as training samples, and recording a sample set as T ((X)1，Y1，X2，Y2，c)1,…,(X1，Y1，X2，Y2，c)N) Wherein X is1，Y1，X2，Y2Respectively representing data collected by two wearable sensors, X1Representing acceleration data collected by the lumbar sensor, Y1Representing the spirometer data collected by the lumbar sensor; x2Representing acceleration data collected by wrist sensors, Y2Representing the gyroscope data collected by the wrist sensor and c representing the motion class.
Inputting the data collected from waist and wrist into modelTraining, and finally combining the two classifiers for final classification. In the subsequent steps, the waist data is used as input, and the methods adopted by the wrist and waist sensor data are the same. Whereby the subsequent step will be X1，Y1Abbreviated x, y.
Both data (acceleration data, gyroscope data) are represented as a sequence of length T.
Figure BDA0002197714580000101
And
Figure BDA0002197714580000103
representing the m-dimensional characteristics of the X mode at time sequence t. For simplicity, the superscripts m and n are omitted subsequently.
(2) Time-series multi-modal learning
The two modes are fused at t, and the extension is expressed as
Figure BDA0002197714580000104
Andl is the historical timing range. Given a multi-modal data sequence pairTraining a feature learning model M to learn multi-dimensional joint representation
Figure BDA0002197714580000107
(3) CorrRNN model structure
The model enables encoder-decoder operation, sequence-to-sequence learning, and learning of the sequence representation in an unsupervised manner. The model includes two regenerated neural networks: a multi-modal encoder and multi-modal decoder, the model flow chart is shown in fig. 3. The multi-modal encoder maps the two input sequences to a common space. The multi-modal decoder reconstructs the two input sequences from the joint representation obtained by the encoder. Both the encoder and decoder are two-layer networks. The multi-modal input is first mapped to a separate hidden layer before being passed to a common layer called the fusion layer. Similarly, prior to reconstructing the multimodal input, the joint representation is first decoded to separate the hidden layers.
Using a pair of multimodal inputs to train the model, the activation of the fusion layer in the last step encoder is output as a sequence feature representation. Two types of feature representations can be obtained from the model input: if both input modalities exist, obtaining a joint representation of them; if only one form exists, an "enhanced" single-modality representation will be obtained.
(4) Multi-modal encoder
The multi-modal encoder fuses the input modality sequences into a common representation such that one coherent input is given greater importance and the correlation between the inputs is maximized. Thus, the multimodal encoder uses three main modules at each time.
Dynamic weighting module (DW): the signals of the two modalities are dynamically weighted by evaluating the input signal's conformity with the recent history.
GRU module (GRU): the input modalities are fused to generate a fused representation. The module also uses the timing structure of the reset and update gate capture sequences.
Correlation module (Corr): intermediate states generated by the GRU module are used as inputs to compute the correlation-based penalty.
Wherein the dynamic weighting module assigns a weight to each modal input at a given timing step based on its consistency assessment over time. The dynamic weights assigned to the input modalities represent the correspondence between the current input and the fused data of the previous time-series step. Using bilinear function to evaluate consistency scores of two modalities
Figure BDA0002197714580000111
And
Figure BDA0002197714580000112
namely:
Figure BDA0002197714580000113
here, the
Figure BDA0002197714580000114
Are parameters learned during the module training process. The weights for these two modes are obtained by normalizing the scores using Laplace smoothing:
Figure BDA0002197714580000115
the GRU module contains various gate units for modulating the flow of information within the module. The GRU module will x at timing step ttAnd ytAs input and tracking three quantities, i.e. fused representations
Figure BDA0002197714580000116
And modality specific representation
Figure BDA0002197714580000117
Fused representation
Figure BDA0002197714580000118
A single representation of the historical multimodal input is constructed. Modality specific representation
Figure BDA0002197714580000119
Considered as projections that maintain modal input, a measure of their correlation is computed. The calculations in this module can formally be expressed as follows:
Figure BDA00021977145800001110
Figure BDA00021977145800001111
Figure BDA0002197714580000122
Figure BDA0002197714580000124
Figure BDA0002197714580000125
Figure BDA0002197714580000126
where sigma is a logical sigmoid function,
Figure BDA00021977145800001215
is a hyperbolic tangent function, r and z are the inputs to the reset and update gates, h and
Figure BDA0002197714580000127
represents activation and candidate activation of a standard GRU, respectively, b is the bias of the neuron; u shaperIs the input to the r-th neuron of the output layer.
The model uses separate weights for different inputs X and Y so that the model can capture modality relationships and specific aspects of each modality.
A Correlation module calculates modal inputs obtained from the GRU moduleCorrelation between the projections of (a). Formally, two modalities are given at time t
Figure BDA0002197714580000129
And
Figure BDA00021977145800001210
the correlation is calculated as follows:
Figure BDA00021977145800001211
wherein the content of the first and second substances,
Figure BDA00021977145800001212
expressing a correlation-based loss function as
Figure BDA00021977145800001213
The correlation between the two modalities is maximized by this function.
(5) Multi-modal decoder
Joint representation computed by multi-modal decoder from multi-modal encoderThe respective modality input sequences x and y are reconstructed simultaneously. By minimizing the reconstruction loss during training, the resulting joint representation retains as much information as possible from both modalities. In order to share information in the modalities, two additional reconstruction loss terms are introduced in the multi-modal decoder: cross reconstruction and self reconstruction. The joint representation is facilitated and the performance of the model can be improved also in the presence of only one of the modalities. Finally, the multimode decoder includes three reconstruction losses:
fusion reconstruction loss from joint representationError of reconstruction:
Figure BDA0002197714580000132
loss of self-reconstruction from
Figure BDA0002197714580000133
Middle reconstruction
Figure BDA0002197714580000134
From
Figure BDA0002197714580000135
Middle reconstruction
Figure BDA0002197714580000136
Error:
cross reconstruction loss fromMiddle reconstruction
Figure BDA0002197714580000139
From
Figure BDA00021977145800001310
Middle reconstruction
Figure BDA00021977145800001311
Error:
Figure BDA00021977145800001312
where β is a hyper-parameter that balances the relative proportions of the loss function values of the two input modalities, and f, g represent the feature maps implemented by the multi-modal encoder and decoder, respectively. Thus, the objective function used to train the model can be expressed as:
Figure BDA00021977145800001313
where λ is the hyper-parameter used to scale the contribution of the correlated loss term and N is the mini-batch size used in the training phase. Thus, the objective function incorporates different forms of reconstruction loss computed by the decoder, with the associated loss computed as part of the encoding process. The objective function is optimized using a stochastic gradient descent algorithm with an adaptive learning rate.
(6) Classifying collected sensor data
Sensor features are obtained by computing the output of the last convolutional layer (layer 5) of the lstm (dcl) network, and features are extracted from the acquired normalized data. Sampling is performed from the persistent timing using a sliding window based on the extracted features. These activity sequences were used to train the CorrRNN model, using random gradient descent, to train the fall detection model.
Further, the step four includes the steps of respectively constructing classifiers by collecting data through the waist sensor and the wrist sensor, and obtaining fall type judgment results by weighting and combining classification results, wherein the step four includes the following steps:
the classifiers for the waist and the wrist are respectively constructed, the two classifiers are combined in a weighting way, the single classification precision of the two classifiers is obtained, and the waist classification precision is O1Wrist classification accuracy of O2The weight of the waist classifier is O1/(O1+O2) The weight of the wrist classifier is O2/(O1+O2) And the two classifiers are combined in a weighting mode to form a final classifier for fall detection.
The invention provides a fall detection method based on a waist and wrist sensor combination. The classifier employs a CorrRNN model that learns the temporal dependencies between modality joint representations and modalities, and can dynamically weigh different input modalities so that more useful signals can be emphasized, as is applicable in sensors. The two part classifiers are combined in a weighting mode to provide higher detection precision and adapt to more human activity environments.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.

Claims (7)

1. A human body falling detection method based on movable sensor combination equipment is characterized by comprising the following steps:
acquiring sensor data of a human body user based on a wearable sensor system;
secondly, carrying out numerical value normalization processing on the acquired sensor data;
classifying the acquired sensor data by adopting a CorrRNN model based on time sequence multi-modal learning;
and step four, respectively constructing classifiers through data collected by the waist sensor and the wrist sensor, and performing weighted combination on classification results to obtain falling type judgment results.
2. A personal fall detection method based on a mobile sensor combination as claimed in claim 1, wherein the first step of collecting the personal user sensor data based on the wearable sensor system specifically comprises:
(1) the wearable sensor system for collecting user data is an Inertial Measurement Unit (IMU), the IMU measures three-axis acceleration and three-axis angular velocity in daily actions, the IMU and a Personal Computer (PC) are connected through Bluetooth, wireless real-time transmission of data is achieved, and the sampling frequency is set to be 20 Hz;
(2) the two sensors are respectively placed on the wrist and the waist, the wearing position of the wrist is that the radius of the right hand is close to the carpal bone, and the wearing position of the waist is that the front middle position of the hip bone;
(3) action type: the movements are divided into falls and daily movements, which include basic movements of sitting, standing, lying, jogging, running, going up and down stairs, and daily activities of making and receiving calls, squatting.
3. The human body fall detection method based on the movable sensor combination device as claimed in claim 1, wherein the performing of the numerical normalization processing on the acquired sensor data in the second step specifically comprises:
normalizing the signal value, and mapping all data to a range of 0-1; by using a linear function normalization method, the conversion function is as follows:
Figure FDA0002197714570000011
wherein max is the maximum value of the sample data, and min is the minimum value of the sample data; x is the raw sensor signal data and x is the normalized result.
4. A human fall detection method based on a mobile sensor combination as claimed in claim 1, wherein the step three of classifying the collected sensor data by using a CorrRNN model based on temporal multimodal learning specifically comprises:
(1) inputting data, collecting data of falling and daily actions of N testers as training samples, and recording a sample set as T ((X)1，Y1，X2，Y2，c)1...，(X1，Y1，X2，Y2，c)N) Wherein X is1，Y1，X2，Y2Respectively representing data collected by two wearable sensors, X1Representing acceleration data collected by the lumbar sensor, Y1Representing the spirometer data collected by the lumbar sensor; x2Representing acceleration data collected by wrist sensors, Y2Representing the spirometer data collected by the wrist sensor, c representing the motion classification;
respectively inputting data collected by the waist and the wrist into a model for training, and finally combining the two classifiers for final classification; in the subsequent steps, waist data are used as input, and the data of the wrist sensor and the waist sensor are in the same method; whereby the subsequent step will be X1，Y1Abbreviated as x, y;
representing the acceleration data and the gyroscope data as a sequence of length T;
Figure FDA0002197714570000021
and
Figure FDA0002197714570000023
representing m-dimensional characteristics of the X mode at a time sequence t;
(2) time-series multi-modal learning
The two modes are fused at t, and the extension is expressed as
Figure FDA0002197714570000024
Andl is the historical timing range; given a multi-modal data sequence pair
Figure FDA0002197714570000026
Training a feature learning model M to learn multi-dimensional joint representation
Figure FDA0002197714570000027
(3) CorrRNN model structure
Two regenerated neural networks are included: a multi-modal encoder and a multi-modal decoder;
the multi-modal encoder maps the two input sequences to a common space; the multi-modal decoder reconstructs the two input sequences from the joint representation obtained by the encoder; both the encoder and decoder are two-layer networks;
the multimodal input is first mapped to a separate hidden layer before being sent to a common layer called the fusion layer; prior to reconstructing the multimodal input, the joint representation is first decoded to separate the hidden layers;
training a model using a pair of multimodal inputs, the activation of the fusion layer in the last encoder step being output as a sequence feature representation; two types of feature representations can be obtained from the model input: if both input modalities exist, obtaining a joint representation of them; if only one form exists, an "enhanced" single-modality representation will be obtained;
(4) multi-modal encoder
The multimodal encoder fuses the input modality sequences into a common representation, the multimodal encoder using three main modules at each time;
dynamic weighting module DW: dynamically weighting the signals of the two modalities by evaluating the consistency of the input signal with the recent history;
GRU module GRU: fusing the input modalities to generate a fused representation; the module also uses the timing structure of the reset and update gate capture sequences;
correlation module Corr: taking the intermediate state generated by the GRU module as an input to calculate a correlation-based penalty;
wherein the dynamic weighting module assigns a weight to each modal input at a given timing step size based on its consistency assessment with timing; the dynamic weights assigned to the input modalities represent the consistency between the current input and the fused data of the previous time sequence step; using bilinear function to evaluate consistency scores of two modalities
Figure FDA00021977145700000313
And
Figure FDA0002197714570000031
Figure FDA0002197714570000032
Figure FDA0002197714570000033
is a parameter learned during the module training process; the weights for these two modes are obtained by normalizing the scores using Laplace smoothing:
Figure FDA0002197714570000034
the GRU module comprises different gate units and is used for modulating information flow in the module; the GRU module will x at time ttAnd ytAs input and tracking three quantities, i.e. fused representations
Figure FDA0002197714570000035
And modality specific representation
Figure FDA0002197714570000036
Fused representation
Figure FDA0002197714570000037
Constructing a single representation of the historical multimodal input; modality specific representation
Figure FDA0002197714570000038
Projections considered to maintain modal input, compute a measure of their correlation; the calculations in this module are expressed as follows:
Figure FDA0002197714570000039
Figure FDA00021977145700000310
Figure FDA00021977145700000311
Figure FDA00021977145700000312
Figure FDA0002197714570000041
Figure FDA0002197714570000042
Figure FDA0002197714570000043
Figure FDA0002197714570000044
where sigma is a logical sigmoid function,
Figure FDA0002197714570000045
is a hyperbolic tangent function, r and z are the inputs to the reset and update gates, h andrespectively representing activation and candidate activation of a standard GRU; b is the bias of the neuron; u shaperIs the input of the r-th neuron of the output layer;
the model uses separate weights for different inputs X and Y, such that the model captures modal relationships and specific aspects of each modality;
a Correlation module calculates modal inputs obtained from the GRU module
Figure FDA0002197714570000047
Correlation between the projections of (a); formally, two modalities are given at time tAnd
Figure FDA0002197714570000049
the correlation is calculated as follows:
Figure FDA00021977145700000410
wherein;
Figure FDA00021977145700000411
expressing a correlation-based loss function as
Figure FDA00021977145700000412
The correlation between the two modalities is maximized by this function;
(5) multi-modal decoder
Joint representation computed by multi-modal decoder from multi-modal encoder
Figure FDA00021977145700000413
Simultaneously reconstructing input sequences x and y of each mode; by minimizing reconstruction loss during training, the resulting joint representation retains as much information as possible from both modalities; two additional reconstruction loss terms are introduced in the multi-modal decoder: cross reconstruction and self reconstruction; finally, the multimode decoder includes three reconstruction losses:
fusion reconstruction loss from joint representation
Figure FDA00021977145700000414
Error of reconstruction:
Figure FDA00021977145700000415
loss of self-reconstruction from
Figure FDA00021977145700000416
Middle reconstruction
Figure FDA00021977145700000417
From
Figure FDA00021977145700000418
Middle reconstruction
Figure FDA00021977145700000419
Error:
Figure FDA0002197714570000051
cross reconstruction loss from
Figure FDA0002197714570000052
Middle reconstruction
Figure FDA0002197714570000053
From
Figure FDA0002197714570000054
Middle reconstructionError:
where β is a hyper-parameter that balances the relative proportions of the loss function values for the two input modalities, and f, g represent the feature maps implemented by the multi-modal encoder and decoder, respectively; the objective function used to train the model is represented as:
Figure FDA0002197714570000057
where λ is the hyper-parameter used to scale the contribution of the correlated loss term, and N is the size of the small batch of data used in the training phase; the objective function incorporates different forms of reconstruction loss computed by the decoder, with the correlation loss computed as part of the encoding process; optimizing the objective function using a stochastic gradient descent algorithm with an adaptive learning rate;
(6) classifying collected sensor data
Obtaining sensor characteristics by calculating the output of the last convolutional layer of the LSTM network, and extracting characteristics from the obtained normalized data; sampling from a continuous time sequence using a sliding window according to the extracted features; the motion sequence is used to train a CorrRNN model, and a fall detection model is trained using stochastic gradient descent.
5. The human body fall detection method based on the mobile sensor combination device as claimed in claim 1, wherein in the fourth step, the data collected by the waist and wrist sensors are used to respectively construct classifiers, and the weighted combination of the classification results to obtain the fall classification judgment result specifically comprises:
the classifiers for the waist and the wrist are respectively constructed, the two classifiers are combined in a weighting way, the single classification precision of the two classifiers is obtained, and the waist classification precision is O1Wrist classification accuracy of O2The weight of the waist classifier is O1/(O1+O2) The weight of the wrist classifier is O2/(O1+O2) And the two classifiers are combined in a weighting mode to form a final classifier for fall detection.
6. A wearable sensor applying the human body falling detection method based on the movable sensor combination equipment according to any one of claims 1 to 5.
7. An electronic information detection system applying the human body falling detection method based on the movable sensor combination equipment as claimed in any one of claims 1 to 5.
CN201910853810.4A 2019-09-10 2019-09-10 Human body falling detection method based on movable sensor combination equipment Pending CN110659677A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910853810.4A CN110659677A (en) 2019-09-10 2019-09-10 Human body falling detection method based on movable sensor combination equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910853810.4A CN110659677A (en) 2019-09-10 2019-09-10 Human body falling detection method based on movable sensor combination equipment

Publications (1)

Publication Number Publication Date
CN110659677A true CN110659677A (en) 2020-01-07

Family

ID=69036903

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910853810.4A Pending CN110659677A (en) 2019-09-10 2019-09-10 Human body falling detection method based on movable sensor combination equipment

Country Status (1)

Country Link
CN (1) CN110659677A (en)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111681389A (en) * 2020-06-12 2020-09-18 电子科技大学 Old people falling behavior detection method based on blind source separation
CN111931568A (en) * 2020-06-30 2020-11-13 青岛科技大学 Human body falling detection method and system based on enhanced learning
CN112998697A (en) * 2021-02-22 2021-06-22 电子科技大学 Tumble injury degree prediction method and system based on skeleton data and terminal
CN113286309A (en) * 2021-05-18 2021-08-20 合肥工业大学 Heterogeneous communication method and system based on CSI
WO2021212883A1 (en) * 2020-04-20 2021-10-28 电子科技大学 Fall detection method based on intelligent mobile terminal
CN115731602A (en) * 2021-08-24 2023-03-03 中国科学院深圳先进技术研究院 Human body activity recognition method, device, equipment and storage medium based on topological representation
CN117173795A (en) * 2023-11-03 2023-12-05 赋之科技(深圳)有限公司 Dangerous action detection method and terminal

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106539587A (en) * 2016-12-08 2017-03-29 浙江大学 A kind of fall risk assessment and monitoring system and appraisal procedure based on sensor of doing more physical exercises
CN106887115A (en) * 2017-01-20 2017-06-23 安徽大学 A kind of Falls Among Old People monitoring device and fall risk appraisal procedure
US20170220854A1 (en) * 2016-01-29 2017-08-03 Conduent Business Services, Llc Temporal fusion of multimodal data from multiple data acquisition systems to automatically recognize and classify an action
CN109171738A (en) * 2018-07-13 2019-01-11 杭州电子科技大学 Fall detection method based on human body acceleration multiple features fusion and KNN

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170220854A1 (en) * 2016-01-29 2017-08-03 Conduent Business Services, Llc Temporal fusion of multimodal data from multiple data acquisition systems to automatically recognize and classify an action
CN106539587A (en) * 2016-12-08 2017-03-29 浙江大学 A kind of fall risk assessment and monitoring system and appraisal procedure based on sensor of doing more physical exercises
CN106887115A (en) * 2017-01-20 2017-06-23 安徽大学 A kind of Falls Among Old People monitoring device and fall risk appraisal procedure
CN109171738A (en) * 2018-07-13 2019-01-11 杭州电子科技大学 Fall detection method based on human body acceleration multiple features fusion and KNN

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
XITONG YANG等: "Deep Multimodal Representation Learning from Temporal Data", 《IEEE CVPR》 *

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021212883A1 (en) * 2020-04-20 2021-10-28 电子科技大学 Fall detection method based on intelligent mobile terminal
CN111681389A (en) * 2020-06-12 2020-09-18 电子科技大学 Old people falling behavior detection method based on blind source separation
CN111681389B (en) * 2020-06-12 2021-02-26 电子科技大学 Old people falling behavior detection method based on blind source separation
CN111931568A (en) * 2020-06-30 2020-11-13 青岛科技大学 Human body falling detection method and system based on enhanced learning
CN112998697A (en) * 2021-02-22 2021-06-22 电子科技大学 Tumble injury degree prediction method and system based on skeleton data and terminal
CN112998697B (en) * 2021-02-22 2022-06-14 电子科技大学 Tumble injury degree prediction method and system based on skeleton data and terminal
CN113286309A (en) * 2021-05-18 2021-08-20 合肥工业大学 Heterogeneous communication method and system based on CSI
CN115731602A (en) * 2021-08-24 2023-03-03 中国科学院深圳先进技术研究院 Human body activity recognition method, device, equipment and storage medium based on topological representation
CN117173795A (en) * 2023-11-03 2023-12-05 赋之科技(深圳)有限公司 Dangerous action detection method and terminal
CN117173795B (en) * 2023-11-03 2024-02-23 赋之科技(深圳)有限公司 Dangerous action detection method and terminal

Similar Documents

Publication Publication Date Title
CN110659677A (en) Human body falling detection method based on movable sensor combination equipment
Zhu et al. Using deep learning for energy expenditure estimation with wearable sensors
Atallah et al. Sensor placement for activity detection using wearable accelerometers
CN110236550B (en) Human gait prediction device based on multi-mode deep learning
Sikder et al. Log-sum distance measures and its application to human-activity monitoring and recognition using data from motion sensors
CN104598880A (en) Behavior identification method based on fuzzy support vector machine
Khodabandelou et al. Attention-based gated recurrent unit for gesture recognition
CN111274998A (en) Parkinson's disease finger knocking action identification method and system, storage medium and terminal
CN113171080B (en) Energy metabolism evaluation method and system based on wearable sensing information fusion
Malshika Welhenge et al. Human activity classification using long short-term memory network
Lubina et al. Artificial neural networks in accelerometer-based human activity recognition
Padmaja et al. A novel random split point procedure using extremely randomized (Extra) trees ensemble method for human activity recognition
Kale et al. Human posture recognition using artificial neural networks
Chuang et al. A wearable activity sensor system and its physical activity classification scheme
Liu et al. Deep-learning-based signal enhancement of low-resolution accelerometer for fall detection systems
CN113974612B (en) Automatic evaluation method and system for upper limb movement function of stroke patient
CN110414590A (en) Physical activity recognition methods based on Intelligent mobile equipment and convolutional neural networks
Tarekegn et al. Enhancing human activity recognition through sensor fusion and hybrid deep learning model
CN110598789A (en) Human fatigue state prediction method and system based on fuzzy perceptron
CN114521900B (en) Arrhythmia classification and identification method based on transfer learning
Kabir et al. Secure Your Steps: A Class-Based Ensemble Framework for Real-Time Fall Detection Using Deep Neural Networks
CN115554674A (en) Motion energy consumption prediction method and device
Mekruksavanich et al. Position-aware Human Activity Recognition with Smartphone Sensors based on Deep Learning Approaches
Sowmiya et al. A hybrid approach using bidirectional neural networks for human activity recognition
Kongsil et al. Physical activity recognition using streaming data from wrist-worn sensors

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20200107

RJ01 Rejection of invention patent application after publication