CN113963442A - Fall-down behavior identification method based on comprehensive body state features - Google Patents
Fall-down behavior identification method based on comprehensive body state features Download PDFInfo
- Publication number
- CN113963442A CN113963442A CN202111243199.7A CN202111243199A CN113963442A CN 113963442 A CN113963442 A CN 113963442A CN 202111243199 A CN202111243199 A CN 202111243199A CN 113963442 A CN113963442 A CN 113963442A
- Authority
- CN
- China
- Prior art keywords
- human body
- fall
- height
- skeleton
- method based
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 44
- 238000001514 detection method Methods 0.000 claims abstract description 25
- 238000004364 calculation method Methods 0.000 claims abstract description 19
- 238000012545 processing Methods 0.000 claims abstract description 18
- 230000008859 change Effects 0.000 claims abstract description 14
- 238000007781 pre-processing Methods 0.000 claims abstract description 9
- 238000000605 extraction Methods 0.000 claims abstract description 6
- 238000010276 construction Methods 0.000 claims abstract description 4
- 230000006399 behavior Effects 0.000 claims description 35
- 230000000877 morphologic effect Effects 0.000 claims description 9
- 230000008569 process Effects 0.000 claims description 8
- 230000009471 action Effects 0.000 claims description 7
- 238000010586 diagram Methods 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 claims description 5
- 230000002708 enhancing effect Effects 0.000 claims description 4
- 238000004880 explosion Methods 0.000 claims description 4
- 230000006870 function Effects 0.000 claims description 4
- 230000009467 reduction Effects 0.000 claims description 4
- 238000004458 analytical method Methods 0.000 claims description 3
- 238000013136 deep learning model Methods 0.000 claims description 3
- 230000010365 information processing Effects 0.000 claims description 3
- 238000012549 training Methods 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 claims description 2
- 239000000284 extract Substances 0.000 claims description 2
- 238000010606 normalization Methods 0.000 claims description 2
- 238000005259 measurement Methods 0.000 claims 1
- 238000011478 gradient descent method Methods 0.000 abstract description 2
- 210000000988 bone and bone Anatomy 0.000 description 4
- 238000013528 artificial neural network Methods 0.000 description 3
- 210000000245 forearm Anatomy 0.000 description 3
- 230000007774 longterm Effects 0.000 description 3
- 230000008033 biological extinction Effects 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 210000003414 extremity Anatomy 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 208000027418 Wounds and injury Diseases 0.000 description 1
- 230000032683 aging Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000005452 bending Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000006378 damage Effects 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000009432 framing Methods 0.000 description 1
- 230000036449 good health Effects 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000005802 health problem Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 208000014674 injury Diseases 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 239000013598 vector Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a falling behavior identification method based on a posture comprehensive characteristic, which comprises the following steps: video data acquisition → extraction of a human body skeleton image → preprocessing of skeleton image data → construction of a binary image circumscribed rectangle → LSTM network unit processing → calculation of the effective area of a human body → calculation of the aspect ratio → calculation of the distance from the center of mass of the human body to the ground → calculation of the height change rate → occurrence of a fall. According to the method, on the basis of accurately extracting the skeleton information of the human body, various posture characteristics of the human body are effectively combined to comprehensively judge the falling behavior, the detection accuracy is improved, and the lightweight openpit is used, so that the data volume is light, and then an enhanced version gradient descent method and an SGD (generalized gradient descent) method are used, so that the accuracy can be ensured while the convergence speed of the model is accelerated, and the trained optimal model can be conveniently deployed to each edge end and applied to places such as homes, public places and the like.
Description
Technical Field
The invention relates to the technical field of artificial intelligence, in particular to a falling behavior identification method based on comprehensive body state features.
Background
The social reality of the rapid aging of the population at present enables the old people to have a desire for safety and health problems, and factors threatening the health of the old people not only belong to internal factors such as diseases and psychology, but also are more important to be hurt by falling caused by the combination of internal and external factors. The old people are the main groups which are dangerous and fatal due to falling, and researches show that 20% of old people in China are seriously injured after falling, and even the old people with very good health condition at ordinary times still have the serious injury after falling by 17.7% of the old people. Especially, in the application scene of solitary old people, how to accurately detect the falling behavior in real time and send out an alarm has become one of the subjects of urgent research by domestic and foreign scholars.
Falls are themselves a more complex activity under the dual influence of both internal and external factors: the internal factors are mainly reflected in that people can judge wrong similarity actions by eyes such as squatting and falling, lying and falling, and the like; the external factor body is easily influenced by complex environments such as illumination, shadow, furniture shielding and the like in the current application scene. The influence of internal and external factors can cause the detection accuracy to be reduced, so that the method for realizing fall detection is particularly important. Common methods generally include methods based on wearable devices, environmental sensors, computer vision and the like, and the wearable detection method generally brings serious discomfort to users, and particularly causes inconvenience in actions when facing heavy wearable devices of the elderly; the detection method based on the environmental sensor usually needs to install expensive detection equipment on the floor or the wall, and has poor flexibility and portability and high false detection rate; with the rapid development of deep learning, people naturally focus on a computer vision method, the defects brought by the two methods can be well avoided by adopting the computer vision method, and the computer vision method has the advantages of low cost, rapid improvement and maintenance, strong portability, good flexibility and the like, so that a tumble behavior identification method based on the comprehensive body state characteristics is provided.
Disclosure of Invention
The invention aims to provide a falling behavior identification method based on comprehensive body state features, and the method solves the problems.
In order to achieve the purpose, the invention is realized by the following technical scheme:
the invention provides a falling behavior identification method based on a posture comprehensive characteristic, which comprises the following steps: video data acquisition → extraction of a human body skeleton image → preprocessing of skeleton image data → construction of a binary image circumscribed rectangle → LSTM network unit processing → calculation of human body effective area → calculation of aspect ratio → calculation of distance from human body center of mass to ground → calculation of height change rate → occurrence of a fall, wherein:
s1, video data acquisition: the system is used for acquiring a video sequence from a video file or a camera and initializing the video sequence;
s2, extracting a human skeleton diagram: detecting a human body target according to the key frame image provided by the video sequence read in the S1, and extracting human body skeleton information for subsequent steps;
s3, skeleton image data preprocessing: the system is used for eliminating the extracted irrelevant information in the human skeleton module, enhancing the detectability of the relevant information and simplifying the data to the maximum extent;
s4, constructing a binary image circumscribed rectangle: the human body skeleton point information processing device is used for converting the preprocessed human body skeleton point information into a binary image;
s5, LSTM network unit processing: classification training for images;
s6, calculating the effective area of the human body: a first feature for fall detection;
s7, calculating the aspect ratio: a second morphological feature for fall detection;
s8, calculating the distance between the center of mass of the human body and the ground: a third morphological feature for fall detection;
s9, calculating the height change rate: a fourth morphological feature for fall detection;
s10, occurrence of a fall: the actions of the old are analyzed and processed through S1-S5, and whether the fall behavior occurs is judged through S6-S9.
Preferably, in S1, the collected video data is used as a sample input through the home video surveillance video, and the time-monitoring video stream is intercepted at a preset time interval by using a key frame interception technique, so as to obtain a video frame that can be sent to the deep learning model for processing.
Preferably, in S2, the skeleton of the human body is extracted using openpos, a cuff model of openpos is called using a dnn library of opencv, and then the output is post-processed, where the openpos uses an output mode of 18 joints, and the step of S2 is as follows: (1) the network structure analysis (2) extracts all key points (3) to distinguish the key points, and the relation among the key points is calculated according to an affinity field, wherein the calculation formula of the affinity field is as follows:
Preferably, S3 is normalized.
Preferably, the method of S4 is to convert the human bone point information into a binary image, then use the minimum bounding rectangle frame to find out the processed human bone binary image, and use the OpenCV function to find out the bounding rectangle, and the main steps are as follows: (1) the mathematical basis is that a point (x1, y1) on the plane is set, and a point (x2, y2) which rotates anticlockwise by theta angle around (x0, y0) is set as
x2＝(x1-x0)cosθ-(y1-y0)sinθ+x0
y2＝(x1-x0)sinθ-(y1-y0)cosθ+y0(ii) a (2) Rotating the original polygon, circulating for-90 degrees, adjusting the distance by balancing the precision and the operation efficiency, solving a simple external rectangle of the polygon after rotating each degree, and recording the area, the vertex coordinates and the degree of rotation of the simple external rectangle; (3) comparing all simple circumscribed rectangles obtained by the polygon in the rotating process to obtain a simple circumscribed rectangle with the smallest area, and obtaining the vertex coordinates and the rotating angle of the simple circumscribed rectangle; (4) and (4) rotating the circumscribed rectangle, and rotating the opposite direction of the simple circumscribed rectangle with the smallest area obtained in the previous step by the same angle opposite to the direction in the step 3 to obtain the smallest circumscribed rectangle.
Preferably, the output at the time t is merged into the output at the time t +1 in S5, so that the falling behavior recognition can be performed by integrating the time and space dimensions, the problems of gradient loss and gradient explosion caused by gradual reduction in the gradient back propagation process can be solved, and LSTM provides a reference for setting the threshold value through learning.
Preferably, in S6, the effective area of the human body is calculated by using an image of a minimum circumscribed rectangular image after binarization processing, and after binarization processing, the effective area of the human body is equal to the number of pixels with a pixel value of 1 in the minimum circumscribed rectangular frame image, and the total area of the rectangular frames is the sum of pixels with pixel values of 1 and 0, and the specific formula is as follows:
preferably, the human body width-height ratio in S7 is the width-height ratio corresponding to the minimum circumscribed rectangular frame, and when the human body is walking or standing normally, the human body width-height ratio is much less than 1, and when the human body falls down or squats, the width-height ratio is much greater than 1.
Preferably, in S8, the centroid of the human body is a diagonal intersection O (X0, Y0) of the minimum circumscribed rectangle, and the distance from the centroid to the bottom surface can well reflect the motion state of the human body at that time, and coordinates a (X1, Y1) and B (X2, Y2) of two end points of the bottom surface of the circumscribed rectangle are defined, then a is Y1-Y2, B is X1-X2, C is X1Y2-Y1X2, and the distance from the centroid O to the bottom surface ab is h, where the formula is as follows:
preferably, in S9, the feature of height change rate is introduced, and the height is measured by a ratio of a real-time height to an average height of a human body in a video, where height represents the real-time height of the human body, height average represents the average height of the human body, and height average is defined as follows:
the invention has the beneficial effects that:
firstly, extracting human skeleton information through Openpse, drawing a human skeleton external rectangle by using OpenCV, enhancing human posture estimation by fusing human width-height ratio, Hu moment characteristics, human effective area ratio, centroid change rate, outline eccentricity, height change rate and other multi-modal characteristics, and carrying out falling behavior identification from time-space and multi-modal characteristic multi-dimensionality through a long-short term memory (LSTM) neural network, processing time and spatial characteristics by combining a plurality of LSTM classifiers, sending the extracted human skeleton information into a plurality of branch loop judgments, describing time-space associated sequence data by using an autoregressive model LSTM with a time loop structure, and reducing misjudgment rate and missed judgment rate by combining video front and rear frame changes, thereby realizing real-time and accurate falling behavior detection; according to the method, on the basis of accurately extracting the skeleton information of the human body, various posture characteristics of the human body are effectively combined to comprehensively judge the falling behavior, the detection accuracy is improved, and the lightweight openpit is used, so that the data volume is light, and then an enhanced version gradient descent method and an SGD (generalized gradient descent) method are used, so that the accuracy can be ensured while the convergence speed of the model is accelerated, and the trained optimal model can be conveniently deployed to each edge end and applied to places such as homes, public places and the like.
Drawings
FIG. 1 is an inventive flow chart;
FIG. 2 is a diagram of a human skeleton according to the present invention;
FIG. 3 is a network structure analysis diagram in the step of extracting the skeleton diagram of the human body in the invention.
Detailed Description
The following further describes embodiments of the present invention with reference to examples. The following examples are only for illustrating the technical solutions of the present invention more clearly, and the protection scope of the present invention is not limited thereby.
Referring to fig. 1 to 3, a method for recognizing a falling behavior based on a comprehensive body state feature includes the following steps: video data acquisition → extraction of a human body skeleton image → preprocessing of skeleton image data → construction of a binary image circumscribed rectangle → LSTM network unit processing → calculation of human body effective area → calculation of aspect ratio → calculation of distance from human body center of mass to ground → calculation of height change rate → occurrence of a fall, wherein:
s1, video data acquisition: the system is used for acquiring a video sequence from a video file or a camera and initializing the video sequence;
s2, extracting a human skeleton diagram: detecting a human body target according to the key frame image provided by the video sequence read in the S1, and extracting human body skeleton information for subsequent steps;
s3, skeleton image data preprocessing: the system is used for eliminating the extracted irrelevant information in the human skeleton module, enhancing the detectability of the relevant information and simplifying the data to the maximum extent;
s4, constructing a binary image circumscribed rectangle: the human body skeleton point information processing device is used for converting the preprocessed human body skeleton point information into a binary image;
s5, LSTM network unit processing: classification training for images;
s6, calculating the effective area of the human body: a first feature for fall detection;
s7, calculating the aspect ratio: a second morphological feature for fall detection;
s8, calculating the distance between the center of mass of the human body and the ground: a third morphological feature for fall detection;
s9, calculating the height change rate: a fourth morphological feature for fall detection;
s10, occurrence of a fall: the actions of the old are analyzed and processed through S1-S5, and whether the fall behavior occurs is judged through S6-S9.
In this embodiment, in S1, the collected video data is input as a sample through the home video surveillance video, and the key frame capture technology is used to capture the time-monitoring video stream at a preset time interval to obtain a video frame that can be sent to the deep learning model for processing, and the video data collection module is connected to the home live video surveillance, and collects the real-time video by increasingly popular home surveillance, and captures and inputs the real-time monitoring video stream at a preset time interval, using the collected video sequence as a sample input.
In this embodiment, in S2, openpos is used to extract human skeleton, openpos' S dnn library is directly used to call the caffe model, and then post-processing is performed on the output, so that the important point in the implementation process is to analyze the affinity field representing the articulation intimacy, and as openpos has two output modes of human skeleton articulation points, namely 18 articulation points and 25 articulation points, the most important difference between the two is foot articulation points, and in the case of a fall, the foot articulation points seem to have no influence on the fall, the invention adopts 18 articulation points, and the specific process is as follows: (1) network structure resolution, as shown in FIG. 3, the two layers being spliced have 19 and 38 profiles, two stages, each having two bridges: the first branch outputs 19 feature maps, which respectively represent 18 human key points and the background; the second branch has 38 signatures representing the Affinity fields (PAF), i.e., the joint-to-joint anterior association; (2) then, extracting the key points of the limb by using the 18 characteristic graphs, calling key point codes, extracting the position and the corresponding confidence coefficient of the current joint, and traversing the characteristic graph of each joint if the positions and the confidence coefficients of all the joints are extracted; (3) all the key points are extracted, and then the key points are distinguished, but which key point belongs to which person is not calculated, and at the moment, the relation among the key points needs to be calculated according to the affinity field. For example, the affinity fields of the respective forearm key points of the first person's forearm to the three persons are definitely different, and the affinity fields are only distinguished by the affinity fields of the respective forearm key points belonging to the person. The calculation of the affinity field is to directly connect two key points, and a line is made in the middle to calculate the value on the line on the affinity field, and the formula is as follows:
(4) and finally, storing separately, wherein the principle is simple, namely, edges with connection are placed in a set, and the next node which can be connected with the previous node is continuously plugged to the corresponding position.
In this embodiment, S3 is to perform mean subtraction normalization on the video keyframe images one by one, so as to ensure that the images can be better extracted with features, and since the extracted skeleton information has no coordinate information or no framed object, there is no way to be directly applied, so that image data preprocessing is extremely necessary.
In this embodiment, S4 converts the pre-human skeleton point information into a binary image, that is, the human skeleton is a white pixel point, and the other unrelated backgrounds are black pixel points, and then the processed human skeleton binary image is framed with a minimum bounding rectangle. The problem of the minimum circumscribed rectangle is that the rectangle which circumscribes the polygon and has the minimum area is solved when the vertex of a convex polygon is given, and the human body detection → the skeleton information extraction → the framing of the circumscribed rectangle is actually realized. Finding out a circumscribed rectangle by using an OpenCV function, which mainly comprises the following steps: (1) and (3) realizing an algorithm that a certain point on the plane rotates around a fixed point by a certain angle. The mathematical basis is that, assuming that points (x1, y1) on a plane and points (x2, y2) rotated counterclockwise by an angle theta around (x0, y0) are (x2, y2), the points have
x2＝(x1-x0)cosθ-(y1-y0)sinθ+x0
y2＝(x1-x0)sinθ-(y1-y0)cosθ+y0
(2) Rotating the original polygon (circulation, the distance can balance the precision and the operation efficiency to adjust, and the distance can be adjusted), solving the simple external rectangle of the polygon after rotating each degree, and recording the area, the vertex coordinates and the degree of rotation of the simple external rectangle.
(3) And comparing all the simple external rectangles obtained by the polygon in the rotation process to obtain the simple external rectangle with the minimum area, and obtaining the vertex coordinates and the rotation angle of the simple external rectangle.
(4) The circumscribed rectangle is rotated. And (4) rotating the simple circumscribed rectangle with the smallest area obtained in the previous step by the same angle in the opposite direction (opposite to the direction of the step 3) to obtain the minimum circumscribed rectangle.
The method is used for converting the preprocessed human skeleton point information into a binary image, so that on one hand, foreground and background can be better distinguished, and on the other hand, the dimension reduction of data can be realized. And then, the processed human body bone binary image is displayed by using a minimum external rectangle frame, all the steps realize human body detection, bone extraction and preprocessing functions, and a good foundation is laid for the next gesture recognition.
In this embodiment, the LSTM network in S5 is a special type of RNN, and can learn long-term dependence information, and in a scene of fall detection, human behavior is closely linked to a timestamp, and a previous second of behavior can provide a reference for predicting a next second of behavior, so that the invention introduces processing time and spatial features of LSTM network elements. LSTM avoids the long term dependency problem by being deliberately designed, and long term information is in practice the default behavior of LSTM, rather than the ability to be obtained at great expense. All RNNs have a form of a chain of repeating neural network modules. In a standard RNN, this duplicated module has only a very simple structure, such as a tanh layer. LSTM is also such a structure, but the duplicated modules have a different structure. Unlike a single neural network layer, there are four, interacting in a very specific way. Lines that are taken together represent concatenation of vectors, and separate lines represent content that is copied and then distributed to different locations. Because the falling behavior is highly related to the time sequence, the LSTM network module can not only integrate time and space dimensions to identify the falling behavior, but also solve the problems of gradient extinction and gradient explosion caused by gradual reduction in the gradient backward propagation process, and avoid the problems of gradient explosion and gradient extinction in the standard RNN, so that the method is better in use and faster in learning speed; in addition, the output of the front unit is used as the input of the rear unit in the same layer, and the output of the front layer is used as the input of the rear layer, so that the time relation and the context information before and after the video can be connected.
In this embodiment, in S6, the effective area of the human body is calculated by using an image of the minimum circumscribed rectangular image after binarization processing, and after binarization processing, the effective area of the human body is equal to the number of pixels whose pixel values are 1 in the minimum circumscribed rectangular frame image, and the total area of the rectangular frame is the total sum of pixels whose pixel values are 1 and 0, and the specific formula is as follows:
In this embodiment, the human body width-height ratio in S7 is the width-height ratio corresponding to the minimum circumscribed rectangular frame, and when the human body is walking or standing normally, the human body width-height ratio is much less than 1, and when the human body falls down or squats, the width-height ratio is much greater than 1; the second body aspect characteristic used for falling detection is that the aspect ratio corresponding to the minimum external rectangular frame is far less than 1 when a human body normally walks or stands, and when the human body falls down or squats, the aspect ratio is far greater than that which is the most important characteristic module for posture identification.
In this embodiment, in S8, the distance from the center of mass to the bottom surface of the human body, that is, the intersection O (X0, Y0) of the diagonal line of the minimum circumscribed rectangle, can well reflect the motion state of the human body at this time, and coordinates a (X1, Y1) and B (X2, Y2) of two end points of the bottom surface of the circumscribed rectangle are defined, then a is Y1-Y2, B is X1-X2, C is X1Y2-Y1X2, and the distance from the center O to the bottom surface ab is h, where the following formula:
in this embodiment, before and after the fall action occurs in S9, the height of the human body is constantly changed, but the most easily neglected point is that the height change is still very obvious in many non-fall actions such as squat, stooping, lying down, creeping and the like, so that the feature of the height change rate is introduced, and the height change rate is measured by using the ratio of the real-time height to the average height of the human body in the video. height represents the real-time height of the human body, height average represents the average height of the human body, and height average is defined as follows:
because the position of the camera has non-fixity and the size of the human body at different positions of the camera can be changed, the invention introduces an image invariance characteristic-Hu moment characteristic. The low-order moment value of the Hu moment has definite geometric properties, wherein the zero-order moment m0 can represent the total area of a target, the first-order moment values m10 and m01 can represent the size of a circumscribed outline of the target, and the second-order moments m20, m11 and m02 are also called moments of inertia and represent the size and the direction of an image. The Hu moment translation invariance is realized by a mode of a centroid coordinate. Here, the first four Hu moments of the falling behaviors (M1-M4) are obviously different from the first four Hu moments of the squat, bending and sitting behaviors, so that the falling behaviors can be used as one of the characteristics for representing the four behaviors.
The foregoing shows and describes the general principles, essential features, and advantages of the invention. It will be understood by those skilled in the art that the present invention is not limited to the embodiments described above, which are merely illustrative of the principles of the invention, but that various changes and modifications may be made without departing from the spirit and scope of the invention, which fall within the scope of the invention as claimed. The scope of the invention is defined by the appended claims and equivalents thereof.
Claims (10)
1. A falling behavior identification method based on a posture comprehensive characteristic is characterized in that: the method comprises the following steps: video data acquisition → extraction of a human body skeleton image → preprocessing of skeleton image data → construction of a binary image circumscribed rectangle → LSTM network unit processing → calculation of human body effective area → calculation of aspect ratio → calculation of distance from human body center of mass to ground → calculation of height change rate → occurrence of a fall, wherein:
s1, video data acquisition: the system is used for acquiring a video sequence from a video file or a camera and initializing the video sequence;
s2, extracting a human skeleton diagram: detecting a human body target according to the key frame image provided by the video sequence read in the S1, and extracting human body skeleton information for subsequent steps;
s3, skeleton image data preprocessing: the system is used for eliminating the extracted irrelevant information in the human skeleton module, enhancing the detectability of the relevant information and simplifying the data to the maximum extent;
s4, constructing a binary image circumscribed rectangle: the human body skeleton point information processing device is used for converting the preprocessed human body skeleton point information into a binary image;
s5, LSTM network unit processing: classification training for images;
s6, calculating the effective area of the human body: a first feature for fall detection;
s7, calculating the aspect ratio: a second morphological feature for fall detection;
s8, calculating the distance between the center of mass of the human body and the ground: a third morphological feature for fall detection;
s9, calculating the height change rate: a fourth morphological feature for fall detection;
s10, occurrence of a fall: the actions of the old are analyzed and processed through S1-S5, and whether the fall behavior occurs is judged through S6-S9.
2. A fall behavior recognition method based on posture-integrated features as claimed in claim 1, wherein: and S1, the collected video data is used as a sample input through the household video monitoring video, and the time monitoring video stream is intercepted by using a preset time interval by using a key frame intercepting technology to obtain a video frame which can be sent to the deep learning model for processing.
3. A fall behavior recognition method based on posture-integrated features as claimed in claim 1, wherein: in S2, human skeletons are extracted by using openpos, a cuff model of the openpos is called by using a dnn library of opencv, and then output is post-processed, wherein the openpos adopts an 18-joint output mode, and S2 comprises the following steps: (1) the network structure analysis (2) extracts all key points (3) to distinguish the key points, and the relation among the key points is calculated according to an affinity field, wherein the calculation formula of the affinity field is as follows:
4. A fall behavior recognition method based on posture-integrated features as claimed in claim 1, wherein: s3 adopts a normalization method.
5. A fall behavior recognition method based on posture-integrated features as claimed in claim 1, wherein: the method of S4 is to convert the human skeleton point information into a binary image, then use the minimum bounding rectangle frame to find out the processed human skeleton binary image, and use the OpenCV function to find out the bounding rectangle, and the main steps are as follows: (1) the mathematical basis is that a point (x1, y1) on the plane is set, and a point (x2, y2) which rotates anticlockwise by theta angle around (x0, y0) is set as
x2＝(x1-x0)cosθ-(y1-y0)sinθ+x0
y2＝(x1-x0)sinθ-(y1-y0)cosθ+y0(ii) a (2) Rotating the original polygon, cycleThe distance between the rings and the ring is 90-90 degrees, the precision and the operation efficiency can be balanced, the simple external rectangle of the polygon rotated by each degree is obtained, and the area, the vertex coordinates and the rotating degree of the simple external rectangle are recorded; (3) comparing all simple circumscribed rectangles obtained by the polygon in the rotating process to obtain a simple circumscribed rectangle with the smallest area, and obtaining the vertex coordinates and the rotating angle of the simple circumscribed rectangle; (4) and (4) rotating the circumscribed rectangle, and rotating the opposite direction of the simple circumscribed rectangle with the smallest area obtained in the previous step by the same angle opposite to the direction in the step 3 to obtain the smallest circumscribed rectangle.
6. A fall behavior recognition method based on posture-integrated features as claimed in claim 1, wherein: and S5, the output at the time t is fused into the output at the time t +1, so that the falling behavior can be identified by integrating time and space dimensions, the problems of gradient loss and gradient explosion caused by gradual reduction in the gradient back propagation process can be solved, and the LSTM provides a reference for setting a threshold value through learning.
7. A fall behavior recognition method based on posture-integrated features as claimed in claim 1, wherein: in the S6, the effective area of the human body is calculated by using the image of the minimum circumscribed rectangular image after binarization processing, and after binarization processing, the effective area of the human body is equal to the number of pixels with a pixel value of 1 in the minimum circumscribed rectangular frame image, and the total area of the rectangular frames is the sum of pixels with pixel values of 1 and 0, and the specific formula is as follows:
8. a fall behavior recognition method based on posture-integrated features as claimed in claim 1, wherein: the human body width-height ratio in S7 is the width-height ratio corresponding to the minimum circumscribed rectangular frame, and when the human body normally walks or stands, the human body width-height ratio is far less than 1, and when the human body falls down or squats, the width-height ratio is far greater than 1.
9. A fall behavior recognition method based on posture-integrated features as claimed in claim 7, wherein: in S8, the centroid of the human body is a diagonal intersection O (X0, Y0) of the minimum circumscribed rectangle, the distance from the centroid to the bottom surface can well reflect the motion state of the human body at that time, coordinates a (X1, Y1) and B (X2, Y2) of two end points of the bottom surface of the circumscribed rectangle are defined, then a is Y1-Y2, B is X1-X2, C is X1Y2-Y1X2, and the distance from the centroid O to the bottom surface ab is h, and the formula is as follows:
10. a fall behavior recognition method based on posture-integrated features as claimed in claim 1, wherein: the characteristic of height change rate is introduced into S9, the ratio of real-time height to human body average height of human body in video is used for measurement, height represents the real-time height of human body, height average represents the human body average height, and height average is defined as follows:
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111243199.7A CN113963442A (en) | 2021-10-25 | 2021-10-25 | Fall-down behavior identification method based on comprehensive body state features |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111243199.7A CN113963442A (en) | 2021-10-25 | 2021-10-25 | Fall-down behavior identification method based on comprehensive body state features |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113963442A true CN113963442A (en) | 2022-01-21 |
Family
ID=79466894
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111243199.7A Pending CN113963442A (en) | 2021-10-25 | 2021-10-25 | Fall-down behavior identification method based on comprehensive body state features |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113963442A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114463788A (en) * | 2022-04-12 | 2022-05-10 | 深圳市爱深盈通信息技术有限公司 | Fall detection method, system, computer equipment and storage medium |
| CN114973403A (en) * | 2022-05-06 | 2022-08-30 | 广州紫为云科技有限公司 | Efficient behavior prediction method based on space-time dual-dimension feature depth network |
| CN115376237A (en) * | 2022-07-28 | 2022-11-22 | 北京航天名翰网络科技有限公司 | Non-contact self-service registration system |
| CN117017276A (en) * | 2023-10-08 | 2023-11-10 | 中国科学技术大学 | Real-time human body tight boundary detection method based on millimeter wave radar |
| CN117132949A (en) * | 2023-10-27 | 2023-11-28 | 长春理工大学 | All-weather fall detection method based on deep learning |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107729876A (en) * | 2017-11-09 | 2018-02-23 | 重庆医科大学 | Fall detection method in old man room based on computer vision |
| CN109820515A (en) * | 2019-03-01 | 2019-05-31 | 中南大学 | The method of more sensing fall detections on TensorFlow platform based on LSTM neural network |
| CN110490171A (en) * | 2019-08-26 | 2019-11-22 | 睿云联(厦门)网络通讯技术有限公司 | A kind of danger gesture recognition method, device, computer equipment and storage medium |
| CN110555394A (en) * | 2019-08-19 | 2019-12-10 | 西安理工大学 | Fall risk assessment method based on human body shape characteristics |
| CN111161320A (en) * | 2019-12-30 | 2020-05-15 | 浙江大华技术股份有限公司 | Target tracking method, target tracking device and computer readable medium |
| CN111274954A (en) * | 2020-01-20 | 2020-06-12 | 河北工业大学 | Embedded platform real-time falling detection method based on improved attitude estimation algorithm |
| CN113469116A (en) * | 2021-07-20 | 2021-10-01 | 河南科技大学 | Face expression recognition method combining LBP (local binary pattern) features and lightweight neural network |
-
2021
- 2021-10-25 CN CN202111243199.7A patent/CN113963442A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107729876A (en) * | 2017-11-09 | 2018-02-23 | 重庆医科大学 | Fall detection method in old man room based on computer vision |
| CN109820515A (en) * | 2019-03-01 | 2019-05-31 | 中南大学 | The method of more sensing fall detections on TensorFlow platform based on LSTM neural network |
| CN110555394A (en) * | 2019-08-19 | 2019-12-10 | 西安理工大学 | Fall risk assessment method based on human body shape characteristics |
| CN110490171A (en) * | 2019-08-26 | 2019-11-22 | 睿云联(厦门)网络通讯技术有限公司 | A kind of danger gesture recognition method, device, computer equipment and storage medium |
| CN111161320A (en) * | 2019-12-30 | 2020-05-15 | 浙江大华技术股份有限公司 | Target tracking method, target tracking device and computer readable medium |
| CN111274954A (en) * | 2020-01-20 | 2020-06-12 | 河北工业大学 | Embedded platform real-time falling detection method based on improved attitude estimation algorithm |
| CN113469116A (en) * | 2021-07-20 | 2021-10-01 | 河南科技大学 | Face expression recognition method combining LBP (local binary pattern) features and lightweight neural network |
Non-Patent Citations (6)
| Title |
|---|
| ZHE CAO 等: "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields", COMPUTER VISION AND PATTERN RECOGNITION, 24 November 2016 (2016-11-24), pages 10 * |
| 小灰灰超: "最小外接矩形", Retrieved from the Internet <URL:https://zhuanlan.zhihu.com/p/97855964> * |
| 尹志成 等: "基于OpenPose的摔倒行为检测技术研究", 长春理工大学学报（自然科学版）, vol. 44, no. 3, 15 June 2021 (2021-06-15), pages 1 - 2 * |
| 彭帅;黄宏博;陈伟骏;胡志颖;袁铭阳;: "基于卷积神经网络的人体姿态估计算法综述", 北京信息科技大学学报(自然科学版), no. 03, 15 June 2020 (2020-06-15), pages 62 - 69 * |
| 彭玉青;高晴晴;刘楠楠;宋初柏;张媛媛;: "基于多特征融合的跌倒行为识别与研究", 数据采集与处理, no. 05, 15 September 2016 (2016-09-15), pages 50 - 62 * |
| 高苗 等: "基于多特征分析的摔倒检测算法设计", 上海师范大学学报（自然科学版）, vol. 47, no. 2, 15 April 2018 (2018-04-15), pages 1 - 2 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114463788A (en) * | 2022-04-12 | 2022-05-10 | 深圳市爱深盈通信息技术有限公司 | Fall detection method, system, computer equipment and storage medium |
| CN114973403A (en) * | 2022-05-06 | 2022-08-30 | 广州紫为云科技有限公司 | Efficient behavior prediction method based on space-time dual-dimension feature depth network |
| CN114973403B (en) * | 2022-05-06 | 2023-11-03 | 广州紫为云科技有限公司 | Behavior prediction method based on space-time double-dimension feature depth network |
| CN115376237A (en) * | 2022-07-28 | 2022-11-22 | 北京航天名翰网络科技有限公司 | Non-contact self-service registration system |
| CN115376237B (en) * | 2022-07-28 | 2024-02-20 | 北京航天名翰网络科技有限公司 | Non-contact self-service reporting system |
| CN117017276A (en) * | 2023-10-08 | 2023-11-10 | 中国科学技术大学 | Real-time human body tight boundary detection method based on millimeter wave radar |
| CN117017276B (en) * | 2023-10-08 | 2024-01-12 | 中国科学技术大学 | Real-time human body tight boundary detection method based on millimeter wave radar |
| CN117132949A (en) * | 2023-10-27 | 2023-11-28 | 长春理工大学 | All-weather fall detection method based on deep learning |
| CN117132949B (en) * | 2023-10-27 | 2024-02-09 | 长春理工大学 | All-weather fall detection method based on deep learning |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113963442A (en) | Fall-down behavior identification method based on comprehensive body state features | |
| Rougier et al. | Fall detection from human shape and motion history using video surveillance | |
| Feng et al. | Floor pressure imaging for fall detection with fiber-optic sensors | |
| Kwolek et al. | Improving fall detection by the use of depth sensor and accelerometer | |
| Bobick et al. | The recognition of human movement using temporal templates | |
| Derpanis et al. | Efficient action spotting based on a spacetime oriented structure representation | |
| CN111368810A (en) | Sit-up detection system and method based on human body and skeleton key point identification | |
| US9183431B2 (en) | Apparatus and method for providing activity recognition based application service | |
| Arivazhagan et al. | Human action recognition from RGB-D data using complete local binary pattern | |
| CN113239797B (en) | Human body action recognition method, device and system | |
| Chang et al. | The model-based human body motion analysis system | |
| Rani et al. | Kinematic joint descriptor and depth motion descriptor with convolutional neural networks for human action recognition | |
| CN113378649A (en) | Identity, position and action recognition method, system, electronic equipment and storage medium | |
| CN111783702A (en) | Efficient pedestrian tumble detection method based on image enhancement algorithm and human body key point positioning | |
| Cardile et al. | A vision-based system for elderly patients monitoring | |
| Yan et al. | Human-object interaction recognition using multitask neural network | |
| CN109784230A (en) | A kind of facial video image quality optimization method, system and equipment | |
| Zhang et al. | Visual surveillance for human fall detection in healthcare IoT | |
| Geng | Research on athlete’s action recognition based on acceleration sensor and deep learning | |
| CN114724251A (en) | Old people behavior identification method based on skeleton sequence under infrared video | |
| Nouredanesh et al. | Chasing feet in the wild: a proposed egocentric motion-aware gait assessment tool | |
| Liu et al. | Key algorithm for human motion recognition in virtual reality video sequences based on hidden markov model | |
| Zhang | Applications of Google MediaPipe Pose Estimation Using a Single Camera | |
| Shu et al. | The research and implementation of human posture recognition algorithm via OpenPose | |
| Li et al. | Application of thermal infrared imagery in human action recognition |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |