CN114067358B - Human body posture recognition method and system based on key point detection technology - Google Patents
Human body posture recognition method and system based on key point detection technology Download PDFInfo
- Publication number
- CN114067358B CN114067358B CN202111287237.9A CN202111287237A CN114067358B CN 114067358 B CN114067358 B CN 114067358B CN 202111287237 A CN202111287237 A CN 202111287237A CN 114067358 B CN114067358 B CN 114067358B
- Authority
- CN
- China
- Prior art keywords
- human body
- key points
- key point
- gesture
- relative position
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 77
- 238000001514 detection method Methods 0.000 title claims abstract description 57
- 238000005516 engineering process Methods 0.000 title claims abstract description 51
- 238000009825 accumulation Methods 0.000 claims abstract description 16
- 210000000988 bone and bone Anatomy 0.000 claims abstract description 6
- 238000007781 pre-processing Methods 0.000 claims abstract description 6
- 230000006399 behavior Effects 0.000 claims description 38
- 238000004422 calculation algorithm Methods 0.000 claims description 34
- 230000036544 posture Effects 0.000 claims description 34
- 230000005021 gait Effects 0.000 claims description 20
- 238000013528 artificial neural network Methods 0.000 claims description 19
- 238000012545 processing Methods 0.000 claims description 17
- 238000012549 training Methods 0.000 claims description 17
- 230000008569 process Effects 0.000 claims description 15
- 239000013598 vector Substances 0.000 claims description 7
- 238000013527 convolutional neural network Methods 0.000 claims description 6
- 230000001186 cumulative effect Effects 0.000 claims description 6
- 238000012795 verification Methods 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 4
- 238000005070 sampling Methods 0.000 claims description 4
- 238000010276 construction Methods 0.000 claims description 3
- 238000009792 diffusion process Methods 0.000 claims description 3
- 238000003062 neural network model Methods 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 2
- 210000003414 extremity Anatomy 0.000 description 46
- 238000012544 monitoring process Methods 0.000 description 6
- 210000000689 upper leg Anatomy 0.000 description 6
- 210000003423 ankle Anatomy 0.000 description 5
- 210000003127 knee Anatomy 0.000 description 5
- 244000309466 calf Species 0.000 description 4
- 230000009471 action Effects 0.000 description 2
- 230000004913 activation Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000033001 locomotion Effects 0.000 description 2
- 230000000877 morphologic effect Effects 0.000 description 2
- 238000012706 support-vector machine Methods 0.000 description 2
- 210000000707 wrist Anatomy 0.000 description 2
- 230000003542 behavioural effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 210000000629 knee joint Anatomy 0.000 description 1
- 210000003141 lower extremity Anatomy 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012806 monitoring device Methods 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012552 review Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computational Linguistics (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a human body posture recognition method and a system based on a key point detection technology, comprising the following steps: (1) Acquiring continuous frame images through video data and preprocessing; (2) Detecting key points of human bones of the acquired image; (3) Setting a preset value, and determining whether to convert coordinates or not by comparing the number of the detected key points of the human body with the preset value to perform gesture recognition; (4) Acquiring the relative position characteristics of limbs mapped by the key points based on the three-dimensional coordinates of the key points; (5) Calculating the confidence coefficient accumulation weight of the key points, comparing the accumulation weight with a preset threshold value, and judging whether the information of the key points of the human body representing the gesture is enough; if the relative position relation of limbs is enough, the gesture recognition is carried out by adopting a relative position relation judging method. The invention can recognize the gesture according to the content of the video image.
Description
Technical Field
The invention relates to the technical field of image recognition, in particular to a human body posture recognition method and system based on a key point detection technology.
Background
Human body gesture detection is one of the most challenging directions in the field of computer vision, and is widely applied to the fields of security monitoring of key areas, behavior detection of public places, behavior supervision of workplaces, behavior detection of drivers, security monitoring of solitary old people, security management of public security judicial departments and the like. At present, most of human body behaviors are detected in the forms of field watch-dog and video picture monitoring of supervisory personnel, and the traditional behavior monitoring mode has the following problems due to the large number of monitoring devices, various data types and increasingly complex monitoring objects: on one hand, the relevant supervisory personnel need to monitor too much video picture information, the supervisory energy is insufficient, and the working efficiency is low; on the other hand, the judgment and supervision of people have a certain subjective factor, and are difficult to review and realize standardization in a later period. In the prior art, in the process of carrying out gesture recognition detection by utilizing a key point technology, the video content is often directly recognized, whether the number of the identifiable key points or the characteristic information expressed by the key points is enough or not is not considered, so that the gesture recognition result is inaccurate, even gesture recognition cannot be carried out, and the detection efficiency is low.
Disclosure of Invention
The invention aims to: the invention aims to provide a human body gesture recognition method and system based on a key point detection technology, which can automatically select a gesture recognition algorithm according to the content of a video image and improve the efficiency of human body gesture recognition.
The technical scheme is as follows: the invention discloses a human body posture identification method based on a key point detection technology, which comprises the following steps of:
(1) Acquiring continuous frame images through video data and preprocessing;
(2) Detecting key points of human bones of the acquired image, acquiring two-dimensional image coordinates of each key point and associating the key points with a human body;
(3) Setting a preset value, judging whether the number of detected key points of the human body exceeds the preset value, and if not, not carrying out gesture recognition; if the two-dimensional coordinates of the key points exceed the preset value, converting the acquired two-dimensional image coordinates of the key points into three-dimensional coordinates;
(4) Acquiring the relative position characteristics of limbs mapped by the key points based on the three-dimensional coordinates of the key points, wherein the relative position characteristics comprise the characteristic limb distance and limb included angle among the key points and the distance difference of the key points in the directions of x, y and z axes;
(5) Calculating the confidence coefficient accumulation weight of the key points, comparing the accumulation weight with a preset threshold value, and judging whether the information of the key points of the human body representing the gesture is enough; if the cumulative weight is greater than a preset threshold, the human body key point information representing the gesture is enough, gesture recognition is carried out by using a relative position relation judgment method of limbs, and a detection result is output; if the accumulated weight is smaller than a preset threshold, indicating that the information of the key points of the human body representing the gesture is insufficient, and executing the step (6);
The method for judging the relative position relationship of limbs comprises the following steps:
(51) Acquiring the relative position characteristics of limbs under each posture through various preset human body postures, and taking the range of the relative position characteristics of the limbs acquired by different postures as a set threshold value;
(52) Comparing the relative position characteristics of the limbs obtained in the step (4) with a set threshold value, and if the relative position characteristics are within the set threshold value range, correspondingly obtaining a preset gesture; otherwise, not the gesture;
the beneficial effects are that: compared with the prior art, the invention has the advantages that: and judging whether the quantity of all the acquired human body key points representing the action is enough or not through the setting of the preset value and the key point confidence coefficient accumulation weight, extracting the relative position features between limbs to identify the gesture on the basis of enough acquired key point information, comparing the relative position features with the relative position features of the preset gesture, and finally identifying the gesture.
Further, the method further comprises the step (6) of carrying out gesture recognition by adopting a neural network gesture recognition technology and outputting a detection result if the accumulated weight is smaller than a preset threshold value and the information of key points of the human body representing the gesture is insufficient; the neural network gesture recognition technology described in the step (6) specifically includes the following steps:
(61) Extracting the area, perimeter, aspect ratio and eccentricity characteristics of the human body target outline by adopting an image processing technology;
(62) Combining and normalizing the target contour features and the relative position features of the limb mapped by the key points to form feature vectors, and forming a training pattern library;
(63) And building a neural network gesture classifier to recognize the gesture through training of the neural network model.
The detection condition of the key point information is judged by setting the comparison of the accumulated weight and the preset threshold value, and the human body gesture is identified by adopting which algorithm through the autonomous switching of the detection condition, so that the detection efficiency is improved. Aiming at the condition that the shooting angle is poor, namely, the individual key points cannot be detected, the outline edge features of the target are extracted through an image processing technology, and normalized after being combined with the relative position features of the key points, the gesture recognition is realized by adopting a neural network, so that the recognition accuracy is improved.
Further, the method further comprises the steps of: (7) Inputting the detection result of gesture recognition into a non-compliance behavior judging module; if the person is determined to be non-compliant, acquiring gait information of the target in the image by combining a target tracking algorithm, and verifying the identity of the person with the non-compliant behavior in the image by adopting a gait recognition technology; the input of the non-compliance behavior judging module is a gesture, the output is a result of whether compliance or not, and the user sets partial gestures as non-compliance through setting parameters. The identity information verification of the non-compliance person is realized through the gait recognition algorithm, and the problem that the identity is difficult to verify through face recognition due to the shooting angle or the shooting distance is solved.
Further, the method further comprises the steps of: (8) Judging the position of the person who does not act properly by adopting a stereoscopic vision technology algorithm and an image processing technology; and (3) tracking the position information of the non-compliance behavior personnel in real time by combining a target tracking algorithm, counting the duration time of the same person and the same behavior, and uploading the result to a background management system at regular time.
Further, in the step (5), the calculated confidence coefficient accumulation weight of the key point is as follows:
Where E is the keypoint confidence, e= (E 1,E2,…,EJ), Q is the keypoint weight, q= (Q 1,Q2,…,QJ), J represents the number of keypoints to be detected.
Further, the predetermined value in step (3) is 4.
Further, the key point detection of the human skeleton in the step (2) adopts a key point detection algorithm from bottom to top; the method specifically comprises the following steps:
(2.1) construction of a double-branched convolutional neural network
Inputting the preprocessed picture into a double-branch depth convolutional neural network VGG-19 comprising 16 convolutional layers and 3 full-connection layers, wherein the first 10 layers of the neural network are used for creating feature mapping for an input image to obtain a feature F; f is respectively input into two branches, wherein the first branch is used for predicting a key point confidence map S, and the second branch is used for predicting a key point affinity field L; wherein s= (s 1,s2,…,SJ), J represents the number of key points to be detected; l= (L 1,L2,…,LC), C denotes the logarithm of the joint to be detected; the inputs to the network for each stage are:
Where S t,Lt represents the result of the t-th round of training, ρ t represents the t-th round of confidence training process, Representing a t-th round of affinity training process;
(2.2) keypoint confidence map prediction
Confidence map of key pointThe method comprises the following steps:
In the method, in the process of the invention, A confidence that the jth keypoint of the kth person is present at the p-pixel; x j,k represents the jth keypoint of the kth person; sigma is used to control the degree of diffusion of the gaussian distribution; setting a key point confidence coefficient threshold value, and if the key point confidence coefficient exceeds the threshold value, reserving the key point; the confidence of the whole human body is the maximum value in the confidence of all the components of the human body, and the following formula is adopted:
(2.3) Critical Point affinity field prediction
The key point affinity field is:
In the method, in the process of the invention, Indicating whether a certain pixel point p exists on the joint of the kth person connected in pairs; a unit vector indicating that position j 1 points to position j 2, The real coordinates of j 1、j2 respectively; if the following conditions are met, judging that the pixel point p is on a limb formed by connecting the joint pair;
Wherein, l c,k is the length of the limb formed by the connection of the c joint pair of the kth person, and sigma l is the width of the limb;
(2.4) Critical Point clustering
Performing bipartite graph matching by using a maximum-side-weight Hungary algorithm to obtain an optimal multi-user key point connection result, and corresponding each key point to different users;
The goal of the hungarian algorithm is to find the edge weights and the largest combinations in the set of C joints connected pairwise, as follows:
Wherein E mn is the edge weight of the m-th and n-th key point types, D J is the set of J types of key points, The method is used for judging whether the two key points are connected or not; for any two key pointsAndThe correlation of the keypoint pair is characterized by calculating the integral of the affinity field as follows:
In the method, in the process of the invention, For predicted coordinates of the key point j 1,j2, C is a limb formed by connecting the key point j 1,j2, p (u) is sampling on the key point, and L c (p (u)) is a PAF predicted value of the limb C at the p (u) point.
In addition, the invention also provides a human body gesture recognition system based on the key point detection technology, which comprises the following steps:
The image and processing module is used for acquiring continuous frame images through video data and preprocessing the continuous frame images;
The key point detection module is used for detecting key points of the bones of the human body on the acquired images, acquiring two-dimensional image coordinates of each key point and associating the key points with the human body; the method comprises the steps of setting a preset value, judging whether the number of detected key points of a human body exceeds the preset value, and if the number of the detected key points of the human body does not exceed the preset value, not carrying out gesture recognition; if the two-dimensional coordinates of the key points exceed the preset value, converting the acquired two-dimensional image coordinates of the key points into three-dimensional coordinates;
the gesture recognition module is used for acquiring the relative position characteristics of the limb mapped by the key points based on the three-dimensional coordinates of the key points, wherein the relative position characteristics comprise the limb distance and the limb included angle represented among the key points and the distance difference of the key points in the directions of x, y and z axes; calculating the confidence coefficient accumulation weight of the key points, comparing the accumulation weight with a preset threshold value, and judging whether the information of the key points of the human body representing the gesture is enough; if the cumulative weight is greater than a preset threshold, the human body key point information representing the gesture is enough, gesture recognition is carried out by using a relative position relation judgment method of limbs, and a detection result is output; if the accumulated weight is smaller than a preset threshold, representing that the human body key point information representing the gesture is insufficient, performing gesture recognition by adopting a neural network gesture recognition technology, and outputting a detection result; the method for judging the relative position relationship of the limbs comprises the following steps: acquiring the relative position characteristics of limbs under each posture through various preset human body postures, and taking the range of the relative position characteristics of the limbs acquired by different postures as a set threshold value; comparing the acquired relative position characteristics of the limbs with a set threshold value, and if the relative position characteristics are within the set threshold value range, correspondingly setting the relative position characteristics as a preset gesture; otherwise, not the gesture;
The identity verification module is used for inputting the detection result of gesture recognition into the non-compliance behavior judgment module; if the person is determined to be non-compliant, acquiring gait information of the target in the image by combining a target tracking algorithm, and verifying the identity of the person with the non-compliant behavior in the image by adopting a gait recognition technology; the input of the non-compliance behavior judging module is a gesture, the output is a result of whether compliance or not, and a user sets partial gestures as non-compliance through setting parameters;
The position tracking module is used for judging the position of the person who does not act in compliance by adopting a stereoscopic vision technology algorithm and an image processing technology; and (3) tracking the position information of the non-compliance behavior personnel in real time by combining a target tracking algorithm, counting the duration time of the same person and the same behavior, and uploading the result to a background management system at regular time.
The invention also provides a computer readable storage medium storing a computer program which when executed by a processor implements the above method steps.
The invention also provides a human body gesture recognition debugging device, a memory, a processor and a program stored and operable on the memory, wherein the program realizes the steps of the method when being executed by the processor.
Drawings
FIG. 1 is a flow chart of the method of the present invention;
FIG. 2 is a graph showing the distribution of key points of a human body in the method of the present invention;
FIG. 3 is a flow chart of determining whether the content of the key point information is sufficient in the method of the present invention;
FIG. 4 is a flow chart of gesture recognition based on the relative position relationship of limbs in the method of the present invention;
FIG. 5 is a schematic front view of a squat gesture;
FIG. 6 is a side schematic view of a squat gesture;
FIG. 7 is a flowchart of a neural network gesture recognition algorithm in the method of the present invention;
FIG. 8 is a flowchart of a person authentication algorithm performed by the gait recognition technology in the method of the present invention;
FIG. 9 is a stereotactic model;
FIG. 10 is a flow chart of a three-dimensional coordinate calculation of a target;
Fig. 11 is a flow chart of the same person's same behavior detection.
Detailed Description
The technical scheme of the invention is further described below with reference to the accompanying drawings.
As shown in fig. 1, the human body posture recognition method based on the key point detection technology of the invention comprises the following steps:
Logging in a stereoscopic vision camera through software, obtaining a video stream in a channel, extracting frames, decoding, and performing format conversion to finally obtain continuous frame images in RGB format; and carrying out image enhancement and denoising on continuous frame images, and improving the image quality. Considering that in an actual application scene, an image acquired by a camera contains more non-target objects, and in order to improve detection efficiency and accuracy, the image is cut according to a monitoring range and scaled to a specified size;
Step two, detecting human skeleton key points of the obtained image by adopting a bottom-up human key point detection algorithm, obtaining two-dimensional image coordinates of each key point and associating the key points with a human body; the human skeleton key point detection algorithm comprises the following steps:
(1) Construction of a double-branch convolutional neural network
The preprocessed picture is input into a double-branch depth convolutional neural network VGG-19 comprising 16 convolutional layers and 3 full-connection layers, the convolutional kernel size is 3 multiplied by 3, the activation function adopts a ReLU, and the training adopts a Dropout mechanism. The first 10 layers of the neural network are used to create a feature map for the input image, resulting in feature F. F is respectively input into two branches, wherein the first branch is used for predicting a key point confidence map S, and the second branch is used for predicting a key point affinity field L. Wherein s= (S 1,S2,…,SJ), J represents the number of key points to be detected; l= (L 1,L2,…,LC), C denotes the joint logarithm to be detected. The inputs to the network for each stage are:
Where S t,Lt represents the result of the t-th round of training, ρ t represents the t-th round of confidence training process, Indicating the t-th round of affinity training process.
(2) Key point confidence map prediction
Confidence map of key pointThe j-th keypoint representing the kth person is present at the confidence level of p pixels. The calculation method comprises the following steps:
Where x j,k represents the jth key point of the kth person and σ is used to control the degree of diffusion of the gaussian distribution. And setting the confidence coefficient threshold value of the key point to be 0.5, and if the confidence coefficient of the key point exceeds the threshold value, reserving the key point.
The confidence of the whole human body is the maximum value in the confidence of all the components of the human body, and the following formula is adopted:
(3) Keypoint affinity field prediction
Using critical affinity fieldsThe following are the cases where a pixel point p exists on the joint of the kth person, which is connected two by two:
In the method, in the process of the invention, A unit vector representing position j 1 pointing to position j 2; The real coordinates of j 1、j2 respectively;
if the following conditions are met, judging that the pixel point p is on a limb formed by connecting the joint pair;
Where l c,k is the length of the limb formed by the connection of the c-th joint pair of the kth person, and σ l is the width of the limb.
(4) Key point clustering
And carrying out bipartite graph matching by using a maximum-side-weight Hungary algorithm to obtain an optimal multi-user key point connection result, and corresponding each key point with different users.
The goal of the hungarian algorithm is to find the edge weights and the largest combinations in the set of C joints connected pairwise, as follows:
Wherein E mn is the edge weight of the m-th and n-th key point types, D J is the set of J types of key points, For determining whether two keypoints are connected.
For any two key pointsAndThe correlation of the keypoint pair is characterized by calculating the integral of the affinity field as follows:
In the method, in the process of the invention, For predicted coordinates of the key point j 1,j2, C is a limb formed by connecting the key point j 1,j2, p (u) is sampling on the key point, and L c (p (u)) is a PAF predicted value of the limb C at the p (u) point.
To improve the efficiency of computation, the correlation between two keypoints is approximated by sampling 10 pixels at equal intervals for integration and re-summation. Through the above steps, as shown in fig. 2, 18 human key point two-dimensional coordinate information of a nose, a neck, a left shoulder, a left elbow, a left wrist, a right shoulder, a right elbow, a right wrist, a left crotch, a left knee, a left ankle, a right crotch, a right knee, a right ankle, a left eye, a right eye, a left ear, a right ear, etc. of a single person or a plurality of persons in an image can be obtained.
Step three, setting a preset value as 4, judging whether the number of detected key points of the human body exceeds 4, and if the number of the detected key points does not exceed 4, not carrying out gesture recognition; if the number of the detected key points exceeds 4, converting the acquired two-dimensional image coordinates of each key point into three-dimensional coordinates;
Acquiring a limb relative position characteristic value mapped by key points based on three-dimensional coordinates of the key points, wherein the relative position characteristic comprises a characteristic limb distance between the key points, a limb included angle and a distance difference between the key points in the directions of x, y and z;
For example, the left crotch coordinate is (x 1,y1,z1), the left knee coordinate is (x 2,y2,z2), the left ankle coordinate is (x 3,y3,z3), the left thigh limb distance is II (x 1,y1,z1)-(x2,y2,z2)‖2, the left calf limb distance is II (x 3,y3,z3)-(x2,y2,z2)‖2, the angle between the left thigh and the left calf is II The distance difference from the left crotch joint to the left knee joint in each of the x, y, and z directions is x 1-x2,y1-y2,z1-z2, respectively. In order to enhance the robustness and practicability of the system, the calculated distance and included angle are subjected to data normalization processing.
And fifthly, calculating the confidence coefficient accumulation weight of the key points to be used as an index for measuring whether the key point information is enough. According to the confidence coefficient of each joint point, the weight value of each key point set by combining with each preset gesture is shown in table 1, and the calculation accumulated weight formula is as follows:
Where E is the keypoint confidence, e= (E 1,E2,…,EJ), Q is the keypoint weight q= (Q 1,Q2,…,QJ), and J represents the number of keypoints to be detected.
TABLE 1 weight of key points for each gesture part
As shown in fig. 3, comparing the calculated accumulated weight with a preset threshold value, and judging whether the detected human body key point information representing the gesture is enough or not; if the cumulative weight is greater than a preset threshold, the human body key point information representing the gesture is enough, gesture recognition is carried out by using a relative position relation judgment method of limbs, and a detection result is output; if the accumulated weight is smaller than a preset threshold, representing that the human body key point information representing the gesture is insufficient, performing gesture recognition by adopting a neural network gesture recognition technology, and outputting a detection result; different poses have different preset thresholds.
As shown in fig. 4, the method for determining the relative positional relationship of the limbs includes the steps of:
(a) Acquiring the relative position characteristics of limbs under each posture through various preset human body postures, and taking the range of the relative position characteristic values of the limbs acquired by different postures as a set threshold value;
(b) Comparing the characteristic value of the relative position of the limb obtained in the step (4) with a set threshold value, and if the characteristic value is within the set threshold value range, correspondingly obtaining a preset gesture; otherwise, not the gesture.
As shown in fig. 5 to 6, for detection and recognition of squatting motions, according to coordinate information of key points such as a trunk, a left knee, a left ankle, a left waist, a right knee, a right ankle, a right waist, and the like, an included angle α between a left calf and a left thigh, an included angle δ between a right calf and a right thigh, an included angle β between the trunk and the left thigh, and an included angle γ between the trunk and the right thigh are calculated, and if α <100 ° n δ <100 ° n β <90 ° n γ <90 °, the squatting motions are determined.
As shown in fig. 7, the neural network gesture recognition technology specifically includes the following steps:
(a) The method is characterized in that the characteristics of the area, perimeter, aspect ratio, eccentricity and the like of the outline of the human body target are extracted by adopting image processing technologies such as Gaussian modeling, foreground detection, connected domain analysis, morphological processing and the like;
(b) Combining and normalizing the target contour features and the relative position features of the key points to form feature vectors, and forming a training pattern library;
(c) And building a neural network gesture classifier to recognize the gesture through training of the neural network model.
The gesture recognition method comprises the steps of recognizing gestures, namely extracting features such as area, perimeter, aspect ratio, eccentricity and the like of a human body target outline, combining and normalizing the target outline features and the relative position features of key points, recognizing the gestures by using a neural network gesture classifier, and outputting detection results;
The neural network is trained in advance and is based on a keras built seven-layer network structure, wherein the first three layers are relu activation layers, the middle three layers are BatchNormalization layers, and the last layer is a softmax output layer. softmax maps the output of multiple neurons into a (0, 1) interval, expressed as:
Wherein V is an array of various action result values output by the network, V g is a value of behavior g, g epsilon {0,1, …, K-1}, K is the number of behaviors, S g is the probability value of the g-th behavior, and e is the bottom of natural logarithm.
Step six, inputting the detection result of gesture recognition into an irregular behavior judging module; if the person is determined to be non-compliant, acquiring gait information of the target in the image by combining a target tracking algorithm, and verifying the identity of the person with the non-compliant behavior in the image by adopting a gait recognition technology; the program of the non-compliance judging module comprises a function, the input is a gesture, the output is a result of whether the gesture is compliance or not, and a user can set which gestures are non-compliance according to different scenes by setting parameters in the program.
As shown in fig. 8, the gait recognition technology has the following steps:
(a) Acquiring a gait outline drawing of a human body when walking at an asynchronous walking speed;
(b) Acquiring a lower limb angle change track of a gait cycle, taking the contour width, the perimeter, the area and the like as gait characteristics, constructing a support vector machine gait recognition model library, training a model, and finally obtaining a vector machine gait classifier;
(c) And (3) adopting a trained support vector machine classifier to realize identity verification for people with non-compliance behavior in the acquired image.
Step seven, judging the positions of the non-compliance behavioral staff by adopting a stereoscopic vision technology algorithm and an image processing technology; and (3) tracking the position information of the non-compliance behavior personnel in real time by combining a target tracking algorithm, counting the duration time of the same person and the same behavior, and uploading the result to a background management system at regular time.
As shown in fig. 9 to 10, the specific requirements of the position of the non-compliance person are obtained, through the image processing technologies such as gaussian modeling, foreground detection, connected domain analysis, morphological processing, template matching and the like, the centroid pixel coordinates of the human body target connected domain in the image shot by the left camera and the right camera are extracted, the physical length corresponding to the unit pixels is combined, the image center is taken as an origin vertical coordinate system, the key point pixel coordinates are converted into image coordinates, then a coordinate system is established by taking the midpoint of the optical center connecting line of the left camera and the right camera as the origin according to the parallax principle and the triangulation principle, and the coordinates of the key points in the three-dimensional space are calculated, so that the position information is obtained.
As shown in fig. 11, if the pose and position of the target are found in the nth frame image, determining the pose and position of the corresponding target in the (n+1) th frame through a target tracking algorithm, and judging whether the pose changes in real time, if so, saving and uploading the picture to a background management system to realize real-time tracking of the pose of the same person; if the position is not changed and the position is not changed, the time for keeping the gesture is judged to be compared with a set threshold value, and if the time exceeds the set threshold value, the background management system is uploaded, so that the real-time statistics of the same person and the same behavior is realized.
In addition, the invention also provides a human body gesture recognition system based on the key point detection technology, which comprises the following steps:
The image and processing module is used for acquiring continuous frame images through video data and preprocessing the continuous frame images;
The key point detection module is used for detecting key points of the bones of the human body on the acquired images, acquiring two-dimensional image coordinates of each key point and associating the key points with the human body; the method comprises the steps of setting a preset value, judging whether the number of detected key points of a human body exceeds the preset value, and if the number of the detected key points of the human body does not exceed the preset value, not carrying out gesture recognition; if the two-dimensional coordinates of the key points exceed the preset value, converting the acquired two-dimensional image coordinates of the key points into three-dimensional coordinates;
The gesture recognition module is used for acquiring the relative position characteristics of the limb mapped by the key points based on the three-dimensional coordinates of the key points, wherein the relative position characteristics comprise the limb distance and the limb included angle represented among the key points and the distance difference of the key points in the directions of x, y and z axes; calculating the confidence coefficient accumulation weight of the key points, comparing the accumulation weight with a preset threshold value, and judging whether the information of the key points of the human body representing the gesture is enough; if the cumulative weight is greater than a preset threshold, the human body key point information representing the gesture is enough, gesture recognition is carried out by using a relative position relation judgment method of limbs, and a detection result is output; if the accumulated weight is smaller than a preset threshold, representing that the human body key point information representing the gesture is insufficient, performing gesture recognition by adopting a neural network gesture recognition technology, and outputting a detection result; the method for judging the relative position relationship of the limbs comprises the following steps: acquiring the relative position characteristics of limbs under each posture through various preset human body postures, and taking the range of the relative position characteristic values of the limbs acquired by different postures as a set threshold value; comparing the acquired characteristic value of the relative position of the limb with a set threshold value, and if the characteristic value is within the set threshold value range, correspondingly setting the characteristic value as a preset gesture; otherwise, not the gesture;
The identity verification module is used for inputting the detection result of gesture recognition into the non-compliance behavior judgment module; if the person is determined to be non-compliant, acquiring gait information of the target in the image by combining a target tracking algorithm, and verifying the identity of the person with the non-compliant behavior in the image by adopting a gait recognition technology; the input of the non-compliance behavior judging module is a gesture, the output is a result of whether compliance or not, and a user sets partial gestures as non-compliance through setting parameters;
The position tracking module is used for judging the position of the person who does not act in compliance by adopting a stereoscopic vision technology algorithm and an image processing technology; and (3) tracking the position information of the non-compliance behavior personnel in real time by combining a target tracking algorithm, counting the duration time of the same person and the same behavior, and uploading the result to a background management system at regular time.
The invention also provides a computer readable storage medium storing a computer program which, when executed by a processor, performs the steps of the method of the invention.
The invention also provides a human body gesture recognition debugging device, a memory, a processor and a program stored and operable on the memory, wherein the program realizes the steps of the method when being executed by the processor.
Claims (10)
1. The human body posture recognition method based on the key point detection technology is characterized by comprising the following steps of:
(1) Acquiring continuous frame images through video data and preprocessing;
(2) Detecting key points of human bones of the acquired image, acquiring two-dimensional image coordinates of each key point and associating the key points with a human body;
(3) Setting a preset value, judging whether the number of detected key points of the human body exceeds the preset value, and if not, not carrying out gesture recognition; if the two-dimensional coordinates of the key points exceed the preset value, converting the acquired two-dimensional image coordinates of the key points into three-dimensional coordinates;
(4) Acquiring the relative position characteristics of limbs mapped by the key points based on the three-dimensional coordinates of the key points, wherein the relative position characteristics comprise limb distances and limb included angles represented among the key points and distance differences of the key points in the directions of x, y and z;
(5) Calculating the confidence coefficient accumulation weight of the key points, comparing the accumulation weight with a preset threshold value, and judging whether the information of the key points of the human body representing the gesture is enough; if the cumulative weight is greater than a preset threshold, the human body key point information representing the gesture is enough, gesture recognition is carried out by using a relative position relation judgment method of limbs, and a detection result is output; the method for judging the relative position relationship of limbs comprises the following steps:
(51) Acquiring the relative position characteristics of limbs under each posture through various preset human body postures, and taking the range of the relative position characteristics of the limbs acquired by different postures as a set threshold value;
(52) Comparing the relative position characteristics of the limbs obtained in the step (4) with a set threshold value, and if the relative position characteristics are within the set threshold value range, correspondingly obtaining a preset gesture; otherwise, not the gesture.
2. The human body posture recognition method based on the key point detection technology according to claim 1, further comprising the step of (6) if the accumulated weight is smaller than a preset threshold value, representing that the human body key point information representing the posture is insufficient, performing posture recognition by adopting a neural network posture recognition technology, and outputting a detection result; the neural network gesture recognition technology described in the step (6) specifically includes the following steps:
(61) Extracting the area, perimeter, aspect ratio and eccentricity characteristics of the human body target outline by adopting an image processing technology;
(62) Combining and normalizing the target contour features and the relative position features of the limb mapped by the key points to form feature vectors, and forming a training pattern library;
(63) And building a neural network gesture classifier to recognize the gesture through training of the neural network model.
3. The human body posture recognition method based on the key point detection technology according to claim 2, further comprising the steps of:
(7) Inputting the detection result of gesture recognition into a non-compliance behavior judging module; if the person is determined to be non-compliant, acquiring gait information of the target in the image by combining a target tracking algorithm, and verifying the identity of the person with the non-compliant behavior in the image by adopting a gait recognition technology.
4. The human body posture recognition method based on the key point detection technology of claim 3, further comprising the steps of:
(8) Judging the position of the person who does not act properly by adopting a stereoscopic vision technology algorithm and an image processing technology; and (3) tracking the position information of the non-compliance behavior personnel in real time by combining a target tracking algorithm, counting the duration time of the same person and the same behavior, and uploading the result to a background management system at regular time.
5. The human body posture recognition method based on the key point detection technology according to claim 1, wherein the calculation of the key point confidence coefficient accumulation weight in the step (5) is as follows:
Where E is the keypoint confidence, e= (E 1,E2,…,EJ), Q is the keypoint weight, q= (Q 1,Q2,…,QJ), J represents the number of keypoints to be detected.
6. The human body posture identifying method based on the key point detecting technique according to claim 1, characterized in that the predetermined value in step (3) is 4.
7. The human body posture recognition method based on the key point detection technology according to claim 1, wherein the human body skeleton key point detection in the step (2) adopts a key point detection algorithm from bottom to top; the method specifically comprises the following steps:
(2.1) construction of a double-branched convolutional neural network
Inputting the preprocessed picture into a double-branch depth convolutional neural network VGG-19 comprising 16 convolutional layers and 3 full-connection layers, wherein the first 10 layers of the neural network are used for creating feature mapping for an input image to obtain a feature F; f is respectively input into two branches, wherein the first branch is used for predicting a key point confidence map S, and the second branch is used for predicting a key point affinity field L; wherein s= (S 1,S2,…,SJ), J represents the number of key points to be detected; l= (L 1,L2,…,LC), C denotes the logarithm of the joint to be detected; the inputs to the network for each stage are:
Where S t,Lt represents the result of the t-th round of training, ρ t represents the t-th round of confidence training process, Representing a t-th round of affinity training process;
(2.2) keypoint confidence map prediction
Confidence map of key pointThe method comprises the following steps:
In the method, in the process of the invention, A confidence that the jth keypoint of the kth person is present at the p-pixel; x j,k represents the jth keypoint of the kth person; sigma is used to control the degree of diffusion of the gaussian distribution; setting a key point confidence coefficient threshold value, and if the key point confidence coefficient exceeds the threshold value, reserving the key point; the confidence of the whole human body is the maximum value in the confidence of all the components of the human body, and the following formula is adopted:
(2.3) Critical Point affinity field prediction
The key point affinity field is:
In the method, in the process of the invention, Indicating whether a certain pixel point p exists on the joint of the kth person connected in pairs; a unit vector indicating that position j 1 points to position j 2, The real coordinates of j 1、j2 respectively; if the following conditions are met, judging that the pixel point p is on a limb formed by connecting the joint pair;
Wherein, l c,k is the length of the limb formed by the connection of the c joint pair of the kth person, and sigma l is the width of the limb;
(2.4) Critical Point clustering
Performing bipartite graph matching by using a maximum-side-weight Hungary algorithm to obtain an optimal multi-user key point connection result, and corresponding each key point to different users;
The goal of the hungarian algorithm is to find the edge weights and the largest combinations in the set of C joints connected pairwise, as follows:
Wherein E mn is the edge weight of the m-th and n-th key point types, D J is the set of J types of key points, The method is used for judging whether the two key points are connected or not; for any two key pointsAndThe correlation of the keypoint pair is characterized by calculating the integral of the affinity field as follows:
In the method, in the process of the invention, For predicted coordinates of the key point j 1,j2, C is a limb formed by connecting the key point j 1,j2, p (u) is sampling on the key point, and L c (p (u)) is a PAF predicted value of the limb C at the p (u) point.
8. A human body posture recognition system based on a key point detection technology, comprising:
The image and processing module is used for acquiring continuous frame images through video data and preprocessing the continuous frame images;
The key point detection module is used for detecting key points of the bones of the human body on the acquired images, acquiring two-dimensional image coordinates of each key point and associating the key points with the human body; the method comprises the steps of setting a preset value, judging whether the number of detected key points of a human body exceeds the preset value, and if the number of the detected key points of the human body does not exceed the preset value, not carrying out gesture recognition; if the two-dimensional coordinates of the key points exceed the preset value, converting the acquired two-dimensional image coordinates of the key points into three-dimensional coordinates;
The gesture recognition module is used for acquiring the relative position characteristics of limbs mapped by the key points based on the three-dimensional coordinates of the key points, wherein the relative position characteristics comprise limb distances and limb included angles represented among the key points and the distance differences of the key points in the directions of x, y and z; calculating the confidence coefficient accumulation weight of the key points, comparing the accumulation weight with a preset threshold value, and judging whether the information of the key points of the human body representing the gesture is enough; if the cumulative weight is greater than a preset threshold, the human body key point information representing the gesture is enough, gesture recognition is carried out by using a relative position relation judgment method of limbs, and a detection result is output; if the accumulated weight is smaller than a preset threshold, representing that the human body key point information representing the gesture is insufficient, performing gesture recognition by adopting a neural network gesture recognition technology, and outputting a detection result; the method for judging the relative position relationship of the limbs comprises the following steps: acquiring the relative position characteristics of limbs under each posture through various preset human body postures, and taking the range of the relative position characteristics of the limbs acquired by different postures as a set threshold value; comparing the acquired relative position characteristics of the limbs with a set threshold value, and if the relative position characteristics are within the set threshold value range, correspondingly setting the relative position characteristics as a preset gesture; otherwise, not the gesture;
The identity verification module is used for inputting the detection result of gesture recognition into the non-compliance behavior judgment module; if the person is determined to be non-compliant, acquiring gait information of the target in the image by combining a target tracking algorithm, and verifying the identity of the person with the non-compliant behavior in the image by adopting a gait recognition technology;
The position tracking module is used for judging the position of the person who does not act in compliance by adopting a stereoscopic vision technology algorithm and an image processing technology; and (3) tracking the position information of the non-compliance behavior personnel in real time by combining a target tracking algorithm, counting the duration time of the same person and the same behavior, and uploading the result to a background management system at regular time.
9. A computer-readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the method steps of any one of claims 1 to 7.
10. A human body posture recognition debugging device, characterized by a memory, a processor and a program stored and executable on said memory, which program, when being executed by the processor, realizes the steps of the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111287237.9A CN114067358B (en) | 2021-11-02 | 2021-11-02 | Human body posture recognition method and system based on key point detection technology |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111287237.9A CN114067358B (en) | 2021-11-02 | 2021-11-02 | Human body posture recognition method and system based on key point detection technology |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114067358A CN114067358A (en) | 2022-02-18 |
| CN114067358B true CN114067358B (en) | 2024-08-13 |
Family
ID=80236415
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111287237.9A Active CN114067358B (en) | 2021-11-02 | 2021-11-02 | Human body posture recognition method and system based on key point detection technology |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114067358B (en) |
Families Citing this family (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114623400B (en) * | 2022-03-22 | 2023-10-20 | 广东卫明眼视光研究院 | Sitting posture identification desk lamp system and identification method based on remote intelligent monitoring |
| CN115105821A (en) * | 2022-07-04 | 2022-09-27 | 合肥工业大学 | Gymnastics training auxiliary system based on OpenPose |
| CN115331153B (en) * | 2022-10-12 | 2022-12-23 | 山东省第二人民医院(山东省耳鼻喉医院、山东省耳鼻喉研究所) | Posture monitoring method for assisting vestibule rehabilitation training |
| CN115497596B (en) * | 2022-11-18 | 2023-04-07 | 深圳聚邦云天科技有限公司 | Human body motion process posture correction method and system based on Internet of things |
| CN115540875B (en) * | 2022-11-24 | 2023-03-07 | 成都运达科技股份有限公司 | Method and system for high-precision detection and positioning of train vehicles in station track |
| CN116129524B (en) * | 2023-01-04 | 2024-06-14 | 长沙观谱红外科技有限公司 | Automatic gesture recognition system and method based on infrared image |
| CN116206369B (en) * | 2023-04-26 | 2023-06-27 | 北京科技大学 | Human body posture data acquisition method and device based on data fusion and machine vision |
| CN116269355B (en) * | 2023-05-11 | 2023-08-01 | 江西珉轩智能科技有限公司 | Safety monitoring system based on figure gesture recognition |
| CN116703227B (en) * | 2023-06-14 | 2024-05-03 | 快住智能科技（苏州）有限公司 | Guest room management method and system based on intelligent service |
| CN117255451B (en) * | 2023-10-24 | 2024-05-03 | 快住智能科技（苏州）有限公司 | Intelligent living guest control method and system for hotel guest room management |
| CN117953591B (en) * | 2024-03-27 | 2024-06-14 | 中国人民解放军空军军医大学 | Intelligent limb rehabilitation assisting method and device |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110598554A (en) * | 2019-08-09 | 2019-12-20 | 中国地质大学（武汉） | Multi-person posture estimation method based on counterstudy |
| CN111680562A (en) * | 2020-05-09 | 2020-09-18 | 北京中广上洋科技股份有限公司 | Human body posture identification method and device based on skeleton key points, storage medium and terminal |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110502986B (en) * | 2019-07-12 | 2024-09-17 | 平安科技（深圳）有限公司 | Method, device, computer equipment and storage medium for identifying positions of persons in image |
| CN112633196A (en) * | 2020-12-28 | 2021-04-09 | 浙江大华技术股份有限公司 | Human body posture detection method and device and computer equipment |
-
2021
- 2021-11-02 CN CN202111287237.9A patent/CN114067358B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110598554A (en) * | 2019-08-09 | 2019-12-20 | 中国地质大学（武汉） | Multi-person posture estimation method based on counterstudy |
| CN111680562A (en) * | 2020-05-09 | 2020-09-18 | 北京中广上洋科技股份有限公司 | Human body posture identification method and device based on skeleton key points, storage medium and terminal |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114067358A (en) | 2022-02-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114067358B (en) | Human body posture recognition method and system based on key point detection technology | |
| CN106650687B (en) | Posture correction method based on depth information and skeleton information | |
| CN109829436B (en) | Multi-face tracking method based on depth appearance characteristics and self-adaptive aggregation network | |
| CN110135375B (en) | Multi-person attitude estimation method based on global information integration | |
| CN107590452A (en) | A kind of personal identification method and device based on gait and face fusion | |
| CN111753747B (en) | Violent motion detection method based on monocular camera and three-dimensional attitude estimation | |
| CN114220176A (en) | Human behavior recognition method based on deep learning | |
| CN104200200B (en) | Fusion depth information and half-tone information realize the system and method for Gait Recognition | |
| CN112633196A (en) | Human body posture detection method and device and computer equipment | |
| CN113516005B (en) | Dance action evaluation system based on deep learning and gesture estimation | |
| CN111144165B (en) | Gait information identification method, system and storage medium | |
| CN108875586B (en) | Functional limb rehabilitation training detection method based on depth image and skeleton data multi-feature fusion | |
| CN112200074A (en) | Attitude comparison method and terminal | |
| CN109919137B (en) | Pedestrian structural feature expression method | |
| Kusakunniran et al. | Automatic gait recognition using weighted binary pattern on video | |
| CN112989889A (en) | Gait recognition method based on posture guidance | |
| CN113920326A (en) | Tumble behavior identification method based on human skeleton key point detection | |
| Arif et al. | Human pose estimation and object interaction for sports behaviour | |
| Bhargavas et al. | Human identification using gait recognition | |
| Yan et al. | Human-object interaction recognition using multitask neural network | |
| CN110334609B (en) | Intelligent real-time somatosensory capturing method | |
| Yamao et al. | Development of human pose recognition system by using raspberry pi and posenet model | |
| Chai et al. | Human gait recognition: approaches, datasets and challenges | |
| CN116343325A (en) | Intelligent auxiliary system for household body building | |
| CN111444374B (en) | Human body retrieval system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |