CN117253290A - Rope skipping counting implementation method and device based on yolopose model and storage medium - Google Patents

Rope skipping counting implementation method and device based on yolopose model and storage medium Download PDF

Info

Publication number
CN117253290A
CN117253290A CN202311331557.9A CN202311331557A CN117253290A CN 117253290 A CN117253290 A CN 117253290A CN 202311331557 A CN202311331557 A CN 202311331557A CN 117253290 A CN117253290 A CN 117253290A
Authority
CN
China
Prior art keywords
rope skipping
counting
human body
key points
value
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202311331557.9A
Other languages
Chinese (zh)
Other versions
CN117253290B (en
Inventor
韩宇娇
倪非非
张波
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Scenery Wisdom Beijing Information Technology Co ltd
Original Assignee
Scenery Wisdom Beijing Information Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Scenery Wisdom Beijing Information Technology Co ltd filed Critical Scenery Wisdom Beijing Information Technology Co ltd
Priority to CN202311331557.9A priority Critical patent/CN117253290B/en
Publication of CN117253290A publication Critical patent/CN117253290A/en
Application granted granted Critical
Publication of CN117253290B publication Critical patent/CN117253290B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/23—Recognition of whole body movements, e.g. for sport training
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Multimedia (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Human Computer Interaction (AREA)
  • Social Psychology (AREA)
  • Psychiatry (AREA)
  • Image Analysis (AREA)

Abstract

The application discloses a rope skipping counting realization method, device and storage medium based on a yolopose model, which belong to the technical field of AI body-building exercises and comprise the following steps: collecting video data in real time through video data collecting equipment, and carrying out human body posture assessment on each frame of data by utilizing a yollopic related algorithm and storing the human body posture assessment; determining the position of the tested person by utilizing the key point information of the human body; constructing a motion feature vector by utilizing key point features of a human body; classifying the human body key point characteristic motion construction, judging whether the rope skipping action is performed or not, and then counting. The counting method utilizes the excellent characteristics of the yolopose series model, realizes simultaneous detection of multiple people, greatly reduces calculation resources, simultaneously counts the rope skipping for the first time by utilizing the characteristic engineering method, improves the counting robustness, ensures the counting accuracy, and solves the defects of inaccurate counting of counting personnel and large number of manual counting in the traditional mode.

Description

Rope skipping counting implementation method and device based on yolopose model and storage medium
Technical Field
The application relates to the technical field of AI sports, in particular to a rope skipping counting implementation method, device and storage medium based on a yolopose model.
Background
With the continuous maturation and development of artificial intelligence technology and big data analysis technology, ai+others have become a trend, wherein ai+sports are now developing better and better. With the proposal of openpost, simple and efficient human feature point algorithms develop better and better, and the algorithms can better help to realize AI+sports.
In rope skipping field, rope skipping belongs to quick cyclic motion, to traditional people's count mode, can appear personnel's wasting of resources and personnel's error of counting, leads to the inaccurate scheduling problem of count, adopts present AI technique to count and can significantly reduce the manpower, and the degree of accuracy of simultaneous count is higher.
The common AI implementation modes are two, one is based on cloud computing, the other is based on edge end, the cloud computing has the defects that a network is needed, the network is required to transmit video, and the missing of information transmitted by the network can cause miscounting or missing. Network delay can be well solved based on edge end realization, but the computational power requirement for edge calculation is high, so that selecting an efficient algorithm to realize rope skipping is also an important step.
AI skipping ropes are currently counted based on single features, such as height features, and human body posture assessment algorithms for single person detection are used in many cases, so that it is difficult to realize multi-person skipping ropes at the edge. The single characteristic can lead to lower counting robustness, namely, the situation that a person walks back and forth and the situation that skeleton points shake can lead to miscounting, and different positions of the person in a lens are difficult to set a threshold value even if normalization is carried out, and the limitation can lead to difficulty in realizing rope skipping of multiple persons. Such as: patent CN116563922a, an artificial intelligence-based automatic counting method for rope skipping, successfully uses key point information of human body to obtain height characteristics, and uses the height characteristics of shoulders to count; the patent CN115471906A discloses a multi-rope skipping mode identification counting method, wherein actions of a person are shot through two cameras, jump counting points are carried out by utilizing heights, and jump counting is corrected by combining rope skipping identification with time, so that the counting accuracy is improved; patent CN116311523a, a fancy rope skipping recognition algorithm based on image recognition, performs smoothing and normalization on height information by dynamically selecting key points, counting the height information of key points of a human body and determining fancy actions. In the patent CN116416551A, a tracking algorithm-based video image multi-person self-adaptive rope skipping intelligent counting method is used, a two-stage algorithm is adopted, namely, target detection is firstly carried out, then human body posture evaluation is carried out, rope skipping counting is carried out, the counting algorithm is judged only through height characteristics, the multi-person rope skipping counting method is successfully realized, but the human body posture evaluation can be carried out for n times during multi-person counting, the calculation resource consumption is high, and the counting characteristic is single.
Disclosure of Invention
The utility model provides a rope skipping count realization method, device and storage medium based on a yolopose model, solves the counting problem existing in the prior art, successfully realizes multi-person rope skipping at the edge end, avoids interference of other actions of a human body on rope skipping counting, and improves counting accuracy.
In a first aspect, a rope skipping counting implementation method based on a yollope model, the method comprising:
collecting single rope skipping video data, generating rope skipping pictures, marking key points of the rope skipping pictures by using lableme, and training a pre-established yollopic model by using the rope skipping picture data marked by the key points;
acquiring video data of a plurality of people skipping ropes through a camera, and dividing the video data according to character areas; wherein, each divided video data is single rope skipping video data;
aiming at the divided single rope skipping video data, acquiring human body key point features corresponding to each region according to a yollopic model after training; wherein the human body key points are leg key points;
according to the obtained human body key point characteristics, determining the motion state of rope skipping in each video frame, and constructing corresponding characteristic vectors according to the motion states of all the video frames;
and determining the man-made rope skipping times in each region according to the feature vectors corresponding to each region.
Optionally, in the training of the pre-established yolopose model by the rope skipping picture data marked by the key points, the leg key points are identified by modifying the weights in the OKS in the loss function;
the loss function of OKS specifically includes:
wherein k is n The specific weight of the key point to be adjusted; d, d n The Euclidean distance between the predicted point and the actual point; delta (v) n >0) For key point visibility, v n >The 0 condition is satisfied, delta (v) n >0) =1, condition is not satisfied, δ (v n >0)=0；v n Indicating a visibility flag, i.e., 0 unlabeled, 1 labeled not occluded, 2 labeled occluded; s is a scale factor, the value of which is the square root of the detection frame area.
Optionally, determining a motion state of rope skipping in each video frame, wherein the motion state specifically includes jump, jump hover, drop, two-foot landing, one-foot landing, and others, wherein the specific state is defined by displacement and speed of actions in adjacent frames, specifically:
the take-off is defined as: the y value of the key points of the human body characteristics of the adjacent frames is smaller, and the upward speed is higher, wherein the y value is used for representing the vertical height in the image frames;
the jump-up hovering is as follows: the y value difference value of the key points of the human body features of the adjacent frames is basically unchanged, and the speed is smaller than a set speed threshold;
the drop is defined as: the y value of key points of human body characteristic frames of adjacent frames is increased, and the speed in the downward direction is increased;
the bipedal landing is defined as: the y value and the x value of the human body characteristic key points of the adjacent frames are unchanged, the speed is smaller than a set speed threshold, and the height difference of the y values of the coordinates of the ankle of the two feet is smaller than a height threshold; the x value is used for representing the horizontal distance in the image frame;
single foot landing is defined as: the y value and the x value of the human body characteristic key points of the adjacent frames are unchanged, the speed is smaller than a set speed threshold, and the height difference of the coordinate y values of the ankles of the feet is larger than the height threshold;
other states are defined as: other states than the above state.
Optionally, constructing the corresponding feature vector according to the motion states of all video frames includes:
mapping the motion state in each video frame to a corresponding number;
arranging the numbers mapped by the motion states of all the video frames in time sequence to obtain a state sequence corresponding to the video;
and determining the number of each state in the state sequence, and taking the number of each state as a characteristic value, thereby obtaining the characteristic vectors of all video frames.
Optionally, feature vectors of all video frames are obtained, and the method further comprises:
corresponding weights are set for different eigenvalues in the eigenvector.
Optionally, determining the number of rope skipping of the person in each region according to the feature vector corresponding to each region includes counting by similarity:
when counting is carried out through the similarity, cosine similarity calculation is carried out on the current feature vector and the feature vector mean value of which the historical counting is successful;
and when the calculation result is greater than a preset threshold value, performing rope skipping counting.
Optionally, determining the number of rope skipping of the people in each region according to the feature vector corresponding to each region includes counting by a classifier:
when counting is carried out through the classifier, a counting classifier is constructed; the count classifier may include a decision tree and a support vector machine;
and (3) performing two-classification by inputting feature vectors corresponding to all video frames, and performing rope skipping counting according to classification results.
In a second aspect, a rope skipping count implementation device based on a yollope model, the device comprising:
the training module is used for collecting single rope skipping video data, generating rope skipping pictures, marking key points of the rope skipping pictures by using lableme, and training a pre-established yolopose model by using the rope skipping picture data marked by the key points;
the acquisition module is used for acquiring video data of the multi-person rope skipping through the camera and dividing the video data according to the character areas; wherein, each divided video data is single rope skipping video data;
the first processing module is used for acquiring human body key point characteristics corresponding to each region according to the trained yolopose model aiming at the divided single rope skipping video data; wherein the human body key points are leg key points;
the second processing module is used for determining the motion state of rope skipping in each video frame according to the obtained human body key point characteristics and constructing corresponding feature vectors according to the motion states of all the video frames;
and the determining module is used for determining the man-made rope skipping times in each region according to the feature vectors corresponding to each region.
In a third aspect, there is provided an electronic device comprising a memory and a processor, the memory storing a computer program, the processor implementing the rope skipping count implementing method of any of the first aspects described above when executing the computer program.
In a fourth aspect, a computer readable storage medium is provided, on which a computer program is stored, which when executed by a processor implements the rope skipping count implementation method of any of the first aspects above.
Compared with the prior art, the application has the following beneficial effects:
according to the method, through a yolopose series algorithm, target detection and human body posture evaluation of multiple persons can be achieved simultaneously, and computing resources of edge end computing are greatly reduced.
The method for constructing the jump rope feature vector is provided for the first time, and the method can better integrate the movement features of the jump rope and has stronger feature expression.
The method and the device for counting the rope skipping by using the cosine similarity or the classifier for counting are provided for the first time, the interference of other redundant information on rope skipping counting can be greatly eliminated, and meanwhile, multiple people can stand at different positions to realize multi-person asynchronous rope skipping.
The multi-person rope skipping method based on the yollope algorithm successfully realizes multi-person rope skipping and improves counting accuracy.
Drawings
FIG. 1 is a flow chart of steps of a method of counting rope skipping by multiple persons in the application;
fig. 2 is a flow chart of a counting method of a multi-person rope skipping according to an embodiment of the present application;
fig. 3 is a schematic diagram of an actual scene of a multi-person rope skipping according to an embodiment of the present application;
FIG. 4 is a schematic diagram of characteristics of an abnormal state and a normal state of rope skipping counting according to an embodiment of the present application;
fig. 5 is a block diagram of a module architecture of a rope skipping count implementation device according to an embodiment of the present application;
fig. 6 is an internal structural diagram of a computer device in one embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.
In the description of the present application: the expressions "comprising", "including", "having", etc. also mean "not limited to" (certain units, components, materials, steps, etc.).
In one embodiment, as shown in fig. 1, a rope skipping count implementation method based on a yollopic model is provided, which includes the following steps:
step one, collecting single rope skipping video data, generating rope skipping pictures, marking key points of the rope skipping pictures by using lableme, and training a pre-established yollopic model through the rope skipping picture data marked by the key points.
In this step, a pre-training of the yolopose model is first performed: the yollose model refers to a yollose model of a part of the object detection models yollov 5, yolox, yollov 7 and yollov 8 series models, is used for human body posture estimation, provides a pre-training model based on a COCO data set, but the data in the jumping process can generate unrecognized conditions, so 17 key point data sets of the COCO are built through lableme, and the model is trained. The training process specifically comprises the following steps:
the specific mode is as follows:
collecting rope skipping video data of a single person or multiple persons, and generating pictures, wherein the main environment is a plurality of indoor and outdoor conditions;
then, marking key points and marking bbox by using lableme;
processing the lableme annotation data to obtain a yolo series data format required by model training;
in rope skipping, the facial key point information of the person is non-key information, so that the weight in the OKS in the loss function is modified to better identify the leg characteristics in order to improve the identification accuracy of the key points of the legs during training.
The OKS-based loss function is as follows:
wherein: k (k) n Specific weights that are critical; d, d n Europe as a predicted point and a true pointA formula distance; delta is the visibility of each keypoint.
Adjusting the weight to be k n Is a value of (2); delta (v) n >0) For key point visibility, v n >The 0 condition is satisfied, delta (v) n >0) =1, condition is not satisfied, δ (v n >0)=0；v n Indicating a visibility flag, i.e., 0 unlabeled, 1 labeled not occluded, 2 labeled occluded; s is a scale factor, the value of which is the square root of the detection frame area.
And secondly, acquiring video data of the multi-person rope skipping through a camera, and dividing the video data according to the character areas.
Wherein, each divided video data is single rope skipping video data. In the step, the identification of key points of the human body is mainly realized, namely, the trained model is utilized, the data is acquired through the camera, and the bone point information of all people in the video is identified.
In the step, the rope skipping area is set: the area setting means that in the multi-person jump rope, the positions of each person station are different, so that the area of the camera is divided, then the key points of the legs of the yolopose are associated with the positions of the divided areas based on the positions of the areas, and keywords of only one are given, namely, only one person can stand in one area.
And thirdly, aiming at the divided single rope skipping video data, acquiring human body key point characteristics corresponding to each region according to the yolopose model after training.
Wherein, human body key points are leg key points.
And step four, determining the motion state of rope skipping in each video frame according to the obtained human body key point characteristics, and constructing corresponding feature vectors according to the motion states of all the video frames.
In the step, a feature vector is constructed, specifically, the obtained key point features of the human body are obtained, and a feature vector is constructed according to the action and the state of the rope skipping. The specific construction method is as follows:
the motion characteristics of a jump rope can be defined as jump, jump hover, drop, two-foot landing, one-foot landing and other 6 states, each action is determined by displacement and velocity:
the take-off is defined as: the Y value of the key point of the human body characteristic of the adjacent frame becomes smaller, namely Y t -Y t-1 <θ y0 The upward velocity becomes greater V 0 <θ v0 The downward direction in the image is positive, where the y value is used to characterize the vertical height in the image frame;
the jump-up hovering is as follows: y-value difference of human body characteristic key points of adjacent frames is basically unchanged Y t -Y t-1 |<θ y1 The speed is about 0, |V 1 |<θ v1 ；
The drop is defined as: y value of key point of human body characteristic frame of adjacent frame is enlarged by Y t -Y t-1 >θ y2 The velocity in the downward direction becomes greater, V 2 >θ v2 ；
The bipedal landing is defined as: the y value and the X value of the key points of the human body characteristics of the adjacent frames are unchanged, |X t -X t-1 |<θ x3 ，|Y t -Y t-1 |<θ y3 And a speed of about 0, i.e. |V 3 |<θ v3 The height difference of the coordinate Y value of the ankle of the two feet is smaller than a threshold value Y t [0]-Y t [1]|<δ 0 The method comprises the steps of carrying out a first treatment on the surface of the The x value is used for representing the horizontal distance in the image frame;
single foot landing is defined as: the y value and the X value of the key points of the human body characteristics of the adjacent frames are unchanged, |X t -X t-1 |<θ x3 ，|Y t -Y t-1 |<θ y3 And the speed is about 0, the height difference of the coordinate Y value of the ankle of the two feet is larger than a threshold value |Y t [0]-Y t [1]|>δ 1 ；
Other states are defined as: other states than the above state.
The key points of the human body characteristics are several or all 11-16 corresponding to the COCO key point data set of the human body.
Wherein each video frame corresponds to a state, including [0,1,2,3,4,5], respectively. I.e., a video frame is determined by the above formula, specifically, jump hover, drop, two-foot landing, one-foot landing, and other corresponding to 0,1,2,3,4,5, respectively.
The feature vector is defined as starting from the detected take-off state to the next take-off state and is assumed to have a state sequence of [0,0,0,1,1,2,4,3,3] for a period of time, the feature vector for this period of time being [3,2,1,2,1,0], i.e. the first bit of the feature vector represents the number of states of 0, i.e. the number of 0's of the state sequence has 3, the second bit and so on.
The feature vector may set a weight vector w= [ W0, W1, W2, W3, W4, W5 ]]The feature vectors are as follows: f=a×w T
Fifthly, determining the man-made rope skipping times in each area according to the feature vectors corresponding to each area.
In this step, counting is specifically implemented: the counting can be performed by two methods, namely, the counting by similarity and the counting by using a classifier;
similarity count: a markov independent assumption is made here that the current rope-jump status is to be correlated only with the previous ones. The cosine similarity is selected for similarity calculation.
Current feature vector F t As A and vector F of the previous n counts successful 1 ，F 2 ,. the mean value is taken as the eigenvector B.
The similarity calculation value is greater than a threshold value, and counting is performed.
The classification algorithm realizes counting: no markov assumption is made here, but a classifier needs to be constructed, the input of which is the features constructed above, and the classification algorithm selects a traditional machine learning algorithm, such as decision tree and support vector machine, and the classification adopts two classifications, namely a successful jump and a failure (1, 0). The data set is collected through the method, normal data and abnormal data are divided into approximately 2:1 for better classification, and for better data collection, all the videos are abnormal as far as possible and all the actions of normal jump are performed when the videos are shot.
In summary, it can be seen that the method for automatically counting a plurality of skipping ropes based on yolopose belongs to the field of edge calculation, belongs to the related technical field of AI body-building exercise, and comprises the following steps: collecting video data in real time through video data collecting equipment, and carrying out human body posture assessment on each frame of data by utilizing a yollopic related algorithm and storing the human body posture assessment; determining the position of the tested person by utilizing the key point information of the human body; constructing a motion feature vector by utilizing key point features of a human body; classifying the human body key point characteristic motion construction, judging whether the rope skipping action is performed or not, and then counting.
The counting method of the invention utilizes the excellent characteristics of the yolopose series model, realizes simultaneous detection of multiple people, greatly reduces the calculation resources, simultaneously counts the rope skipping for the first time by utilizing the characteristic engineering method, improves the counting robustness and ensures the counting accuracy. The defects of inaccurate counting of counting personnel and large number of manual counting in the traditional mode are overcome.
Meanwhile, skeleton point information of all people in the image is grasped through yolopose at one time, calculated amount is reduced, communication delay is greatly reduced through edge end calculation, and counting is realized by utilizing motion characteristics of a characteristic engineering construction rope skipping, rather than only counting through height. The multi-person asynchronous rope skipping at the edge end is successfully realized, the calculation accuracy is improved by fusing a plurality of characteristics, and the setting difficulty of the threshold value is reduced.
A specific example of the application of the above method is given below:
as shown in fig. 2, the method for implementing the method according to the embodiment of the present invention includes the following steps:
initializing parameters, firstly setting parameters of positions, and demarcating areas on the ground, wherein 5 areas are demarcated in the camera area as shown in fig. 3. Next, a start feature f0= [2,1,2,1,0,0] is initialized. The final initialization weight W is [1,1,1,1,0.1,1], where this is equivalent to a single-foot jump being turned off.
Training a yollopic model, wherein a yollov 5 model is selected for fine tuning;
the input of the selected model in the second step is the model structure of yolov5-s, and the input size of the model is (640 x 640);
the weight of the key points for training in the second step is set as follows: [0.26, 0.25, 25, 0.35, 0.35, 0.79, 0.79, 0.72,0.72, 0.62, 0.62, 0.7, 0.7, 0.6, 0.6, 0.6, 0.6, ];
the training data in the second step is derived from labelme labeling data, so that the labeling of the motion data of people is mainly increased, and the total number of the labeling is 5630;
and step two, training the super parameters of the model, wherein the batch-size is 8, training 100epochs, and lr is 0.001, and adding data for enhancing.
In the second step, in order to better verify the fine tuning effect, 1000 test set pictures are constructed, are also directly obtained from the rope skipping video, AP (Average Precision) is calculated by using the weight of the OKS, and the AP is improved from 41.2% to 43.6%, so that the AP is improved to a certain extent;
the face recognition authentication is firstly carried out, and the hand lifting can be carried out to indicate the beginning of rope skipping only after the authentication is successful.
In the third step, the snapshot of the face is based on a target detection frame of yolov5, and 1/3 of the image information is taken from top to bottom;
and step three, face recognition adopts a disclosed algorithm, such as a rainbow soft open api, and hand recognition is carried out by judging by using key points, namely, the skeleton points of the wrist are higher than the head.
The face recognition process needs to input user information in advance and then compare faces.
The data for each frame was recorded after hand lifting using the yolov 5-phase algorithm and the feature engineering above.
The parameters of the feature engineering in step four are shown in table 1:
TABLE 1 characteristic engineering parameter set table
The size of the picture input by the parameter in the fourth step is 640 x 640.
In the third step, the human body key points mainly include 17 key points of human body, including (no, left_eye, right_eye, left_ear, right_ear, left_hand, right_hand, right_elbox, left_write, right_write, left_shift, right_shift, left_knee, right_knee, left_ankle, right_ankle).
In the fourth step, the key points of the jump and drop human body characteristics are the information with the serial numbers of 15 and 16, namely (left_ankle, right_ankle), and the other key points are all the information with the serial numbers of 11-16. The sequence numbers start from 0.
And entering the recorded characteristics, entering a counter to calculate the similarity, and calculating by using the initialized characteristics F0. And if the similarity is greater than 0.9, judging that the jump is successful.
The embodiment of the invention runs on the edge equipment of Jetson Orin 70T, and the DS-2CD7A4XYZ123A is adopted by the camera. The yolov 5-phase performs the quantization processing of FP16, and the tensor R engine is utilized to perform reasoning, so that the FPs of the execution of the yolov 5-phase can reach about 30, and the whole program can run at 20-25 FPs.
According to the implementation scheme, the method realizes multi-person rope skipping and accurate counting through a yollopic algorithm and a characteristic construction method.
Fig. 4 is a schematic diagram showing the characteristics of the abnormal state and the normal state of rope skipping counting, and can be used for recognizing the skeleton points of the corresponding position people correctly and counting when a plurality of people skip the rope. The upper part of the figure 4 shows that the feet can be lifted and moved forward when the jump is interrupted, the lower part of the figure is the characteristic number when the jump is normal, and the similarity of the two characteristics is very low, so that the invention can better avoid abnormal actions and better ensure the accuracy of the jump rope.
In one embodiment, as shown in fig. 5, there is provided a rope skipping count implementation apparatus, including the following program modules: training module, acquisition module, first processing module, second processing module and confirm the module, wherein:
the training module is used for collecting single rope skipping video data, generating rope skipping pictures, marking key points of the rope skipping pictures by using lableme, and training a pre-established yolopose model by using the rope skipping picture data marked by the key points;
the acquisition module is used for acquiring video data of the multi-person rope skipping through the camera and dividing the video data according to the character areas; wherein, each divided video data is single rope skipping video data;
the first processing module is used for acquiring human body key point characteristics corresponding to each region according to the trained yolopose model aiming at the divided single rope skipping video data; wherein, the key points of the human body are key points of the legs;
the second processing module is used for determining the motion state of rope skipping in each video frame according to the obtained human body key point characteristics and constructing corresponding feature vectors according to the motion states of all the video frames;
and the determining module is used for determining the man-made rope skipping times in each region according to the feature vectors corresponding to each region.
The specific implementation content of each module can be referred to the limitation of the rope skipping counting implementation method, and is not repeated here.
In one embodiment, a computer device is provided, which may be a terminal, and the internal structure of which may be as shown in fig. 6. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected by a system bus. The processor of the computer device is used for providing computing and control capabilities, and the communication interface is used for conducting wired or wireless communication with an external terminal, wherein the wireless communication can be realized through WIFI, an operator network, NFC (near field communication) or other technologies. The computer equipment runs a computer program by loading to realize the rope skipping counting realization method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, the input device of the computer equipment can be a camera, and the video data of the rope skipping is obtained through the camera; the memory is used for process data in the rope skipping counting implementation method and result data of the rope skipping counting implementation method.
It will be appreciated by those skilled in the art that the structure shown in fig. 6 is merely a block diagram of some of the structures associated with the present application and is not limiting of the computer device to which the present application may be applied, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
In an embodiment, a computer readable storage medium is also provided, on which a computer program is stored, involving all or part of the flow of the method of the above embodiment.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

Claims (10)

1. The rope skipping counting implementation method based on the yolopose model is characterized by comprising the following steps of:
collecting single rope skipping video data, generating rope skipping pictures, marking key points of the rope skipping pictures by using lableme, and training a pre-established yollopic model by using the rope skipping picture data marked by the key points;
acquiring video data of a plurality of people skipping ropes through a camera, and dividing the video data according to character areas; wherein, each divided video data is single rope skipping video data;
aiming at the divided single rope skipping video data, acquiring human body key point features corresponding to each region according to a yollopic model after training; wherein the human body key points are leg key points;
according to the obtained human body key point characteristics, determining the motion state of rope skipping in each video frame, and constructing corresponding characteristic vectors according to the motion states of all the video frames;
and determining the man-made rope skipping times in each region according to the feature vectors corresponding to each region.
2. The method according to claim 1, wherein in training the pre-established yolopose model by the rope skipping picture data marked by the key points, the leg key points are identified by modifying weights in OKS in a loss function;
the loss function of OKS specifically includes:
wherein k is n The specific weight of the key point to be adjusted; d, d n The Euclidean distance between the predicted point and the actual point; delta (v) n >0) For key point visibility, v n >The 0 condition is satisfied, delta (v) n >0) =1, condition is not satisfied, δ (v n >0)=0；v n Indicating a visibility flag, i.e., 0 unlabeled, 1 labeled not occluded, 2 labeled occluded; s is a scale factor, the value of which is the square root of the detection frame area.
3. The method according to claim 1, wherein the motion state of the rope being jumped in each video frame is determined, in particular comprising jump, jump hover, drop, two-foot landing, one-foot landing and others, wherein the particular state is defined by the displacement and speed of the motion in the adjacent frame, in particular:
the take-off is defined as: the y value of the key points of the human body characteristics of the adjacent frames is smaller, and the upward speed is higher, wherein the y value is used for representing the vertical height in the image frames;
the jump-up hovering is as follows: the y value difference value of the key points of the human body features of the adjacent frames is basically unchanged, and the speed is smaller than a set speed threshold;
the drop is defined as: the y value of key points of human body characteristic frames of adjacent frames is increased, and the speed in the downward direction is increased;
the bipedal landing is defined as: the y value and the x value of the human body characteristic key points of the adjacent frames are unchanged, the speed is smaller than a set speed threshold, and the height difference of the y values of the coordinates of the ankle of the two feet is smaller than a height threshold; the x value is used for representing the horizontal distance in the image frame;
single foot landing is defined as: the y value and the x value of the human body characteristic key points of the adjacent frames are unchanged, the speed is smaller than a set speed threshold, and the height difference of the coordinate y values of the ankles of the feet is larger than the height threshold;
other states are defined as: other states than the above state.
4. A method according to claim 3, wherein constructing corresponding feature vectors from the motion states of all video frames comprises:
mapping the motion state in each video frame to a corresponding number;
arranging the numbers mapped by the motion states of all the video frames in time sequence to obtain a state sequence corresponding to the video;
and determining the number of each state in the state sequence, and taking the number of each state as a characteristic value, thereby obtaining the characteristic vectors of all video frames.
5. The method of claim 4, wherein feature vectors are derived for all video frames, the method further comprising:
corresponding weights are set for different eigenvalues in the eigenvector.
6. The method of claim 1, wherein determining the number of rope jumps of the person in each region based on the feature vector corresponding to each region comprises counting by similarity:
when counting is carried out through the similarity, cosine similarity calculation is carried out on the current feature vector and the feature vector mean value of which the historical counting is successful;
and when the calculation result is greater than a preset threshold value, performing rope skipping counting.
7. The method of claim 1, wherein determining the number of rope jumps for the person in each region based on the feature vector for each region comprises counting by a classifier:
when counting is carried out through the classifier, a counting classifier is constructed; the count classifier may include a decision tree and a support vector machine;
and (3) performing two-classification by inputting feature vectors corresponding to all video frames, and performing rope skipping counting according to classification results.
8. A rope skipping count implementation device based on a yolopose model, the device comprising:
the training module is used for collecting single rope skipping video data, generating rope skipping pictures, marking key points of the rope skipping pictures by using lableme, and training a pre-established yolopose model by using the rope skipping picture data marked by the key points;
the acquisition module is used for acquiring video data of the multi-person rope skipping through the camera and dividing the video data according to the character areas; wherein, each divided video data is single rope skipping video data;
the first processing module is used for acquiring human body key point characteristics corresponding to each region according to the trained yolopose model aiming at the divided single rope skipping video data; wherein the human body key points are leg key points;
the second processing module is used for determining the motion state of rope skipping in each video frame according to the obtained human body key point characteristics and constructing corresponding feature vectors according to the motion states of all the video frames;
and the determining module is used for determining the man-made rope skipping times in each region according to the feature vectors corresponding to each region.
9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 7 when the computer program is executed.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method according to any one of claims 1 to 7.
CN202311331557.9A 2023-10-13 2023-10-13 Rope skipping counting implementation method and device based on yolopose model and storage medium Active CN117253290B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311331557.9A CN117253290B (en) 2023-10-13 2023-10-13 Rope skipping counting implementation method and device based on yolopose model and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311331557.9A CN117253290B (en) 2023-10-13 2023-10-13 Rope skipping counting implementation method and device based on yolopose model and storage medium

Publications (2)

Publication Number Publication Date
CN117253290A true CN117253290A (en) 2023-12-19
CN117253290B CN117253290B (en) 2024-05-10

Family

ID=89132925

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311331557.9A Active CN117253290B (en) 2023-10-13 2023-10-13 Rope skipping counting implementation method and device based on yolopose model and storage medium

Country Status (1)

Country Link
CN (1) CN117253290B (en)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112464808A (en) * 2020-11-26 2021-03-09 成都睿码科技有限责任公司 Rope skipping posture and number identification method based on computer vision
CN114100103A (en) * 2021-10-28 2022-03-01 杭州电子科技大学 Rope skipping counting detection system and method based on key point identification
CN115171019A (en) * 2022-07-15 2022-10-11 浙江大学 Rope skipping counting method based on semi-supervised video target segmentation
CN115205750A (en) * 2022-07-05 2022-10-18 北京甲板智慧科技有限公司 Motion real-time counting method and system based on deep learning model
CN115346149A (en) * 2022-06-21 2022-11-15 浙江大沩人工智能科技有限公司 Rope skipping counting method and system based on space-time diagram convolution network
CN115359566A (en) * 2022-08-23 2022-11-18 深圳市赛为智能股份有限公司 Human behavior identification method, device and equipment based on key points and optical flow
CN115512439A (en) * 2022-09-23 2022-12-23 燕山大学 Real-time analysis method for spontaneous gait of mouse in long and narrow runway

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112464808A (en) * 2020-11-26 2021-03-09 成都睿码科技有限责任公司 Rope skipping posture and number identification method based on computer vision
CN114100103A (en) * 2021-10-28 2022-03-01 杭州电子科技大学 Rope skipping counting detection system and method based on key point identification
CN115346149A (en) * 2022-06-21 2022-11-15 浙江大沩人工智能科技有限公司 Rope skipping counting method and system based on space-time diagram convolution network
CN115205750A (en) * 2022-07-05 2022-10-18 北京甲板智慧科技有限公司 Motion real-time counting method and system based on deep learning model
CN115171019A (en) * 2022-07-15 2022-10-11 浙江大学 Rope skipping counting method based on semi-supervised video target segmentation
CN115359566A (en) * 2022-08-23 2022-11-18 深圳市赛为智能股份有限公司 Human behavior identification method, device and equipment based on key points and optical flow
CN115512439A (en) * 2022-09-23 2022-12-23 燕山大学 Real-time analysis method for spontaneous gait of mouse in long and narrow runway

Also Published As

Publication number Publication date
CN117253290B (en) 2024-05-10

Similar Documents

Publication Publication Date Title
WO2021129064A1 (en) Posture acquisition method and device, and key point coordinate positioning model training method and device
CN107168527B (en) The first visual angle gesture identification and exchange method based on region convolutional neural networks
CN111931585A (en) Classroom concentration degree detection method and device
CN108629306A (en) Human posture recognition method and device, electronic equipment, storage medium
EP4053791A1 (en) Image processing device, image processing method, and non-transitory computer-readable medium having image processing program stored thereon
CN107767335A (en) A kind of image interfusion method and system based on face recognition features&#39; point location
EP3716212A1 (en) Image coding method, action recognition method, and computer device
WO2017161734A1 (en) Correction of human body movements via television and motion-sensing accessory and system
CN109685037A (en) A kind of real-time action recognition methods, device and electronic equipment
CN105912126B (en) A kind of gesture motion is mapped to the adaptive adjusting gain method at interface
CN108053420A (en) A kind of dividing method based on the unrelated attribute dynamic scene of limited spatial and temporal resolution class
CN110956141B (en) Human body continuous action rapid analysis method based on local recognition
CN110032932A (en) A kind of human posture recognition method based on video processing and decision tree given threshold
CN109299641A (en) A kind of train dispatcher&#39;s fatigue monitoring image adaptive Processing Algorithm
Yang et al. Human exercise posture analysis based on pose estimation
CN117253290B (en) Rope skipping counting implementation method and device based on yolopose model and storage medium
Chang et al. Vision-based tracking and interpretation of human leg movement for virtual reality applications
CN111353347B (en) Action recognition error correction method, electronic device, and storage medium
CN113707271B (en) Fitness scheme generation method and system based on artificial intelligence and big data
CN111860031A (en) Face pose estimation method and device, electronic equipment and readable storage medium
Zeng et al. Deep learning approach to automated data collection and processing of video surveillance in sports activity prediction
CN107749942A (en) Suspension image pickup method, mobile terminal and computer-readable recording medium
Zhang et al. [Retracted] Research on Pose Recognition Algorithm for Sports Players Based on Machine Learning of Sensor Data
CN115294623B (en) Human body whole body motion capturing method, device, storage medium and terminal
CN117423166B (en) Motion recognition method and system according to human body posture image data

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant