CN115188062B

CN115188062B - User running posture analysis method and device, running machine and storage medium

Info

Publication number: CN115188062B
Application number: CN202110369149.7A
Authority: CN
Inventors: 贾玮
Original assignee: Guangzhou Shiyuan Electronics Thecnology Co Ltd; Guangzhou Yuandong Smart Sports Technology Co Ltd
Current assignee: Guangzhou Shiyuan Electronics Thecnology Co Ltd; Guangzhou Yuandong Smart Sports Technology Co Ltd
Priority date: 2021-04-06
Filing date: 2021-04-06
Publication date: 2024-02-27
Anticipated expiration: 2041-04-06
Also published as: CN115188062A

Abstract

The application discloses a user running gesture analysis method, a device, a running machine and a storage medium, wherein the method comprises the steps of extracting a human body lower body image with a set frame number from a user running video captured by a camera; constructing a key point coordinate matrix according to the bone key points determined from the lower body images of the human body; determining semantic feature information corresponding to each bone key point in the lower body of the human body according to the key point coordinate matrix; through analysis of each piece of semantic feature information, the running posture of the body part associated with the user in running is determined. By using the method, the multi-frame human body lower body image in the video stream is used as the input of the gesture analysis, so that a more robust gesture analysis result in the dynamic running process of a relative user is ensured, meanwhile, the skeleton key point detection on the multi-frame human body lower body image in the gesture analysis implementation is realized, the probability of the key point prediction error is effectively reduced, and the accuracy and the effectiveness of the gesture analysis result are improved.

Description

User running posture analysis method and device, running machine and storage medium

Technical Field

The present disclosure relates to the field of image recognition technologies, and in particular, to a method and an apparatus for analyzing running postures of a user, a running machine, and a storage medium.

Background

With the progress of society, people pay more and more attention to exercise and fitness, wherein a running machine becomes the most popular fitness equipment. However, there is a lack of one-to-one instruction by the coach, and the wrong running posture cannot perform exercise, but damages the knee and ankle of the athlete. Analyzing the running gesture of the user and providing correction comments in time becomes a function application which is popular with the user on the running machine.

In the existing running posture analysis method, the running posture is generally classified based on analysis of a single image, such as calculating an angle by correlating key points of knees or ankles and performing a threshold value determination.

However, the human body is dynamic during running, and the result determined by the existing running posture analysis method is not representative; in addition, the misprediction of key points in the existing analysis method can have a great influence on the analysis result.

Disclosure of Invention

In view of this, the embodiments of the present application provide a method, an apparatus, a treadmill and a storage medium for analyzing a running gesture of a user, which can realize effective analysis of the running gesture of the user, so as to improve the user experience of the treadmill.

In a first aspect, an embodiment of the present application provides a method for analyzing a running gesture of a user, including:

extracting a lower body image of a human body with a set frame number from a running video of a user captured by a camera;

constructing a key point coordinate matrix according to the bone key points determined from the lower body images of the human body;

determining semantic feature information corresponding to each bone key point in the lower body of the human body according to the key point coordinate matrix;

through analysis of each piece of semantic feature information, the running posture of the body part associated with the user in running is determined.

Further, the constructing a key point coordinate matrix according to the bone key points determined from each lower body image of the human body comprises:

aiming at each human body lower body image, taking the human body lower body image as input data, inputting a skeleton point extraction network model, and obtaining an output skeleton key point thermodynamic diagram;

decoding the skeletal key point thermodynamic diagram through a peak value calculation method to obtain skeletal key point coordinates of a set number corresponding to the lower body image of the human body;

and summarizing skeleton key point coordinates of the lower body images of the human body to form a key point coordinate matrix taking the skeleton key point coordinates as elements, the set frame number as row numbers and the set number as column numbers.

Further, the determining, according to the key point coordinate matrix, semantic feature information corresponding to each bone key point in the lower body of the human body includes:

carrying out space division on the key point coordinate matrixes according to the determined key point association relation to obtain key point coordinate sub-matrixes of the determined number of matrixes;

and aiming at each key point coordinate submatrix, inputting the key point coordinate submatrix serving as input data into a corresponding neural network model, obtaining semantic feature vectors corresponding to each associated skeleton point included in the key point coordinate submatrix, and recording the semantic feature vectors as semantic feature information.

Further, the space division is performed on the key point coordinate matrix according to the determined key point association relationship to obtain key point coordinate submatrices with the determined number of matrices, including:

determining the associated body parts associated with the lower body of the user during running, and determining the number of the determined associated body parts as the matrix number;

determining a set of associated bone points constituting each associated body part, wherein the set of associated bone points comprises all associated bone points having a key point association relationship with respect to the associated body part;

Extracting column information corresponding to each associated bone point in the associated bone point set from the key point coordinate matrix aiming at each associated bone point set, and forming a corresponding key point coordinate sub-matrix based on each column information;

and obtaining the key point coordinate submatrices of the matrix number.

Further, the value of the number of the matrixes is 4;

the key point coordinate submatrices of the matrix number are respectively as follows: a first keypoint coordinate sub-matrix characterizing the left knee, a second keypoint coordinate sub-matrix characterizing the right knee, a third keypoint coordinate sub-matrix characterizing the left ankle, and a fourth keypoint coordinate sub-matrix characterizing the right ankle.

Further, the neural network model corresponding to each key point coordinate sub-matrix is an adaptive graph convolution network model;

adopting adjacent matrixes to represent the relation among all the associated skeleton points in the operation of the self-adaptive graph convolution network model, and determining semantic feature vectors of all the associated skeleton points through determining element values in all the adjacent matrixes;

the values of the elements in the adjacency matrix are characterized and determined by at least three means:

characterizing element values based on physical connections of human bones;

Characterizing element values based on node association strength determined after training and learning of sample data in the training set;

the element values are characterized based on key point features determined from current input data entered into the adaptive graph rolling network model.

Further, the first adaptive graph rolling network model and the second adaptive graph rolling network model which respectively correspond to the first key point coordinate submatrix and the second key point coordinate submatrix are 3*3 in the size of an adjacent matrix adopted in operation;

and the size of an adjacent matrix adopted in the operation is 5*5.

Further, the determining the running posture of the body part associated with the running of the user through analyzing each piece of semantic feature information comprises the following steps:

inputting corresponding semantic feature information into a pre-trained gesture classifier aiming at each key point coordinate submatrix to obtain a matching score value of a body part associated with the key point coordinate submatrix relative to each running gesture;

determining the running gesture corresponding to the highest matching score as the current running gesture of the corresponding body part;

Wherein the running posture comprises normal running, eversion running and inner buckle running.

Further, the gesture classifier at least includes: a cascade full connection layer, an activation layer, a batch normalization layer and a logistic regression layer.

In a second aspect, embodiments of the present application provide a user running posture analysis apparatus, including:

the image extraction module is used for extracting the lower half body image of the human body with a set frame number from the running video of the user captured by the camera;

the matrix determining module is used for constructing a key point coordinate matrix according to the bone key points determined from the lower body images of the human body;

the feature determining module is used for determining semantic feature information corresponding to each bone key point in the lower body of the human body according to the key point coordinate matrix;

and the gesture determining module is used for determining the running gesture of the body part related to the running of the user through analysis of the semantic feature information.

In a third aspect, embodiments of the present application further provide a treadmill, including: base, stand, set up in the intelligent interaction device on stand top, still include: the storage, one or more controllers and the camera are arranged at the appointed position of the upright post;

The camera is used for capturing video streams when a user runs;

the memory is used for storing one or more programs;

when the one or more programs are executed by the one or more controllers, the one or more controllers are caused to implement the methods as described in any of the embodiments of the present invention.

In a fourth aspect, embodiments of the present application also provide a storage medium containing computer executable instructions, which when executed by a computer controller, are for performing the user running profile analysis method as described in the first aspect.

The method extracts the lower half body image of the human body with the set frame number from the running video of the user captured by the camera; then constructing a key point coordinate matrix according to the bone key points determined from the lower half body images of the human bodies; and finally, determining the running gesture of the body part related to the running of the user through analysis of the semantic feature information. According to the technical scheme, the multi-frame human body lower body image in the video stream is used as the input of the gesture analysis, so that a more robust gesture analysis result in the dynamic running process of a relative user is ensured, meanwhile, the skeleton key point detection on the multi-frame human body lower body image in the gesture analysis implementation is also effectively reduced in probability of the key point prediction error, and the accuracy and the effectiveness of the gesture analysis result are improved.

Drawings

Other features, objects and advantages of the present application will become more apparent upon reading of the detailed description of non-limiting embodiments, made with reference to the following drawings, in which:

fig. 1 is a flow chart of a method for analyzing running gesture of a user according to an embodiment of the present application;

FIG. 1a is a diagram showing an example of skeletal key points provided in the method for analyzing a running posture of a user according to the first embodiment;

FIG. 1b is a flowchart illustrating an example of a method for analyzing a running gesture of a user according to an embodiment of the present application;

fig. 2 is a flow chart of a method for analyzing running gesture of a user according to a second embodiment of the present application;

FIG. 2a is a flowchart illustrating the determination of a sub-matrix of coordinates of a keypoint in a second embodiment of the present application;

FIG. 2b is a diagram illustrating an example of a set of associated skeletal points included in each associated body part in a method for analyzing a running profile of a user according to a second embodiment of the present application;

fig. 3 is a block diagram of a user running gesture analysis apparatus according to a third embodiment of the present application;

fig. 4 is a schematic structural diagram of a treadmill according to a fourth embodiment of the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the following detailed description of the embodiments of the present application will be given with reference to the accompanying drawings. It should be understood that the described embodiments are merely some, but not all, of the embodiments of the present application. All other embodiments, based on the embodiments herein, which would be apparent to one of ordinary skill in the art without making any inventive effort, are intended to be within the scope of the present application.

When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the accompanying claims.

In the description of this application, it should be understood that the terms "first," "second," "third," and the like are used merely to distinguish between similar objects and are not necessarily used to describe a particular order or sequence, nor should they be construed to indicate or imply relative importance. The specific meaning of the terms in this application will be understood by those of ordinary skill in the art as the case may be. Furthermore, in the description of the present application, unless otherwise indicated, "a plurality" means two or more. "and/or", describes an association relationship of an association object, and indicates that there may be three relationships, for example, a and/or B, and may indicate: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship.

Example 1

Fig. 1 is a flowchart of a method for analyzing a running gesture of a user according to an embodiment of the present application, where the method is suitable for analyzing the gesture of the user running on a treadmill. The method may be performed by a user running profile analysis device, which may be implemented in hardware and/or software and is typically integrated into a treadmill.

It can be known that the existing running gesture analysis method adopts a single image for analysis, the determined running gesture is often not representative, and meanwhile, if the single image is detected by a key point, the analysis result of the running gesture is directly affected. The first embodiment provides a method for analyzing a running gesture of a user, which can realize effective analysis of the running gesture of the user.

As shown in fig. 1, the method for analyzing a running gesture of a user according to the first embodiment specifically includes the following steps:

s101, extracting a lower half body image of a human body with a set frame number from a running video of a user captured by a camera.

In this embodiment, the camera may be specifically considered to be disposed at a position of the treadmill, and the camera may capture a running video of the user when the user runs on the treadmill. At the same time, the captured user running video needs to include at least the relevant running motion of the user's lower body.

For example, the camera may be disposed on the treadmill at a position capable of clearly capturing a portion below the hip joint of the user, thereby ensuring that each extracted image of the lower body of the human body includes at least the hip joint portion, the knee portion, the ankle portion, the toe portion, and the heel portion of the user.

The embodiment considers that the analysis is performed on a plurality of frames of images containing running actions of a user, so that the problem that an analysis result obtained by analyzing a single image is inaccurate is solved. Specifically, the embodiment can continuously capture images of a set frame number in a frame unit from the running video of the user captured by the camera in real time, and record the images as images of the lower half of the human body.

Or, the embodiment can also intercept the lower half body image of the human body with a set frame number from the running video of the user at a set time interval in real time; likewise, it may be an off-line user running video captured by the camera, whereby a set number of frames of human lower body images are acquired continuously or at set time intervals in the off-line user running video.

S102, constructing a key point coordinate matrix according to the bone key points determined from the lower body images of the human body.

In this embodiment, the bone keypoints are specifically understood as keypoints that can characterize the human bone joint sites. The step can adopt a pre-trained skeleton point extraction network model to extract skeleton key points contained in the lower body image of the human body. For example, in this step, all the lower body images of the human body may be input together into the skeleton point extraction network model, then, the key point thermodynamic diagrams including the key points of the skeleton corresponding to all the lower body images of the human body may be obtained, and finally, the coordinates of the key points of the skeleton included in the key point thermodynamic diagrams may be obtained from the key point thermodynamic diagrams.

In addition, the step can also process one human body lower body image firstly, input the human body lower body image into a skeleton point extraction network model, directly determine the contained skeleton key point coordinates after obtaining a key point thermodynamic diagram, and then process the rest human body lower body images sequentially until all the human body lower body images correspond to the contained skeleton key point coordinates.

In this embodiment, the step may be to aggregate all the bone key point coordinates according to a certain order after determining the bone key point coordinates corresponding to the lower body image of each human body, thereby forming a key point coordinate matrix including all the bone key point coordinates. For example, when forming the key point coordinate matrix, the sequence of the step can be to collect different skeleton key point coordinates in the lower half body image of the same human body to form element values of the same row in the matrix; meanwhile, the element values of each column in the defined matrix can be regarded as coordinate values of skeleton key points of the same body part in different lower body images of the human body.

S103, determining semantic feature information corresponding to each bone key point in the lower body of the human body according to the key point coordinate matrix.

In this embodiment, the above S102 corresponds to obtaining a fusion matrix composed of key points of human bones included in a plurality of human lower body images, that is, a key point coordinate matrix. It will be appreciated that the image of the lower body of a person contains all skeletal keys that represent hip joint locations of the lower body of the person, knee bone keys that represent knee joint locations, ankle and toe bone keys that represent the entire foot.

By way of example, fig. 1a shows an example representation of skeletal keypoints provided in the user running posture analysis method provided in this embodiment. As shown in fig. 1a, the bone contour of the lower body of the human body can be characterized by 12 bone keypoints. Specifically, the skeletal keypoints labeled 0 and 1 represent the right and left hip, the skeletal keypoints labeled 2 and 3 represent the right and left knee, the skeletal keypoints labeled 4 and 5 represent the right and left ankle, the skeletal keypoints labeled 6 and 7 represent the right and left big toe, the skeletal keypoints labeled 8 and 9 represent the right and left small toe, and the skeletal keypoints labeled 10 and 11 represent the right and left heel, respectively.

In this embodiment, a feature extraction network model may be used to obtain semantic feature information of each skeletal key using the key coordinate matrix as input data. However, it should be noted that not all skeletal keys are associated during a user's running, for example, hip skeletal keys are not in close relationship with toe skeletal keys of the foot.

Based on the above, when determining the semantic feature information of each bone key point based on the key point coordinate matrix, the embodiment may consider that the bone key points which are relatively closely related are summarized, then extract the corresponding key point coordinate submatrices from the key point coordinate matrix based on the summarized bone key point subset, and finally determine the semantic feature information of each bone key point based on each formed key point coordinate submatrices.

Wherein, for the tightly-connected skeletal key point summary, the embodiment can be considered to be determined according to the relevance of each body part in the lower body of the human body when the user runs. For example, the hip skeletal keypoints, knee skeletal keypoints, and ankle skeletal keypoints may together represent the state of the knee when the user is running, and thus may be considered to be in close association with each other.

According to the embodiment, after the plurality of key point coordinate submatrices are formed through dividing, the key point coordinate submatrices are respectively combined with the trained feature extraction network model, and then corresponding semantic feature information is obtained relative to skeleton key points contained in each key point coordinate submatrix.

S104, determining the running posture of the body part related to the running of the user through analysis of the semantic feature information.

In this embodiment, with respect to the process of obtaining the semantic feature information in S103, this embodiment may consider that, with respect to each skeletal key point included in the lower body of the human body, corresponding semantic feature information exists, where the semantic feature information may be represented in the form of a semantic feature vector, and each semantic feature information includes a state that the corresponding skeletal key point is presented by the body part associated with the user when running.

It will be appreciated that in performing the analysis of the user's running position, the determination of the user's running position may be translated into a position determination of the body parts associated with the user's running, which are mainly the left/right knee and the left/right ankle, and that in this embodiment there is a set of skeletal keypoints (in addition to the left/right knee skeletal keypoints, other skeletal keypoints in close connection therewith) that characterize the left/right knee, and a set of skeletal keypoints (in addition to the left/right ankle keypoints, other skeletal keypoints in close connection therewith) of the left/right ankle.

On the basis of the above, the embodiment can summarize the semantic feature information corresponding to the skeleton key point set representing the left/right knee, and perform gesture classification processing on the summarized semantic feature information, so as to determine the running gesture currently presented by the left/right knee; similarly, the embodiment can further collect semantic feature information corresponding to the skeleton key point set of the left/right ankle, and perform gesture classification processing on the collected semantic feature information, so as to determine the running gesture currently presented by the left/right ankle. The gesture classification processing can be realized by adopting a pre-trained gesture classifier, and the presented gesture classification result can comprise a normal gesture, an everting gesture and an inner buckling gesture.

For a better understanding of the specific implementation of the method for analyzing a running gesture of a user according to the present embodiment, fig. 1b is a flowchart illustrating an example of the method for analyzing a running gesture of a user according to the first embodiment of the present application. As shown in fig. 1b, first, the lower body image 12 of the human body with the set number of frames extracted can be obtained from the video stream of the capturing operation performed by the camera 11, then, all the skeletal key points 13 after the skeletal point extraction performed on each lower body image of the human body can be obtained, and then, the posture result 14 of the body part associated with the running of the user can be obtained through the semantic feature and the classification analysis operation based on all the skeletal key points.

According to the running gesture analysis method for the user, provided by the embodiment of the invention, the multi-frame human body lower body image in the video stream is used as the input of gesture analysis, so that a more robust gesture analysis result in the dynamic running process of the user is ensured, meanwhile, bone key points are detected on the multi-frame human body lower body image in the gesture analysis implementation, the probability of the prediction errors of the key points is effectively reduced, and the accuracy and the effectiveness of the gesture analysis result are improved.

Example two

Fig. 2 is a flow chart of a method for analyzing running posture of a user according to a second embodiment of the present application, where the present embodiment is based on the foregoing embodiments, and in the present embodiment, a key point coordinate matrix may be constructed according to bone key points determined from each lower body image of the human body, and the key point coordinate matrix may be specifically expressed for each lower body image of the human body, where the lower body image of the human body is used as input data, and the input bone point extraction network model is used to obtain an output bone key point thermodynamic diagram; decoding the skeletal key point thermodynamic diagram through a peak value calculation method to obtain skeletal key point coordinates of a set number corresponding to the lower body image of the human body; and summarizing skeleton key point coordinates of the lower body images of the human body to form a key point coordinate matrix taking the skeleton key point coordinates as elements, the set frame number as row numbers and the set number as column numbers.

Meanwhile, according to the key point coordinate matrix, the method can further specifically express that semantic feature information corresponding to each bone key point in the lower body of the human body is determined as follows: carrying out space division on the key point coordinate matrixes according to the determined key point association relation to obtain key point coordinate sub-matrixes of the determined number of matrixes; and aiming at each key point coordinate submatrix, inputting the key point coordinate submatrix serving as input data into a corresponding neural network model, obtaining semantic feature vectors corresponding to each associated skeleton point included in the key point coordinate submatrix, and recording the semantic feature vectors as semantic feature information.

In addition, the running gesture of the body part related to the running of the user can be determined by analyzing each piece of semantic feature information in the embodiment as follows: inputting corresponding semantic feature information into a pre-trained gesture classifier aiming at each key point coordinate submatrix to obtain a matching score value of a body part associated with the key point coordinate submatrix relative to each running gesture; determining the running gesture corresponding to the highest matching score as the current running gesture of the corresponding body part; wherein the running posture comprises normal running, eversion running and inner buckle running.

As shown in fig. 2, a method for analyzing running postures of a user according to a second embodiment of the present application specifically includes the following operations:

s201, extracting a lower half body image of a human body with a set frame number from running video of a user captured by a camera.

Illustratively, this step may capture a video stream in real time by a camera provided on a column of the treadmill and extract therefrom a set number of human lower body images, each comprising a body part of the user's lower body such as a hip, knee, foot, etc.

The following S202 to S204 in this embodiment provide a specific construction procedure of the key point coordinate matrix.

S202, inputting skeleton point extraction network models by taking the lower body image of the human body as input data aiming at the lower body image of the human body, and obtaining an output skeleton key point thermodynamic diagram.

In this embodiment, the implementation process of the skeleton point extraction operation for each human lower body image is the same, and first, a skeleton point extraction network model is adopted through this step to obtain a skeleton key point thermodynamic diagram corresponding to each human lower body image. The skeletal keypoint thermodynamic map comprises skeletal keypoints that characterize various body parts in the lower body of the human body.

Illustratively, the process of obtaining a skeletal keypoint thermodynamic diagram through a skeletal point extraction network model may be expressed as: and identifying skeleton key points of all people in the lower body image of the human body, carrying out matching connection on the skeleton key points, regarding the skeleton key points in the range as skeleton key points belonging to the same person through a given matching degree range, distributing the skeleton key points into the same group, and finally outputting a key point coordinate thermodynamic diagram formed based on the skeleton key points of the same group. In this procedure, the skeletal point extraction network model used employs a convolution cascade of 1 x 1,3 x 3, and 3*3 convolution layers. The skeleton point extraction network model has the characteristics of small calculated amount, high speed and high efficiency.

S203, decoding the skeletal key point thermodynamic diagram through a peak value calculation method to obtain skeletal key point coordinates of the set number corresponding to the lower body image of the human body.

It can be known that the obtained bone key point thermodynamic diagram cannot directly obtain coordinate information of the bone key points, and the bone key point thermodynamic diagram needs to be decoded through the step to obtain the bone key point coordinates. In addition, as the number of skeleton key points corresponding to the body part of the human body is basically limited, the set number of skeleton key point coordinates can be obtained from each lower body image of the human body in the step, and each skeleton key point coordinate is a two-dimensional coordinate, and the set number can be preferably 12.

S204, summarizing skeleton key point coordinates of the lower body images of the human body to form a key point coordinate matrix taking the skeleton key point coordinates as elements, the set frame number as row number and the set number as column number.

In this embodiment, the ranking mode adopted in the step of summarizing the bone key point coordinates to form the key frame coordinate matrix is to consider the bone key point coordinates corresponding to the lower body image of a single human body as one element row, and each element is one bone key point coordinate in the lower body image of the human body. Assuming that the number of frames is set to N, the number is set to V, and the skeletal key point coordinate dimension is set to C, the size of the formed key point coordinate matrix may be expressed as: n×v×c.

The implementation operation of the semantic feature information determination is given in the following S205 and S206 in this embodiment.

S205, carrying out space division on the key point coordinate matrixes according to the determined key point association relation to obtain key point coordinate submatrices with the determined number of matrixes.

In this embodiment, the coordinate matrix of the key points includes coordinate information of the bone key points included in the lower body image of the human body of the set frame number. In this embodiment, considering bone key points adopted to represent relevant body parts during running of a user, so that the degree of close connection between different bone key points is different, if running gesture analysis is directly performed on the obtained key point coordinate matrix, semantic feature information of each bone key point is obtained, and accuracy of running gesture analysis results is affected.

Therefore, in the embodiment, the key point coordinate matrix is divided through the step, and the bone key points with compact key point association relations are divided together to form a certain number of key point coordinate submatrices. Wherein the key point association relationship can be determined by bone key points associated with the body parts involved in running of the user.

Illustratively, for a body part of the knee, the associated skeletal keypoints include knee skeletal keypoints that characterize the knee, hip skeletal keypoints that characterize the hip, and ankle skeletal keypoints that characterize the ankle when the user is running, whereby it can be determined that the knee skeletal keypoints, the hip skeletal keypoints, and the ankle skeletal keypoints form a keypoint relationship with respect to the body part of the knee. The step can consider the key point association relation of the relative knee as a space division item, and extract the related key point coordinate submatrix from the key point coordinate matrix based on the space division item.

It should be noted that, the lower body of the human body is mainly two legs, and each leg has a body part such as a hip, a knee, and an ankle, and the body parts referred to in this embodiment are equivalent to a part on one leg, such as a corresponding left knee, left ankle, or left hip on the left leg, and a corresponding right knee, right ankle, or right hip on the right leg, which are respectively regarded as one body part.

Considering that the body parts associated with the running of the user mainly comprise knees and ankles, which is equivalent to the body parts with four key point association relations to be determined, namely, the step can consider that the four key point association relations exist, and the corresponding key point coordinate submatrix can be extracted from the key point coordinate matrix based on each key point association relation. And the number of matrices determined may correspond to the number of body parts associated with the user running.

Fig. 2a shows a flowchart for implementing determination of a key point coordinate submatrix in the second embodiment of the present application, as shown in fig. 2a, in the second embodiment, the key point coordinate submatrix for obtaining the number of the determined matrices is specifically expressed as:

s2051, determining the associated body parts associated with the lower body of the user during running, and determining the number of the determined associated body parts as the matrix number.

The step realizes the determination of the number of the associated body parts and the matrix. The determined relevant body parts preferably exist around, namely, the left knee, the right knee, the left ankle and the right ankle, and further, the embodiment preferably determines that the number of the matrices has a value of 4.

S2052, determining an associated skeleton point set forming each associated body part, wherein the associated skeleton point set comprises all associated skeleton points with key point association relations relative to the associated body parts.

For example, when the associated body part is a right ankle, the skeletal key points having a key point association relationship with respect to the right ankle may be a right ankle skeletal point, a right big toe skeletal point, a right small toe skeletal point, and a right heel skeletal point. Thus, the associated skeletal points with respect to the right ankle are concentrated to include the right ankle skeletal point, the right big toe skeletal point, the right small toe skeletal point, and the right heel skeletal point as the associated skeletal points.

Fig. 2b is a diagram illustrating an example of a set of associated skeletal points included in each associated body part in a method for analyzing a running posture of a user according to a second embodiment of the present application. As shown in fig. 2b, the first region 21 includes a left knee and associated body parts including linkage parts (e.g., left hip and left ankle); the second region 22 includes the associated body parts of the right knee and the linkage parts (e.g., the right hip and the right ankle); the third region 23 includes the left ankle and the associated body part of the linkage (left heel, left big toe, and left small toe, if any); the fourth region 24 includes the right ankle and the associated body part of the interlocking parts (right heel, right big toe and right small toe, if any)

S2053, extracting column information corresponding to each associated bone point in the associated bone point set from the key point coordinate matrix aiming at each associated bone point set, and forming a corresponding key point coordinate submatrix based on each column information.

An associated set of skeletal points for each associated body part may be obtained through S2052 described above. The step provides a specific implementation of the key point coordinate submatrix, and the key point coordinate submatrix is mainly formed by extracting each associated bone point included in the associated bone point set from the key point coordinate matrix. The above description can be used to know that the row information in the coordinate matrix of the key points is the coordinate information of each bone key point contained in one lower half image of the human body, and the column information is the coordinate information of the bone key point with the same attribute in different lower half images of the human body.

In this embodiment, with respect to an associated bone point, the step may find its column index in the key point coordinate matrix, and extract the column information with the column index. For each associated bone point in the associated bone point set, corresponding column information can be extracted through the step, and based on the column information, corresponding key point coordinate submatrices can be formed by arranging according to a column index sequence. Thereby realizing the space division of the key point coordinate matrix.

S2054, obtaining the key point coordinate submatrices of the matrix number.

In this embodiment, since the relevant body parts of the user during running include at least the left knee, the right knee, the left ankle and the right ankle, the number of key point coordinate sub-matrices formed after the above-mentioned spatial division of the key point coordinate matrices can be respectively expressed as: a first keypoint coordinate sub-matrix characterizing the left knee, a second keypoint coordinate sub-matrix characterizing the right knee, a third keypoint coordinate sub-matrix characterizing the left ankle, and a fourth keypoint coordinate sub-matrix characterizing the right ankle.

S206, aiming at each key point coordinate submatrix, inputting the key point coordinate submatrix serving as input data into a corresponding neural network model, obtaining semantic feature vectors corresponding to each associated bone point included in the key point coordinate submatrix, and recording the semantic feature vectors as semantic feature information.

In this embodiment, for each key point coordinate sub-matrix, there is a neural network model for performing semantic feature extraction processing, and in this step, each key point coordinate sub-matrix may be respectively regarded as input data to a corresponding neural network model, and by processing of the corresponding neural network model, a semantic feature vector may be obtained with respect to each associated bone point in the key point coordinate sub-matrix. In this embodiment, the set of semantic feature vectors determined by a corresponding key point coordinate sub-matrix is regarded as semantic feature information corresponding to the key point coordinate sub-matrix, where the neural network models corresponding to different key point coordinate sub-matrices may be different in network structure.

In this embodiment, the adaptive graph rolling network model may be preferably a neural network model corresponding to each of the key point coordinate sub-matrices; the key point of extracting semantic features by the self-adaptive graph convolution network model is that: the object facing the network model is each associated skeleton point, and the process of extracting the semantic features of each associated skeleton point is realized by relying on the relationship between the associated skeleton point and other associated skeleton points in the key point coordinate submatrix.

However, the relation between any two associated skeleton points can be represented by an adjacent matrix with a determined row and column size, that is, the adjacent matrix is adopted to characterize the relation between the associated skeleton points in the operation of the adaptive graph rolling network model, and the semantic feature vector of each associated skeleton point is determined by determining the element value in each adjacent matrix.

In this embodiment, for an adjacency matrix with a determined rank size, it is necessary to determine the values of each element included, and the determination manner adopted may be considered from different dimensions, where the dimensions considered may be: physical connection relation of human bones, sample data for training of the adaptive graph rolling neural network model, and current input data as the adaptive graph rolling neural network model.

In particular, the element values in the adjacency matrix can be characterized and determined at least preferably by the following three ways:

1) The element values are characterized based on the physical connection of the human bones.

For example, if there is a physical connection between the associated skeleton point i and the associated skeleton point j, the element value corresponding to the associated skeleton point i and the associated skeleton point j may be determined to be 1, and if there is no physical connection between the associated skeleton point i and the associated skeleton point j, the element value may be determined to be 0.

2) The element values are characterized based on node association strengths determined after training and learning of the sample data in the training set.

In this way, the element values in the adjacency matrix are mainly considered to be learnable in the training process of the adaptive graph convolution neural network, even if two associated skeleton points are not physically connected, as long as the two associated skeleton points are associated, the association between the two associated skeleton points can be represented by association strength, and the determined association strength can be used as the element value. Since this approach is learnable, the element values in the adjacency matrix are related to the sample data of the training set and the entire learning process.

3) The element values are characterized based on the input data and independent of the key point features of the training set data.

It will be appreciated that in this manner the values of the elements in the adjacency matrix do not need to take into account whether or not there is a physical connection between two associated bone points. The method mainly relates to the determination of characteristic data of the related skeleton points input in the iterative processing of the graph convolution neural network, and the determination of element values in the mode mainly depends on the key point characteristics determined under one iteration.

In this embodiment, in the implementation of obtaining, by using a corresponding adaptive graph convolution network model, semantic feature information of each associated skeleton key point included in each key point coordinate sub-matrix, determining, by using the three ways, an adjacent matrix representing a relationship between any two associated skeleton key points, respectively, and performing weighting processing on the adjacent matrices determined by using the three ways; and then, representing the relation of any two associated skeleton key points by adopting an adjacent matrix formed after weighting, and finally perfecting the semantic features of each associated skeleton key point according to the adjacent matrix of each two associated skeleton key points so that the extracted semantic feature vector effectively contains the information required by the running gesture.

Meanwhile, in this embodiment, considering that the number of associated skeleton points included in the key point coordinate submatrices of different associated body parts is different, when the adaptive graph rolling network model is set, the size of the adjacent matrix adopted in the operation in the corresponding adaptive graph rolling network model can be set to be different.

Preferably, the first adaptive graph rolling network model corresponding to the first key point coordinate sub-matrix and the second adaptive graph rolling network model corresponding to the second key point coordinate sub-matrix may be set to 3*3 in the row and column size of the adjacent matrix adopted in the operation; meanwhile, the third adaptive graph rolling network model corresponding to the third key point coordinate sub-matrix and the fourth adaptive graph rolling network model corresponding to the fourth key point coordinate sub-matrix can be set to 5*5 in the row and column size of the adjacent matrix adopted in the operation.

The following steps S207 and S208 correspond to the running posture analysis implementation steps of the body part associated with the running of the user in this embodiment.

S207, inputting corresponding semantic feature information into a pre-trained gesture classifier aiming at each key point coordinate sub-matrix to obtain a matching score value of a body part related to the key point coordinate sub-matrix relative to each running gesture.

In this embodiment, the gesture classifier includes at least: the system comprises a cascade full-connection layer, an activation layer, a batch normalization layer and a logistic regression layer, wherein semantic feature information comprises semantic feature vectors of all associated skeleton points determined by the key point coordinate submatrix through the steps.

Based on the analysis of the semantic features, a matching score value of the body part associated with the key point coordinate sub-matrix relative to each running gesture can be determined, wherein the running gesture can comprise normal running, eversion running and inward buckling running.

S208, determining the running gesture corresponding to the highest matching score value as the current running gesture of the corresponding body part.

Based on the above embodiment, the running gesture with the highest matching score value can be selected as the target running gesture of the body part associated with the key point coordinate sub-matrix in this step.

The second embodiment of the present application provides a method for analyzing a running gesture of a user, which embodies a framework process of a key point coordinate matrix, a determining process of semantic feature information and an analysis process of the running gesture, and the method provided in the present embodiment, unlike the gesture analysis implemented by the existing method based on the judgment of the angle threshold value of the skeletal points extracted from the large vibration image, specifically adopts a method for implementing the dynamic analysis of the running gesture by using the skeletal key points obtained from multiple frames as input data, effectively improves the robustness and accuracy of the running gesture analysis function; meanwhile, unlike the prior method that feature information extracted from key points of all bones of a human body is adopted to determine a running gesture classification result, the obtained bone points are specifically formed into a large key point coordinate matrix and are divided into a plurality of submatrices through the thought of body part association, semantic feature extraction is respectively carried out, adverse effects of other invalid bone points on the gesture classification result in running gesture analysis are effectively reduced, flexibility of association inside the body part during running is enhanced, and accuracy of the running classification result is further improved.

Example III

Fig. 3 is a block diagram of a user running posture analysis device according to a third embodiment of the present application, where the device is suitable for analyzing a posture of a user running on a treadmill. The device may be implemented in hardware and/or software and is typically integrated into a treadmill. As shown in fig. 3, the apparatus includes: an image extraction module 31, a matrix determination module 32, a feature determination module 33, and a pose determination module 34.

An image extraction module 31 for extracting a lower body image of the human body of a set frame number from the running video of the user captured by the camera;

a matrix determining module 32, configured to construct a key point coordinate matrix according to the bone key points determined from each of the lower body images of the human body;

the feature determining module 33 is configured to determine semantic feature information corresponding to each skeletal key point in the lower body of the human body according to the key point coordinate matrix;

the gesture determining module 34 is configured to determine a running gesture of the body part associated with the user running through analysis of each of the semantic feature information.

According to the user running gesture analysis device provided by the third embodiment, multiple frames of human body lower body images in the video stream are used as input of gesture analysis, so that a more robust gesture analysis result in the dynamic running process of a user is ensured, meanwhile, bone key points are detected on the multiple frames of human body lower body images in the gesture analysis implementation, the probability of key point prediction errors is effectively reduced, and the accuracy and the effectiveness of the gesture analysis result are improved.

Further, the matrix determination module 32 may specifically be configured to:

Further, the feature determination module 33 may include:

the submatrix determining unit is used for carrying out space division on the key point coordinate matrixes according to the determined key point association relation to obtain key point coordinate submatrices with the determined number of matrixes;

the information determining unit is used for inputting the key point coordinate submatrices serving as input data into the corresponding neural network model aiming at each key point coordinate submatrix, obtaining semantic feature vectors corresponding to each associated bone point included in the key point coordinate submatrices, and recording the semantic feature vectors as semantic feature information.

Further, the submatrix determination unit may be specifically configured to:

and obtaining the key point coordinate submatrices of the matrix number.

Further, the value of the number of the matrixes is 4;

On the basis of the optimization, the neural network model corresponding to each key point coordinate sub-matrix is an adaptive graph convolution network model;

characterizing element values based on physical connections of human bones;

and the size of an adjacent matrix adopted in the operation is 5*5.

Further, the gesture determination module 34 may be specifically configured to:

Based on the optimization, the gesture classifier at least comprises: a cascade full connection layer, an activation layer, a batch normalization layer and a logistic regression layer.

Example IV

Fig. 4 is a schematic structural diagram of a treadmill according to a fourth embodiment of the present application. The running machine comprises: the intelligent interaction device comprises a base 41, a stand column 42, an intelligent interaction device 43 arranged at the top end of the stand column 42, a memory, one or more controllers and a camera 44 arranged at a designated position of the stand column. The number of controllers in the treadmill may be one or more, and at the same time, the number of memories in the treadmill may be one or more, and in this embodiment, one memory and one controller are taken as an example, respectively, but the specific locations of the memories and the controllers are not shown in fig. 4 of this embodiment.

Specifically, the main body of the running machine consists of the base 41, the upright 42 and the intelligent interaction device 43, it may be preferable to consider that the intelligent interaction device 43 has a display screen, an input module and an output module, and the controller and the memory may also be disposed in the intelligent interaction device 43 or may be disposed in the base, where the controller, the memory, the display screen, the input module and the output module included in the running machine may be connected by a bus or other manners, and fig. 4 is an example of connection through the bus. In addition, a camera 44 is provided on the upright for capturing images of the user while running, and the camera 44 may also be connected to the controller and memory via a bus or other connection.

The memory is used as a computer readable storage medium for storing software programs, computer executable programs and modules corresponding to the running machine according to any embodiment of the present invention (e.g., the image extraction module 31, the matrix determination module 32, the feature determination module 33 and the posture determination module 34 in the running posture analysis device of the user). The memory may mainly include a memory program area and a memory data area, wherein the memory program area may store an operating system, at least one application program required for a function; the storage data area may store data created according to the use of the device, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some examples, the memory may further include memory remotely located with respect to the controller, the remote memory being connectable to the device through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The display screen may be a display screen with touch functionality, which may be a capacitive screen, an electromagnetic screen, or an infrared screen. Generally, the display screen is used for displaying data according to an instruction of the controller, and is also used for receiving a touch operation applied to the display screen and transmitting a corresponding signal to the controller or other devices.

The input module may be used to receive input digital or character information and to generate key signal inputs related to user settings and function control of the display device, as well as cameras for capturing images and pickup devices for capturing audio data. The output module may include an audio device such as a speaker. It should be noted that the specific composition of the input module and the output module may be set according to the actual situation.

The controller executes various functional applications of the device and data processing by running software programs, instructions and modules stored in the memory, namely, the user running gesture analysis method is realized.

The running machine provided by the above can be used for executing the running gesture analysis method of the user provided by any embodiment, and has corresponding functions and beneficial effects.

Example five

A fifth embodiment of the present invention also provides a storage medium containing computer-executable instructions, which when executed by a computer controller, are for performing a user running posture analysis method, comprising:

Of course, the storage medium containing the computer executable instructions provided by the embodiment of the invention is not limited to the operation of the user running gesture analysis method described above, and can also execute the related operation in the user running gesture analysis method provided by any embodiment of the invention, and has corresponding functions and beneficial effects.

From the above description of embodiments, it will be clear to a person skilled in the art that the present application may be implemented by means of software and necessary general purpose hardware, but of course also by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a FLASH Memory (FLASH), a hard disk, or an optical disk of a computer, etc., including several instructions for causing a treadmill (which may be a robot, a personal computer, a server, or a network device, etc.) to perform the user running posture analysis method according to any embodiment of the present application. It should be noted that, in the above-mentioned user running gesture analysis apparatus, each unit and module included are only divided according to the functional logic, but not limited to the above-mentioned division, so long as the corresponding functions can be implemented; in addition, the specific names of the functional units are also only for distinguishing from each other, and are not used to limit the protection scope of the present application.

It is to be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.

Note that the above is only a preferred embodiment of the present application and the technical principle applied. Those skilled in the art will appreciate that the present application is not limited to the particular embodiments described herein, but is capable of numerous obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the present application. Therefore, while the present application has been described in connection with the above embodiments, the present application is not limited to the above embodiments, but may include many other equivalent embodiments without departing from the spirit of the present application, the scope of which is defined by the scope of the appended claims.

Claims

1. A method for analyzing a running posture of a user, comprising:

determining the running posture of the body part related to the running of the user through analysis of the semantic feature information;

the determining semantic feature information corresponding to each bone key point in the lower body of the human body according to the key point coordinate matrix comprises the following steps:

aiming at each key point coordinate submatrix, inputting the key point coordinate submatrix serving as input data into a corresponding neural network model, obtaining semantic feature vectors corresponding to each associated skeleton point included in the key point coordinate submatrix, and marking the semantic feature vectors as semantic feature information;

the key point association relationship is determined by the skeletal key points associated with the body parts involved in running of the user;

The space division is carried out on the key point coordinate matrixes according to the determined key point association relation to obtain key point coordinate submatrices with the determined matrix number, and the method comprises the following steps:

obtaining a key point coordinate submatrix of the matrix number;

the row information in the key point coordinate matrix is the coordinate information of each bone key point contained in one human body lower body image, and the column information is the coordinate information of the bone key points with the same attribute in different human body lower body images.

2. The method of claim 1, wherein constructing a key point coordinate matrix from bone key points determined from each of the lower body images of the human body comprises:

3. The method according to claim 1, wherein the number of matrices has a value of 4;

4. A method according to claim 3, wherein the neural network model corresponding to each of the key point coordinate sub-matrices is an adaptive graph rolling network model;

characterizing element values based on physical connections of human bones;

the element values are characterized based on the input data and independent of the key point features of the training set data.

5. The method of claim 4, wherein the step of determining the position of the first electrode is performed,

a first self-adaptive graph rolling network model corresponding to the first key point coordinate submatrix and a second self-adaptive graph rolling network model corresponding to the second key point coordinate submatrix, wherein the row and column sizes of adjacent matrixes adopted in operation are set as 3*3;

and setting the row and column sizes of adjacent matrixes adopted in the operation to 5*5 in the third self-adaptive graph rolling network model corresponding to the third key point coordinate submatrix and the fourth self-adaptive graph rolling network model corresponding to the fourth key point coordinate submatrix.

6. The method according to any one of claims 1-5, wherein determining the running posture of the body part associated with the user running through analysis of each of the semantic feature information comprises:

7. The method of claim 6, wherein the gesture classifier comprises at least: a cascade full connection layer, an activation layer, a batch normalization layer and a logistic regression layer.

8. A method for analyzing a running posture of a user, comprising:

the neural network model corresponding to each key point coordinate sub-matrix is an adaptive graph convolution network model;

characterizing element values based on physical connections of human bones;

9. A user running posture analysis apparatus, comprising:

the gesture determining module is used for determining the running gesture of the body part related to the running of the user through analysis of the semantic feature information;

the feature determination module includes:

The information determining unit is used for inputting the key point coordinate submatrices serving as input data into the corresponding neural network model for each key point coordinate submatrix, obtaining semantic feature vectors corresponding to each associated bone point included in the key point coordinate submatrices, and recording the semantic feature vectors as semantic feature information;

the submatrix determining unit is specifically configured to:

obtaining a key point coordinate submatrix of the matrix number;

10. A user running posture analysis apparatus, comprising:

the feature determination module includes:

characterizing element values based on physical connections of human bones;

11. A treadmill, comprising: base, stand, set up in the intelligent interaction device on stand top, its characterized in that still includes: the storage, one or more controllers and the camera are arranged at the appointed position of the upright post;

the camera is used for capturing video streams when a user runs;

the memory is used for storing one or more programs;

when the one or more programs are executed by the one or more controllers, the one or more controllers are caused to implement the methods of any of claims 1-8.

12. A storage medium containing computer executable instructions which, when executed by a computer controller, are for performing the method of any of claims 1-8.