WO2022110453A1

WO2022110453A1 - Running posture recognition method and apparatus, and computer device

Info

Publication number: WO2022110453A1
Application number: PCT/CN2020/139929
Authority: WO
Inventors: 栗晓燕; 薇静初
Original assignee: 广州源动智慧体育科技有限公司
Priority date: 2020-11-26
Filing date: 2020-12-28
Publication date: 2022-06-02
Also published as: CN114627546A

Abstract

A running posture recognition method and apparatus, and a computer device. The method comprises: collecting a lower body part image when a user is running (S1); calling an underlying skeleton extraction algorithm to parse the lower body part image, so as to obtain a plurality of key point thermodynamic diagrams, a single key point thermodynamic diagram corresponding to a single key point (S2); decoding the key point thermodynamic diagrams, so as to obtain key point coordinates corresponding to the key point thermodynamic diagrams (S3); and generating a current key point pair according to the key point coordinates, comparing the current key point pair with a pre-constructed reference key point pair, and performing analysis to obtain a running posture of the user (S4). By means of the method, a collected lower body part image comprises a foot image of a user, so that key point thermodynamic diagrams that comprise the feet of the user are obtained by performing parsing by means of an underlying skeleton extraction algorithm, the prediction of key points of the feet when the user is running is added, and analysis can be then performed to obtain a foot posture of the user, such that a finally recognized running posture is more comprehensive and accurate.

Description

Running posture recognition method, device and computer equipment

This application claims the priority of the Chinese patent application with the application number 202011349062.5 and the application title "Method, Apparatus and Computer Equipment for Recognition of Running Posture" filed with the China Patent Office on November 26, 2020, the entire contents of which are incorporated herein by reference Applying.

technical field

The present application relates to the technical field of gesture recognition, and in particular, to a method, device and computer equipment for recognition of running gestures.

Background technique

As the treadmill enters the public's field of vision and becomes a convenient choice for more and more people to exercise, the correct running posture plays an important role in the user's exercise effect and body shaping. Existing gesture recognition for users running on a treadmill usually extracts the key points of the human body with the help of computer vision, and further analyzes the human posture formed by the key points (such as outputting images through a low-cost common RGB camera, and analyzing the image Extract the key points of the human skeleton; or capture the depth information of the human body through three-dimensional sensors such as Kinect (somatosensory sensor), so as to further model the human body and obtain posture information). However, such gesture recognition methods usually cannot accurately extract the skeleton of the lower body through the only image information of the lower body, and the recognition accuracy of the user's running posture is low.

For the above problems, no effective solution has been proposed yet.

SUMMARY OF THE INVENTION

At least some embodiments of the present application provide a running posture recognition method, apparatus and computer equipment, to at least solve the existing technical problem of low posture recognition accuracy when a user is running on a treadmill.

According to an aspect of one of the embodiments of the present application, a method for identifying a running posture is provided, which includes: collecting a lower body image of a user when running, the lower body image including a foot image; calling an underlying skeleton extraction algorithm to parse the lower body image, and obtaining Multiple key point heatmaps, a single keypoint heatmap corresponds to a single keypoint; decode each keypoint heatmap to obtain the keypoint coordinates corresponding to each keypoint heatmap; generate the current keypoint pair according to the keypoint coordinates , and compare the current key point pair with the pre-built benchmark key point pair, and analyze the running posture of the user.

According to another aspect of one of the embodiments of the present application, there is also provided a running posture recognition device, including: a collection module, configured to collect images of the lower body of the user when running, and the images of the lower body include images of feet; an analysis module, configured as Call the underlying skeleton extraction algorithm to parse the lower body image, and obtain multiple key point heat maps. A single key point heat map corresponds to a single key point; the decoding module is set to decode each key point heat map and obtain each key point heat map. The corresponding key point coordinates respectively; the analysis module is set to generate the current key point pair according to the coordinates of each key point, and compare the current key point pair with the pre-built benchmark key point pair, and analyze the running posture of the user.

Optionally, the parsing module includes: an extraction unit, which is set to input the lower body image into an encoder for encoding, and extracts to obtain semantic features. The encoder is composed of multiple depthwise separable convolutions and one hole convolution; the prediction unit is set to Through the superposition of convolution layers, multiple rough key point heat maps are predicted from semantic features; the refining unit is set to refine each rough key point heat map to obtain each key point heat map.

Optionally, the refining unit includes: extracting subunits, which are set to pool the rough key point heatmap through the hollow space pyramid, and extract the secondary keypoint heatmap; refine the subunit, which is set to be based on global spatial attention The mechanism refines the secondary key point heat map to obtain the key point heat map.

Optionally, the decoding module includes: a first calculation unit, set to use the wave peak point method to calculate the heatmap of each key point respectively, to obtain the Gauss point peak value corresponding to the heatmap of each key point; a judgment unit, set to judge Whether the corresponding Gauss point peak value of each key point heat map is one; mark unit, set to if the corresponding Gauss point peak value of each key point heat map is one, the coordinate of the Gauss point peak value is taken as the corresponding key point coordinate .

Optionally, the decoding module further includes: a screening unit, configured to select a first Gaussian point peak and a second Gaussian point peak from the plurality of Gaussian point peaks if there are multiple Gaussian point peaks corresponding to each of the key point heatmaps. Point peak, the first Gauss point peak is the largest Gauss point peak, and the second Gauss point peak is only smaller than the first Gauss point peak; the second calculation unit is set to calculate the coordinates of the first Gauss point peak and the second Gauss point peak. The distance between the coordinates; the adjustment unit is set to offset and adjust the coordinate values of the first Gaussian point peak in the X direction and the Y direction according to the distance, and obtain the key point coordinates corresponding to the key point heat map.

Optionally, the coordinates of each key point include leg shape key point coordinates and foot shape key point coordinates, the reference key point pair includes a preset leg shape key point pair and a preset foot shape key point pair, and the current key point pair includes the current leg shape. The key point pair and the key point pair of the current foot shape, the running posture includes the running leg shape and the running foot shape, and the analysis module includes: a construction unit, which is set to construct the current leg shape key point pair according to the coordinates of each leg shape key point, and according to each leg shape key point pair. The coordinates of the key points of the foot shape construct the current foot shape key point pair; the comparison unit is set to compare the current leg shape key point pair with the preset leg shape key point pair, and calculate the first included angle information in the vertical direction. ; Compare the key point pair of the current foot shape with the preset key point pair of the leg shape, and calculate the second included angle information in the vertical direction; the analysis unit is set to be based on the first included angle information and the first angle threshold. The running leg shape is obtained by analysis, and the running foot shape is obtained by analyzing the second included angle information and the second angle threshold.

Optionally, the preset leg shape key point pair includes a preset left thigh key point pair, a preset right thigh key point pair, a preset left calf key point pair, and a preset right calf key point pair; the current leg shape key point pair; Including the current left thigh key point pair, the current right thigh key point pair, the current left calf key point pair and the current right calf key point pair; the first included angle information includes the current left thigh key point pair and the preset left thigh key point pair The included angle a1, the included angle between the current right thigh key point pair and the preset right thigh key point pair a2, the included angle between the current left calf key point pair and the preset left calf key point pair a3, the current right The included angle a4 between the pair of key points of the lower leg and the pair of preset key points of the right lower leg; the first angle threshold is e; the analysis unit includes: a first analysis subunit, set to be when a1>e°, a2>-e°, When a3>-e° and a4>e°, it is determined that the running leg is an X-shaped leg.

Optionally, the preset foot type key point pair includes a preset left sole key point pair and a preset right sole key point pair; the current foot type key point pair includes the current left sole key point pair and the current right sole key point pair; The second included angle information includes the included angle b1 between the current left sole key point pair and the preset left sole key point pair, and the included angle b2 between the current right sole key point pair and the preset right sole key point pair; the second angle The threshold is β; the analysis unit further includes: a second analysis subunit, configured to determine that the running foot type is inner eight feet when b1>β° and b2>-β°.

Optionally, the identification device further includes: an output module configured to generate running posture analysis information according to the current leg shape key point pair, the current foot shape key point pair, the running leg shape and the running foot shape, and output it to the display interface in real time.

According to another aspect of one of the embodiments of the present application, a computer device is further provided, including a memory and a processor, a computer program is stored in the memory, and the processor implements the steps of any one of the above methods when executing the computer program.

According to another aspect of one of the embodiments of the present application, there is also provided a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any one of the above methods are implemented.

In at least some embodiments of the present application, a treadmill is used to collect images of the user's lower body while running, and then an underlying skeleton extraction algorithm is invoked to parse the lower body images, thereby obtaining multiple key point heatmaps. Among them, a single keypoint heatmap corresponds to a single keypoint. The treadmill decodes the heat map of each key point, and obtains the coordinates of the key points corresponding to the heat map of each key point. Finally, the treadmill generates the current key point pair according to the coordinates of each key point, compares the current key point pair with the pre-built benchmark key point pair, and analyzes the running posture of the user. The lower body image collected by the treadmill in this application includes the user's foot image, so that the underlying skeleton extraction algorithm is used to parse and obtain a heat map of key points including the user's feet, which increases the prediction of the key points of the user's feet when running, and further The user's foot posture can be obtained by analysis, so that the final recognized running posture is more comprehensive and accurate.

Description of drawings

The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The schematic embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation of the present application. In the attached image:

1 is a schematic diagram of steps of a method for recognizing running posture in an embodiment of the present application;

2 is a block diagram of the overall structure of a running posture recognition device in an embodiment of the present application;

3a-3d are schematic structural diagrams of key point pairs in an embodiment of the present application;

FIG. 4 is a schematic structural block diagram of a computer device according to an embodiment of the present application.

The realization, functional characteristics and advantages of the purpose of the present application will be further described with reference to the accompanying drawings in conjunction with the embodiments.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application.

1, 3a and 3b, an embodiment of the present application provides a method for identifying a running posture, including:

S1: collect the lower body image of the user when running, and the lower body image includes the foot image;

S2: call the underlying skeleton extraction algorithm to analyze the lower body image, and obtain multiple key point heat maps, and a single key point heat map corresponds to a single key point;

S3: Decode each key point heat map to obtain the key point coordinates corresponding to each key point heat map respectively;

S4: Generate a current key point pair according to the coordinates of each key point, compare the current key point pair with a pre-built reference key point pair, and analyze the running posture of the user.

In this embodiment, the running posture recognition method is specifically applied to a treadmill, and the control system of the treadmill (hereinafter referred to as the system) captures the lower body image of the user when running in real time through the camera, and the lower body image covers the hip, knee, ankle, Finally, the lower body of the human body to the toes. The system calls the underlying skeleton extraction algorithm to parse the captured lower body image, thereby obtaining multiple key point heat maps, of which a single key point heat map corresponds to a single key point (for example, the key point is the left hip bone corresponding to the key point heat map A). , the key point is the heat map of the key point corresponding to the right hip bone B). Specifically, in order to ensure real-time performance, the system inputs the lower body image into a lightweight encoder for encoding, and extracts the corresponding semantic features. Then, through simple convolutional layer stacking, multiple rough keypoint heatmaps are predicted from the extracted semantic features (each rough keypoint heatmap also corresponds to a keypoint). The system refines each rough key point heat map by cascading empty spatial pyramid pooling and adding a global spatial attention mechanism, and obtains the corresponding accurate key point heat map. The system uses the wave peak point method to calculate the heatmaps of each key point respectively, and obtains the corresponding Gauss point peak of each key point heatmap. The system judges whether each key point heatmap has a Gaussian point peak. There is a Gauss point peak in the key point heat map, and the coordinates of the Gauss point peak are the coordinates of the key point corresponding to the corresponding key point heat map in the original image (ie, the lower body image). Therefore, the system takes the coordinates of the peaks of each Gaussian point as the coordinates of the corresponding key points (ie, the coordinates of the key points), and the coordinates of the key points corresponding to each key point constitute the key point coordinate information. The system generates the current key point pair according to the coordinates of each key point, and combines the current key point pair and the pre-built benchmark key point pair (the benchmark key point pair is pre-built by the developer and stored in the internal database of the treadmill as benchmark information) Compare and analyze the running posture of the user. Specifically, the key point coordinate information includes a plurality of leg shape key point coordinates (left hip bone key point coordinates, right hip bone key point coordinates, left knee key point coordinates, right knee key point coordinates, left ankle key point coordinates and right ankle key point coordinates key point coordinates) and multiple foot shape key point coordinates (left ankle key point coordinates, right ankle key point coordinates, left toe key point coordinates and right toe key point coordinates); Preset leg type key point pair, preset foot type key point pair corresponding to normal foot type skeleton; current key point pair includes current foot type key point pair and current leg type key point pair, running posture includes running leg type and running Foot shape. The system builds the user's current leg key point pair according to the coordinates of each leg key point, that is, according to the left hip key point coordinate, the left knee key point coordinate and the left ankle key point coordinate to construct the user's left leg key point pair, and The shape of the user's current left leg skeleton corresponds; the leg shape key point pair of the user's right leg is constructed according to the coordinates of the right hip bone key point, the right knee key point coordinate and the right ankle key point coordinate, which corresponds to the user's current right leg skeleton shape. At the same time, the system also constructs the user's current foot type key point pair according to the coordinates of each foot type key point, that is, the user's left foot key point pair is constructed according to the left ankle key point coordinate and the left toe key point coordinate, which is consistent with the user's current left foot type key point pair. The shape of the foot skeleton corresponds; the key point pair of the user's right foot is constructed according to the coordinates of the right ankle key point and the right toe key point, which corresponds to the user's current right foot skeleton shape. The system compares the current leg shape key point pair with the corresponding preset leg shape key point pair, and calculates the first angle information between the two in the vertical direction (including the left thigh key point pair corresponding to the user's current left thigh skeleton). The included angle a1 between the preset left thigh key point corresponding to the normal left thigh skeleton, the included angle between the right thigh key point corresponding to the user's current right thigh skeleton and the preset right thigh key point corresponding to the normal right thigh skeleton a2, the angle between the left calf key point pair corresponding to the user's current left calf skeleton and the preset left calf key point corresponding to the normal left calf skeleton a3, the right calf key point pair corresponding to the user's current right calf skeleton and the corresponding normal right calf skeleton The included angle between the preset right calf key points of the calf skeleton a4); and compare the current foot shape key point pair with the preset leg shape key point pair, and calculate the second included angle between the two in the vertical direction Information (including the angle b1 between the left foot key point pair corresponding to the user's current left foot skeleton and the preset left foot key point corresponding to the normal left foot skeleton, the right foot key point pair corresponding to the user's current right foot skeleton and the corresponding normal The included angle between the preset right sole key points of the right sole skeleton b2). The system analyzes and obtains the user's current running leg shape according to the size relationship between the first included angle information and the first angle threshold; and analyzes and obtains the user's current running leg shape according to the size relationship between the second included angle information and the second angle threshold. Running feet. For example, when a1>e°, a2>-e°, a3>-e°, and a4>e°, the system determines that the user's current running leg is an X-shaped leg (as shown in Figure 3b); when a1(- When e°, e°), a2(-e°, e°), a3>-e°, and a4>e°, the system determines that the user's current running leg is an XO leg. The system integrates the running leg shape and the running foot shape to obtain the running posture of the user.

In the method for identifying a running posture provided in this embodiment, the system collects the lower body image of the user while running, and then invokes the underlying skeleton extraction algorithm to analyze the lower body image, thereby obtaining multiple key point heat maps. Among them, a single keypoint heatmap corresponds to a single keypoint. The system decodes the heat map of each key point, and obtains the coordinates of the key points corresponding to the heat map of each key point. Finally, the system generates the current key point pair according to the coordinates of each key point, compares the current key point pair with the pre-built benchmark key point pair, and analyzes the running posture of the user. The lower body image collected by the system in the present application includes the user's foot image, so that the heat map of key points including the user's feet is obtained through the analysis of the underlying skeleton extraction algorithm, which increases the prediction of the key points of the user's feet when running, and further can The user's foot posture is obtained by analysis, so that the final recognized running posture is more comprehensive and accurate.

Optionally, the steps of invoking the underlying skeleton extraction algorithm to parse the lower body image to obtain multiple key point heatmaps, including:

S201: input the lower body image into the encoder for encoding, and extract the semantic features, and the encoder is composed of a plurality of depthwise separable convolutions and a hollow convolution;

S202: By stacking convolutional layers, multiple rough key point heatmaps are predicted from the semantic features;

S203: Refine the heat map of each rough key point to obtain a heat map of each key point.

In this embodiment, in order to ensure the real-time recognition of the user's running posture, the system inputs the captured image of the lower body into a lightweight encoder for encoding. Among them, the encoder is composed of multiple depthwise separable convolutions and an atrous convolution, which is encoded through the above convolutional layers, and then obtains the corresponding semantic features from the lower body image (semantic features can be learned from the network finally from the included Semantic information of the human body is captured in pictures with complex backgrounds). At the initial stage of each keypoint heatmap generation, the system predicts a rough keypoint heatmap from the semantic features extracted in the previous step by simply stacking convolutional layers. In order to improve the accuracy of the key point heat map, the system first pools the rough key point heat map through the empty space pyramid, and extracts the secondary key point heat map. Then, the secondary keypoint heatmap is refined based on the global spatial attention mechanism to obtain an accurate keypoint heatmap. The encoder of this embodiment is composed of multiple depthwise separable convolutions and one atrous convolution, and is a lightweight encoder, which can improve the processing speed of information. Moreover, by performing empty spatial pyramid pooling on the rough keypoint heatmap and a global-based spatial attention mechanism, the accuracy of the final keypoint heatmap can be improved.

Optionally, in the step of refining each rough key point heat map to obtain each key point heat map, the refining step of a single rough key point heat map includes:

S2031: Pooling the rough key point heat map through the empty space pyramid, and extracting the secondary key point heat map;

S2032: Refine the secondary key point heat map based on the global spatial attention mechanism to obtain the key point heat map.

In this embodiment, the system refines the rough key point heat map based on the rough key point heat map. First, in order to make full use of the information in the rough key point heat map and ensure real-time performance, the system uses spatial pyramid pooling (ASPP ) are cascaded to obtain a larger receptive field, so as to extract more effective information, that is, the secondary key point heat map. In addition, in order to ignore the influence of interfering factors such as complex background, a global spatial attention mechanism is added at the end of each ASPP module, and the more effective information (secondary key point heat map) extracted is continuously refined to obtain A more accurate keypoint heatmap, namely the keypoint heatmap. In this embodiment, by cascading the empty space pyramid pooling, a larger receptive field can be obtained, thereby extracting more effective information. On this basis, a global-based spatial attention mechanism is added at the end of each empty spatial pyramid pooling module, which can refine the secondary key point heat map, thereby obtaining a more accurate key point heat map.

Optionally, the steps of decoding each key point heat map to obtain the key point coordinates corresponding to each key point heat map respectively include:

S301: Calculate the heatmap of each key point by using the wave peak point method to obtain the Gauss point peak value corresponding to the heatmap of each key point;

S302: Determine whether the corresponding Gaussian point peak value of each key point heat map is one;

S303: If each of the key point heat maps has one Gauss point peak corresponding to each, the coordinates where each Gauss point peak is located are taken as the corresponding key point coordinates.

In this embodiment, a single key point heat map corresponds to a single key point (for example, the key point is the heat map A corresponding to the key point of the left hip bone, and the key point is the heat map B corresponding to the key point of the right hip), and the system adopts the wave peak point selection method. , calculate each key point heat map separately, so as to obtain the corresponding Gauss point peaks of each key point heat map. Since there may be more than one Gauss point peak in the key point heat map, the system needs to first determine whether the corresponding Gauss point peak of each key point heat map is only one. If there is only one Gauss point peak in each key point heat map, the coordinates of the Gauss point peak are the coordinates of the key point corresponding to the key point heat map in the original image. Therefore, the system directly uses the coordinates of the peaks of each Gaussian point as the key point coordinates of the corresponding key points, so as to obtain the key point coordinate information by combination. In this embodiment, by identifying the number of Gauss point peaks corresponding to each key point heat map, in the case that the number of Gauss point peaks corresponding to a single key point heat map is only one, the selected Gauss point peak is located. The coordinates are the key point coordinates, which ensures the accuracy of the selected key point coordinates.

Optionally, after the step of judging whether the corresponding Gaussian point peak value of each key point heat map is one, the method further includes:

S304: If there are multiple Gauss point peaks corresponding to each of the key point heatmaps, screen the first Gauss point peak and the second Gauss point peak from the multiple Gauss point peaks, and the first Gauss point peak is the largest Gauss point peak , the peak value of the second Gaussian point is only smaller than the peak value of the first Gaussian point;

S305: calculate the distance between the coordinates of the first Gauss point peak and the coordinates of the second Gauss point peak;

S306: Offset and adjust the coordinate values of the first Gauss point peak in the X direction and the Y direction according to the distance, to obtain the key point coordinates corresponding to the key point heat map.

In this embodiment, if there are multiple Gauss point peaks in a certain key point heat map, the system traverses all the Gauss point peaks corresponding to the key point heat map, so as to filter out the first Gauss point peak with the maximum value, and save it There is a second Gaussian peak that overlaps the first Gaussian peak. The system calculates according to the coordinates between the two to obtain the distance between the coordinates of the first Gaussian peak and the coordinates of the second Gaussian peak, so that the system can use the coordinates of the second Gaussian peak to compare the coordinates of the first Gaussian peak. Fine tune for more precise keypoint coordinates. Specifically, according to the distance obtained by the above calculation, in the X direction, the system takes 1/4 of the offset in the X direction (that is, the distance between the peak coordinates of the first Gaussian point and the peak coordinates of the second Gaussian point). The value of X in the peak coordinate of the first Gaussian point; in the Y direction, take 1/4 of the offset in the Y direction to adjust the value of Y in the peak coordinate of the first Gaussian point. In this embodiment, the distance between the coordinates of the first Gaussian peak and the coordinates of the second Gaussian peak is calculated as the offset, and then the coordinates of the first Gaussian peak are adjusted based on the offset, so that the final obtained Keypoint coordinates are more accurate.

3a-3d, optionally, the coordinates of each key point include leg shape key point coordinates and foot shape key point coordinates, and the reference key point pair includes a preset leg shape key point pair and a preset foot shape key point pair, and the current key The point pair includes the current leg type key point pair and the current foot type key point pair, and the running posture includes the running leg type and the running foot type. The current key point pair is generated according to the coordinates of each key point, and the current key point pair and the pre-built benchmark The key points are compared, and the steps to obtain the user's running posture are analyzed, including:

S401: constructing a current leg type key point pair according to the coordinates of each leg type key point, and constructing a current foot type key point pair according to the coordinates of each foot type key point;

S402: Compare the current leg shape key point pair with the preset leg shape key point pair, and calculate the first included angle information in the vertical direction; pair the current foot shape key point pair with the preset leg shape key point pair Carry out the comparison, and calculate the second included angle information in the vertical direction;

S403: Analyzing and obtaining the running leg shape according to the first included angle information and the first angle threshold, and obtaining the running foot shape according to the second included angle information and the second angle threshold analysis.

Preferably, the preset leg shape key point pair includes a preset left thigh key point pair, a preset right thigh key point pair, a preset left calf key point pair, and a preset right calf key point pair;

The current leg shape key point pair includes the current left thigh key point pair, the current right thigh key point pair, the current left calf key point pair and the current right calf key point pair;

The first included angle information includes the included angle a1 between the current left thigh key point pair and the preset left thigh key point pair, the included angle a2 between the current right thigh key point pair and the preset right thigh key point pair, and the current left thigh key point pair. The angle a3 between the key point pair of the calf and the preset left calf key point pair, and the angle a4 between the current right calf key point pair and the preset right calf key point pair;

The first angle threshold is e;

According to the first included angle information and the first angle threshold analysis, the steps of obtaining the running leg shape include:

S4031: When a1>e°, a2>-e°, a3>-e°, and a4>e°, determine that the running leg type is an X-shaped leg.

Preferably, the preset foot shape key point pair includes a preset left sole key point pair and a preset right sole key point pair;

The current foot type key point pair includes the current left foot key point pair and the current right foot key point pair;

The second included angle information includes the included angle b1 between the current left sole key point pair and the preset left sole key point pair, and the included angle b2 between the current right sole key point pair and the preset right sole key point pair;

The second angle threshold is β;

The steps of obtaining the running foot shape by analyzing the second included angle information and the second angle threshold include:

S4032: When b1>β° and b2>-β°, determine that the running foot type is inner eight feet.

In this embodiment, the key point coordinate information includes a plurality of leg shape key point coordinates (left hip key point coordinates, right hip key point coordinates, left knee key point coordinates, right knee key point coordinates, left ankle key point coordinates and right ankle keypoint coordinates) and multiple foot shape keypoint coordinates (left ankle keypoint coordinates, right ankle keypoint coordinates, left toe keypoint coordinates, and right toe keypoint coordinates); pre-built datum keypoint pairs include The preset leg shape key point pair corresponding to the leg skeleton (as shown in Figure 3a), the preset foot shape key point pair corresponding to the normal foot skeleton (as shown in Figure 3c); the current key point pair includes the current foot shape The key point pair and the key point pair of the current leg shape, the running posture includes the running leg shape and the running foot shape. The system builds the user's current leg key point pair according to the coordinates of each leg key point, that is, according to the left hip key point coordinate, the left knee key point coordinate and the left ankle key point coordinate to construct the user's left leg key point pair, and The shape of the user's current left leg skeleton corresponds; the leg shape key point pair of the user's right leg is constructed according to the coordinates of the right hip bone key point, the right knee key point coordinate and the right ankle key point coordinate, which corresponds to the user's current right leg skeleton shape. At the same time, the system also constructs the user's current foot type key point pair according to the coordinates of each foot type key point, that is, the user's left foot key point pair is constructed according to the left ankle key point coordinate and the left toe key point coordinate, which is consistent with the user's current left foot type key point pair. The shape of the foot skeleton corresponds; the key point pair of the user's right foot is constructed according to the coordinates of the right ankle key point and the right toe key point, which corresponds to the user's current right foot skeleton shape. The system compares the current leg shape key point pair with the corresponding preset leg shape key point pair, and calculates the first angle information between the two in the vertical direction (including the left thigh key point pair corresponding to the user's current left thigh skeleton). The included angle a1 between the preset left thigh key point corresponding to the normal left thigh skeleton, the included angle between the right thigh key point corresponding to the user's current right thigh skeleton and the preset right thigh key point corresponding to the normal right thigh skeleton a2, the angle between the left calf key point pair corresponding to the user's current left calf skeleton and the preset left calf key point corresponding to the normal left calf skeleton a3, the right calf key point pair corresponding to the user's current right calf skeleton and the corresponding normal right calf skeleton The included angle between the preset right calf key points of the calf skeleton a4); and compare the current foot shape key point pair with the preset leg shape key point pair, and calculate the second included angle between the two in the vertical direction Information (including the angle b1 between the left foot key point pair corresponding to the user's current left foot skeleton and the preset left foot key point corresponding to the normal left foot skeleton, the right foot key point pair corresponding to the user's current right foot skeleton and the corresponding normal The included angle between the preset right sole key points of the right sole skeleton b2). The system analyzes and obtains the user's current running leg shape according to the size relationship between the first included angle information and the first angle threshold; and analyzes and obtains the user's current running leg shape according to the size relationship between the second included angle information and the second angle threshold. Running feet. Specifically, the first angle threshold is e. When a1>e°, a2>-e°, a3>-e°, and a4>e°, the system determines that the user's current running leg is an X-shaped leg (as shown in Figure 3a shown); when a1>-e°, a2>e°, a3>e° and a4>-e°, the system determines that the user's current running leg is an X-shaped leg; when a1(-e°, e° ), a2(-e°, e°), a3>-e°, and a4>e°, the system determines that the user's current running leg is an XO leg; otherwise, it is determined to be a normal leg. The second angle threshold is β. When b1>β° and b2>-β°, the system determines that the user's current running foot type is the inner eight feet (as shown in Figure 3d); when b1>-β° and b2>β °, the system determines that the user's current running foot type is the outer eight feet; in other cases, it is determined to be a normal foot type. In this embodiment, the first angle information between the current leg shape key point pair and the preset leg shape key point pair is calculated and obtained, and the first angle information between the two is calculated based on the current foot shape key point pair and the preset leg shape key point pair. information about the second angle between them. Then, the first included angle information is compared with the first angle threshold, and the second included angle information is compared with the second angle threshold, so as to accurately identify the running posture of the user.

Optionally, generating a current key point pair according to the coordinates of each key point, comparing the current key point pair with a pre-built benchmark key point pair, and analyzing the steps of obtaining the running posture of the user, including:

S5: Generate running posture analysis information according to the current leg shape key point pair, the current foot shape key point pair, the running leg shape and the running foot shape, and output it to the display interface in real time.

In this embodiment, the running posture analysis information includes an image of the skeleton posture of the user's lower body when running and the running posture obtained by the above analysis. The system generates an image of the user's lower body skeleton posture when running (including leg image and foot image) according to the current leg shape key point pair and the current foot shape key point pair, and at the same time analyzes the running leg shape (such as normal leg shape or X-legs) and running feet (such as normal feet or outer eight feet) are used as text information, which are output to the display interface of the system (such as the display screen of the treadmill) in real time, so as to provide fitness users intuitively with running posture information. . This embodiment enables the user to adjust his running posture in real time through the running posture analysis information, thereby improving the fitness effect.

2, 3a, 3b, an embodiment of the present application also provides a running posture recognition device, including:

The acquisition module 1 is configured to collect the lower body image of the user when running, and the lower body image includes the foot image;

The parsing module 2 is set to call the underlying skeleton extraction algorithm to parse the lower body image, and obtain multiple key point heat maps, and a single key point heat map corresponds to a single key point;

The decoding module 3 is set to decode each key point heat map, and obtain the key point coordinates corresponding to each key point heat map respectively;

The analysis module 4 is configured to generate the current key point pair according to the coordinates of each key point, compare the current key point pair with the pre-built reference key point pair, and analyze the running posture of the user.

Optionally, parsing module 2, including:

The extraction unit is set to input the lower body image into the encoder for encoding, and extract the semantic features. The encoder is composed of multiple depthwise separable convolutions and one hole convolution;

The prediction unit, which is set to be superimposed by convolutional layers, predicts multiple rough keypoint heatmaps from semantic features;

The refining unit is set to refine the heat map of each rough key point to obtain the heat map of each key point.

Optional, refining unit, including:

Extract subunits, set to pool the rough key point heat map through the empty space pyramid, and extract the secondary key point heat map;

The refinement subunit is set to refine the secondary keypoint heatmap based on the global spatial attention mechanism to obtain the keypoint heatmap.

Optionally, decoding module 3, including:

The first calculation unit is set to use the wave peak point method to calculate the heat map of each key point respectively, and obtain the Gauss point peak value corresponding to the heat map of each key point;

The judgment unit is set to judge whether the corresponding Gauss point peak value of each key point heat map is one;

The marking unit is set so that if the corresponding Gauss point peak of each key point heat map is one, the coordinates of the Gauss point peak are taken as the corresponding key point coordinates.

Optionally, the decoding module 3 further includes:

Screening unit, set so that if there are multiple Gauss point peaks corresponding to each key point heatmap, the first Gauss point peak and the second Gauss point peak are selected from the multiple Gauss point peaks, and the first Gauss point peak is the largest. Gauss point peak, the second Gauss point peak is only smaller than the first Gauss point peak;

The second calculation unit is set to calculate the distance between the coordinates of the first Gaussian peak and the coordinates of the second Gaussian peak;

The adjustment unit is set to offset and adjust the coordinate values of the first Gauss point peak in the X direction and the Y direction according to the distance, so as to obtain the key point coordinates corresponding to the key point heat map.

3a-3d, optionally, the coordinates of each key point include leg shape key point coordinates and foot shape key point coordinates, and the reference key point pair includes a preset leg shape key point pair and a preset foot shape key point pair, and the current key The point pair includes the key point pair of the current leg shape and the key point pair of the current foot shape. The running posture includes the running leg shape and the running foot shape. The analysis module 4 includes:

The construction unit is set to construct the current leg key point pair according to the coordinates of each leg type key point, and construct the current foot type key point pair according to the coordinates of each foot type key point;

The comparison unit is set to compare the current leg shape key point pair with the preset leg shape key point pair, and calculate the first included angle information in the vertical direction; compare the current foot shape key point pair with the preset leg shape key point pair Compare the key points of the type, and calculate the second included angle information in the vertical direction;

The analyzing unit is configured to analyze and obtain the running leg shape according to the first included angle information and the first angle threshold, and obtain the running foot shape according to the second included angle information and the second angle threshold analysis.

Optionally, the preset leg shape key point pair includes a preset left thigh key point pair, a preset right thigh key point pair, a preset left calf key point pair, and a preset right calf key point pair;

The first included angle information includes the included angle a1 between the current left thigh key point pair and the preset left thigh key point pair, the included angle a2 between the current right thigh key point pair and the preset right thigh key point pair, and the current left thigh key point pair. The included angle a3 between the key point pair of the calf and the preset left calf key point pair, and the included angle a4 between the current right calf key point pair and the preset right calf key point pair;

The first angle threshold is e;

The analysis unit includes: a first analysis subunit, configured to determine that the running leg is an X-shaped leg when a1>e°, a2>-e°, a3>-e°, and a4>e°.

Optionally, the preset foot type key point pair includes a preset left sole key point pair and a preset right sole key point pair;

The second angle threshold is β;

The analysis unit further includes: a second analysis subunit, configured to determine that the running foot type is inner eight feet when b1>β° and b2>-β°.

Optionally, as shown in Figure 2, the identification device further includes:

The output module 5 is configured to generate running posture analysis information according to the current leg shape key point pair, the current foot shape key point pair, the running leg shape and the running foot shape, and output it to the display interface in real time.

In this embodiment, each module, unit and sub-unit of the recognition device are configured to perform corresponding steps in the above-mentioned running posture recognition method, and the specific implementation process thereof will not be described in detail here.

This embodiment provides a running posture recognition device. The recognition device collects the lower body image of the user when running, and then invokes the underlying skeleton extraction algorithm to analyze the lower body image, thereby obtaining multiple key point heat maps. Among them, a single keypoint heatmap corresponds to a single keypoint. The identification device decodes the heat map of each key point, and obtains the coordinates of the key point corresponding to the heat map of each key point. Finally, the identification device generates the current key point pair according to the coordinates of each key point, compares the current key point pair with the pre-built reference key point pair, and analyzes the running posture of the user. The lower body image collected by the identification device in the present application includes the user's foot image, so the heatmap of key points including the user's feet is obtained through analysis by the underlying skeleton extraction algorithm, which increases the prediction of the key points of the user's feet when running, and further The user's foot posture can be obtained by analysis, so that the final recognized running posture is more comprehensive and accurate.

Referring to FIG. 4 , an embodiment of the present application further provides a computer device. The computer device may be a server, and its internal structure may be as shown in FIG. 4 . The computer device includes a processor, memory, a network interface, and a database connected by a system bus. Among other things, the computer is designed with a processor arranged to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium, an internal memory. The nonvolatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the execution of the operating system and computer programs in the non-volatile storage medium. The computer device's database is configured to store pre-built benchmark keypoint peer-to-peer data. The network interface of the computer device is configured to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for recognizing a running posture is realized.

The above-mentioned processor executes the steps of the above-mentioned running posture recognition method:

The first angle threshold is e;

The second angle threshold is β;

An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, a method for identifying a running posture is implemented, and the method for identifying a running posture is specifically:

S1: collect the lower body image of the user when running;

S201: input the lower body image into the encoder for encoding, and extract to obtain semantic features;

S302: Determine whether each key point heat map has only one Gauss point peak;

S303: If each key point heat map has only one Gauss point peak, the coordinates where each Gauss point peak is located are used as the key point coordinates of the corresponding key point.

Optionally, after the step of judging whether each key point heatmap has only one Gaussian point peak, including:

S304: If there are multiple Gauss point peaks in the key point heat map, screen out the first Gauss point with the maximum value from each Gauss point peak, and the second Gauss point overlapping with the first Gauss point, and the second Gauss point corresponds to The peak value of the Gaussian point is only smaller than the peak value of the Gaussian point corresponding to the first Gaussian point;

S305: calculate the distance between the peak coordinates of the first Gaussian point and the peak coordinates of the second Gaussian point;

S306: Adjust the peak coordinates of the first Gaussian point according to the distance to obtain the key point coordinates corresponding to the key point heat map.

The first angle threshold is e;

The second angle threshold is β;

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage In the medium, when the computer program is executed, it may include the processes of the above-mentioned method embodiments. Wherein, any reference to memory, storage, database or other medium provided in this application and used in the embodiments may include non-volatile and/or volatile memory. Nonvolatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

It should be noted that, herein, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, device, article or method comprising a series of elements includes not only those elements, It also includes other elements not expressly listed or inherent to such a process, apparatus, article or method. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, apparatus, article, or method that includes the element.

The above descriptions are only preferred embodiments of the present application, and are not intended to limit the scope of the patent of the present application. Any equivalent structure or equivalent process transformation made by using the contents of the description and drawings of the present application, or directly or indirectly applied to other related The technical field is similarly included in the scope of patent protection of this application.

Industrial Applicability

As shown above, the method, device and computer equipment for running posture recognition provided by at least some of the embodiments of the present application have the following beneficial effects: collecting the lower body image of the user when running through the treadmill, and then calling the underlying skeleton extraction algorithm to analyze the lower body image, Thereby, multiple key point heatmaps are obtained. Among them, a single keypoint heatmap corresponds to a single keypoint. The treadmill decodes the heat map of each key point, and obtains the coordinates of the key points corresponding to the heat map of each key point. Finally, the treadmill generates the current key point pair according to the coordinates of each key point, compares the current key point pair with the pre-built benchmark key point pair, and analyzes the running posture of the user. The lower body image collected by the treadmill in this application includes the user's foot image, so the underlying skeleton extraction algorithm is used to parse and obtain the key point heat map including the user's feet, which increases the prediction of the key points of the user's feet when running, and further The user's foot posture can be obtained by analysis, so that the final recognized running posture is more comprehensive and accurate.

Claims

A method for identifying a running posture, comprising:

collecting a lower body image of the user when running, the lower body image including a foot image;

Call the underlying skeleton extraction algorithm to analyze the lower body image, and obtain multiple key point heat maps, and a single key point heat map corresponds to a single key point;

Decoding each of the key point heat maps to obtain the key point coordinates corresponding to each of the key point heat maps;

A current key point pair is generated according to the coordinates of each key point, and the current key point pair is compared with a pre-built reference key point pair, and the running posture of the user is obtained by analysis.
The method for recognizing running posture according to claim 1, wherein the step of invoking an underlying skeleton extraction algorithm to analyze the lower body image to obtain a plurality of key point heatmaps includes:

Inputting the lower body image into an encoder for encoding, and extracting semantic features, the encoder is composed of a plurality of depthwise separable convolutions and a hole convolution;

By stacking convolutional layers, multiple rough keypoint heatmaps are predicted from the semantic features;

Each of the rough key point heat maps is refined to obtain each of the key point heat maps.
The method for recognizing running posture according to claim 2, wherein, in the step of refining each of the rough key point heat maps to obtain each of the key point heat maps, the refining of a single rough key point heat map Steps include:

Pooling the rough key point heat map through the empty space pyramid, and extracting the secondary key point heat map;

The secondary key point heat map is refined based on a global spatial attention mechanism to obtain the key point heat map.
The method for identifying a running posture according to claim 1, wherein the step of decoding each of the key point heat maps to obtain the key point coordinates corresponding to each of the key point heat maps comprises:

Using the wave peak point method to calculate the heat map of each of the key points respectively, and obtain the corresponding Gauss point peak of each of the heat maps of the key points;

Determine whether the corresponding Gauss point peak of each of the key point heatmaps is one;

If there is one Gauss point peak corresponding to each of the key point heatmaps, the coordinates where the Gauss point peak is located are taken as the corresponding key point coordinates.
The method for recognizing running posture according to claim 4, wherein after the step of judging whether the corresponding Gauss point peak value of each of the key point heat maps is one, the method further comprises:

If there are multiple Gaussian point peaks corresponding to each of the key point heatmaps, the first Gaussian point peak and the second Gaussian point peak are selected from the multiple Gaussian point peaks, and the first Gaussian point peak is the maximum Gaussian point peak value, the second Gaussian point peak value is only smaller than the first Gaussian point peak value;

calculating the distance between the coordinates of the first Gaussian peak and the coordinates of the second Gaussian peak;

Offset adjustment is performed on the coordinate values of the first Gauss point peak in the X direction and the Y direction according to the distance, so as to obtain the key point coordinates corresponding to the key point heat map.
The method for identifying a running posture according to claim 1, wherein each of the key point coordinates includes leg shape key point coordinates and foot shape key point coordinates, and the reference key point pair includes a preset leg shape key point pair and a preset leg shape key point pair. Set foot type key point pair, the current key point pair includes the current leg type key point pair and the current foot type key point pair, and the running posture includes running leg type and running foot type; The steps of generating the current key point pair, comparing the current key point pair with the pre-built benchmark key point pair, and analyzing and obtaining the running posture of the user include:

The current leg shape key point pair is constructed according to the coordinates of each of the leg shape key points, and the current foot shape key point pair is constructed according to the coordinates of each of the foot shape key points;

Compare the current leg shape key point pair with the preset leg shape key point pair, and calculate the first included angle information in the vertical direction; and compare the current foot shape key point pair with the preset key point pair. Set the leg shape key point pairs to compare, and calculate the second included angle information in the vertical direction;

The running leg shape is obtained by analyzing the first included angle information and the first angle threshold, and the running foot shape is obtained by analyzing the second included angle information and the second angle threshold.
The method for recognizing a running posture according to claim 6, wherein the preset leg shape key point pair comprises a preset left thigh key point pair, a preset right thigh key point pair, a preset left calf key point pair, and a preset left calf key point pair Set the right calf key point pair; the current leg shape key point pair includes the current left thigh key point pair, the current right thigh key point pair, the current left calf key point pair and the current right calf key point pair; the first included angle The information includes the angle a1 between the current left thigh key point pair and the preset left thigh key point pair, the angle between the current right thigh key point pair and the preset right thigh key point pair a2, the angle between the current left calf key point pair and the preset left calf key point pair a3, the included angle between the current right calf key point pair and the preset right calf key point pair a4; the first angle threshold is e; the step of analyzing and obtaining the running leg shape according to the first included angle information and the first angle threshold includes:

When a1>e°, a2>-e°, a3>-e°, and a4>e°, it is determined that the running leg type is an X-shaped leg.
The method for recognizing running posture according to claim 6, wherein the preset key point pair of foot shape includes a preset left foot key point pair and a preset right foot key point pair; the current foot shape key point pair includes The current left sole key point pair and the current right sole key point pair; the second included angle information includes the angle b1 between the current left sole key point pair and the preset left sole key point pair, the current The included angle b2 between the right sole key point pair and the preset right sole key point pair; the second angle threshold is β; the second angle information and the second angle threshold are analyzed to obtain the Steps for running foot shape, including:

When b1>β° and b2>-β°, it is determined that the running foot type is inner eight feet.
The method for recognizing running posture according to claim 6, wherein the current key point pair is generated according to the coordinates of each key point, and the current key point pair is compared with a pre-built reference key point pair, After the step of analyzing and obtaining the running posture of the user, it also includes:

The running posture analysis information is generated according to the current leg shape key point pair, the current foot shape key point pair, the running leg shape and the running foot shape, and output to the display interface in real time.
A running posture recognition device, comprising:

a collection module, configured to collect images of the lower body of the user when running, wherein the images of the lower body include images of feet;

The parsing module is set to call the underlying skeleton extraction algorithm to parse the lower body image to obtain multiple key point heat maps, and a single key point heat map corresponds to a single key point;

a decoding module, configured to decode each of the key point heatmaps to obtain key point coordinates corresponding to each of the key point heatmaps;

The analysis module is configured to generate a current key point pair according to the coordinates of each key point, compare the current key point pair with a pre-built reference key point pair, and analyze the running posture of the user.
A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein the processor implements the steps of the method of any one of claims 1 to 9 when the processor executes the computer program.
A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method of any one of claims 1 to 9.