WO2021000708A1

WO2021000708A1 - Fitness teaching method and apparatus, electronic device and storage medium

Info

Publication number: WO2021000708A1
Application number: PCT/CN2020/095369
Authority: WO
Inventors: 汪孔桥
Original assignee: 安徽华米信息科技有限公司
Priority date: 2019-07-04
Filing date: 2020-06-10
Publication date: 2021-01-07
Also published as: CN110418205A

Abstract

Provided are a fitness teaching method and apparatus, a device, and a computer readable storage medium. The method comprises: acquiring action video frames of a user, the action video frames comprising video frames generated upon the actions of the user; acquiring coach video frames played back for the user and a learning position corresponding to the user, the learning position being the display position of the user action in the coach video frames; combining the user actions in the action video frames of the user into a corresponding learning position in the coach video frames played back for the user, so as to generate enhanced coach video frames; and playing back the enhanced coach video frames for the user. The present invention provides a high-quality exercise fitness method.

Description

Fitness teaching method, device, electronic equipment and storage medium

Related cross references

This patent application claims the priority of the Chinese patent application filed on July 4, 2019 with application number 2019105993901, the full text of which is incorporated herein by reference.

Technical field

The present disclosure relates to the field of augmented reality technology, and in particular to a fitness teaching method, device, electronic equipment, and computer-readable storage medium.

Background technique

With the improvement of people's living standards and the strengthening of sports awareness, more and more people are willing to invest more time and energy in fitness or different sports such as basketball and yoga. With the development of science and technology, it is becoming more and more common for people to use virtual teaching such as video teaching to perform fitness or learn different sports. However, compared with the actual teaching process, the current virtual teaching process is not very interactive, and it is difficult for learners to get the true feeling of visiting the fitness site, and the effect needs to be improved.

Summary of the invention

According to a first aspect of the embodiments of the present disclosure, there is provided a fitness teaching method, the method including:

Acquiring a user's action video frame; the action video frame includes a video frame generated when the user moves;

Acquiring a coaching video frame played for the user and a learning position corresponding to the user; the learning position is the display position of the user action in the coaching video frame;

Synthesize the user actions in the user's action video frame to the corresponding learning position in the coach video frame played for the user to generate an enhanced coach video frame;

Playing the enhanced coaching video frame for the user.

Optionally, the coach video frame is a video frame that includes a coach demonstration action; the method further includes: acquiring a coach video frame corresponding to the user's action video frame; wherein, the time of acquiring the user's action video frame The time of playing the coach video frame is the same or within a preset time difference; and, comparing the user action in the user's action video frame with the coach demonstration action in the coaching video frame to generate a measure of the user action The evaluation score.

Optionally, the comparing the user action in the user's action video frame with the coach demonstration action in the coaching video frame, and generating an evaluation score that measures the user action includes: from the user's action video Obtain the coordinate data of the user's skeleton feature point in the frame as user action data, and obtain the coordinate data of the coach's skeleton feature point from the coach video frame as the coach demonstration action data; and, according to the user action The difference between the data and the coach demonstration action data generates an evaluation score that measures the user's action.

Optionally, the generating an evaluation score for measuring the user action according to the difference between the user action data and the coach demonstration action data includes: obtaining at least one user vector angle based on the user action data, And obtaining at least one standard vector angle based on the coach demonstration action data; wherein the user vector angle is the angle between two vectors formed by the coordinate data of any three adjacent skeleton feature points corresponding to the user Angle; the standard vector angle is the angle between the two vectors formed by the coordinate data of any three adjacent skeleton feature points corresponding to the coach; and, according to the user vector angle and the corresponding standard vector The included angle determines a similarity parameter to determine the evaluation score corresponding to the user action; the similarity parameter at least includes a standard deviation result or a variance result of the included angle between the at least one user vector and the corresponding standard vector.

Optionally, the determining the similarity parameter according to the included angle of the at least one user vector and the corresponding standard vector includes:

For the included angle between each user vector and the corresponding standard vector, calculate the difference vector; where, suppose the number of standard vector included angle and user vector included angle are both N (N is an integer greater than 0), and the i-th (1≤i≤N) The included angle of the standard vectors is α _i , the ith user vector included angle is β _i , and the difference vector is Δα _i , then Δα _i = α _i- β _i ;

Calculate the average difference vector according to the difference vector; where, if the average difference vector is Δr, then

Use the difference vector and the average difference vector to calculate the similarity parameter; where, suppose the similarity parameter is S, then

or

Optionally, the method further includes: synthesizing the evaluation score into the enhanced coach video frame.

Optionally, the synthesizing the evaluation scores into the enhanced coaching video frame includes: if the same coaching video frame is played for multiple users at the same time, and correspondingly generated based on the action video frames of the multiple users Evaluation scores, combining a plurality of evaluation scores into the enhanced coaching video frame based on the size ranking result of the evaluation scores.

Optionally, the method further includes: receiving a selection request sent by a user; the selection request includes at least one of the following: an identification of the coaching video selected by the user or an identification of a learning location selected by the user.

Optionally, the coach video is a video that includes a coach demonstration action; the coach video includes a template coach video and an enhanced coach video; wherein the template coach video does not synthesize any user action video frames, the enhanced coach video Pre-composite one or more user action video frames.

Optionally, the acquiring the coach video frame played for the user includes: acquiring the coach video frame played for the user based on the start time and current time of playing the coach video for the user.

Optionally, each coaching video includes one or more learning positions; before said generating an enhanced coaching video frame, it also includes: if the same coaching video is played for multiple users at the same time, and there are unavailability in the coaching video The selected learning location, the user corresponding to the unselected learning location is selected from users who have never selected the learning location to receive the action video frame of the selected user, and the user action corresponding to the selected user Synthesize to the unselected learning position in the coach video.

Optionally, the user corresponds to a user account; the learning location corresponds to asset information; and the fitness teaching method further includes: acquiring asset information corresponding to the learning location selected by the user, and comparing the asset information to the asset information according to the asset information. The user account initiates asset processing operations.

Optionally, the obtaining the user's action video frame includes: detecting a human body in a video frame shot by a camera; based on the detected human body, performing background segmentation on the video frame to extract the human body, and generating a generation that includes only the human body Action video frames for action.

According to a second aspect of the embodiments of the present disclosure, there is provided a fitness teaching device, the device including:

An action video frame acquisition module, configured to acquire a user's action video frame; the action video frame includes a video frame generated when the user moves;

An acquiring module, configured to acquire the coaching video frame played for the user and the learning position corresponding to the user; the learning position is the display position of the user's action in the coaching video frame;

An enhanced coaching video frame generation module, configured to synthesize user actions in the user's action video frames to corresponding learning positions in the coaching video frames played for the user to generate enhanced coaching video frames;

The enhanced coach video frame playback unit is used to play the enhanced coach video frame for the user.

According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, including:

processor;

A memory for storing processor executable instructions;

Wherein, the processor is configured to perform the operations in the method described above.

According to a fourth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon, which when executed by one or more processors, causes the processor to perform the operations in the method described above.

The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects:

In the present disclosure, the user action in the user's action video frame can be synthesized to the corresponding learning location in the coach video frame played for the user according to the user's corresponding learning position, thereby generating an enhanced coaching video frame. The present disclosure is based on AR technology (Visual Augmented Reality Technology) allows users to see their actions and the actions of the coach at the same time, so that users can correct their actions by comparison, and increase the interactive effect and improve the user's motivation for exercise.

In this disclosure, users can freely select fitness videos and learning locations based on their own needs, thereby generating a selection request. The selection request includes the identification of the coach video selected by the user or the identification of the learning location selected by the user, which is beneficial to improve the user's use Experience.

In the present disclosure, if the same coaching video is played for multiple users at the same time, and there is an unselected learning position in the coaching video, the server may also select the unselected learning position from users who have never selected a learning position. The user corresponding to the learning position of the user, so as to receive the action video frame sent by the selected user, and synthesize the user action corresponding to the selected user to the unselected learning position in the coach video, so as to avoid Users who choose the learning position can see their actions, which is conducive to improving the user experience.

In the present disclosure, the user action in the action video frame can also be compared with the coach’s demonstration action in the coach video frame to generate an evaluation score that measures the user action sent to the user terminal, and the The evaluation score is synthesized into the enhanced coaching video frame, so that the user has a clear judgment on the standard degree of his own actions, and improves the user experience.

In the present disclosure, if the same coaching video frame is played for multiple users at the same time, and corresponding evaluation scores are respectively generated based on the action video frames of the multiple users, the multiple evaluation scores are combined into all based on the ranking result of the evaluation scores. The enhanced coaching video frame allows users to not only see their own scores, but also the scores of other users, enhancing the interaction of fitness.

In this disclosure, the user corresponds to a user account, and the learning location also corresponds to asset information. After acquiring asset information corresponding to the learning location selected by the user, an asset processing operation can be initiated on the user account based on the asset information, Provide corresponding economic value for businesses.

It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure.

Description of the drawings

The drawings herein are incorporated into the specification and constitute a part of the specification, show embodiments in accordance with the disclosure, and together with the specification are used to explain the principle of the disclosure.

Fig. 1 is a structural diagram of a fitness teaching system according to an exemplary embodiment of the present disclosure.

Fig. 2 is a schematic diagram showing one frame of a template coach video according to an exemplary embodiment of the present disclosure.

Fig. 3 is a schematic diagram showing one frame of an enhanced coach video according to an exemplary embodiment of the present disclosure.

Fig. 4 is a schematic diagram showing one frame of another enhanced coach video according to an exemplary embodiment of the present disclosure.

Fig. 5 is a structural diagram of another fitness teaching system according to an exemplary embodiment of the present disclosure.

Fig. 6 is a skeleton feature point and vector angle of a human body according to an exemplary embodiment of the present disclosure.

Fig. 7 is a structural diagram of another fitness teaching system according to an exemplary embodiment of the present disclosure.

Fig. 8 is a flowchart of a fitness teaching method according to an exemplary embodiment of the present disclosure.

Fig. 9 is a schematic structural diagram of a fitness teaching device according to an exemplary embodiment of the present disclosure.

Fig. 10 is a structural diagram of an electronic device according to an exemplary embodiment of the present disclosure.

Detailed ways

Here, exemplary embodiments will be described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings indicate the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

The terms used in the present disclosure are only for the purpose of describing specific embodiments, and are not intended to limit the present disclosure. The singular forms "a", "said" and "the" used in the present disclosure and appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and/or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

It should be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various information, the information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as second information, and similarly, the second information may also be referred to as first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determination".

Fitness teaching in related technologies usually includes the following methods: the first type, the learner obtains the coach video, the coach video is a video that includes the coach’s demonstration actions, and then the learner watches the coach’s demonstration in the coach video while following the learning; the second type, One or more learners and the coach are connected to the same video channel. In this channel, the learners follow the coach’s real-time presentation to learn, and the learners and the learners, and the learners and the coach can communicate.

In the process of implementing the embodiments of the present disclosure, the inventor found that the above-mentioned first fitness teaching method lacks interactivity, the learner may not be able to keep up with the coach’s movement rhythm, and the movement of learning is not sure whether it is standardized or not, and it is difficult to learn. ; Although the above-mentioned second fitness teaching method can obtain a certain degree of online guidance from the coach, it is difficult for the learner to get the true feeling of visiting the fitness site, and the time needs to be matched with the start time of the coach, and training cannot be carried out at any time. limitation.

Therefore, in order to solve the problems in related technologies, the embodiments of the present disclosure provide a fitness teaching method, which can be applied to a local terminal, and the terminal can be an electronic device such as a computer, a tablet, a smart TV, or a mobile phone. The fitness teaching method can also be applied to the server, the server can be a server, a computer, and other electronic devices that can provide computing services.

The following fitness teaching system includes the server and the user. The user can be a smart TV, a smart phone, a computer, a personal digital assistant (PDA) or a tablet and other electronic devices with camera functions and audio and video display functions. The server executes the fitness teaching method as an example for description.

Please refer to FIG. 1. FIG. 1 is a structural diagram of a fitness teaching system according to an exemplary embodiment of the present disclosure. In the embodiment shown in Figure 1, the system includes a server and a user.

The user terminal is used to obtain and send the user's action video frame to the server when the coach video selected by the user is played.

The server is configured to receive the action video frame sent by the user, and obtain the coach video frame played for the user and the corresponding learning position of the user; the learning position is the user action in the The display position in the coach video frame.

The server is also used to synthesize the user actions in the action video frames to the corresponding learning positions in the coaching video frames played for the client, to generate an enhanced coaching video frame and send it to the client.

The user end is also used to receive and play the enhanced coach video frame sent by the server end.

In an example, a storage module (database, ROM, etc.) is provided on the server to store coach videos. The coach videos are videos that include coach demonstration actions. Each coach video can correspond to one or more learning The specific number of positions can be specifically set based on actual conditions, and the learning position is the display position of the user's action in the coach video. Please refer to FIG. 2 and FIG. 3. FIG. 2 shows a scene where the coach video corresponds to 4 learning positions, and FIG. 3 shows a schematic diagram of a user action displayed on one of the learning positions of the coach video frame. In addition, the coaching video includes a template coaching video and an enhanced coaching video. The template coach video is a video that has not synthesized any user's action video frames, as shown in FIG. 2. The enhanced coaching video is a video pre-synthesized with one or more user's action video frames, as shown in FIG. 3, which has been pre-synthesized with one user's action video frame.

In one embodiment, if the user of the user terminal needs to select a coach video before exercising or exercising based on the fitness teaching system of the present disclosure, the server will obtain the coach video from the database and send it to the user The terminal pushes the corresponding coach video for display on the client terminal. The user terminal can select the coach video to be played and the learning position to be displayed, or one of the two on the user terminal based on its actual needs, and then the user terminal detects that the user has a specific position, the coach video, and the learning position. Or to select one of the two, a selection request is initiated to the server, and then the server receives the selection request sent by the user and records accordingly.

Wherein, based on the user's selection, the selection request may include at least one of the identification of the coaching video selected by the user or the identification of the learning location selected by the user; the user may select one or more learning locations based on their own needs. The disclosed embodiment does not impose any restriction on this. In addition, the disclosed embodiment does not impose any restriction on the type of coach video selected by the user. It can be a template coach video or an enhanced coach video. Under the premise of user privacy, the embodiments of the present disclosure do not impose any restrictions on the specific form of the enhanced coach video frame selected by the user. For example, the enhanced coach video frame may be the enhanced coach video frame generated during the user's last exercise or fitness. Or, the enhanced coaching video frame may also be an enhanced coaching video frame shared by other users.

In another embodiment, the user may not select the coach video to be played and/or the learning location, and the server terminal determines it. For example, the user clicks a random play button on the user terminal, such as a virtual button or a physical button, and the user terminal sends a random play request, and then the server randomly determines a coaching video and learning location for the user; or The server can also automatically determine a coaching video and learning location according to the user's historical playback data or user preferences.

In a possible implementation manner, the client corresponds to a user account, and the server can combine the coach video (template coach video and/or enhanced coach video) and the generated enhanced coach video selected and played by the user terminal. And the selected learning location is associated with the user account, which is convenient for the user to obtain the associated coach video, which can help the user to re-learn and improve the accuracy of the follow-up exercise or fitness exercise; and if the user selects the same coach multiple times When performing sports or fitness videos, you can directly obtain the learning location selected by the user associated with the user account, without the user having to repeat the selection, reducing user operation steps. Of course, the user can also re-select the learning location based on their own needs. Embodiments of the present disclosure There are no restrictions on this.

In the embodiment of the present disclosure, during exercise or fitness, the user terminal plays the coach video selected by the user, and the user makes corresponding actions according to the demonstration actions of the coach in the played coach video, and the user terminal synchronizes through the camera The user action is captured to generate an action video frame including the user action, and then the action video frame is sent to the server.

Wherein, the user terminal may generate action video frames including user actions through the following two possible implementation modes:

In an implementation manner, the camera may be a 2D RGB camera, and the user terminal obtains RGB video frames captured by the 2D RGB camera, detects the human body in the RGB video frame, and then, based on the detected human body, The video frame undergoes background segmentation to extract the human body, and an action video frame that only includes the human body motion is generated.

In another implementation manner, the camera may be a 2D RGB camera and a 3D depth camera, and the user terminal obtains RGB video frames captured by the 2D RGB camera and depth vision field frames captured by the 3D depth camera, based on the The depth vision field frame calculates the visual depth field, and detects the human body in the RGB video frame, and then divides the human body detected in the RGB video frame from the background according to the visual depth field, and retains the human body foreground, thereby generating only the human body Action video frames for action.

It can be seen that, in the embodiment of the present disclosure, the user terminal sends motion video frames that only include human motion (ie, user motion) to the server, which is beneficial to reduce the amount of video transmission, thereby speeding up the transmission.

In the embodiment of the present disclosure, after receiving the action video frame sent by the client, the server acquires the coach video frame played for the client, that is, the coach video frame currently played by the client. In one example, for example, the user terminal plays the content of the coach video frame in the first frame, and uses the camera to capture the user's actions corresponding to the coach demonstration action in the first frame, because the user makes the corresponding actions based on the coach video frame in the first frame The user side needs to process the captured video frames and transmit them to the server. This may take a certain amount of time. When the user side generates the action video frame corresponding to the first frame of the coach video frame When uploading to the server, the coach video frame of the first frame has already been played on the user end. It is possible that the content of the third coach video frame is played for the user at this time. Therefore, the server needs to obtain this At this time, the coach video frame played on the client side is combined with the action video frame, so that the user can see his own actions based on the progress of the playback; it should be noted that the coach video frame played on the user side and the action The coach video frames corresponding to the user actions in the video frames are not the same frame.

In an implementation manner, since the user terminal sends a corresponding play request to the server when playing the selected coach video, the play request includes a timestamp so that the server can obtain When the user terminal plays the start time of the selected coach video, the server may determine the coach video frame to be played by the user terminal based on the start time and current time of playing the selected coach video for the client terminal.

In the fitness teaching scenario, you can choose one-to-one (that is, one user selects a coach video for learning), or multiple-to-one (that is, multiple users select a coach video for learning at the same time), then this embodiment is for the user There is no restriction on the number of terminals.

In a possible scenario, if the server detects that only one client currently selects the coaching video to play, the server receives the action video frame sent by the client, and then obtains the The coach video frame played by the user end and the learning position corresponding to the user end, where the learning position is the display position of the user action in the coach video frame, and finally the server converts the user in the action video frame The action is synthesized to the corresponding learning position in the coaching video frame played for the client at this time, and an enhanced coaching video frame is generated and sent to the client, so that the client can play the enhanced coaching video frame. The coach’s demonstration actions and the user’s own actions are present in the enhanced coach video frame. Referring to Figure 3, if the coach video that the user chooses to play is a template coach video, only the coach's demonstration actions and the user's own actions exist in the generated enhanced video frame. The embodiments of the present disclosure are based on AR technology (Visual Augmented Reality Technology) so that the user can simultaneously see the actions done by himself and the actions done by the coach.

The embodiment of the present disclosure does not impose any restriction on the number of action video frames uploaded by the client at the same time. That is, the embodiment of the present disclosure is not limited to a scenario where a client can only upload one user's action video frame. If two or more learners learn based on the coach video played in the same place, the user terminal will capture the user's actions through the camera, and at least two human actions can be obtained according to each video frame captured, thereby correspondingly generated At least two action video frames are sent to the server.

If the user terminal selects a learning position corresponding to the number of multiple learners on the corresponding user terminal before the coaching video is played, the user terminal can obtain multiple and single learning positions based on the captured video frames. The corresponding action video frames that only include human actions are sent to the server. The server receives a plurality of action video frames sent by the client, and synthesizes the human body movements in each of the action video frames to the corresponding learning position in the coach video frame played for the client at this time to generate an enhanced Coach video frame. If the client selects only one learning position before the coach video is played, then among the multiple action video frames sent by the client, the server selects one of the action video frames and transfers the action video frame The human body action in is synthesized to the corresponding learning position in the coach video frame played for the client at this time to generate an enhanced coach video frame. The embodiment of the present disclosure does not impose any restriction on the way the server selects the action video frame. For example, it can be selected randomly, or it can be identified from the multiple action video frames based on the user's face image pre-stored on the user end. The action video frame corresponding to the user's face image is displayed.

In an example, if the user chooses to play an enhanced coaching video, and the selected learning position has been synthesized with other user's actions, the server will receive the action video frame sent by the client before synthesizing it. , It is possible to remove the synthesized actions of other users, and then synthesize the user actions in the action video frame of the user end to the corresponding learning position in the coach video frame played for the user end at this time.

In another possible scenario, if the server detects that multiple clients are playing the same coaching video at the same time, the server receives the action video frames sent by the multiple clients, and then obtains the current status of the user The coach video frame played by the client terminal and the learning position information corresponding to each client terminal. Finally, the server terminal synthesizes the user actions in the multiple action video frames into the coach video frames that are simultaneously played for the multiple client terminals at this time. At the corresponding learning position, an enhanced coach video frame is generated, and the enhanced coach video frame contains a demonstration action of the coach and corresponding actions of multiple users. Referring to FIG. 4, if the coach video that the user chooses to play is a template coach video, the generated enhanced video frame contains the coach's demonstration actions and the actions corresponding to multiple users. In the embodiments of the present disclosure, the user can also see the actions made by other users, which enhances the interaction of the exercise, provides a high-quality way of mutual encouragement and joint exercise and fitness, and improves the user's motivation for exercise. In an example, the action video frames sent by multiple clients correspond to the same coaching video frame.

In a possible implementation manner, the server may only receive the action video frames of the client with the selected learning position to improve the receiving efficiency, and if it is detected that the same coaching video frame is played for multiple users at the same time, and the There is an unselected learning position in the coaching video, and the server may select the user terminal corresponding to the unselected learning position from the user terminals that have not selected the learning position to receive the selected user terminal. The action video frame is combined with the user action corresponding to the selected user terminal to the unselected learning position in the coach video, thereby enhancing user interaction.

The embodiment of the present disclosure does not impose any restriction on the manner in which the server selects the user terminal corresponding to the unselected learning position from the user terminal that has never selected the learning position. For example, the server may never select the user terminal. The user terminal of the learning position is randomly selected from the user terminal corresponding to the unselected learning position, or the server may also select the user terminal corresponding to the unselected learning position based on a preset rule. For the user terminal corresponding to the location, the preset rule may be the user terminal that has the largest number of videos played among the user terminals that have not selected the learning location, or the user terminal that has selected the learning location for other coach videos.

In the embodiment of the present disclosure, after generating the enhanced coaching video frame, the server sends the enhanced coaching video frame to the client, so that the client receives and plays the enhanced coaching video frame; The electronic device of the user terminal may include a display screen, so that the user terminal can play the enhanced coach video frame through the display screen. In an example, the user end uploads the action video frame through streaming media technology, and the server sends the enhanced coach video frame through streaming media technology to ensure fast transmission of the video frames.

In one embodiment, after the server generates an enhanced coaching video frame corresponding to the coaching video, it saves the generated enhanced coaching video to the storage module, and if the user has a corresponding account, it can also save the generated enhanced coaching video frame. Associating the enhanced coaching video with the user’s account can help the user’s replay learning, facilitate the improvement of the accuracy of follow-up exercises or fitness exercises, and enrich the coach’s video resources.

In one embodiment, referring to FIG. 2, the learning location may also correspond to asset information, and the asset information represents the value of the learning location. For example, the asset information may be virtual currency, points, or real currency, etc. When the server receives the learning position selected by the user, it initiates an asset processing operation on the user account corresponding to the user terminal according to the asset information corresponding to the learning position, and then the user terminal executes the service to itself An asset processing operation initiated by a user account; for example, the asset processing operation can be an operation of deducting virtual currency, points, or balance in the user account, or deducting the balance in the bank card or third-party payment account bound to the user account Operation; so as to provide businesses with corresponding economic value.

Please refer to FIG. 5, which is a structural diagram of another fitness teaching system according to an exemplary embodiment of the present disclosure. In the embodiment shown in Figure 5, the system includes a server and a user.

The server is also used to synthesize the user actions in the action video frame to the corresponding learning position in the currently playing coach video frame, generate an enhanced coach video frame and send it to the user terminal.

The server is also used to obtain the coach video frame corresponding to the action video frame. Wherein, the time when the user terminal obtains the action video frame is the same as the time when the coach video frame is played or is within a preset time difference; compare the user action in the action video frame with the coach presentation in the coach video frame Action, generating an evaluation score measuring the user's action; sending the evaluation score to the user terminal.

The user terminal is also used to receive the enhanced coach video frame and the evaluation score sent by the server, and synthesize the evaluation score into the enhanced coach video frame and play it.

In the embodiment of the present disclosure, the server may obtain the coach video frame corresponding to the action video frame for comparison. In one example, for example, the user terminal plays the content of the coach video frame in the first frame, and uses the camera to capture the user's action corresponding to the coach demonstration action in the first frame, and the user terminal uploads the action video frame generated based on the action To the server, the first coach video frame corresponds to the action video frame at this time. In a possible implementation manner, the shooting time when the user terminal uses the camera to capture the user action is the same as the time when the coach video frame is played. In an ideal state, the server can be based on the shooting time of the action video frame , Determine the corresponding coach video frame, that is, the time when the user terminal obtains the action video frame is the same as the time when the user terminal plays the coach video frame. In another possible implementation manner, considering that there is a certain reaction delay time for the user's action, the server may also determine the corresponding coaching video frame based on the shooting time of the action video frame and the preset time difference, That is, the time when the user terminal obtains the action video frame and the time when the user terminal plays the coach video frame are within a preset time difference.

In the embodiment of the present disclosure, after obtaining the coach video frame corresponding to the action video frame, the server may compare the user action in the action video frame with the coach demonstration action in the coach video frame to generate The evaluation score of the user action is measured. Specifically, the server may obtain the coordinate data of the user's skeleton feature points from the action video frame as user action data, and obtain the coordinate data of the coach's skeleton feature points from the coach video frame as the coach Demonstration action data, the coordinate data of the skeleton feature point may be two-dimensional coordinate data or three-dimensional coordinate data, and then based on the difference between the user action data and the coach demonstration action data, an evaluation measuring the user action is generated fraction. The calculation process of the embodiment of the present disclosure is beneficial to improve the accuracy of the evaluation score, wherein the coach action data may be offline data obtained in advance, thereby speeding up the processing speed of the server and improving response efficiency.

In a specific implementation manner, the server may obtain all user vector angles based on the user action data, and obtain all standard vector angles based on the coach demonstration action data. The included angle of the user vector may be the included angle of two vectors formed by the coordinate data of any three adjacent skeleton feature points corresponding to the user, and the included angle of the standard vector may be any three corresponding to the coach. The included angle of two vectors formed by the coordinate data of adjacent skeleton feature points, and the included angle of the user vector corresponds to the included angle of the standard vector one-to-one. Please refer to Figure 6. Figure 6 shows 14 skeleton feature points (the black mark points in Figure 6). Two vectors formed by any three adjacent skeleton feature points determine a vector angle, and a total of 13 vector clips Then, the similarity parameter is determined according to the included angle of all user vectors and the included angle of all corresponding standard vectors, so as to determine the evaluation score corresponding to the user action. The similarity parameter includes at least a standard deviation result or a variance result of the included angle between the vector and the standard vector. The smaller the standard deviation result and the variance result, and the greater the similarity, the higher the evaluation score. The standard vector included angle may be offline data obtained in advance, thereby speeding up the processing speed of the server and improving response efficiency.

In another example, the server may obtain the included user vector angle of the part based on the user action data, and obtain the standard vector included angle of the corresponding part based on the coach demonstration action data. The included angle of each user vector corresponds to the included angle of each standard vector one-to-one. Then, the similarity parameter is determined according to the included angle of the partial user vector and the corresponding partial standard vector, thereby determining the evaluation score corresponding to the user action.

In an example, the calculation of the similarity parameter by the server may include: for the corresponding included angle between the standard vector and the user vector, calculating the difference vector; where the number of the included angle between the standard vector and the included angle of the user vector Are all N (N is an integer greater than 0), set the i-th (1≤i≤N) standard vector angle as α _i , the i-th user vector angle as β _i , and the difference vector as Δα _i , then Δα _i = α _i- β _i ; calculate the average difference vector according to all the difference vectors; where, suppose the average difference vector is Δr, then

Use all the difference vectors and the average difference vector to calculate the similarity parameter; among them, set the similarity parameter to S, then

or

In a possible scenario, if the server detects that a coaching video is played for a single user, the server sends the generated evaluation score corresponding to the user to the user, so that the user The terminal receives the evaluation score sent by the server, and generates a score image based on the evaluation score to synthesize with the enhanced coach video frame to generate and play an enhanced coach video frame including the evaluation score, so that the user is There is a clear judgment on the standard degree of the action, which improves the user experience.

In another possible scenario, if the server detects that multiple clients are playing the same coaching video synchronously, and generates corresponding evaluation scores based on the action video frames sent by the multiple clients, then the The server sends the evaluation scores of multiple clients to each client so that each client receives multiple evaluation scores, and sorts the multiple evaluation scores based on the size of the evaluation scores, and generates a score image based on the sorted evaluation scores , To synthesize with the enhanced coaching video frame to generate and play an enhanced coaching video frame including evaluation scores, so that users can not only see their own scores, but also the scores of other users, and enhance the interactivity of fitness. In another example, the user terminal may also only receive its own corresponding evaluation scores for synthetic playback based on the user's selection, which is not limited in the embodiment of the present disclosure.

Please refer to FIG. 7, which is a structural diagram of another fitness teaching system according to an exemplary embodiment of the present disclosure. In the embodiment shown in Figure 7, the system includes a server and a user.

The server is also used to obtain the coach video frame corresponding to the action video frame. Wherein, the time when the user terminal obtains the action video frame is the same as the time when the coach video frame is played or is within a preset time difference; compare the user action in the action video frame with the coach presentation in the coach video frame Action, generating an evaluation score measuring the user's actions; synthesizing the evaluation score into the enhanced coach video frame, generating an enhanced coach video frame including the evaluation score, and sending it to the user terminal.

The user terminal is also used to receive and play the enhanced coaching video frame including the evaluation score sent by the server.

In the embodiment of the present disclosure, after obtaining the coach video frame corresponding to the action video frame, the server may compare the user action in the action video frame with the coach demonstration action in the coach video frame to generate The evaluation score of the user action is measured.

In a possible implementation scenario, if the server detects that a coach video is played for a single user, the server generates a score image according to the generated evaluation score corresponding to the user, and then compares the score image with the Synthesizing the enhanced coaching video frame to generate an enhanced coaching video frame including the evaluation score and send it to the user terminal, so that the user terminal receives and plays the enhanced coaching video frame including the evaluation score sent by the server; Thereby, the user has a clear judgment on the standard degree of his own actions, and the user experience is improved.

In another possible implementation scenario, if the server detects that multiple clients are playing the same coaching video synchronously, and generate corresponding evaluation scores based on the action video frames sent by the multiple clients, then The server sorts a plurality of the evaluation scores based on the size of the evaluation scores, generates a score image according to the sorted evaluation scores, and then superimposes and synthesizes the score image and the enhanced coach video frame to generate an evaluation score The enhanced coaching video frame is sent to the client, so that the client receives and plays the enhanced coaching video frame including the evaluation score sent by the server, so that the user can not only see his own score, but also To the scores of other users, enhance the interaction of fitness. In another example, the server may also generate an enhanced coaching video frame that only includes the user's evaluation score based on the user's selection, and send it to the user terminal. The embodiment of the present disclosure does not impose any limitation on this.

The following is an example of applying the fitness teaching method to a local terminal:

As shown in Fig. 8, Fig. 8 is a flowchart of a fitness teaching method according to an exemplary embodiment of the present disclosure.

In the embodiment shown in FIG. 8, the method includes the following steps.

In step S101, a user's action video frame is obtained; the action video frame includes a video frame generated when the user moves.

In step S102, a coach video frame played for the user and a learning position corresponding to the user are acquired; the learning position is the display position of the user action in the coach video frame.

In step S103, the user action in the user's action video frame is synthesized to the corresponding learning position in the coach video frame played for the user to generate an enhanced coach video frame.

In step S104, the enhanced coaching video frame is played for the user.

In an embodiment, a storage module (database, ROM, etc.) may be provided on the terminal to store coach videos. The coach videos are videos that include coach demonstration actions. Each coach video may correspond to one or more The specific number of learning positions can be specifically set based on the actual situation. The learning position is the display position of the user's actions in the coaching video; in addition, the coaching video includes a template coaching video and an enhanced coaching video. The coach video is a video that has not synthesized any user's action video frames, and the enhanced coach video is a video that has one or more users' action video frames synthesized in advance; wherein the terminal can be connected to a predetermined cloud to update the Coach video.

In an embodiment, the terminal may push a coach video to the user based on the storage module, and when the user wants to exercise, receive a selection request sent by the user, the selection request includes at least one of the following: a coach video selected by the user Or the identification of the learning location selected by the user.

Optionally, the user may not select the coaching video to be played and/or the learning location, and the terminal determines it; for example, the user clicks a random button on the terminal (it can be a virtual button or a physical button) , Then the terminal randomly determines a coaching video and learning location for the user; or the terminal can also automatically determine a coaching video and learning location according to the user's historical playback data or user preferences.

In the embodiment of the present disclosure, when exercising or fitness, the terminal plays the coach video selected by the user, the user makes a corresponding action according to the coach’s demonstration action in the played coach video, and the user terminal synchronously shoots through the camera User actions to generate action video frames including user actions.

Then, after generating the action video, the terminal obtains the coaching video frame played for the user at this time and the corresponding learning position of the user, and then synthesizes the user action in the user's action video frame into In the corresponding learning position in the coaching video frame played by the user, an enhanced coaching video frame is generated, and the enhanced coaching video frame is played for the user; wherein, the terminal may be based on the start of playing the coaching video for the user Time and current time, the coach video frame played for the user at this time is obtained; the embodiment of the present disclosure is based on AR technology (visual augmented reality technology) so that the user can see the actions done by himself and the actions by the coach at the same time, improving Interactivity.

In an embodiment, the coaching video frame is a video frame that includes a coach demonstration action, and the terminal may obtain the coaching video frame corresponding to the user's action video frame; wherein The time is the same as the time of playing the coach video frame or within a preset time difference, and then the user action in the user action video frame is compared with the coach demonstration action in the coach video frame to generate a measurement of the user action And then synthesize the evaluation scores into the enhanced coaching video frame.

In an example, after the terminal generates an enhanced coach video frame corresponding to the coach video, it saves the generated enhanced coach video to the storage module, and if the user has a corresponding account, the generated enhanced coach video frame may also be saved Associating the coaching video with the user's account can help the user's replay learning, facilitate the improvement of the accuracy of subsequent sports or fitness exercises, and enrich the coaching video resources.

In an embodiment, the learning location may also correspond to asset information, and the asset information represents the value of the learning location. For example, the asset information may be virtual currency, points, or actual currency, etc., then the terminal is receiving When the user selects the learning location, according to the asset information corresponding to the learning location, an asset processing operation is initiated on the user’s account. For example, the asset processing operation may be an operation of deducting virtual currency, points or balance in the user’s account. It can also be the operation of deducting the balance in the bank card or third-party payment account bound to the user's account; thereby providing the merchant with corresponding economic value.

In an implementation manner, the terminal may obtain coordinate data of the user's skeleton feature points from the user's action video frame as user action data, and obtain the coach's skeleton from the coach video frame The coordinate data of the feature point is used as the coach’s demonstration action data. The coordinate data of the skeleton feature point is two-dimensional coordinate data (photographed by a 2D camera) or three-dimensional coordinate data (photographed by a combination of 2D and 3D cameras). The coach demonstrates the difference between the action data and generates an evaluation score that measures the user's actions.

In an implementation manner, the terminal obtains at least one user vector angle based on the user action data, and obtains at least one standard vector angle based on the coach demonstration action data, and the user vector angle is the user Corresponding to the angle between two vectors formed by the coordinate data of any three adjacent skeleton feature points, the standard vector angle is formed by the coordinate data of any three adjacent skeleton feature points corresponding to the coach And then determine the similarity parameter according to the included angle of the at least one user vector and the corresponding standard vector to determine the evaluation score corresponding to the user action. The similarity parameter at least includes the The standard deviation result or the variance result of the included angle between at least one user vector and the corresponding standard vector. The smaller the standard deviation result and the variance result and the greater the similarity, the higher the evaluation score.

In an implementation manner, the terminal determining the similarity parameter may include: calculating the difference vector between the included angle of each user vector and the corresponding standard vector; where, the standard vector included angle and the user vector included angle The number of included angles are all N (N is an integer greater than 0). Let the i-th standard vector included angle (1≤i≤N) be α _i , the i-th user vector included angle is β _i , and the difference vector is Δα _i , then Δα _i =α _i- β _i ; calculate the average difference vector according to all the difference vectors; where, suppose the average difference vector is Δr, then

or

In one embodiment, if the same coaching video frame is played for multiple users at the same time, and corresponding evaluation scores are generated based on the action video frames of the multiple users, the terminal may sort the results based on the size of the evaluation scores. The evaluation scores are synthesized into the enhanced coaching video frame, so that users can not only see their own scores, but also the scores of other users, and enhance the interactivity of fitness.

As shown in FIG. 9, FIG. 9 is a schematic structural diagram of a fitness teaching device according to an exemplary embodiment of the present disclosure.

In the embodiment shown in FIG. 9, the device includes: an action video frame acquisition module 21, an acquisition module 22, an enhanced coach video frame generation module 23 and an enhanced coach video frame playback unit 24.

The action video frame acquisition module 21 is used to acquire a user's action video frame; the action video frame includes a video frame generated when the user moves.

The acquiring module 22 is configured to acquire the coaching video frame played for the user and the corresponding learning position of the user; the learning position is the display position of the user action in the coaching video frame.

The enhanced coaching video frame generating module 23 is used for synthesizing the user actions in the user's action video frames to the corresponding learning positions in the coaching video frames played for the user to generate an enhanced coaching video frame.

The enhanced coach video frame playing unit 24 is configured to play the enhanced coach video frame for the user.

Optionally, the coach video frame is a video frame including a coach demonstration action. Then the device further includes: an evaluation score generating module.

The acquisition module is also used to acquire the coach video frame corresponding to the user's action video frame; wherein, the time of acquiring the user's action video frame is the same as the time of playing the coach video frame or within a preset time difference.

The evaluation score generating module is configured to compare the user action in the user's action video frame with the coach demonstration action in the coach video frame to generate an evaluation score that measures the user action.

Optionally, the evaluation score generation module includes: an action data acquisition sub-module and an evaluation score generation sub-module.

The action data acquisition sub-module is used to acquire the coordinate data of the user’s skeleton feature points from the user’s action video frame as user action data, and to acquire the coach’s skeleton feature points from the coach video frame The coordinate data is used as the coach's demonstration action data.

The evaluation score generating sub-module is configured to generate an evaluation score for measuring the user action according to the difference between the user action data and the coach demonstration action data.

Optionally, the evaluation score generation sub-module includes: a vector included angle acquisition unit and an evaluation score generation unit.

The vector included angle acquiring unit is configured to acquire at least one user vector included angle based on the user action data, and acquire at least one standard vector included angle based on the coach demonstration action data; wherein the user vector included angle is the user Corresponding to the angle between two vectors formed by the coordinate data of any three adjacent skeleton feature points; the standard vector angle is formed by the coordinate data of any three adjacent skeleton feature points corresponding to the coach The angle between the two vectors.

The evaluation score generating unit is configured to determine a similarity parameter according to the included angle between the at least one user vector and the corresponding standard vector, thereby determining the evaluation score corresponding to the user action; the similarity parameter includes at least the at least one The standard deviation result or the variance result of the included angle between the user vector and the corresponding standard vector.

Optionally, the evaluation score generating unit includes: a difference vector calculation subunit, an average difference vector calculation subunit, a similarity parameter calculation subunit, and an evaluation score determination subunit.

The difference vector calculation subunit is used to calculate the difference vector between the included angle of each user vector and the corresponding standard vector; where the number of standard vector included angles and user vector included angles are all N (N is greater than 0), set the i-th (1≤i≤N) standard vector angle as α _i , the i-th user vector angle as β _i , and the difference vector as Δα _i , then Δα _i = α _i- β _i .

The average difference vector calculation subunit is used to calculate the average difference vector according to the difference vector; wherein, if the average difference vector is Δr, then

The similarity parameter calculation subunit is used to calculate the similarity parameter by using the difference vector and the average difference vector; wherein, if the similarity parameter is S, then

or

The evaluation score determination subunit is used to determine the evaluation score corresponding to the user action according to the similarity parameter.

Optionally, the device further includes: an evaluation score synthesis module, configured to synthesize the evaluation score into the enhanced coach video frame.

Optionally, the evaluation score synthesis module includes: if the same coaching video frame is played for multiple users at the same time, and corresponding evaluation scores are respectively generated based on the action video frames of the multiple users, the sorting results based on the size of the evaluation scores Multiple evaluation scores are synthesized into the enhanced coach video frame.

Optionally, the device further includes: a selection request receiving module configured to receive a selection request sent by the user; the selection request includes at least one of the following: an identification of the coaching video selected by the user or an identification of the learning location selected by the user.

Optionally, the step of acquiring the coach video frame played for the user in the acquiring module 22 includes: acquiring the coach video frame played for the user based on the start time and current time of playing the coach video for the user .

Optionally, each coaching video includes one or more learning positions. Before generating the enhanced coaching video frame, the device further includes: a learning position allocation module, which is used to play the same coaching video for multiple users at the same time, and there is an unselected learning position in the coaching video, never selected Select the user corresponding to the unselected learning position from the users of the learning position to receive the action video frame of the selected user, and synthesize the user action corresponding to the selected user into the coach video Describe the unselected learning position.

Optionally, the user corresponds to a user account; the learning location corresponds to asset information. The fitness teaching method further includes: an asset operation initiation module, configured to obtain asset information corresponding to the learning location selected by the user, and initiate an asset processing operation on the user account according to the asset information.

Optionally, the action video frame acquisition module 21 includes: a human body detection sub-module and an action video frame generation sub-module.

The human body detection sub-module is used to detect the human body in the video frame shot by the camera.

The action video frame generation sub-module is used to perform background segmentation on the video frame based on the detected human body to extract the human body, and generate an action video frame that only includes human body actions.

For the realization process of the functions and functions of each module in the fitness teaching device, please refer to the realization process of the corresponding steps in the fitness teaching system or fitness teaching method for details, which will not be repeated here.

For the device embodiment, since it basically corresponds to the method embodiment, the relevant part can refer to the part of the description of the method embodiment. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in One place, or it can be distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the objectives of the solutions of the present disclosure. Those of ordinary skill in the art can understand and implement it without creative work.

Correspondingly, the present disclosure also provides an electronic device, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the operations in the method described above.

Fig. 10 is a schematic structural diagram showing an electronic device applied by a fitness teaching device according to an exemplary embodiment.

As shown in FIG. 10, an electronic device 300 is shown according to an exemplary embodiment. The electronic device 300 may be a computing device such as a computer, a server, a mobile phone, or a tablet.

10, the electronic device 300 may include one or more of the following components: processing component 301, memory 302, power supply component 303, multimedia component 304, audio component 305, input/output (I/O) interface 306, sensor component 307 , And communication component 308.

The processing component 301 generally controls the overall operations of the device 300, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 301 may include one or more processors 309 to execute instructions to complete all or part of the steps of the foregoing method. In addition, the processing component 301 may include one or more modules to facilitate the interaction between the processing component 301 and other components. For example, the processing component 301 may include a multimedia module to facilitate the interaction between the multimedia component 304 and the processing component 301.

The memory 302 is configured to store various types of data to support operations in the electronic device 300. Examples of these data include instructions for any application or method operating on the electronic device 300, contact data, phone book data, messages, pictures, videos, etc. The memory 302 can be implemented by any type of volatile or nonvolatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable and Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic Disk or Optical Disk.

The power supply component 303 provides power for various components of the electronic device 300. The power supply component 303 may include a power management system, one or more power supplies, and other components associated with the generation, management, and distribution of power for the electronic device 300.

The multimedia component 304 includes a screen that provides an output interface between the electronic device 300 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, sliding, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure related to the touch or slide operation. In some embodiments, the multimedia component 304 includes a front camera and/or a rear camera. When the electronic device 300 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

The audio component 305 is configured to output and/or input audio signals. For example, the audio component 305 includes a microphone (MIC), and when the electronic device 300 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive external audio signals. The received audio signal may be further stored in the memory 302 or transmitted via the communication component 303. In some embodiments, the audio component 303 further includes a speaker for outputting audio signals.

The I/O interface 302 provides an interface between the processing component 301 and a peripheral interface module. The peripheral interface module may be a keyboard, a click wheel, a button, and the like. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

The sensor component 307 includes one or more sensors for providing the electronic device 300 with various aspects of state evaluation. For example, the sensor component 307 can detect the on/off status of the electronic device 300 and the relative positioning of the components. For example, the component is the display and the keypad of the electronic device 300. The sensor component 307 can also detect the electronic device 300 or the electronic device 300. The position of the component changes, the presence or absence of contact between the user and the electronic device 300, the orientation or acceleration/deceleration of the electronic device 300, and the temperature change of the electronic device 300. The sensor assembly 307 may include a proximity sensor configured to detect the presence of nearby objects when there is no physical contact. The sensor component 307 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 307 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, a heart rate signal sensor, an electrocardiogram sensor, a fingerprint sensor, or a temperature sensor.

The communication component 308 is configured to facilitate wired or wireless communication between the electronic device 300 and other devices. The electronic device 300 can access a wireless network based on a communication standard, such as WiFi, 2G, or 3G, or a combination thereof. In an exemplary embodiment, the communication component 308 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 308 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

In an exemplary embodiment, the electronic device 300 may be used by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field A programmable gate array (FPGA), controller, microcontroller, microprocessor, or other electronic components are implemented to implement the above methods.

In an exemplary embodiment, there is also provided a non-transitory computer-readable storage medium including instructions, such as the memory 302 including instructions, which may be executed by the processor 309 of the electronic device 300 to complete the foregoing method. For example, the non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

Wherein, when the instructions in the storage medium are executed by the processor 309, the device 300 can execute the aforementioned fitness teaching method.

Those skilled in the art will easily think of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptive changes of the present disclosure. These variations, uses, or adaptive changes follow the general principles of the present disclosure and include common knowledge or conventional technical means in the technical field not disclosed in the present disclosure. . The description and the embodiments are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are pointed out by the following claims.

It should be understood that the present disclosure is not limited to the precise structure that has been described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is limited only by the appended claims.

The above descriptions are only preferred embodiments of the present disclosure, and are not intended to limit the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the present disclosure Within the scope of protection.

Claims

A fitness teaching method, characterized in that it includes:

Acquiring a user's action video frame; the action video frame includes a video frame generated when the user moves;

Acquiring a coaching video frame played for the user and a learning position corresponding to the user; the learning position is the display position of the user action in the coaching video frame;

Synthesize the user actions in the user's action video frame to the corresponding learning position in the coach video frame played for the user to generate an enhanced coach video frame;

Playing the enhanced coaching video frame for the user.
The fitness teaching method according to claim 1, wherein the coach video frame is a video frame including a coach demonstration action;

The method further includes:

Acquiring a coach video frame corresponding to the user's action video frame; wherein the time of acquiring the user's action video frame is the same as the time of playing the coach video frame or within a preset time difference;

Compare the user action in the user's action video frame with the coach demonstration action in the coach video frame to generate an evaluation score measuring the user action.
The fitness teaching method according to claim 2, wherein the user action in the user action video frame is compared with the coach demonstration action in the coach video frame to generate an evaluation measuring the user action Score, including:

Acquiring the coordinate data of the user's skeleton feature point from the user's action video frame as user action data, and acquiring the coordinate data of the coach's skeleton feature point from the coaching video frame as coach demonstration action data;

According to the difference between the user action data and the coach demonstration action data, an evaluation score for measuring the user action is generated.
The fitness teaching method according to claim 3, wherein the generating an evaluation score for measuring the user's action according to the difference between the user's action data and the coach's demonstration action data comprises:

At least one user vector included angle is acquired based on the user action data, and at least one standard vector included angle is acquired based on the coach demonstration action data; wherein the user vector included angle is any three adjacent ones corresponding to the user The angle between two vectors formed by the coordinate data of the skeleton feature points; the standard vector angle is the angle between the two vectors formed by the coordinate data of any three adjacent skeleton feature points corresponding to the coach;

The similarity parameter is determined according to the included angle between the at least one user vector and the corresponding standard vector to determine the evaluation score corresponding to the user action; the similarity parameter includes at least the included angle between the at least one user vector and the The standard deviation result or variance result of the corresponding standard vector included angle.
The fitness teaching method according to claim 4, wherein the determining the similarity parameter according to the included angle of the at least one user vector and the corresponding standard vector comprises:

For the included angle between each user vector and the corresponding standard vector, calculate the difference vector; where, suppose the number of standard vector included angle and user vector included angle are both N (N is an integer greater than 0), and the i-th (1≤i≤N) The included angle of the standard vectors is α i , the ith user vector included angle is β i , and the difference vector is Δα i , then Δα i = α i- β i ;

Calculate the average difference vector according to the difference vector; where, if the average difference vector is Δr, then

Use the difference vector and the average difference vector to calculate the similarity parameter; where, suppose the similarity parameter is S, then
or
The fitness teaching method according to claim 2, characterized in that it further comprises:

The evaluation score is synthesized into the enhanced coach video frame.
The fitness teaching method according to claim 6, wherein said synthesizing said evaluation score into said enhanced coach video frame comprises:

If the same coaching video frame is played for multiple users at the same time, and corresponding evaluation scores are respectively generated based on the action video frames of the multiple users, the multiple evaluation scores are combined into the enhanced coaching video frame based on the sorting result of the evaluation scores in.
The fitness teaching method of claim 1, further comprising:

A selection request sent by a user is received; the selection request includes at least one of the following: the identification of the coaching video selected by the user or the identification of the learning location selected by the user.
The fitness teaching method according to claim 8, wherein the coach video is a video that includes a coach’s demonstration actions; the coach video includes a template coach video and an enhanced coach video; wherein the template coach video does not synthesize any The user's action video frame, and the enhanced coach video is pre-combined with one or more user's action video frames.
The fitness teaching method according to claim 1, wherein said acquiring the coach video frame played for the user comprises:

Based on the start time and current time of playing the coach video for the user, obtain the coach video frame played for the user.
The fitness teaching method according to claim 9, wherein each coach video includes one or more learning positions;

Then, before generating the enhanced coaching video frame, it also includes:

If the same coaching video is played for multiple users at the same time, and there is an unselected learning position in the coaching video, select the user corresponding to the unselected learning position from users who have never selected a learning position to receive all The selected user's action video frame, and the user action corresponding to the selected user is synthesized to the unselected learning position in the coach video.
The fitness teaching method according to claim 1, wherein the user corresponds to a user account; the learning location corresponds to asset information;

The fitness teaching method also includes:

Acquire asset information corresponding to the learning location selected by the user, and initiate an asset processing operation on the user account according to the asset information.
The fitness teaching method according to claim 1, wherein:

The obtaining the user's action video frame includes:

Detect the human body in the video frame shot by the camera;

Based on the detected human body, background segmentation is performed on the video frame to extract the human body, and an action video frame including only the human body motion is generated.
A fitness teaching device, characterized in that the method includes:

An action video frame acquisition module, configured to acquire a user's action video frame; the action video frame includes a video frame generated when the user moves;

An acquiring module, configured to acquire the coaching video frame played for the user and the learning position corresponding to the user; the learning position is the display position of the user's action in the coaching video frame;

An enhanced coaching video frame generation module, configured to synthesize user actions in the user's action video frames to corresponding learning positions in the coaching video frames played for the user to generate enhanced coaching video frames;

The enhanced coach video frame playback unit is used to play the enhanced coach video frame for the user.
The fitness teaching device according to claim 14, wherein the coach video frame is a video frame including a coach demonstration action;

The device further includes:

The acquiring module is further configured to acquire the coach video frame corresponding to the user's action video frame; wherein the time of acquiring the user's action video frame is the same as the time of playing the coach video frame or within a preset time difference ；

The evaluation score generating module is configured to compare the user action in the user's action video frame with the coach demonstration action in the coach video frame to generate an evaluation score that measures the user's action.
The fitness teaching device according to claim 15, wherein the evaluation score generating module comprises:

The action data acquisition sub-module is used to acquire the coordinate data of the user’s skeleton feature points from the user’s action video frame as user action data, and to acquire the coach’s skeleton feature points from the coach video frame The coordinate data is used as the coach's demonstration action data;

The evaluation score generating sub-module is configured to generate an evaluation score for measuring the user action according to the difference between the user action data and the coach demonstration action data.
The fitness teaching device according to claim 16, wherein the evaluation score generating sub-module comprises:

The vector included angle acquiring unit is configured to acquire at least one user vector included angle based on the user action data, and acquire at least one standard vector included angle based on the coach demonstration action data; wherein the user vector included angle is the user Corresponding to the angle between two vectors formed by the coordinate data of any three adjacent skeleton feature points; the standard vector angle is formed by the coordinate data of any three adjacent skeleton feature points corresponding to the coach The angle between the two vectors;

The evaluation score generating unit is configured to determine a similarity parameter according to the included angle between the at least one user vector and the corresponding standard vector, thereby determining the evaluation score corresponding to the user action; the similarity parameter includes at least the at least one The standard deviation result or the variance result of the included angle between the user vector and the corresponding standard vector.
The fitness teaching device according to claim 17, wherein the evaluation score generating unit comprises:

The difference vector calculation subunit is used to calculate the difference vector between the included angle of each user vector and the corresponding standard vector; where the number of standard vector included angles and user vector included angles are all N (N is greater than 0), set the i-th (1≤i≤N) standard vector angle as α i , the i-th user vector angle as β i , and the difference vector as Δα i , then Δα i = α i- β i ；

The average difference vector calculation subunit is used to calculate the average difference vector according to the difference vector; wherein, if the average difference vector is Δr, then

The similarity parameter calculation subunit is used to calculate the similarity parameter by using the difference vector and the average difference vector; wherein, if the similarity parameter is S, then
or

The evaluation score determining subunit is used to determine the evaluation score corresponding to the user action according to the similarity parameter.
The fitness teaching device according to claim 15, further comprising:

The evaluation score synthesis module is used to synthesize the evaluation score into the enhanced coach video frame.
The fitness teaching device according to claim 19, wherein the evaluation score synthesis module comprises:

If the same coaching video frame is played for multiple users at the same time, and corresponding evaluation scores are respectively generated based on the action video frames of the multiple users, the multiple evaluation scores are combined into the enhanced coaching video frame based on the sorting result of the evaluation scores in.
The fitness teaching device according to claim 14, further comprising:

The selection request receiving module is configured to receive a selection request sent by the user; the selection request includes at least one of the following: the identification of the coach video selected by the user or the identification of the learning location selected by the user.
The fitness teaching device according to claim 21, wherein the coach video is a video that includes a coach’s demonstration actions; the coach video includes a template coach video and an enhanced coach video; wherein the template coach video does not synthesize any The user's action video frame, and the enhanced coaching video is pre-combined with one or more user's action video frames.
The fitness teaching device according to claim 14, wherein the acquisition module comprises:

Based on the start time and current time of playing the coach video for the user, obtain the coach video frame played for the user.
The fitness teaching device according to claim 21, wherein each coach video includes one or more learning positions;

Then, before generating the enhanced coaching video frame, it also includes:

The learning location allocation module is used to play the same coaching video for multiple users at the same time, and there are unselected learning locations in the coaching video, select the unselected learning location from users who have never selected a learning location The corresponding user may receive the action video frame of the selected user, and synthesize the user action corresponding to the selected user to the unselected learning position in the coach video.
The fitness teaching device according to claim 14, wherein the user corresponds to a user account; the learning location corresponds to asset information;

The fitness teaching device further includes:

The asset operation initiation module is used to obtain asset information corresponding to the learning location selected by the user, and initiate an asset processing operation on the user account according to the asset information.
The fitness teaching device according to claim 14, wherein the action video frame acquisition module comprises:

The human body detection sub-module is used to detect the human body in the video frame shot by the camera;

The action video frame generation sub-module is used to perform background segmentation on the video frame based on the detected human body to extract the human body, and generate an action video frame that only includes human body actions.
An electronic device, characterized in that it comprises:

processor;

A memory for storing executable instructions of the processor;

among them,

The processor is configured to execute the fitness teaching method according to any one of claims 1 to 13.
A computer-readable storage medium, characterized in that a computer program is stored thereon, which when executed by one or more processors, causes the processor to execute the fitness teaching method according to any one of claims 1 to 13.