CN110070036B

CN110070036B - Method and device for assisting exercise motion training and electronic equipment

Info

Publication number: CN110070036B
Application number: CN201910324190.5A
Authority: CN
Inventors: 刘志康; 张弛
Original assignee: Beijing Megvii Technology Co Ltd
Current assignee: Beijing Megvii Technology Co Ltd
Priority date: 2019-04-22
Filing date: 2019-04-22
Publication date: 2021-05-25
Anticipated expiration: 2039-04-22
Also published as: CN110070036A

Abstract

The application relates to the field of computer vision, and discloses a method and a device for assisting exercise motion training and electronic equipment, wherein the method for assisting exercise motion training comprises the following steps: analyzing the acquired first video of any movement action of the sporter to obtain the shooting angle of the first video; then acquiring a second video of the template aiming at any motion action and meeting the shooting angle; then determining a first initial action frame of a first initial action of any motion action in the first video and a second initial action frame of a second initial action of any motion action in the second video; and then determining the angle deviation of the first skeleton diagram of the first initial motion frame relative to the second skeleton diagram of the second initial motion frame so as to assist the sporters to train any sport motion according to the angle deviation. The method of the embodiment of the application enables the sportsman to train the sports action without a manual coach, and improves the technical level.

Description

Method and device for assisting exercise motion training and electronic equipment

Technical Field

The application relates to the technical field of computer vision, in particular to a method and a device for assisting exercise movement training and electronic equipment.

Background

In the process of learning and training corresponding sports (such as table tennis, basketball, badminton and the like), sports personnel or sports enthusiasts often need the field instruction of a professional coach, and under the field instruction of the professional coach, accurate, skilled and solid basic actions are learned, each action link is mastered, and a solid foundation is laid for improving the sports level.

However, under the condition of no professional coach for on-site instruction, in the process of repeated exercise, a sporter or a sports enthusiast cannot determine whether the action is standard or not, cannot find problems in the action, cannot correct the action, and seriously influences the improvement of the technical level. Accordingly, there is a need for a method of assisting an athlete or a sports enthusiast to learn and master standard movements without the presence of a professional coach.

Disclosure of Invention

The purpose of the present application is to solve at least one of the above technical drawbacks, and to provide the following solutions:

in a first aspect, a method for assisting exercise movement training is provided, including:

analyzing the acquired first video of any movement action of the sporter to obtain the shooting angle of the first video;

acquiring a second video of a template aiming at any motion action and meeting the shooting angle;

determining a first starting action frame of a first starting action of any motion action in the first video and a second starting action frame of a second starting action of any motion action in the second video;

and determining the angle deviation of the first skeleton drawing of the first initial action frame relative to the second skeleton drawing of the second initial action frame so as to assist the sporters to train any sport action according to the angle deviation.

Specifically, the method for obtaining the shooting angle of the first video by analyzing the obtained first video of the sports personnel for any sports action includes:

analyzing the first video to obtain each first video frame;

and extracting a target video frame from each first video frame according to a first preset time interval, and determining a shooting angle through a second convolutional neural network based on the target video frame.

Further, determining a first start action frame of a first start action of any motion action in the first video, comprises:

determining a video frame with the position information and the depth information of a preset first skeleton key point of any motion action in each first video frame being changed initially, and determining the video frame as a first starting action frame of a first starting action;

determining a second start motion frame of a second start motion of any motion in a second video, comprising:

analyzing the second video to obtain each second video frame, determining the video frame with the initial change of the position information and the depth information of a preset second skeleton key point of any motion action in each second video frame, and determining the video frame as a second initial action frame of a second initial action; or acquiring a second initial motion frame of a second initial motion of any motion in a second video obtained in advance.

Further, before determining the video frame in which the position information and the depth information of the predetermined first skeleton key point of any motion action in each first video frame are initially changed, the method further comprises the following steps:

respectively carrying out human body detection on each first video frame through a first convolutional neural network, and determining human body images of moving personnel in each first video frame;

and respectively identifying the human body images in the first video frames through a third convolutional neural network to obtain the position information and the depth information of the first skeleton key points of the moving personnel in the first video frames.

Further, before determining the video frame in which the position information and the depth information of the predetermined second skeleton key point of any motion action in each second video frame are initially changed, the method further comprises the following steps:

respectively carrying out human body detection on each second video frame through a first convolutional neural network, and determining human body images of the moving personnel in each second video frame;

and respectively identifying the human body images in the second video frames through a third convolutional neural network to obtain the position information and the depth information of each second skeleton key point of the moving person in each second video frame, or obtaining the position information and the depth information of each second skeleton key point of the moving person in each second video frame in the second video which are obtained in advance.

Further, determining an angular deviation of a first skeletal map of the first start action frame relative to a second skeletal map of the second start action frame comprises:

extracting frames from the first initial action frame according to a second preset time interval to obtain each first subframe, and generating a first skeleton map corresponding to each first subframe according to the position information and the depth information of each first skeleton key point corresponding to each first subframe;

extracting frames from the second initial action frame according to a second preset time interval to obtain each second subframe, and generating a second skeleton map corresponding to each second subframe according to the position information and the depth information of each second skeleton key point corresponding to each second subframe, or acquiring a second skeleton map corresponding to each second subframe in the second initial action frame obtained in advance;

and calculating the angle deviation of each first skeleton map relative to each second skeleton map.

Further, according to the position information and the depth information of each first skeleton key point corresponding to each first subframe, a first skeleton map corresponding to each first subframe is generated, including:

and sequentially connecting all first framework key points except the pelvis central point in any first subframe according to a preset sequence by taking the pelvis central point in any first subframe as an origin to obtain a first framework diagram of any first subframe.

Further, calculating an angle deviation of each first skeleton map relative to each second skeleton map includes:

determining each first rotation angle of each edge in any first skeleton graph relative to a parent edge of each edge according to the depth information and the position information of each first skeleton key point in any first skeleton graph, wherein the parent edge of any edge is a first skeleton key point which is common to any edge and is adjacent to the edge with the pelvic bone center point as the origin;

determining each second rotation angle of each edge in any second skeleton graph relative to a parent edge of each edge according to the depth information and the position information of each second skeleton key point in any second skeleton graph corresponding to any first skeleton graph;

and calculating each angle deviation of each first rotating angle relative to each second rotating angle, and determining each angle deviation as the angle deviation of any first skeleton diagram relative to any second skeleton diagram, wherein the angle deviation comprises the angle difference size and the angle difference direction.

In a second aspect, there is provided an apparatus for assisting exercise training, comprising:

the analysis module is used for analyzing the acquired first video of any movement action of the sporter to obtain the shooting angle of the first video;

the acquisition module is used for acquiring a second video of a template which meets the shooting angle and aims at any motion action;

the first determining module is used for determining a first starting action frame of a first starting action of any motion action in the first video and a second starting action frame of a second starting action of any motion action in the second video;

and the second determining module is used for determining the angle deviation of the first skeleton diagram of the first initial action frame relative to the second skeleton diagram of the second initial action frame so as to assist the sporters to train any sport action according to the angle deviation.

Specifically, the analysis module comprises an analysis submodule and an angle determination submodule;

the analysis submodule is used for analyzing the first video to obtain each first video frame;

and the angle determining submodule is used for extracting the target video frame from each first video frame according to the first preset time interval and determining the shooting angle through the second convolutional neural network based on the target video frame.

Further, the first determining module comprises a first initial action frame determining submodule and a second initial action frame determining submodule;

the first starting action frame determining submodule is used for determining a video frame with the initially changed position information and depth information of a preset first skeleton key point of any motion action in each first video frame and determining the video frame as a first starting action frame of the first starting action;

the second initial action frame determining submodule is used for analyzing the second video to obtain each second video frame, determining the video frame with the initial change of the position information and the depth information of a preset second skeleton key point of any motion action in each second video frame, and determining the video frame as the second initial action frame of the second initial action; or a second starting motion frame of a second starting motion for acquiring any motion in a second video obtained in advance.

Further, still include: the first human body determining module and the first identifying module;

the first human body determining module is used for respectively carrying out human body detection on each first video frame through a first convolutional neural network and determining a human body image of a moving person in each first video frame;

and the first identification module is used for respectively identifying the human body images in the first video frames through a third convolutional neural network to obtain the position information and the depth information of the first skeleton key points of the moving personnel in the first video frames.

Further, still include: the second human body determining module and the second identifying module;

the second human body determining module is used for respectively carrying out human body detection on each second video frame through the first convolutional neural network and determining the human body image of the moving person in each second video frame;

and the second identification module is used for respectively identifying the human body images in the second video frames through a third convolutional neural network to obtain the position information and the depth information of each second skeleton key point of the moving person in each second video frame, or obtaining the position information and the depth information of each second skeleton key point of the moving person in each second video frame in the second video which are obtained in advance.

Further, the second determining module comprises a first processing submodule, a second processing submodule and a calculating submodule;

the first processing submodule is used for performing frame extraction on the first initial action frame according to a second preset time interval to obtain each first subframe, and generating a first skeleton map corresponding to each first subframe according to the position information and the depth information of each first skeleton key point corresponding to each first subframe;

the second processing submodule is used for performing frame extraction on the second initial action frame according to a second preset time interval to obtain each second subframe, and generating a second skeleton map corresponding to each second subframe according to the position information and the depth information of each second skeleton key point corresponding to each second subframe, or acquiring the second skeleton map corresponding to each second subframe in the second initial action frame obtained in advance;

and the calculating submodule is used for calculating the angle deviation of each first skeleton diagram relative to each second skeleton diagram.

Further, the first processing submodule is specifically configured to sequentially connect each first skeleton key point in any first subframe except the pelvis central point according to a predetermined sequence, with the pelvis central point in any first subframe serving as an origin, to obtain a first skeleton map of any first subframe.

Further, the calculation submodule comprises a first rotation angle determination unit, a second rotation angle determination unit and a deviation calculation unit;

the first rotation angle determining unit is used for determining each first rotation angle of each edge in any first skeleton image relative to a parent edge of each edge according to the depth information and the position information of each first skeleton key point in any first skeleton image, wherein the parent edge of any edge is a first skeleton key point which is common with any edge and is close to the edge which takes the pelvis center point as the origin;

the second rotation angle determining unit is used for determining each second rotation angle of each edge in any second skeleton image relative to a parent edge of each edge according to the depth information and the position information of each second skeleton key point in any second skeleton image corresponding to any first skeleton image;

and the deviation calculating unit is used for calculating each angle deviation of each first rotating angle relative to each second rotating angle and determining each angle deviation as the angle deviation of any first skeleton diagram relative to any second skeleton diagram, wherein the angle deviation comprises the angle difference size and the angle difference direction.

In a third aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the computer program, the method for assisting exercise movement training is implemented.

In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, which when executed by a processor implements the above-mentioned method of assisted athletic movement training.

The method for assisting exercise training provided by the embodiment of the application, which is used for assisting an exerciser to train any exercise by acquiring a second video of a corresponding template according to a shooting angle of the exerciser for a first video of the exercise according to an angle deviation between a first skeleton diagram and a second skeleton diagram, wherein the first skeleton diagram is a skeleton diagram of a first initial action frame of a first initial action of the any exercise in the first video, and the second skeleton diagram is a skeleton diagram of a second initial action frame of a second initial action of the any exercise in the second video, so that the exerciser can determine whether the action is standard or not without field guidance of a professional trainer, find a deviation between the action and a standard action, actively adjust the action according to the deviation and correct the action, thereby gradually mastering various standard actions and continuously improving the technical level.

Additional aspects and advantages of the present application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the present application.

Drawings

The foregoing and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:

FIG. 1 is a schematic flow chart illustrating a method for assisting exercise training according to an embodiment of the present disclosure;

FIG. 2 is a schematic illustration of a skeletal diagram of an embodiment of the present application;

FIG. 3 is a schematic diagram of a basic structure of an apparatus for assisting exercise training according to an embodiment of the present application;

FIG. 4 is a detailed structural diagram of an apparatus for assisting exercise training according to an embodiment of the present disclosure;

fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

Reference will now be made in detail to embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are exemplary only for the purpose of explaining the present application and are not to be construed as limiting the present application.

As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. As used herein, the term "and/or" includes all or any element and all combinations of one or more of the associated listed items.

To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

The following describes the technical solutions of the present application and how to solve the above technical problems with specific embodiments. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.

Example one

The embodiment of the present application provides a method for assisting exercise movement training, as shown in fig. 1, including:

step S110, analyzing the acquired first video of the sports personnel for any sports action to obtain the shooting angle of the first video.

Specifically, when a sportsman trains any one sport action, the multimedia acquisition device may be started, a video (i.e., the first video) of the sportsman training the any sport action is acquired by the multimedia acquisition device, and the acquired video is transmitted or copied to a terminal device such as a desktop computer, a notebook computer, an iPad, or a smart phone. In contrast, the terminal device acquires a video input by the sporter and used for training any one of the sports actions, and analyzes the acquired video to obtain a shooting angle of the video, that is, analyzes the acquired first video of the sporter and used for any one of the sports actions to obtain a shooting angle of the first video.

And step S120, acquiring a second video of the template for any motion action meeting the shooting angle.

Specifically, the terminal device may obtain, from a video library, a video (that is, the second video) of a template for any motion action, which satisfies an obtained shooting angle of the first video of the motion person for the any motion action, where the video library of the terminal device stores videos of templates for various motion actions at various video angles in advance.

Furthermore, the sporter can input the video of the template for any sport motion, which meets the shooting angle, into the terminal device according to any sport motion that the sporter wishes to practice and by combining the shooting angle of the video of any sport motion. Accordingly, the terminal equipment acquires the video of the template which is input by the sporter and meets the shooting angle and aims at any sport action.

For the convenience of distinction, in the embodiment of the present application, a video of a sporter training for a certain sports motion at a certain shooting angle is denoted as a first video, and a video of a template of the sporter training for the certain sports motion at a certain shooting angle is denoted as a second video, that is, the first video and the second video are shot for the same sports motion at the same shooting angle. If any of the above-mentioned motion actions is action A and the shooting angle of the video for training the motion A by the sporter is front shooting, the second video is the video of the template of the motion A when the shooting angle is front shooting.

Any of the above-mentioned sports actions may be a ping-pong sports action, a basketball sports action, a badminton sports action, or other sports actions or non-sports actions. No matter what kind of sports or non-sports motion the sportsman is doing, it is only necessary that the video library of the terminal device stores in advance the video of the template of various sports motion at various video angles about the sports or non-sports motion, or the sportsman can input the video of the template of the sports motion meeting the shooting angle to the terminal device according to the sports motion that the sportsman wishes to practice by himself and combining the shooting angle of the video of the sports motion.

If the sporter performs the table tennis playing action, the video library of the terminal device may include videos of the following templates at various shooting angles: right-hand cross-bat reverse-glue positive-hand batting, right-hand cross-bat reverse-glue positive-hand pulling, left-hand cross-bat positive-hand batting, left-hand cross-bat positive-hand pulling, right-hand cross-bat reverse-glue positive-hand pulling, left-hand cross-bat reverse-glue positive-hand pulling, and corresponding direct-bat techniques. If the sporter is doing badminton, the video library of the terminal device may include videos of the following templates at various shooting angles: a step-by-step action, a stepping action, a striding action, a jumping action, and the like.

In step S130, a first start motion frame of a first start motion of any motion in the first video and a second start motion frame of a second start motion of any motion in the second video are determined.

Specifically, whether any exercise action is standard or not depends on whether the initial action is standard or not, that is, whether the initial action of any exercise action is standard or not, which is largely determined, so that the correction of any exercise action can be achieved by correcting the initial action.

In general, in the video of any motion action, the initial motion of any motion action corresponds to the corresponding video frame in the video, so that a first initial motion frame of a first initial motion of any motion action in the first video and a second initial motion frame of a second initial motion of any motion action in the second video can be determined, thereby laying a necessary foundation for subsequently determining the skeleton map and the angular deviation.

Step S140, determining an angle deviation of the first skeleton map of the first initial motion frame relative to the second skeleton map of the second initial motion frame, so as to assist the sporter to train any sport motion according to the angle deviation.

Specifically, after a first start action frame of a first start action of any motion action in the first video and a second start action frame of a second start action of any motion action in the second video are determined, a first skeleton diagram of the first start action frame and a second skeleton diagram of the second start action frame can be determined, and an angle deviation of the first skeleton diagram relative to the second skeleton diagram is determined, that is, an angle deviation between the start action of any motion action of the sportsman and the standard start action of any motion action is obtained, so that the sportsman can actively adjust the action to be done and correct the action to be done according to the angle deviation, thereby gradually mastering various standard actions and continuously improving the technical level.

Compared with the prior art, the method for assisting in training the sports motion, provided by the embodiment of the application, is used for assisting a sporter in training any sports motion by acquiring the second video of a corresponding template according to the shooting angle of the sporter for the first video of the any sports motion and according to the angle deviation between the first skeleton diagram and the second skeleton diagram, wherein the first skeleton diagram is the skeleton diagram of the first initial motion frame of the first initial motion of the any sports motion in the first video, and the second skeleton diagram is the skeleton diagram of the second initial motion frame of the second initial motion of the any sports motion in the second video, so that the sporter can determine whether the motion is standard or not without the field guidance of a professional coach, find the deviation between the motion and the standard motion, and actively adjust the motion according to the deviation, The action is corrected, so that various standard actions are gradually mastered, and the technical level is continuously improved.

The embodiment of the present application provides another possible implementation manner, wherein step S110 includes step S1101 (not labeled in the figure) and step S1102 (not labeled in the figure), where:

step S1101: analyzing the first video to obtain each first video frame;

step S1102: and extracting a target video frame from each first video frame according to a first preset time interval, and determining a shooting angle through a second convolutional neural network based on the target video frame.

Step S130 includes step S1302 (not shown) and step S1305 (not shown), wherein:

step S1302: and determining the video frame with the initial change of the position information and the depth information of the preset first skeleton key point of any motion action in each first video frame, and determining the video frame as a first starting action frame of the first starting action.

Step S1305: analyzing the second video to obtain each second video frame, determining the video frame with the initial change of the position information and the depth information of a preset second skeleton key point of any motion action in each second video frame, and determining the video frame as a second initial action frame of a second initial action; or acquiring a second initial motion frame of a second initial motion of any motion in a second video obtained in advance.

Step S1300 (not shown) and step S1301 (not shown) are further included before step S1302, where:

step S1300: and respectively carrying out human body detection on each first video frame through the first convolutional neural network, and determining the human body image of the moving person in each first video frame.

Step S1301: and respectively identifying the human body images in the first video frames through a third convolutional neural network to obtain the position information and the depth information of the first skeleton key points of the moving personnel in the first video frames.

Step S1303 (not shown in the figure) and step S1304 (not shown in the figure) are further included before step S1305, where:

step S1303: and respectively carrying out human body detection on each second video frame through the first convolutional neural network, and determining the human body image of the moving person in each second video frame.

Step S1304: and respectively identifying the human body images in the second video frames through a third convolutional neural network to obtain the position information and the depth information of each second skeleton key point of the moving person in each second video frame, or obtaining the position information and the depth information of each second skeleton key point of the moving person in each second video frame in the second video which are obtained in advance.

The following describes the related contents related to the implementation:

specifically, after the terminal device acquires a video (i.e., the first video) for training a sporter for any sport action, the first video is analyzed to obtain a series of continuous video frames, and for convenience of distinguishing, each obtained video frame is subsequently marked as a first video frame. After a series of first video frames are obtained, inputting each first video frame into a convolutional neural network (i.e., the second convolutional neural network), and performing video shooting angle identification on each first video frame through the convolutional neural network to obtain a shooting angle identification result corresponding to each first video frame; then, the identified shooting angles may be averaged, and the averaged shooting angle may be used as the shooting angle of the first video.

In general, the angle of the multimedia capturing device when capturing the same video is fixed, so that one or more video frames (i.e. the above-mentioned target video frame) can be extracted from each first video frame at a predetermined time interval (i.e. the above-mentioned first predetermined time interval), and the one or more video frames are input into the second convolutional neural network, and the shooting angle recognition result corresponding to the one or more video frames is obtained by performing video shooting angle recognition on the one or more video frames through the second convolutional neural network. If only a certain video frame is extracted when the video frames are extracted at a preset time interval, determining the shooting angle identified for the certain video frame as the shooting angle of the first video; if some video frames are extracted when the video frames are extracted at preset time intervals, the shooting angles identified for the some video frames are correspondingly averaged, and the average value of the shooting angles of the some video frames is used as the shooting angle of the first video.

The video shooting angles include, but are not limited to, front shooting, left-side shooting, right-side shooting, back shooting, and the like.

Specifically, for the purpose of distinguishing, a video frame obtained by parsing the first video is referred to as a first video frame. Therefore, after the first video is analyzed to obtain each corresponding first video frame, each first video frame may be input into one convolutional neural network (i.e., the first convolutional neural network), and the human body of the moving person is detected in each video frame through the first convolutional neural network to obtain the human body image of the moving person, that is, the human body detection is performed on each first video frame through the first convolutional neural network, so as to determine the human body image of the moving person in each first video frame. If a plurality of human body images are detected in the same first video frame, the human body image occupying the largest pixel is taken as the human body image of the moving person.

Further, after obtaining the human body image of the moving person in each first video frame, the human body image in each first video frame may be respectively identified through the third convolutional neural network, so as to obtain each skeleton key point (i.e., the first skeleton key point) of the moving person in each first video frame, and position information and depth information of each first skeleton key point. And the last characteristic layer of the third convolutional neural network is connected with a plurality of fully-connected output layers, and the plurality of fully-connected output layers output the estimated first skeleton key points and the position information and the depth information of the first skeleton key points. If all the first video frames are connected, the first video frames are input into a third convolutional neural network as a video frame sequence, and a string of first skeleton key points of the moving person and position information and depth information of the first skeleton key points can be obtained.

Wherein, the first skeleton key points include but are not limited to head, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, pelvis center point, left knee joint, right knee joint, left ankle, right ankle, etc.

Specifically, a first video for training a sporter for any sport action may be input to the action start determining module, and a start action of any sport action in the first video is obtained by the action start determining module. In the skeleton model, a video frame with a preset first skeleton key point of any motion action and a changed motion direction is generally used as a starting frame of any motion action, that is, a motion action in the starting frame is used as a starting action of any motion action, that is, the starting action of any motion action in a first video is determined, which is equivalent to determining a video frame with a preset first skeleton key point of any motion action and a changed motion direction in the first video, and the video frame is determined as a starting action frame of the starting action. Wherein the initial change of the motion direction of the skeleton key point is generally determined according to the initial changes of the position information and the depth information of the skeleton key point, and for the convenience of distinguishing, a start action in the first video is taken as a first start action, and a video frame including the first start action is taken as a first start action frame.

If the motion is right-handed cross-shot reverse-glue forward-hand-drawing, the video frame with the position information and the depth information of the key point of the right arm changed each time is the initial motion frame of the motion (i.e. right-handed cross-shot reverse-glue forward-hand-drawing) in the skeleton model, and the change of the two continuous motion directions can be regarded as a complete motion.

Similarly, the video frame in which the motion direction of the preset second skeleton key point of any motion in each video frame of the second video is initially changed is determined as the initial motion frame of the initial motion of any motion. For the purpose of distinguishing, each video frame in the second video is referred to as a second video frame, the start action of any motion action in the second video is referred to as a second start action, and the start action frame of the second start action is referred to as a second start action frame.

It should be noted that before determining a second initial motion frame of a second initial motion in a second video, the second video needs to be parsed to obtain each second video frame corresponding to the second video, and then a video frame with an initially changed motion direction (i.e., position information and depth information) of a predetermined second skeleton key point of any motion is determined from each second video frame, and the video frame is determined as the second initial motion frame of the second initial motion.

It should be noted that, if a video (i.e., a second video) of a template of any motion action at any video angle, which is stored in advance in a video library of the terminal device, has been previously parsed into video frames (i.e., respective second video frames), i.e., a second video of the template of any motion action at any video angle, which is stored in the form of respective second video frames, and a video frame (i.e., a start action frame) in which a motion direction of a predetermined skeleton key point of any motion action is initially changed is determined in advance from each second video frame, i.e., a second start action frame of a second start action in the second video frames is determined; the second video frames are obtained without analyzing the second video again, the video frames with the movement direction of the preset skeleton key points of any movement in the second video frames being changed initially are determined, and the video frames are determined as the second starting movement frames of the second starting movement.

Specifically, after the second video is analyzed to obtain each second video frame corresponding to the second video, each second video frame may be input into a convolutional neural network (i.e., the first convolutional neural network), and the human body of the moving person is detected in each second video frame through the first convolutional neural network to obtain the human body image of the moving person, that is, the human body is detected in each second video frame through the first convolutional neural network, so as to determine the human body image of the moving person in each second video frame. And if a plurality of human body images are detected in the same second video frame, taking the human body image occupying the largest pixel as the human body image of the moving person.

Further, after obtaining the human body image of the moving person in each second video frame, the human body image in each second video frame may be respectively identified through the third convolutional neural network, so as to obtain each skeleton key point (i.e., the second skeleton key point) of the moving person in each second video frame, and position information and depth information of each second skeleton key point. For the convenience of distinguishing, the skeleton key points of the moving person in each second video frame are marked as second skeleton key points.

And the last characteristic layer of the third convolutional neural network is connected with a plurality of fully-connected output layers, and the plurality of fully-connected output layers output the estimated second skeleton key points and the position information and the depth information of the second skeleton key points. If all the second video frames are connected, the second video frames are input into a third convolutional neural network as a video frame sequence, and a string of second skeleton key points of the moving person and position information and depth information of the second skeleton key points can be obtained.

Wherein, the second skeleton key points include but are not limited to head, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, pelvis center point, left knee joint, right knee joint, left ankle, right ankle, etc.

It should be noted that, if a video (i.e., a second video) of a template of any motion action at any video angle, which is stored in advance in a video library of the terminal device, has been previously parsed into video frames (i.e., second video frames), that is, the second video of the template of any motion action at any video angle is stored in the form of second video frames, and position information and depth information of second skeleton key points and second skeleton key points of the moving person have been determined in advance from the second video frames; the position information and the depth information of each second skeleton key point of the moving person in each second video frame in the second video, which are obtained in advance, can be directly obtained, and the human body images in each second video frame do not need to be identified through the third convolutional neural network again, so that the position information and the depth information of each second skeleton key point of the moving person in each second video frame and each second skeleton key point are obtained.

In another possible implementation manner, step S140 includes step S1401 (not labeled in the figure), step S1402 (not labeled in the figure), and step S1402 (not labeled in the figure), where:

step S1401: and extracting frames from the first initial action frame according to a second preset time interval to obtain each first subframe, and generating a first skeleton map corresponding to each first subframe according to the position information and the depth information of each first skeleton key point corresponding to each first subframe.

Step S1402: and extracting frames from the second initial action frame according to a second preset time interval to obtain each second subframe, and generating a second skeleton map corresponding to each second subframe according to the position information and the depth information of each second skeleton key point corresponding to each second subframe, or acquiring the second skeleton map corresponding to each second subframe in the second initial action frame obtained in advance.

Step S1403: and calculating the angle deviation of each first skeleton map relative to each second skeleton map.

The method for generating the first skeleton map corresponding to each first subframe according to the position information and the depth information of each first skeleton key point corresponding to each first subframe includes:

Calculating the angle deviation of each first skeleton map relative to each second skeleton map, including:

Specifically, after the first start action frame and the second start action frame are determined, the first start action frame may be decimated at a predetermined time interval (i.e., the second predetermined time interval) to obtain each subframe. Similarly, the second start action frame is decimated at predetermined time intervals (i.e., the second predetermined time intervals mentioned above) to obtain each subframe. For the convenience of distinguishing, the sub-frame obtained by extracting the frame of the first initial action frame is taken as the first sub-frame, and the sub-frame obtained by extracting the frame of the second initial action frame is taken as the second sub-frame, so that each first sub-frame corresponding to the first initial action frame and each second sub-frame corresponding to the second initial action frame are obtained.

The second predetermined time interval may be set according to actual needs, for example, the time scale may be set to 1/5, that is: taking the sub-frames at equal intervals 1/5 in the first start action frame as the respective first sub-frames of the first start action frame; likewise, the subframes at equal intervals 1/5 in the second start action frame are taken as the respective second subframes of the second start action frame. When the time scale is set to 1/5, the first sub-frame is extracted from the first start action frame at a predetermined time interval of 1/5, that is, a total of 5 first sub-frames are obtained, which can be respectively denoted as a1, a2, A3, a4 and a 5; similarly, the second subframes are extracted from the second start action frame at a predetermined time interval of 1/5, that is, a total of 5 second subframes are obtained, and may be referred to as B1, B2, B3, B4, and B5, respectively. Wherein, A1 corresponds to B1, A2 corresponds to B2, A2 corresponds to B2, A3 corresponds to B3, A4 corresponds to B4, and A5 corresponds to B5.

Further, after obtaining each first subframe of the first start action frame, the first skeleton key points corresponding to each first subframe may be obtained according to the position information and the depth information of each first skeleton key point and each first skeleton key point of the moving person in each first video frame obtained by the third convolutional neural network, and the first skeleton map corresponding to each first subframe is generated according to the position information and the depth information of each first skeleton key point corresponding to each first subframe, that is, each first subframe generates the corresponding first skeleton map according to the position information and the depth information of the corresponding first skeleton key point. If 5 first subframes are extracted from the first initial action frame, 5 first skeleton maps can be generated at this time, that is, each first subframe corresponds to one first skeleton map. The skeleton diagram shown on the left side of fig. 2 is a first skeleton diagram corresponding to a certain first subframe.

Similarly, after each second subframe of the first start action frame is obtained, each second skeleton key point corresponding to each second subframe may be obtained according to the position information and the depth information of each second skeleton key point and each second skeleton key point of the moving person in each second video frame obtained by the third convolutional neural network, and a second skeleton map corresponding to each second subframe is generated according to each second skeleton key point position information and the depth information corresponding to each second subframe, that is, each second subframe generates a corresponding second skeleton map according to its corresponding second skeleton key point. If 5 second subframes are extracted from the second initial action frame, 5 second skeleton maps can be generated at this time, that is, each second subframe corresponds to one second skeleton map. The skeleton map on the right side of fig. 2 is a second skeleton map of a second subframe corresponding to the certain first subframe.

It should be noted that, if the first video includes N first start action frames, and 5 first subframes are extracted from each first start action frame at a predetermined time interval 1/5, and the 5 first subframes extracted from the first start action frame are referred to as a1_1, a1_2, a1_3, a1_4 and a1_5, respectively, and the 5 first subframes extracted from the second first start action frame are referred to as a2_1, a2_2, a2_3, a2_4 and a2_5, respectively, and so on, and the 5 first subframes extracted from the nth first start action frame are referred to as AN _1, AN _2, AN _3, AN _4 and AN _5, respectively. At this time, the first subframes of the N first start motion frames may be averaged, and the obtained average values are used as the first subframes of the first start motion frame in the first video, that is, the first subframes of the first start motion frame are a1 ═ a1_1+ a2_1+ … + AN _1)/N, A2 ═ a1_2+ a2_2+ … + AN _2)/N, A3 ═ a1_3+ a2_3+ … + AN _3)/N, A4 ═ a1_4+ a2_4+ … + AN _4)/N and A5 ═ a1_5+ a2_5+ … + AN _5)/N, respectively.

Further, for each first subframe in the first initial action frame or each second subframe in the second initial action frame, when the skeleton map is generated according to the skeleton key points corresponding to a certain subframe, the pelvis center point may be used as the origin point, and the adjacent skeleton key points may be connected in a predetermined order, so as to obtain the skeleton map corresponding to the certain subframe, as shown in fig. 2.

It should be noted that, if each skeleton map of each second subframe after being framed according to a second predetermined time interval is stored in the video library of the terminal device in advance, the second skeleton maps corresponding to each second subframe in the second start action frame obtained in advance may be obtained, and it is not necessary to perform framing on the second start action frame again according to the second predetermined time interval to obtain each second subframe, and the second skeleton maps corresponding to each second subframe are generated according to each second skeleton key point corresponding to each second subframe.

Further, if there are 5 first subframes, the 5 first subframes are a1, a2, A3, a4 and a5, respectively, and the first skeleton maps of a1, a2, A3, a4 and a5 are P1, P2, P3, P4 and P5, respectively; the number of the second subframes is also 5, the 5 second subframes are respectively B1, B2, B3, B4 and B5, and the second skeleton maps corresponding to B1, B2, B3, B4 and B5 are respectively T1, T2, T3, T4 and T5; meanwhile, a1 corresponds to B1, a2 corresponds to B2, A3 corresponds to B3, a4 corresponds to B4, and a5 corresponds to B5. In this case, the calculation of the angle deviation of each first skeleton map from each second skeleton map is to calculate the angle deviation G1 of P1 from T1, calculate the angle deviation G2 of P2 from T2, calculate the angle deviation G3 of P3 from T3, calculate the angle deviation G4 of P4 from T4, and calculate the angle deviation G5 of P5 from T5.

Taking the first skeleton map P1 (i.e., the left skeleton map in fig. 2) and the second skeleton map T1 (i.e., the right skeleton map in fig. 2) corresponding to the first skeleton map as an example, the following describes calculating the angular deviation G1 of the first skeleton map P1 with respect to the second skeleton map T1, wherein the calculation process may specifically be:

first, based on the depth information and position information of each first skeleton key point in P1, each rotation angle of each edge in P1 with respect to its parent edge is determined, and the rotation angle is denoted as a first rotation angle for easy distinction, that is, each first rotation angle of each edge in P1 with respect to its parent edge is determined. The parent edge of any edge is a first skeleton key point which is common with any edge, and is adjacent to the edge taking the center point of the pelvis as the origin.

In the skeleton diagram on the left side of fig. 2, if the point a is the pelvis center point, that is, a is the origin, the sides with the pelvis center point as the origin may be side 1, side 9 and side 10; it can be seen that the edge 1 and the edge 3 have a common first skeleton key point, so that the parent edge 1 of the edge 3, and so on, the parent edge of the edge 2 is the edge 1, the parent edge of the edge 6 is the edge 1, the parent edge of the edge 4 is the edge 3, the parent edge of the edge 5 is the edge 4, the parent edge of the edge 7 is the edge 6, the parent edge of the edge 8 is the edge 7, the parent edge of the edge 11 is the edge 9, the parent edge of the edge 12 is the edge 10, the parent edge of the edge 14 is the edge 11, and the parent edge of the edge 13 is the edge 12, so that each first rotation angle of each edge relative to its parent edge can be determined.

Similarly, in the skeleton diagram on the right side of fig. 2, if the point b is the pelvis center point, that is, b is the origin, the sides with the pelvis center point as the origin may be the side 1 ', the side 9 ', and the side 10 '; it can be seen that the edge 1 'and the edge 3' have a common first skeleton key point, so that the parent edge 1 'of the edge 3' is, and so on, the parent edge of the edge 2 'is the edge 1', the parent edge of the edge 6 'is the edge 1', the parent edge of the edge 4 'is the edge 3', the parent edge of the edge 5 'is the edge 4', the parent edge of the edge 7 'is the edge 6', the parent edge of the edge 8 'is the edge 7', the parent edge of the edge 11 'is the edge 9', the parent edge of the edge 12 'is the edge 10', the parent edge of the edge 14 'is the edge 11', and the parent edge of the edge 13 'is the edge 12', so that each second rotation angle of each edge relative to its parent edge can be determined.

Next, the respective angular deviations of the respective first rotation angles with respect to the respective second rotation angles obtained as described above are calculated, and the respective angular deviations are determined as angular deviations G1 of any of the first skeleton diagrams with respect to any of the second skeleton diagrams, wherein the angular deviations include angular difference magnitudes and angular difference directions.

The process of calculating the angle deviation G2 of the first skeleton map P2 with respect to the second skeleton map T2, the process of calculating the angle deviation G3 of the first skeleton map P3 with respect to the second skeleton map T3, the process of calculating the angle deviation G4 of the first skeleton map P4 with respect to the second skeleton map T4, and the process of calculating the angle deviation G5 of the first skeleton map P5 with respect to the second skeleton map T5 are the same as the process of calculating the angle deviation G1 of the first skeleton map P1 with respect to the second skeleton map T1, and are not repeated herein.

Furthermore, after obtaining each angle deviation, each angle deviation can be output to assist the sportsman to train the corresponding sport action according to the angle deviation, so that the sportsman can determine whether the action is standard or not under the condition of no professional coach field instruction, find the deviation between the action and the standard action, actively adjust the action according to the deviation and correct the action, thereby gradually mastering various standard actions and continuously improving the technical level.

Furthermore, besides outputting the angle deviation, the first skeleton diagrams and the second skeleton diagrams can be output in the same coordinate system, wherein the pelvis center point of each skeleton is located at the origin, and the user can intuitively feel the action difference.

It should be noted that, when outputting each first skeleton diagram and each second skeleton diagram in the same coordinate system, the first skeleton diagram and the second skeleton diagram corresponding to the first skeleton diagram may be displayed side by side as shown in fig. 2, or the first skeleton diagram and the second skeleton diagram corresponding to the first skeleton diagram may be displayed in an overlapping manner, so that the user can easily view the action difference between the two diagrams at a glance.

Example two

Fig. 3 is a schematic diagram of a basic structure of an apparatus for assisting exercise training provided in an embodiment of the present application, and as shown in fig. 3, the apparatus 30 may include an analysis module 31, an acquisition module 32, a first determination module 33, and a second determination module 34, where:

the analysis module 31 is configured to analyze the acquired first video of the sports person for any sports action to obtain a shooting angle of the first video;

the acquisition module 32 is used for acquiring a second video of a template which meets the shooting angle and aims at any motion action;

the first determining module 33 is configured to determine a first start action frame of a first start action of any motion action in the first video and a second start action frame of a second start action of any motion action in the second video;

the second determining module 34 is configured to determine an angle deviation of a first skeleton map of the first start action frame relative to a second skeleton map of the second start action frame, so as to assist a sporter to train any sport action according to the angle deviation.

Compared with the prior art, the device provided by the embodiment of the application acquires the second video of the corresponding template according to the shooting angle of the sporter aiming at the first video of any sport action, and assists the sporter to train any sport action according to the angle deviation between the first skeleton diagram and the second skeleton diagram, wherein the first skeleton diagram is the skeleton diagram of the first initial action frame of the first initial action of any sport action in the first video, and the second skeleton diagram is the skeleton diagram of the second initial action frame of the second initial action of any sport action in the second video, so that the sporter can determine whether the action is standard or not without the field guidance of a professional coach, find the deviation between the action and the standard action, actively adjust the action according to the deviation and correct the action, thereby gradually mastering various standard actions and continuously improving the technical level.

Specifically, fig. 4 is a detailed structural diagram of an apparatus for assisting exercise movement training according to an embodiment of the present application, and the apparatus 40 may include an analysis module 41, an acquisition module 42, a first determination module 43, and a second determination module 44. The function implemented by the parsing module 41 in fig. 4 is the same as the parsing module 31 in fig. 3, the function implemented by the obtaining module 42 in fig. 4 is the same as the obtaining module 32 in fig. 3, the function implemented by the first determining module 43 in fig. 4 is the same as the first determining module 32 in fig. 3, and the function implemented by the second determining module 44 in fig. 4 is the same as the second determining module 34 in fig. 3, which is not described herein again. The device for assisting exercise training shown in fig. 4 will be described in detail below:

specifically, the parsing module 41 includes a parsing sub-module 411 and an angle determining sub-module 412, as shown in fig. 4, wherein:

the parsing submodule 411 is configured to parse the first video to obtain each first video frame;

the angle determining sub-module 412 is configured to extract a target video frame from each first video frame according to a first predetermined time interval, and determine a shooting angle through a second convolutional neural network based on the target video frame.

The first determination module 43 comprises a first start action frame determination submodule 431 and a second start action frame determination submodule 432, wherein:

a first start action frame determining submodule 431, configured to determine a video frame in which position information and depth information of a predetermined first skeleton key point of any motion action in each first video frame are initially changed, and determine the video frame as a first start action frame of a first start action;

the second initial action frame determining submodule 432 is configured to analyze the second video to obtain each second video frame, determine a video frame in which position information and depth information of a predetermined second skeleton key point of any motion action in each second video frame are initially changed, and determine the video frame as a second initial action frame of the second initial action; or a second starting motion frame of a second starting motion for acquiring any motion in a second video obtained in advance.

Further, the apparatus further comprises: a first person determination module 45 and a first identification module 46, wherein:

a first human body determining module 45, configured to perform human body detection on each first video frame through a first convolutional neural network, and determine a human body image of a moving person in each first video frame;

the first identification module 46 is configured to respectively identify the human body images in each first video frame through a third convolutional neural network, so as to obtain position information and depth information of each first skeleton key point of the moving person in each first video frame.

Further, the apparatus further comprises: a second human body determination module 47 and a second identification module 48, wherein:

the second human body determining module 47 is configured to perform human body detection on each second video frame through the first convolutional neural network, and determine a human body image of a moving person in each second video frame;

the second identifying module 48 is configured to respectively identify the human body images in the second video frames through a third convolutional neural network to obtain position information and depth information of each second skeleton key point of the moving person in each second video frame, or obtain position information and depth information of each second skeleton key point of the moving person in each second video frame in the second video, which are obtained in advance.

Further, the second determination module 44 includes a first processing sub-module 441, a second processing sub-module 442 and a calculation sub-module 443, as shown in fig. 4, wherein:

the first processing submodule 441 is configured to perform frame extraction on the first initial action frame according to a second predetermined time interval to obtain each first subframe, and generate a first skeleton map corresponding to each first subframe according to each first skeleton key point corresponding to each first subframe;

the second processing submodule 442 is configured to perform frame extraction on the second initial action frame according to a second predetermined time interval to obtain each second subframe, and generate a second skeleton map corresponding to each second subframe according to each second skeleton key point corresponding to each second subframe, or obtain a second skeleton map corresponding to each second subframe in the second initial action frame obtained in advance;

the calculation submodule 443 is configured to calculate an angular deviation of each first skeleton map with respect to each second skeleton map.

Further, the first processing submodule 441 is specifically configured to, according to a predetermined sequence, sequentially connect each first skeleton key point, except for the pelvis central point, in any first subframe according to a condition that the pelvis central point in any first subframe is an origin point, so as to obtain a first skeleton map of any first subframe.

Further, the calculation sub-module 443 comprises a first rotation angle determination unit 4431, a second rotation angle determination unit 4432 and a deviation calculation unit 4433, as shown in fig. 4, wherein:

the first rotation angle determining unit 4431 is configured to determine, according to the depth information and the position information of each first skeleton key point in any first skeleton diagram, each first rotation angle of each edge in any first skeleton diagram relative to its parent edge, where the parent edge of any edge is a first skeleton key point that is common to any edge and is adjacent to an edge using a pelvis center point as an origin;

the second rotation angle determining unit 4432 is configured to determine, according to the depth information and the position information of each second skeleton key point in any second skeleton diagram corresponding to any first skeleton diagram, each second rotation angle of each edge in any second skeleton diagram relative to its parent edge;

the deviation calculating unit 4433 is configured to calculate each angle deviation of each first rotation angle with respect to each second rotation angle, and determine each angle deviation as an angle deviation of any one of the first skeleton diagrams with respect to any one of the second skeleton diagrams, where the angle deviation includes an angle difference magnitude and an angle difference direction.

It should be noted that the present embodiment is an apparatus embodiment corresponding to the first embodiment (i.e., the method embodiment), and the present embodiment can be implemented in cooperation with the first embodiment. The related technical details mentioned in the first embodiment are still valid in this embodiment, and are not described herein again in order to reduce repetition. Accordingly, the related art details mentioned in the present embodiment can also be applied to the first embodiment.

EXAMPLE III

An embodiment of the present application provides an electronic device, as shown in fig. 5, an electronic device 500 shown in fig. 5 includes: a processor 501 and a memory 503. Wherein the processor 501 is coupled to the memory 503, such as via the bus 502. Further, the electronic device 500 may also include a transceiver 504. It should be noted that the transceiver 504 is not limited to one in practical applications, and the structure of the electronic device 500 is not limited to the embodiment of the present application.

The processor 501 is applied to the embodiment of the present application, and is configured to implement the functions of the parsing module, the obtaining module, the first determining module, and the second determining module shown in fig. 3 or fig. 4.

The processor 501 may be a CPU, general purpose processor, DSP, ASIC, FPGA or other programmable logic device, transistor logic device, hardware component, or any combination thereof. Which may implement or perform the various illustrative logical blocks, modules, and circuits described in connection with the disclosure. The processor 501 may also be a combination of implementing computing functionality, e.g., comprising one or more microprocessors, a combination of DSPs and microprocessors, and the like.

Bus 502 may include a path that transfers information between the above components. The bus 502 may be a PCI bus or an EISA bus, etc. The bus 502 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in FIG. 5, but this is not intended to represent only one bus or type of bus.

The memory 503 may be, but is not limited to, a ROM or other type of static storage device that can store static information and instructions, a RAM or other type of dynamic storage device that can store information and instructions, an EEPROM, a CD-ROM or other optical disk storage, optical disk storage (including compact disk, laser disk, optical disk, digital versatile disk, blu-ray disk, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.

The memory 503 is used for storing application program codes for executing the scheme of the application, and the processor 501 controls the execution. The processor 501 is configured to execute application program codes stored in the memory 503 to implement the actions of the device for assisting exercise action training provided by the embodiment shown in fig. 3 or fig. 4.

The electronic device provided by the embodiment of the application comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, and when the processor executes the program, compared with the prior art, the electronic device can realize that: the second video of the corresponding template is obtained according to the shooting angle of the first video of the sports personnel for any sports action, and assists the sporter to train any sport action according to the angle deviation between the first skeleton diagram and the second skeleton diagram, wherein the first skeleton map is a skeleton map of a first initial motion frame of a first initial motion of the any motion in the first video, the second skeleton map is a skeleton map of a second initial motion frame of a second initial motion of the any motion in the second video, so that the sportsman can determine whether the action is standard or not without the field guidance of a professional coach, find the deviation between the action and the standard action, and actively adjusting and correcting the action according to the deviation, thereby gradually mastering various standard actions and continuously improving the technical level.

The embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the method shown in the first embodiment. Compared with the prior art, the second video of the corresponding template is obtained according to the shooting angle of the sporter aiming at the first video of any sport action, and the sporter is assisted to train any sport action according to the angle deviation between the first skeleton diagram and the second skeleton diagram, wherein the first skeleton diagram is the skeleton diagram of the first initial action frame of the first initial action of any sport action in the first video, and the second skeleton diagram is the skeleton diagram of the second initial action frame of the second initial action of any sport action in the second video, so that the sporter can determine whether the action is standard or not without the field guidance of a professional coach, find the deviation between the action and the standard action, actively adjust the action according to the deviation, correct the action, and gradually master various standard actions, the technical level is continuously improved.

The computer-readable storage medium provided by the embodiment of the application is suitable for any embodiment of the method. And will not be described in detail herein.

It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least a portion of the steps in the flow chart of the figure may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

The foregoing is only a partial embodiment of the present application, and it should be noted that, for those skilled in the art, several modifications and decorations can be made without departing from the principle of the present application, and these modifications and decorations should also be regarded as the protection scope of the present application.

Claims

1. A method of assisting exercise training, comprising:

analyzing an acquired first video of any motion action of the sporter to obtain a shooting angle of the first video;

determining a first start action frame of a first start action of the any motion action in the first video and a second start action frame of a second start action of the any motion action in the second video;

and determining the angle deviation of the first skeleton diagram of the first initial action frame relative to the second skeleton diagram of the second initial action frame so as to assist a sporter to train any sport action according to the angle deviation.

2. The method according to claim 1, wherein the analyzing the acquired first video of any motion action of the sporter to obtain a shooting angle of the first video comprises:

analyzing the first video to obtain each first video frame;

and extracting target video frames from the first video frames according to a first preset time interval, and determining the shooting angle through a second convolutional neural network based on the target video frames.

3. The method of claim 2, wherein determining a first start action frame of a first start action of the any motion action in the first video comprises:

determining a video frame with the position information and the depth information of a preset first skeleton key point of any motion action in each first video frame being changed initially, and determining the video frame as a first starting action frame of the first starting action;

determining a second start action frame of a second start action of the any motion action in the second video, comprising:

analyzing the second video to obtain each second video frame, determining the video frame with the initial change of the position information and the depth information of the preset second skeleton key point of any motion action in each second video frame, and determining the video frame as a second initial action frame of the second initial action; or

And acquiring a second initial action frame of a second initial action of any motion action in the second video, which is obtained in advance.

4. The method of claim 3, further comprising, prior to determining the video frame in which the position information and the depth information of the predetermined first skeletal keypoint of any of the motion actions in the respective first video frame are initially changed:

respectively carrying out human body detection on each first video frame through a first convolutional neural network, and determining human body images of the moving personnel in each first video frame;

and respectively identifying the human body image in each first video frame through a third convolutional neural network to obtain the position information and the depth information of each first skeleton key point of the moving person in each first video frame.

5. The method of claim 3, further comprising, before determining the video frame in which the position information and the depth information of the predetermined second skeleton key point of any one of the motion actions in each second video frame are initially changed:

respectively carrying out human body detection on each second video frame through a first convolutional neural network, and determining the human body image of the moving person in each second video frame;

and respectively identifying the human body images in the second video frames through a third convolutional neural network to obtain the position information and the depth information of each second skeleton key point of the moving person in each second video frame, or obtaining the position information and the depth information of each second skeleton key point of the moving person in each second video frame obtained in advance in the second video.

6. The method of claim 5, wherein determining an angular deviation of a first skeletal map of the first start action frame relative to a second skeletal map of the second start action frame comprises:

7. The method according to claim 6, wherein generating the first skeleton map corresponding to each first subframe according to the position information and the depth information of each first skeleton key point corresponding to each first subframe respectively comprises:

8. The method of claim 6, wherein calculating an angular offset of each first skeletal map relative to each second skeletal map comprises:

determining each first rotation angle of each edge in any first skeleton graph relative to a parent edge of each edge according to depth information and position information of each first skeleton key point in any first skeleton graph, wherein the parent edge of any edge is a first skeleton key point which is common to any edge and is adjacent to an edge with a pelvis center point as an origin;

determining each second rotation angle of each edge in any second skeleton graph relative to a parent edge of each edge in any second skeleton graph according to the depth information and the position information of each second skeleton key point in any second skeleton graph corresponding to any first skeleton graph;

and calculating each angle deviation of each first rotation angle relative to each second rotation angle, and determining each angle deviation as the angle deviation of any first skeleton diagram relative to any second skeleton diagram, wherein the angle deviation comprises the angle difference size and the angle difference direction.

9. An apparatus for assisting exercise training, comprising:

the acquisition module is used for acquiring a second video of the template aiming at any motion action and meeting the shooting angle;

a first determining module, configured to determine a first start motion frame of a first start motion of the any motion in the first video and a second start motion frame of a second start motion of the any motion in the second video;

and the second determination module is used for determining the angle deviation of the first skeleton diagram of the first initial action frame relative to the second skeleton diagram of the second initial action frame so as to assist a sporter to train any one sport action according to the angle deviation.

10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of assisted athletic movement training of any one of claims 1-8 when executing the program.

11. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon a computer program which, when being executed by a processor, carries out the method of assisted motor activity training according to any one of claims 1 to 8.