WO2021244411A1

WO2021244411A1 - Movement self-adaptive synchronization method and electronic device

Info

Publication number: WO2021244411A1
Application number: PCT/CN2021/096640
Authority: WO
Inventors: 马春晖; 黄磊; 陈霄汉; 赵杰; 郁心迪
Original assignee: 华为技术有限公司
Priority date: 2020-06-01
Filing date: 2021-05-28
Publication date: 2021-12-09
Also published as: CN113761966A

Abstract

The present application relates to the technical field of computer applications. Disclosed is a movement self-adaptive synchronization method. The number of frames of each workout movement of a workout video displayed via a display screen by an electronic device corresponds to the collected movement distance of each bone node when each workout movement is completed by a user; for the movement distance covered by the user when each workout movement is completed, a number of frames matching the movement distance is correspondingly played back, this allows the playback speed of a workout movement demonstrated by an instructor in the workout video to be synchronized with the speed at which the user makes the workout movement, thus ensuring the continuity and rhythm of the user making workout movements with the workout video, and enhancing workout experience for the user.

Description

Action adaptive synchronization method and electronic equipment

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on June 1, 2020, the application number is 202010484845.8, and the application name is "an action adaptive synchronization method and electronic equipment", the entire content of which is incorporated by reference In this application.

Technical field

This application relates to the field of computer application technology, and in particular to an action adaptive synchronization method and electronic equipment.

Background technique

With the increasing pressure of people’s lives in contemporary society, there is a serious shortage of fitness and leisure time. In order to reduce the time for users to move to the gym, more and more users choose indoor fitness. When exercising indoors, users often learn fitness actions through fitness video tutorials provided by fitness software on electronic devices, such as mobile phones, computers, smart screens, etc. However, current fitness software can only provide fitness video tutorials. Users in fitness video tutorials It’s difficult to match the speed of the same fitness action with the speed of the coach’s fitness action. This leads to the problem that the coach and the user’s fitness actions in the fitness video tutorial are not synchronized, which destroys the continuity and rhythm of the user’s fitness training, resulting in the user’s fitness experience Low.

Summary of the invention

The purpose of the present invention is to provide an action adaptive synchronization method, which ensures the synchronization of the fitness actions of the coach and the user in the fitness video tutorial, and improves the user's sense of fitness experience.

In the first aspect, an embodiment of the present invention discloses an action adaptive synchronization method, which is applied to an electronic device, and the action adaptive synchronization method includes: the electronic device plays a fitness video. Among them, the fitness video contains information such as the action posture of the fitness coach, the guidance voice of the fitness video, and text information.

The electronic device collects the user's action posture. The image acquisition module of the electronic device can provide the image information used on the electronic device to collect the posture, movement and other image information of the user in front of the electronic device. The image acquisition module can be embedded in the housing of the electronic device, such as the upper frame of the electronic device. , It can also be set independently of the electronic device, and the image acquisition module is set above the display screen of the electronic device. The image acquisition module may be a camera (such as an infrared camera) and the like.

The electronic device matches the fitness action currently played in the fitness video with the user's action posture. Among them, the electronic device determines the key bone nodes involved in the fitness action demonstrated by the coach in the fitness video and the key bone nodes involved in the user's current action posture collected by the camera of the smart screen, and determines the user and the coach in the fitness video based on the key bone nodes. Whether the fitness actions are synchronized.

According to the embodiment in the first aspect of the present application, at least one fitness trainer may demonstrate standard fitness actions in the fitness video.

According to the embodiment in the first aspect of the present application, when determining the key bone node, the following steps can be implemented:

The electronic device obtains one or more bone nodes of at least one fitness coach in the fitness video;

The electronic device obtains one or more bone nodes of the user;

The electronic device determines key bone nodes from one or more bone nodes of at least one fitness coach and one or more bone nodes of the user in the fitness video.

According to the embodiment in the first aspect of the present application, determining the key bone node includes:

The electronic device selects the key bone node of the currently demonstrated fitness action from the bone nodes of the user and at least one fitness trainer, where the key bone node is the bone node involved in the current fitness action.

According to the embodiment in the first aspect of the present application, the electronic device matching the fitness action currently played in the fitness video with the user's action posture includes:

Based on LSTM, the electronic device extracts the key bone nodes of the fitness action currently demonstrated by at least one fitness trainer in the current fitness video and the user's action posture to obtain the third posture feature of the currently demonstrated fitness action and the fourth posture feature of the user, based on the key The skeleton node matches the third posture feature with the fourth posture feature. Among them, the third posture feature and the fourth posture feature are the action postures of the key bone nodes.

According to the embodiment in the first aspect of the present application, the electronic device extracts the key bone nodes of the fitness action currently demonstrated by at least one fitness trainer in the current fitness video and the user's action posture based on the LSTM to obtain the first fitness action currently demonstrated A posture feature and a second posture feature of the user are matched with the first posture feature and the second posture feature. Among them, the first posture feature and the second posture feature are the action postures of the key bone nodes.

According to the embodiment in the first aspect of the present application, video framing is performed on the fitness video to obtain frame images of consecutive moments of the fitness video. Among them, the electronic device sequentially reads each fitness action in the fitness video according to the playback time sequence, and performs video framing for each fitness action. Video framing is a picture that divides each fitness action in the fitness video into one frame by one frame. Each frame in the video is a still picture, so that each fitness action is divided into multiple frames of image pictures.

If the fitness action does not match the user's current action posture, the electronic device adjusts the playback speed of the fitness action in the fitness video to synchronize the fitness action with the user's action posture.

According to the embodiment in the first aspect of the present application, the electronic device determines the target frame image corresponding to the second posture feature of the user. Based on the time relationship between the frame image corresponding to the first posture feature and the target frame image, the playback speed of the fitness video is adjusted to synchronize the fitness action corresponding to the target frame image with the user's action posture.

According to the embodiment in the first aspect of the present application, determining the target frame image corresponding to the second posture feature of the user includes:

The electronic device determines the target frame image based on the total number of frames of the complete fitness action corresponding to the second posture feature and the average moving distance, where the average moving distance is the key bone node corresponding to the second posture feature from the initial position to the end of the complete fitness action The average moving distance of the location. Wherein, the electronic device calculates the speed of the user when doing the fitness action based on the average moving distance and the number of frames of the fitness action. The camera of the electronic device collects the user's action posture in real time. When the user repeats the same fitness action, it calculates the real-time average movement distance of all the bone nodes involved in the fitness action, and correspondingly plays the frame corresponding to the average movement distance to achieve control playback The purpose of fitness video playback speed.

According to the embodiment in the first aspect of the present application, the average moving distance is that the key bone node corresponding to the second posture feature occurs in the three directions of the X axis, the Y axis, and the Z axis from the initial position to the end position in the three-dimensional coordinate system. The average value of the moving distance.

The embodiment of the first aspect of the present application discloses a method for adaptive synchronization of actions. The number of frames of each fitness action of the fitness video displayed on the display screen of the electronic device and the number of frames of each skeletal node when the user completes each fitness action are collected. The movement distance corresponds to the movement distance that occurs when the user completes each fitness movement. The number of frames that matches the movement distance is played correspondingly, so that the playback speed of the fitness movement demonstrated by the trainer in the fitness video is synchronized with the speed of the fitness movement performed by the user, ensuring that This improves the continuity and rhythm of the user's fitness actions with the fitness video, and improves the user's sense of fitness experience.

According to the embodiment in the first aspect of the present application, when the time point corresponding to the target frame image exceeds the time point of the frame image corresponding to the first posture feature, the electronic device adjusts the playback speed of the fitness video Is a target speed, where the target speed is greater than the current playback speed of the fitness video.

The time point corresponding to the target frame image refers to the value of the time corresponding to the target frame image, and the time point of the user's current fitness action refers to the time point corresponding to the first posture feature of the user's current fitness action. The relationship between the time point of the user's current fitness action and the time point corresponding to the target frame image is called the time relationship.

According to the embodiment in the first aspect of the present application, when the time point corresponding to the target frame image is greater than the time point of the frame image corresponding to the first posture feature, the electronic device controls the fitness video to be played The fitness action jumps directly to the target frame image. The time relationship between the time point of the frame image of the fitness action of the currently played fitness video and the time point of the target frame image is: the time of the frame image of the fitness action of the currently played fitness video is earlier than the time of the target frame image.

According to the embodiment in the first aspect of the present application, when the time point corresponding to the target frame image is less than the time point of the frame image corresponding to the first posture feature, the electronic device controls the fitness video playback and is controlled by The currently played fitness action jumps directly to the target frame image.

According to the embodiment in the first aspect of the present application, when the fitness action is not synchronized with the user's action posture, the electronic device provides voice prompts and/or text prompts. Wherein, when the fitness action is not synchronized with the user's action posture, the electronic device displays words such as "action not synchronized" on the display screen or prompts "action not synchronized" through a speaker.

According to the embodiment in the first aspect of the present application, the electronic device displays the fitness video on a full screen.

According to the embodiments in the first aspect of the present application, in some embodiments of the present application, when the video interface is displayed in a full screen, the video interface may occupy all the display area of the display screen. In a possible implementation manner, the full-screen display of the video interface refers to that only the video interface is displayed on the display screen, and no other content is displayed. In another possible implementation manner, the video interface can also only occupy part of the display area of the display screen. For example, the video interface is displayed in the middle part of the display screen, and it can also be viewed when the edge part of one or both sides is white or black. The video interface is displayed on the full screen of the display screen.

According to the embodiment in the first aspect of the present application, the user's action posture is displayed in the floating window of the display screen of the electronic device.

According to the embodiment in the first aspect of the present application, the electronic device displays the user's real-time fitness window on the full screen of the display screen, and displays the fitness video window floating in the lower left corner of the display screen. And the text prompt of the current fitness action is displayed on the display screen.

According to the embodiment in the first aspect of the present application, the electronic device stores the user's action posture; the electronic device determines the user’s action posture based on the standard fitness action corresponding to the user’s action posture. The action posture standard does not match;

Then mark the user’s non-standard action and replay the standard fitness action corresponding to the non-standard action in the fitness video; the electronic device replays the fitness speed at which the user performs the fitness action corresponding to the non-standard action The standard fitness action in the fitness video is played, and the standard fitness action corresponds to the fitness action corresponding to the non-standard action.

The electronic device provides text and/or voice explanation of the non-standard action.

According to the embodiment in the first aspect of the present application, the electronic device determines that the user has not completed the fitness action based on the user's motion posture and motion trajectory collected in real time by the camera, the electronic device pauses playing the fitness video, and displays the fitness video on the electronic device. The display shows the text prompt of the current fitness action being performed by the user until the camera collects in real time that the user has completed the fitness action that has been broadcasted and the user’s current fitness action matches the action posture corresponding to the current pause screen. The video playback is resumed at the current pause time, or the fitness video is played back to the screen corresponding to the user's current action posture to synchronize the fitness action again.

According to the embodiment in the first aspect of the present application, the electronic device plays fitness videos on the display screen and collects the user’s action posture in real time. The electronic device determines whether the user’s fitness action is standard. The movement is regarded as a non-standard fitness movement, and the fitness speed at which the user performs the non-standard fitness movement is calculated. The electronic device plays the standard fitness movement demonstrated by the instructor corresponding to the non-standard fitness movement in the fitness video at the fitness speed, so as to respond to the non-standard fitness movement performed by the user. Standard fitness exercises are corrected.

According to the embodiment in the first aspect of the present application, when the non-standard action video is played, voice explanations and voice prompts are provided to the user to correct the wrong fitness action of the user.

Obviously, the electronic device stores the video of the non-standard fitness actions performed by the user. After the entire fitness video on the display screen is played, it will play the non-standard fitness video that the user did wrong, and perform it with the standard fitness video demonstrated by the instructor. Comparing and explaining, correcting users' wrong fitness actions, is conducive to the user's physical health and further enhances the user's sense of experience.

In the second aspect, an embodiment of the present application discloses an electronic device, including:

A memory, the memory stores synchronization instructions;

The processor, when the processor executes the synchronization instruction, implements the steps of the action adaptive synchronization method mentioned in any one of the above.

In the third aspect, an embodiment of the present application discloses a motion adaptive synchronization device, the motion adaptive synchronization device includes:

Play module, used to play fitness videos;

The collection module is used to collect the user's action posture;

The matching module is used to match the fitness action currently played in the fitness video with the user's action posture.

The adjustment module is used to adjust the playing speed of the fitness action in the fitness video, so that the fitness action is synchronized with the user's posture.

Other features and corresponding beneficial effects of the present invention are described in the latter part of the specification, and it should be understood that at least part of the beneficial effects will become apparent from the description in the specification of the present invention.

Description of the drawings

Figure 1(a) is a schematic structural diagram of a smart screen in an application scenario shown in an embodiment of the application;

Figure 1(b) is a schematic diagram of the positional relationship between the smart screen and the image acquisition module shown in an embodiment of the application;

FIG. 1(c) is a schematic diagram of the use state of the action-adaptive synchronization device shown in an embodiment of this application;

Figure 1(d) is a schematic diagram of a human skeleton model shown in an embodiment of the application;

Figure 1(e) is a schematic diagram of a UI interface of a smart screen shown in an embodiment of the application;

Figure 1(f) is a schematic diagram of a fitness page after a user clicks on a fitness APP according to an embodiment of the application;

2(a) to 2(d) are UI interface diagrams of an action adaptive synchronization method shown in an embodiment of this application;

Figure 3(a) is a schematic flowchart of an action adaptive synchronization method shown in an embodiment of this application;

FIG. 3(b) is a schematic diagram of an action adaptive synchronization state shown in an embodiment of this application;

4(a) to 4(c) are UI interface diagrams of another action adaptive synchronization method shown in an embodiment of this application;

FIG. 5 is a schematic flowchart of another action adaptive synchronization method shown in an embodiment of this application;

6(a) to 6(c) are UI interface diagrams of another method for adaptive synchronization of actions shown in an embodiment of this application;

FIG. 7 is a schematic flowchart of another action adaptive synchronization method shown in an embodiment of the application;

FIG. 8 is a schematic structural diagram of an electronic device shown in an embodiment of the application;

FIG. 9 is a schematic structural diagram of an SOC shown in an embodiment of the application.

detailed description

The following specific examples illustrate the implementation of the present application, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. Although the description of this application will be introduced in conjunction with the embodiments, this does not mean that the features of this application are limited to this embodiment. On the contrary, the purpose of introducing the application in combination with the embodiments is to cover other options or modifications that may be extended based on the claims of this application. In order to provide an in-depth understanding of this application, the following description will contain many specific details. This application can also be implemented without using these details. In addition, in order to avoid confusion or obscuring the focus of this application, some specific details will be omitted in the description. It should be noted that the embodiments in this application and the features in the embodiments can be combined with each other if there is no conflict.

It can be understood that, as used herein, the term "module" can refer to or include an application specific integrated circuit (), an electronic circuit, a processor (shared, dedicated, or group) that executes one or more software or firmware programs, and/ Or memory, combinational logic circuits, and/or other suitable hardware components that provide the described functions, or may be part of these hardware components.

It can be understood that, in the embodiments of the present application, the processor may be a microprocessor, a digital signal processor, a microcontroller, etc., and/or any combination thereof. According to another aspect, the processor may be a single-core processor, a multi-core processor, etc., and/or any combination thereof.

The following describes electronic devices that provide fitness video tutorials according to some embodiments of the present application. Electronic devices include, but are not limited to, mobile phones, tablets, laptop computers, desktop computers, virtual reality or augmented reality devices, smart screens and other electronic devices. For the above electronic devices, in order to implement the technical solutions provided in the embodiments of this application, the electronic devices include but are not limited to display screens and image acquisition devices. One of the purposes of setting the display screens on the electronic devices is to display fitness videos. One of the purposes of setting up a video capture device on an electronic device is to capture the user's movement posture within the collection range of the video capture device of the electronic device. The movement distance of each skeletal node when the user completes each fitness action, the movement distance that the user completes each fitness action and the total number of frames of the fitness action in the fitness video are used to control the playback speed of the fitness video, so that the fitness video The playback speed of the fitness action is synchronized with the speed at which the user performs the fitness action.

Figure 1(a) is a schematic structural diagram of a smart screen in an application scenario shown in an embodiment of this application, and Figure 1(b) is a schematic diagram of a positional relationship between a smart screen and an image acquisition module shown in an embodiment of this application, and Figure 1(c) is A schematic diagram of the use state of an action-adaptive synchronization device shown in an embodiment of this application. Figure 1(d) is a schematic diagram of a human skeleton model shown in an embodiment of this application. Figure 1(e) is a schematic diagram of a smart screen shown in an embodiment of this application Schematic diagram of the UI interface. Fig. 1(f) is a schematic diagram of the fitness page after the user clicks on the fitness APP according to an embodiment of the application.

In some embodiments of the present application, the electronic device is a smart screen as an example to describe the action adaptive synchronization method provided in the embodiments of the present application. As shown in FIG. 1(a), the smart screen 10 includes: a processor 101, a power module 102, a memory 103, a wireless communication module 104, a display screen 105, an image acquisition module 106, and so on.

In some embodiments of the present application, it can be understood that the structure shown in FIG. 1(a) illustrated in the embodiments of the present application does not constitute a specific limitation on the smart screen 10. In other embodiments of the present application, the smart screen 10 may include more or fewer components than shown in FIG. 1(a), or combine certain components, or split certain components, or arrange different components. . The components shown in Figure 1(a) can be hardware, software, or a combination of software and hardware.

In some embodiments of the present application, the processor 101 may include one or more processing units for application on the smart screen 10. For example, the processor 101 includes, but is not limited to, a central processing unit (CPU), and image processing. Processor (Graphics Processing Unit, GPU), digital signal processor (Digital Signal Processing, DSP), microprocessor (Micro-programmed Control Unit, MCU), artificial intelligence (Artificial Intelligence, AI) processor or programmable logic device ( Field Programmable Gate Array, FPGA) and other processing modules or processing circuits. Among them, the different processing units may be independent devices or integrated in one or more processors. A storage unit may be provided in the processor 101 for storing instructions and data. In some embodiments, the storage unit in the processor 101 is a cache memory. It should be noted that the processor 101 may also serve as a controller.

In some embodiments of the present application, the power module 102 may provide a battery, a battery management component, etc., which are applied to the smart screen 10. The power management component is used to manage the charging of the power supply and the power supply of the power supply to other modules. The battery can be a lithium battery, a lead-acid battery and other rechargeable batteries. The shape of the battery can be but not limited to button type, square type, etc.

In some embodiments of the present application, the memory 103 may include one or more tangible, non-transitory computer-readable media for storing data and/or instructions. In some embodiments, the memory 103 includes, but is not limited to, any suitable non-volatile memory such as flash memory and/or any suitable non-volatile storage device, such as a hard disk drive (HDD), and an optical disk (compact disc). , CD) drive, at least one of the digital versatile disc (digital versatile disc, DVD) drive, the fitness video tutorial can be cached in the memory 103, when the user determines to play the fitness video tutorial, the fitness video tutorial can be directly retrieved from the memory 103 And play on the display screen 105. In addition, the memory 103 can also pre-store a human skeleton model. Each bone node of the human skeleton model is numbered or defined with a name. The processor 101 can directly retrieve it from the memory 103 The human skeleton model is applied.

In some embodiments of the present application, the wireless communication module 104 may provide applications on the smart screen 10 including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (bluetooth, BT), global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC) and other wireless communication solutions, smart screen 10 can Communicate with the network, mobile phones, remote controllers, etc. through wireless communication technology. In some embodiments of the present application, the wireless communication module 104 may be a Bluetooth chip, and the communication method is Bluetooth communication, in order to reduce the power consumption of the Bluetooth chip on the smart screen 10 , Bluetooth chip can be selected low-power Bluetooth chip, and its model can be selected as NRF52810-QFAA-R Bluetooth 5.0 low-power Bluetooth chip.

In some embodiments of the present application, the display screen 105 may provide an application on the smart screen 10 for displaying human-computer interaction interfaces, images, fitness APP interaction interfaces, fitness videos, and the like. The display screen 105 includes but is not limited to a display panel. The display panel can use liquid crystal display (LCD), organic light-emitting diode (organic light-emitting diode, OLED), active-matrix organic light-emitting diode or active-matrix organic light-emitting diode (active-matrix organic light-emitting diode). Diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (QLED), etc.

In some embodiments of the present application, the image acquisition module 106 may provide the image information used on the smart screen 10 to collect the posture and actions of the user in front of the smart screen 10. The image acquisition module may be embedded in the shell of the smart screen 10. If it is embedded in the upper frame of the smart screen, it can also be set independently of the smart screen 10. As shown in FIG. 1(b), the image acquisition module 106 is arranged above the display screen of the smart screen 10. The video transmission line connects to the smart screen 10. The image acquisition module 106 may be a camera (such as an infrared camera) or the like.

In some embodiments of the present application, the image acquisition module 106 includes but is not limited to a lens part, a chipset part, a PCB and a component part, etc. The lens part includes but is not limited to the lens. The lens is composed of lenses (convex lens and concave lens), which can be classified into plastic lenses and glass lenses in terms of material; according to color classification, lenses can be divided into color lenses and black and white lenses; according to functional classification, lenses Can be divided into fixed lens, zoom lens; according to the size classification, the lens can be divided into single-plate lens, pinhole lens, CS lens; according to the infrared classification, the lens can be divided into 850nm lens, 940nm lens, 650nm lens. The chip set is the heart of the camera and is the main component that converts light signals into electrical signals. According to the characteristics of the photosensitive chip in the chipset, the main chip components are CCD chips and CMOS chips. The PCB and components are an important part of the camera and the carrier of all devices. Its main function is to provide electrical connections between various parts.

In some embodiments of the present application, as shown in Figure 1(c), the power supply is for the smart screen. After the smart screen is powered on, the user operates the smart screen and uses the fitness video tutorial played by the smart screen shown in the embodiments of this application Get fit.

The image acquisition module 106 is arranged on the upper edge of the display 105 of the smart screen 10. The image acquisition module 106 can be connected to the smart screen 10 through a video transmission line, and the smart screen 10 plays and displays fitness videos of the fitness APP through the display 105.

Among them, the fitness video includes the action posture of the fitness coach, the guidance voice of the fitness video, and text information. Among them, the fitness video can be pre-stored in the memory 103, or can be downloaded and played through the network in real time. The memory 103 also pre-stores The human skeleton model, each skeletal node of the human skeleton model is numbered or defined by name.

In some embodiments of the present application, as shown in Figure 1(d), the bone points of the human body are set to 15 bone points, and the 15 bone points are the center of the head and the center of the neck (for example, the center of the neck). Spine center) 2, left shoulder joint point 3, right shoulder joint point 4, left elbow joint point 5, right elbow joint point 6, left wrist joint point 7, right wrist joint point 8, left hip joint point 9, left knee joint point 10. The left ankle joint point 11, the right hip joint point 12, the right knee joint point 13, the right ankle joint point 14 and the torso center (such as the spine center of the torso) 15. Divided into five body parts, namely the torso, left arm, right arm, left leg and right leg, where the torso includes, for example, the spine center of the torso and the torso center (for example, the spine center of the torso) 15, and the left arm includes the left shoulder joint point 3, Left elbow joint point 5 and left wrist joint point 7, right arm includes right shoulder joint point 4, right elbow joint point 6 and right wrist joint point 8, left leg includes left hip joint point 9, left knee joint point 10 and left ankle At node 11, the right leg includes right hip joint point 12, right knee joint point 13 and right ankle joint point 14. In addition, it should be noted that in addition to numbering the bone nodes to distinguish, you can also define the name of each bone node to distinguish, such as head, neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right For the wrist, left hip, left knee, left ankle, right hip, right knee, right ankle, and torso, the embodiment of the present application does not limit the way in which the bone nodes are defined and distinguished. When the user performs fitness exercises, the torso, left arm, right arm, left leg, and right leg coordinate with each other to complete a series of fitness exercises.

In some embodiments of the present application, the displacement distance and displacement direction of each skeletal node of the human skeleton model can be mapped to the positive direction of the X axis, the negative direction of the X axis, the positive direction of the Y axis, and the negative direction of the Y axis of the three-dimensional coordinate system. The positive direction of the Z axis and the negative direction of the Z axis are calculated through the three-dimensional coordinate system when each bone node is displaced.

In some embodiments of the present application, the fitness actions and voice text displayed by the coach in the fitness video displayed on the display screen 105 are used to prompt the user to perform fitness actions simultaneously. The image collection on the upper edge of the display screen 105 of the smart screen 10 The module 106 collects the posture information of the user’s fitness actions in real time and transmits it to the processor 101 of the smart screen 10 through a video transmission line or wireless communication. The processor 101 reads the fitness video and performs frame processing on the fitness video, and the fitness video is involved All of the fitness actions are divided into segments (for example, a fitness action in the fitness video is to raise the left leg from the initial position to the left side, and then restore the left leg from the raised position to the initial position. This fitness action can be divided There are two segments, that is, the left leg is raised from the initial position to the left side as the first segment, and the left leg is restored from the raised position to the initial position as the second segment), and then the video of each exercise is video Framing processing, video framing processing is to divide each fitness action in the fitness video into a frame by frame. Each frame in the video is a still picture to obtain frame images of consecutive moments of the fitness action. After the video is divided into frames, count the total number of frames after each fitness action is divided into frames (take the left leg raised to the left as an example, the total number of frames after the divided frame processing is 40 frames); 10 Collect the posture information of the user’s initial fitness action collected by the camera, and count the number or name of the bone node (left hip joint) involved when the user initially performs a certain fitness action (take the left leg raised to the left as an example) Point 9, left knee joint point 10 and left ankle joint point 11), when the user initially performs a fitness exercise of raising his left leg to the left, the involved bone nodes are left hip joint point 9, left knee joint point 10 and The distance that the left ankle joint point 11 moves from the original position when the feet are standing to the left knee joint point of the left leg is raised to the final position are: the movement distance of the left hip joint point, the movement distance of the left knee joint point and the left ankle The distance the node moves.

The smart screen 10 calculates the average of the movement distance of the left hip joint point, the movement distance of the left knee joint point and the movement distance of the left ankle joint point of the fitness action of the user's left leg to the left side to complete the left leg to the left. The average movement distance of the joint points involved in the side-lift fitness exercise, and then calculate the ratio of the average movement distance to the total number of frames of the fitness exercise of raising the left leg to the left, and get the left leg to the left. The movement unit distance of the joint points involved in the fitness exercise of the square raising corresponds to the number of frames that the fitness exercise of raising the left leg to the left side should be played. When the user performs the fitness exercise of raising the left leg to the left side again , Calculate the real-time average movement distance of the user’s left hip joint point 9, left knee joint point 10, and left ankle joint point 11 when the user’s left leg is raised to the left side, and control the fitness video to play the left leg to the left side. The number of frames corresponding to the average distance between this fitness action and the real-time movement can be increased to achieve the purpose of synchronization between the fitness action and the fitness video by the user.

Example 1

FIG. 1(e) exemplarily shows an exemplary user interface 10 on the smart screen 10 for displaying applications installed on the smart screen 10.

The user interface 100 may include: a status bar 101 and an application icon 102. in:

The status bar 101 may include: one or more signal strength indicators 101C of wireless fidelity (Wi-Fi) signals, a battery status indicator 101D, and a time indicator 101E.

In some embodiments, the status bar 101 may also include: one or more signal strength indicators of a mobile communication signal (also called a cellular signal), and an operator name (for example, "China Mobile").

The application icon 102 can be: fitness icon 102A, gallery icon 102B, music icon 102C, application icon 102D, contact icon 102E, mailbox icon 102F, cloud sharing icon 102G, memo icon 102H, settings The icon 102I of the camera, the icon 102J of the camera. The user interface 10 may also include a page indicator 103. The icons of other applications may be distributed on multiple pages, and the page indicator 103 may be used to indicate the application in which page the user is currently browsing. The user can swipe the area of other application icons left and right to browse application icons in other pages.

In some embodiments of the present application, the user interface 100 may further include: a navigation bar 104.

The navigation bar 104 may include system navigation keys such as a return key 104A, a home screen key 104B, and a multi-task key 104C. When it is detected that the user clicks the return button 104A, the smart screen 100 can display the previous page of the current page. When it is detected that the user taps the home button 104B, the smart screen 100 can display the home interface. When it is detected that the user clicks on the multitasking button 104C, the smart screen 100 may display the task recently opened by the user. The naming of each navigation key can also be other, which is not limited in this application. Not limited to virtual keys, each navigation key in the navigation bar 104 can also be implemented as a physical key.

In some embodiments, the user interface 100 exemplarily shown in FIG. 1(e) may be a home screen (Home screen).

In some other embodiments, the smart screen 100 may also include a front camera. The front camera may also be called a secondary camera, which is mainly located above the screen of the smart screen 100. The front camera is shown in Figure 1(a) in the embodiment of the present application. The image acquisition module 106 in, which can collect image information such as postures and actions of the user in front of the smart screen 10, as well as self-portraits.

In some other embodiments, the smart screen 10 may also include a home screen key. The main screen key can be a physical key or a virtual key. The home screen key can be used to receive instructions from the user and return the currently displayed UI to the home interface, so that it is convenient for the user to view the home screen at any time. The above instruction can be an operation instruction for the user to press the home screen key once, or an operation instruction for the user to press the home screen key twice in a short period of time, or the user to press and hold the home screen key for a predetermined period of time. Operation instructions. In some other embodiments of the present application, the home screen key can also be integrated with a fingerprint recognizer, so that when the home screen key is pressed, fingerprints are collected and recognized accordingly.

It can be understood that FIG. 1(e) only exemplarily shows the user interface on the smart screen 10, and should not constitute a limitation to the embodiment of the present application.

Exemplarily, as shown in FIG. 1(f), the user can click on the fitness icon 102A on the user interface 100, the smart screen 10 detects the above-mentioned user operation, and in response to the above-mentioned user operation, the smart screen 10 displays the interface 11 of the fitness app.

The user interface 11 may include: an application title bar 201, a search box 202, a setting 203, a function bar 204, and a display area 205. in:

The application title bar 201 can be used to indicate that the current page is used to display the setting interface of the smart screen 10. The presentation form of the application title bar 201 may be text information "smart fitness", icons or other forms.

The search box 202 can be used to search for fitness courses matching the character according to the character input by the user.

The setting 203 may receive a user operation (for example, a touch operation), and in response to the detected user operation, the smart screen 10 may display a setting interface of smart fitness.

The function bar 204 may include: a user center control 204A, a course recommendation control 204B, and multiple course classification controls. The above-mentioned multiple course classification controls may include, but are not limited to: fat burning zone control 204C, shaping zone control 204D, and shaping zone control 204E. in

The user center control 204A can receive a user operation (such as a touch operation), and in response to the detected user operation, the smart screen 10 can display the interface content of the user's personal center in the display area 205.

The course recommendation control 204B can receive a user operation (such as a touch operation), and in response to the detected user operation, the smart screen 10 can display one or more recommended fitness courses in the display area 205 of the display screen 105. For example, as shown in FIG. 1(e), the display area 205 displays the course covers of multiple recommended courses, as well as the course classification, duration, and name of each recommended course.

The controls in the above-mentioned multiple course classification controls can receive user operations (such as touch operations). In response to the detected user operation, the smart screen 10 can display one or more corresponding course classification controls in the display area 205 of the display screen 105. A course cover for a fitness course.

In some embodiments of the present application, the course cover or name of the aforementioned fitness course can receive the user's playback operation (such as touch operation). In response to the aforementioned playback operation detected, the smart screen 10 can display the specific fitness course on the display screen. content.

The fitness app may be started in response to a user's touch operation on the icon of the fitness app, for example, single click, double tap, long press, and so on. In some embodiments of the present application, the display screen is equipped with a touch panel, which can be used to receive a user's touch operation. The touch operation refers to the operation of the user's hand, elbow, stylus, etc., touching the display screen. In specific implementation, there may be other ways to open the interface 11 of the fitness app. There is no limitation here.

For example, the user can open the fitness app interface 11 by pressing the button to turn on the first control mode; or, by detecting the voice input by the user, open the fitness app interface 11; or, by drawing a specific shape (such as Z-shape, rectangle, Circle, etc.) Open the interface 11 of the fitness app. The embodiment of the application does not specifically limit this.

In addition, the user can also control the smart screen 10 to display the fitness app interface 11 through the remote control; the user can also control the smart screen 10 to display the fitness app interface 11 through specific gestures. In the embodiments of the present application, user operations are not specifically limited.

It is understandable that Fig. 1(e) only exemplarily shows the fitness app interface on the smart screen 10, and should not constitute a limitation to the embodiment of the present application.

Fitness courses usually include multiple actions, and there can be a preset rest time between two consecutive actions. The fitness course can be recommended by the smart screen based on the user's historical fitness data, or can be selected by the user according to actual needs. Fitness courses can be played locally or online. There is no specific limitation here.

In some embodiments of the present application, the smart screen 10 can receive the user's playback operation. In response to detecting the above-mentioned playback operation, the smart screen 10 can display the specific content of the fitness course in the fitness course display area of the display screen, and at the same time display the specific content of the fitness course on the display screen. The user’s fitness display area displays the user’s body posture in real time.

In some embodiments of the present application, the smart screen 10 displays the user's fitness video in real time in the user fitness display area according to the video data collected by the camera.

The aforementioned camera is used as the image acquisition module 106 of the smart screen. It may be the front camera of the smart screen 10, and the aforementioned camera may also be an external camera of the smart screen 10, which is not specifically limited here.

In the following, an exercise adaptive synchronization method provided by the embodiment of the present application will be described by taking the fitness action of "straight back and back, and raise the left knee to the left side from the front of the eyes" as an example.

Exemplarily, as shown in FIG. 2(a), the user can click on the recommended course displayed in the display area 205, the smart screen 10 detects the above user operation, and in response to the above user operation, the smart screen 10 displays the training interface 12.

The training interface 12 includes a fitness course window 301 and a user fitness window 302. in:

The fitness course window 301 is used to display specific content of the fitness course.

The user fitness window 302 is used to display the user's fitness actions collected by the camera in real time.

The camera that plays fitness videos and smart screens on the display screen collects the user's movement posture in real time. Before the display screen starts to play the fitness video, the trainer and the user's movements in the fitness video are in the initial state, and the upcoming UI is displayed on the initial UI interface. The text description of the fitness action, the right side shows the fitness exercise demonstrated by the coach, the left side shows the user’s current posture, the middle position of the display 105 shows the fitness exercise essentials of the text prompt of the fitness exercise, and then starts to play the fitness video on the display .

In some embodiments of the present application, the smart screen 10 may display a user's fitness video in a full screen, and display a small window of fitness courses floating on the display screen.

Exemplarily, as shown in FIG. 2(b), the smart screen 10 displays the user fitness window 302 in the full screen of the training interface 12, and displays the fitness course window 301 floating in the lower left corner of the display screen. And the text prompt of the current fitness action is displayed on the training interface 12.

In some embodiments of the present application, the smart screen 10 displays fitness courses on a full screen, and a small window that displays a user's fitness video is floating on the display screen.

In some embodiments of the present application, when the video interface is displayed in a full screen, the video interface may occupy all the display area of the display screen. In a possible implementation manner, the full-screen display of the video interface refers to that only the video interface is displayed on the display screen, and no other content is displayed. In another possible implementation manner, the video interface can also only occupy part of the display area of the display screen. For example, the video interface is displayed in the middle part of the display screen, and it can also be viewed when the edge part of one or both sides is white or black. The video interface is displayed in full screen on the screen.

In other embodiments of the present application, the full-screen display of the video interface may mean that while the video interface is displayed on the display screen, system-level interface elements, such as status bar, floating shortcut menu, etc., can also be displayed.

In addition to the window display modes shown in FIGS. 2(a) and 2(b), the fitness course window 301 and the user fitness window 302 may also be displayed in other display modes, which are not specifically limited in the embodiment of the present application.

After the user performs the fitness action shown in Figure 2(a), the smart screen 10 determines the key bone nodes involved in the fitness action demonstrated by the trainer in the fitness video and the key bones involved in the user’s current action posture collected by the smart screen’s camera Nodes, and based on the key bone nodes, determine whether the fitness actions of the user and the coach in the fitness video are synchronized. If they are not synchronized, the smart screen 10 controls and adjusts the playback speed of the fitness video.

In some embodiments of the present application, as shown in Figure 2(c), the smart screen displays the coach’s action of “straight back and raise the left knee to the left” in real time on the display screen of the smart screen. Posture, the coach’s gesture of "raising left knee to the left" is displayed on the screen at the current moment (the screen shows the 50th frame when the left knee is raised to the standard position), on the upper left of the screen The corner also displays the text prompt of the fitness action "back straight, visually raise the left knee to the left side". The user compares the fitness action and text prompt demonstrated by the trainer on the display to perform fitness.

The lower left corner of the display shows that the user's action posture is "left knee raised to the left" in real time (the display shows the picture of frame 25 when the left knee is raised to half the distance from the standard position), which is obvious What’s more, when the user initially does the fitness action of "back straight, looking forward to the left knee and raising the left knee to the left", the user’s fitness action is not consistent with the fitness action demonstrated by the coach in the fitness video, that is, It is said that the user’s fitness speed is slower than the fitness speed demonstrated by the trainer in the fitness video. In order to remind the user, text prompts can be provided on the display screen, such as the text prompt "Please note that the action does not match".

In some embodiments of the present application, as shown in Figure 2(d), the playback speed of the fitness video is adjusted, that is, the playback speed of the fitness video is adjusted to 0.5 times speed and the text prompt "Currently at 0.5 times speed is displayed on the display screen" "Play", so that when the user's next action is "raise the left knee to half the distance from the standard position", in the fitness video, "back straight, look forward to the left knee to the left side The fitness action “High” plays the 25th frame to ensure that the playback speed of the fitness action demonstrated by the instructor in the fitness video is synchronized with the speed of the fitness action performed by the user.

As shown in Figure 3(a), according to some embodiments of the present application, the process of performing action adaptive synchronization through the present application is as follows:

Step S30: The power module supplies power to the smart screen. The user operates the remote control to control the smart screen to turn on, and clicks on the fitness app on the interactive interface of the smart screen’s display screen to open the fitness video on the fitness app and play it, among which, play on the smart screen In the case of the fitness video, if the fitness video is pre-stored in the memory of the smart screen, the fitness video is directly extracted from the memory and played on the display screen. If the fitness video is not pre-stored in the memory, the fitness video is downloaded through the network and displayed Played on the display screen and buffered in the memory. The fitness video contains multiple fitness exercises demonstrated by the fitness trainer, voice guidance, and text prompts. For example, according to the time sequence, the first fitness exercise is to straighten your back and look visually. Raise the left knee from the front to the left. The second exercise is to bend the left side of the body and touch the elbow and knee. The third exercise is to raise the leg (both the left leg and the right leg are completed once a raised leg is complete When the trainer demonstrates fitness exercises, the trainer will give voice prompts and display the text of the voice on the display screen of the smart screen.

Step S31: The smart screen reads each fitness action in the fitness video in sequence according to the playback time, and performs video framing for each fitness action. Video framing is a picture that divides each fitness action in the fitness video into one frame by one frame. , Each frame in the video is a still image, so that each fitness action is divided into multiple frames of image images, each frame of each continuous fitness action in the fitness video is a still image, for example Divide the first fitness movement into 50 frames from the back straight and the movement from the front to the left knee lifted to the left. The second fitness movement is to bend the left side of the body with the elbow touching the left knee. The action is divided into 25 frames, and the third action is the action of raising the leg, divided into 50 frames.

Step S32: The smart screen recognizes the trajectory of each fitness action demonstrated by the coach of the fitness video, extracts the text description and voice description of each fitness action, and parses the keywords from it to determine the bones involved in each fitness action demonstrated by the coach Information about the node. Among them, the analysis of the motion trajectory and keywords of fitness actions can be realized by machine learning, such as Long Short-Term Memory (LSTM), Support Vector Machine (SVM), and so on. Take, for example, that the first fitness action played in the fitness video is "back straight, look forward to the left knee to the left side" as an example. While the instructor demonstrates the fitness exercises, the instructor will broadcast the prompt voice of "back straight, look forward to the left knee to the left side" and display it in the form of text on the display. The smart screen recognizes fitness based on LSTM Exercise and extract keywords from the prompt voice and text: "lower back", "knee", "left side" and "eye front" to divide the body of the fitness exercise into head, neck, torso and left The four parts of the leg, and the bone nodes involved in the above body parts are identified as the head center 1, the neck center 2, the torso center 15, the left hip joint point 9, the left knee joint point 10, and the left ankle joint point 11. In addition, based on machine learning, each fitness action can also be segmented. For example, the first fitness action played in the fitness video is "back straight, look forward to the left knee to the left side", based on LSTM divides this fitness exercise into two segments, the first segment is "back straight, look forward", and the second segment is "raise the left knee to the left". In this way, a fitness exercise is divided into two segments , Can better synchronize with the user's fitness speed.

Step S33: When the smart screen displays the fitness video, the user stands within the camera range of the camera facing the smart screen, and follows the fitness actions demonstrated by the instructor displayed on the display, text prompts, etc. to perform fitness actions. The user does this for the first time During a certain fitness action, the camera can collect the user's action posture within the camera range at a sampling frequency of 5 times per second and transmit it to the processor. Based on the collected action posture, the processor extracts the fitness action currently displayed on the display from the memory. , And use machine learning (such as LSTM) to identify the user’s bone nodes that match the coach’s bone nodes involved in the fitness action currently played on the display, and calculate the bone nodes of all users involved in the fitness action from the initial position to the end The average moving distance of the location.

In some embodiments of the present application, the first fitness action played in the fitness video is "back straight, the front of the eye is raised to the left knee to the left" as an example, the fitness action is 50 frames in total , The bone nodes involved in this action are head center 1, neck center 2, torso center 15, left hip joint point 9, left knee joint point 10, and left ankle joint point 11, as shown in Figure 3(b), In the three-dimensional coordinate system, when the user performs the movement of raising the left knee to the left for the first time, the displacement directions of the left knee joint point 10 and the left ankle joint point 11 are the positive direction of the Z axis and the positive direction of the Y axis. The displacement distance of the left knee joint point 10 in the positive direction of the Z-axis from the initial position of this fitness exercise is 80cm, the displacement distance of the left knee joint point 10 in the positive direction of the Y-axis is 50cm, and the left ankle joint point 11 from this From the initial position of the exercise, the displacement distance to the positive direction of the Z-axis is 60m, the displacement distance of the left ankle joint point 11 to the positive direction of the Y-axis is 40cm, the center of the head 1, the center of the neck 2, and the left hip joint point. 9. The displacement of the torso center 15 is zero, the displacement distance of the left knee joint point 10 is about 90.2cm, and the displacement distance of the ankle joint point 11 is about 72.1cm. The average displacement distance of the left knee joint point 10 and the left ankle joint point 11 of the fitness exercise "to the left knee raise to the left" is the displacement distance of the left knee joint point 10 and the displacement distance of the ankle joint point 11. The average value is about 81.2cm.

Step S34: The smart screen calculates the speed of the user during the fitness action based on the average moving distance and the number of frames of the fitness action. In some embodiments of the present application, still taking the fitness action of "back straight, looking forward to the left knee to the left side" as an example, the fitness action is a total of 50 frames, then the user completes the "back Straighten up, look forward to the left knee and lift up to the left side." This fitness action, the head center 1, the neck center 2, the left hip joint point 9, and the torso center 15 have zero displacement, and the left knee joint point 10 and the left ankle joint point 11 in the positive direction of the Z-axis and the Y-axis from the initial position of this exercise to the positive direction of the average displacement distance is 81.2cm, then the user is doing "back straight, looking forward to Raise your left knee to the left. The fitness speed during this fitness exercise is

Step S35: The camera of the smart screen collects the user's action posture in real time at a sampling frequency of 5 times per second. When the user repeats the same fitness action, the real-time average movement distance of all the bone nodes involved in the fitness action is calculated, and the corresponding playback The frame corresponding to the average moving distance achieves the purpose of controlling the playback speed of the fitness video. In some embodiments of the present application, still taking the fitness action of "back straight, looking forward to the left knee to the left side" as an example, the user is doing "back straight, looking forward to the left During the fitness exercise of raising the knee to the left, the displacement of the head center 1, the neck center 2, the left hip joint point 9, and the torso center 15 are zero, the left knee joint point 10 and the left ankle joint At point 11, the real-time average displacement distance in the positive direction of the Z-axis and the positive direction of the Y-axis from the initial position of this fitness exercise is 40cm. The speed of the "high" fitness action is 0.62 frames/cm, then the fitness action of the fitness video is "back straight, look forward to the left knee to the left side" should be played to 0.62. ×40 frames, about the 25th frame, that is, when the real-time average displacement of the user’s left knee joint point 10 and left ankle joint point 11 in the positive direction of the Z-axis and the positive direction of the Y-axis is 40cm, the smart screen’s The fitness video played on the screen is "back straight, look forward to the left knee to the left side" this fitness action should play the 25th frame. The user continues to perform the fitness exercise until the completion time (the real-time average displacement of the left knee joint point 10 and the left ankle joint point 11 in the positive direction of the Z-axis and the positive direction of the Y-axis is 81.2cm), the display is played on the smart screen "Straight back, look forward to the left knee lifted to the left side" this fitness action should play the 50th frame.

Among them, when the user’s left knee joint point 10 and left ankle joint point 11 have a real-time average displacement distance of 40 cm in the positive direction of the Z-axis and the positive direction of the Y-axis, "the back is straight and the eyes are forward to the left knee to the left The normal number of frames played on the display screen of the fitness action "Fang Gao" is 30 frames (as the target frame image). In this way, the smart screen correspondingly controls the playback speed of the fitness action of "back straight, look forward to the left knee to the left side" according to the actual movement distance of a certain fitness action by the user, so that the fitness video will be played The ideal number of frames for the user’s current fitness exercise corresponds to the actual distance the user currently moves. As described above, the user’s left knee joint point 10 and left ankle joint point 11 start from the initial position of this fitness exercise. The real-time average displacement starting in the positive direction of the Z-axis and the positive direction of the Y-axis is 40cm. The smart screen controls the action of "back straight, look forward to the left knee to the left side" in the fitness video. 25 frames (obviously, the time point corresponding to the target frame image (the value of the time corresponding to the target frame image) is less than the user's current fitness action (the time point corresponding to the first posture feature)) screen instead of the normal playback speed Play the 30th frame, that is, the smart screen controls "back straight, look forward to the left knee to the left side". The playback speed of the fitness action is slower than "back straight, look forward to the left" The normal playback speed (0.8 times speed) of the fitness exercise "knee raised to the left" is realized on the display screen of "back straight, look forward to the left knee and raise the left knee to the left". The playback speed of a fitness action matches the actual speed at which the user does "back straight and raises the left knee to the left side as seen from the front".

The embodiment of the application discloses an adaptive synchronization method for actions. The number of frames of each fitness action of the fitness video displayed on the display screen of the smart screen is relative to the movement distance of each bone node when the user completes each fitness action. Correspondingly, the movement distance that the user takes to complete each fitness action corresponds to the number of frames that match the movement distance, so that the playback speed of the fitness action demonstrated by the coach in the fitness video is synchronized with the speed at which the user performs the fitness action, ensuring that the user follows The continuity and rhythm of the fitness movement performed by the fitness video improves the user's sense of fitness experience.

Example 2

4(a) to 4(c) are UI interface diagrams of another action adaptive synchronization method shown in an embodiment of the present application.

In some embodiments of the present application, as shown in Figure 4(a), the fitness video is played on the display screen and the user’s action posture is collected in real time. Before the fitness video is played, both the coach and the user’s actions in the fitness video In the initial state, the text description of the upcoming fitness action is displayed on the initial UI interface, the right side displays the fitness action demonstrated by the instructor, the lower left corner displays the user's current posture, and the upper left corner displays the text prompts of the fitness action essentials. Then start playing fitness videos.

The smart screen determines the bone nodes involved in the fitness action demonstrated by the trainer in the fitness video and the bone nodes involved in the user's current posture. The smart screen determines whether the fitness action played by the fitness video matches the fitness action performed by the user. If it matches, it is normal Play the fitness video on the display. If it does not match, the smart screen adjusts the playback status of the fitness video on the display.

In some embodiments of the present application, as shown in Figure 4(b), taking "the left side of the body bends and the elbow touches the knee" as an example, the real-time display shows that the coach says "the left side of the body bends and the elbow touches the knee" in real time. Knee touch" action posture, the current moment the display shows the coach’s action posture of "the left side of the body is bent and the elbow touches the knee" (the screen shows the 25th frame of the elbow and knee touch) In the upper left corner of the display, there is also a text prompt of the fitness action "Bend the left side of the body and touch the elbow and knee". The user compares the fitness action demonstrated by the trainer and the text prompt to perform fitness and collects the user's real-time action posture. And in the lower left corner of the display screen, the user’s action posture is displayed in real time as the third fitness action "leg raise", it is determined that the user has completed the second fitness action at this time, and the text prompt "Please pay attention to the action" is displayed on the display screen. Does not match" to prompt the user.

In some embodiments of the present application, as shown in Figure 4(c), the processor controls the fitness video to skip the second fitness action directly and displays the text prompt "The previous action has been skipped for you" on the display screen. , Go to the third fitness action “leg raise” currently in progress by the user. At this time, as shown in Figure 4(c), the right side of the display shows the exercise demonstrated by the trainer, and the upper left of the display The corner displays the text prompt of raising the leg, and the lower left corner of the display shows the user’s real-time gesture of raising the leg at this time. This ensures that the fitness action of the fitness video currently displayed on the display is consistent with the fitness action currently performed by the user. Synchronization.

Fig. 5 is a schematic flowchart of another method for adaptive synchronization of actions shown in an embodiment of the application. As shown in Fig. 5, in one embodiment, the process of performing interactive fitness through the application is as follows:

Step S50: The user clicks on the fitness APP on the interactive interface of the display screen of the smart screen, and selects a fitness plan or a fitness course in the fitness APP (specifically, the fitness plan may include one or more fitness courses, which is not limited in this application ), among which, if the fitness course is pre-stored on the smart screen, the fitness video is directly extracted and played on the display. The smart screen can also obtain the fitness course through the network and play it and cache it in the memory. The fitness course includes a fitness coach Demonstration of multiple fitness actions, voice guidance and text prompts and other information. For example, according to the time sequence, the first fitness action is to straighten the back and look forward to the left knee to the left side, and the second fitness action In order to bend the left side of the body and touch the elbows and knees, the third action is to raise the leg (both the left leg and the right leg complete one leg raise is a complete leg raise), etc. When the coach demonstrates the fitness exercises, they will Give voice prompts and display the text of the voice on the display screen of the smart screen. It is worth noting that the way of playing fitness videos can also be other ways (for example, the user sends voice commands to the smart screen (such as playing a video) through voice interaction, for example, if the user says fitness, the smart screen automatically opens the fitness app). The application embodiment is not limited to the method of playing the fitness video when the user clicks on the fitness video.

Step S51: The smart screen reads each fitness action in the fitness video in sequence according to the playback time. Each continuous fitness action in the fitness video can be composed of multiple still picture frames. For example, the first fitness action is a backrest Straightly and visually look forward to raising the left knee to the left side, it is divided into 50 frames. The second exercise is the left side of the body and the touch of the elbow and the left knee is divided into 25 frames, and the third movement is the raised leg. Divided into 50 frames.

Step S52: The smart screen recognizes the trajectory of each fitness action demonstrated by the coach of the fitness video, extracts the text description and voice description of each fitness action, and parses keywords from it to determine the bone node information involved in each fitness action (Number or name). Among them, the analysis of the motion trajectory and keywords of fitness actions can be realized by machine learning, such as Long Short-Term Memory (LSTM), Support Vector Machine (SVM), and so on. Take the first exercise played in the fitness video as "back straight, look forward to the left knee and raise the left knee to the left side" as an example. While the coach demonstrates the exercise, the coach will broadcast the voice of "back straight" Straight, look straight ahead to the left knee raised to the left” prompt voice and display it in text form on the display. The smart screen recognizes fitness actions based on LSTM and extracts the keywords “lower back”, “knee”, and “left”. The body is divided into four parts: head, neck, torso and left leg, and the bone nodes involved in the above body parts are identified as head center 1, neck Section center 2, trunk center 15, left hip joint point 9, left knee joint point 10, and left ankle joint point 11.

Step S53: When the display screen of the smart screen plays fitness videos, the user stands within the camera range of the camera facing the smart screen, and follows the fitness actions demonstrated by the coach displayed on the display screen, text prompts, etc., to perform fitness actions, and the user does something During a fitness action, the camera can collect the user's action posture within the camera range at a sampling frequency of 5 times per second and transmit it to the processor. Based on the collected action posture, the processor extracts the fitness action currently being played on the display screen from the memory. And through machine learning (such as LSTM) to identify the user's bone node corresponding to the coach's bone node involved in the fitness action currently displayed on the display screen, use the fitness action "back straight, look forward to the left knee to the left As an example, the user’s skeletal nodes identified by LSTM are head center 1, neck center 2, torso center 15, left hip joint point 9, left knee joint point 10, and left ankle joint point 11.

Step S54: The smart screen captures in real time the motion trajectory and motion posture of the bone node corresponding to the fitness action of the fitness video played on the current display screen, and the motion trajectory and motion trajectory of the bone node involved in the current fitness action collected by the user through the camera. For the action posture, the smart screen judges whether the fitness action played by the fitness video matches the fitness action performed by the user, if it matches, it goes to S55, if it does not match, it goes to S56.

Among them, the fitness action played by the fitness video and the fitness performed by the user can be judged by the bone node number or name of the coach involved in the fitness action currently played on the display screen and the user's bone node name or number involved in the fitness action currently performed by the user. Whether the action matches (action posture), and the motion trajectory, determine whether the fitness action played by the fitness video matches the fitness action performed by the user. For example, according to the time sequence, the first fitness action of the fitness video displayed on the display screen is waist back support Straightly and visually raise the left knee to the left side, the second exercise is to bend the left side of the body and touch the elbow and knee. The camera of the smart screen collects that the user is performing the second exercise "Left side of the body" Bend and touch your elbows and knees" and transmit to the processor.

The processor learns through the LSTM that the user’s current second fitness action "the left side of the body is bent and the elbows and knees touch" the bone nodes involved are the head center 1, the neck center 2, the torso center 15, and the left hip joint. Point 9, left knee joint point 10, left elbow joint point 5, left wrist joint point 7, and left ankle joint point 11. The smart screen analyzes the user's posture and motion trajectory collected by the camera, that is, the user's posture is left flexion, When the elbow and knee touch, the movement trajectory is: the body changes from positive to left flexion, the elbow and knee are close, that is, the head center 1, the neck center 2 and the trunk center 15 are tilted to the left, and the left knee joint point 10 is lifted up. , The left elbow joint point 5 and the left wrist joint point 7 are close to the left knee joint point 10. In this way, the smart screen determines that the user has completed the first fitness action; the current fitness action on the display screen extracted from the memory by the smart screen is " Raise the left knee to the left", the smart screen analyzes the exercise trajectory of the fitness exercise demonstrated by the coach currently displayed on the display, that is, the coach’s posture is raising the left knee, and the trajectory is: the left knee moves from the initial position to the The left side is raised, that is, the left knee joint point 10 and the left ankle joint point 11 are raised from bottom to top. In this way, the smart screen judges that the current screen is playing the first fitness action of the fitness video. At this time, the smart screen It is determined that the fitness action currently performed by the user does not match the fitness action currently displayed on the display screen.

S55: Play the fitness video on the screen normally.

S56: The smart screen adjusts the playback status of the fitness video on the display, as follows: According to the time sequence, if the user has completed the fitness action of the fitness video currently being played on the display (the frame image of the fitness action of the fitness video currently being played) The time point (the time of the fitness action of the currently played fitness video) does not exceed the time point of the target frame image (the time of the fitness action of the target frame image), where the time point of the frame image of the fitness action of the currently played fitness video and The time relationship between the time points of the target frame image is: the time of the frame image of the fitness action of the currently played fitness video is earlier than the time of the target frame image). Then the smart screen controls the fitness video at high speed (for example, 4 times speed is the target speed) to play the fitness action currently played on the display screen or directly skip the fitness action currently played on the display screen (the fitness action currently played is the frame image corresponding to the first posture feature) ) Play the next fitness action (target frame image) that the user is doing. If the display screen has already played the fitness action of the completed fitness video, the smart screen judges that the user has not completed the fitness action based on the user's action posture and motion trajectory collected in real time by the camera, and the smart screen pauses playing the fitness video and displays it on the smart screen The screen displays the text prompt of the current fitness action being performed by the user until the camera collects in real time that the user has completed the fitness action that has been broadcasted and the user’s current fitness action matches the action posture corresponding to the current pause screen. The video playback is resumed at the moment of pause, or the fitness video is played back to the screen corresponding to the user's current posture to perform the fitness action again.

In some embodiments of the present application, the first fitness action of the fitness video displayed on the display screen is to straighten the back and raise the left knee to the left side, and the second The fitness action is bending the left side of the body and touching the elbow and knee. The camera of the smart screen collects that the user is performing the second fitness action "bending the left side of the body and touching the elbow and knee", and transmits it to the processor.

The processor learns through LSTM that the bone nodes involved in the second fitness action currently performed by the user are head center 1, neck center 2, torso center 15, left hip joint point 9, left knee joint point 10, and left elbow joint. Point 5, left wrist joint point 7, left ankle joint point 11, the smart screen analyzes the user's posture and motion trajectory collected by the camera, that is, the user posture is the left side of the body, the elbow and the knee touch, and the motion trajectory is: It is becoming left flexion, the elbow is close to the knee, that is, the head center 1, the neck center 2 and the torso center 15 are tilted to the left, the left knee joint point 10 is lifted up, the left elbow joint point 5 and the left wrist joint point 7 Approach the left knee joint point 10, so the smart screen determines that the user has completed the first fitness action; the current fitness action being played on the display screen extracted from the memory by the smart screen is "raise the left knee to the left", the smart screen Analyze the trajectory of the fitness exercise demonstrated by the coach currently displayed on the display, that is, the coach’s posture is to raise the left knee, and the trajectory is: the left knee is raised from the initial position to the left, that is, the left knee joint point 10 And the left ankle joint point 11 is lifted from bottom to top, so the smart screen judges that the current screen is playing the first fitness action of the fitness video. At this time, the camera of the smart screen will collect the user's current fitness actions in real time, and the smart screen can play fitness videos at 4 times the speed until the fitness action played on the display matches the user's current fitness action, then the normal speed will be restored Play fitness videos.

An action adaptive synchronization method disclosed in the embodiments of the present application recognizes in real time the action posture and movement track of the coach of the fitness video played on the display screen, and recognizes the action posture and movement of the fitness action performed by the user collected by the camera of the smart screen Trajectory, when the two do not match, adjust the playback status of the fitness video on the display screen so that the fitness action of the fitness video currently displayed on the display screen is synchronized with the fitness action currently performed by the user, ensuring the continuity of the user during fitness And fluency.

Example 3

6(a) to 6(c) are UI interface diagrams of another method for adaptive synchronization of actions shown in an embodiment of the present application.

In some embodiments of the present application, as shown in FIG. 6(a), the fitness video is played on the display screen and the user's action posture is collected in real time. Before the fitness video is played, the coach and the user's actions in the fitness video are both In the initial state, the text description of the upcoming fitness action is displayed on the initial UI interface, the right side displays the fitness action demonstrated by the instructor, the lower left corner displays the user's current posture, and the upper left corner displays the text prompts of the fitness action essentials. Then the smart screen control starts to play fitness videos.

The smart screen judges whether the fitness action performed by the user is standard. If it is not standard, the smart screen uses the fitness action as a non-standard fitness action, and calculates the fitness speed of the user performing the non-standard fitness action. The smart screen plays the fitness video at the fitness speed The standard fitness action demonstrated by the coach corresponding to the non-standard fitness action (wrong fitness action) is to correct the wrong fitness action performed by the user.

In some embodiments of the present application, as shown in Figure 6(b), taking "back straight, visually looking forward to the left knee raised to the left side" as an example, the real-time display on the display shows that the coach reads " Keep your back straight and look forward to the left knee lifted to the left. At the current moment, the coach’s motion posture of “left knee lifted to the left” is displayed on the screen (the display shows The 50th frame when the left knee is raised to the standard position), the text prompt of the fitness action is also displayed in the upper left corner of the display screen: "Straight back, look forward to raise the left knee to the left side", The user compares the fitness actions and text prompts demonstrated by the coach on the display to perform fitness, collects the user's real-time action posture and displays the user’s action posture in real time in the lower left corner of the display as "right knee lifted to the right", which is obvious What’s more, when the user is doing the fitness exercise of "back straight, looking forward to the left knee to the left side", the fitness exercise demonstrated by the instructor is "back straight, looking forward to the left knee "Left side elevation" does not match, indicating that the user's fitness exercise is wrong at this time, and the text prompt "Please note that the exercise does not match" is displayed on the display screen.

After the entire fitness video is played, the standard fitness action "Left Knee Raise to the Left" and the user's wrong fitness action "Right Knee Raise to the Right" demonstrated by the coach are simultaneously played on the display screen. In some embodiments of the present application, as shown in FIG. 6(c), the video of the user's wrong fitness action can be displayed on the right side of the display screen, and the standard fitness action demonstrated by the coach can be displayed in the lower left corner of the display screen. , The text prompt of the standard fitness exercise demonstrated by the coach is displayed in the upper left corner, and the standard fitness video demonstrated by the coach is compared and explained, and the text prompt "Wrong action occurred, please watch your action replay" on the display screen. To remind the user, correct the user's wrong fitness action.

FIG. 7 is a schematic flowchart of another method for adaptive synchronization of actions shown in an embodiment of the application. As shown in FIG. 7, in an embodiment, the process of performing interactive fitness through the application is as follows:

Step S70: The power module supplies power to the smart screen. The user operates the remote control to control the smart screen to turn on, and clicks on the fitness app on the interactive interface of the smart screen’s display screen to open the fitness video on the fitness app and play it, among which, play on the smart screen In the case of the fitness video, if the fitness video is pre-stored in the memory of the smart screen, the fitness video is directly extracted from the memory and played on the display screen. If the fitness video is not pre-stored in the memory, the fitness video is downloaded through the network and displayed Played on the display screen and buffered in the memory. The fitness video contains multiple fitness exercises demonstrated by the fitness trainer, voice guidance, and text prompts. For example, according to the time sequence, the first fitness exercise is to straighten your back and look visually. Raise the left knee from the front to the left. The second exercise is to bend the left side of the body and touch the elbow and knee. The third exercise is to raise the leg (both the left leg and the right leg are completed once a raised leg is complete When the trainer demonstrates fitness exercises, the trainer will give voice prompts and display the text of the voice on the display screen of the smart screen.

Step S71: The processor of the smart screen reads each fitness action in the fitness video in sequence according to the playback time. Each continuous fitness action in the fitness video can be composed of multiple still picture frames. For example, the first fitness action is The back is straight, and the left knee is raised from the front to the left. The second exercise is divided into 50 frames. The second exercise is the left side of the body and the touch of the elbow and the left knee is divided into 25 frames. The third movement is The high leg is divided into 50 frames.

Step S72: The smart screen recognizes the trajectory of each fitness action demonstrated by the coach of the fitness video, extracts the text description and voice description of each fitness action, and parses the keywords from it to determine what each fitness action in the fitness video involves The information (number or name) of the bone node. Among them, the analysis of the motion trajectory and keywords of fitness actions can be realized by machine learning, such as Long Short-Term Memory (LSTM), Support Vector Machine (SVM), and so on. Take the first exercise played in the fitness video as "back straight, look forward to the left knee and raise the left knee to the left side" as an example. While the coach demonstrates the exercise, the coach will broadcast the voice of "back straight" Straight, look straight ahead to the left knee raised to the left” prompt voice and display it in text form on the display. The smart screen recognizes fitness actions based on LSTM and extracts the keywords “lower back”, “knee”, and “left”. The body is divided into four parts: head, neck, torso and left leg, and the bone nodes involved in the above body parts are identified as head center 1, neck Section center 2, trunk center 15, left hip joint point 9, left knee joint point 10, and left ankle joint point 11.

Step S73: When the display screen of the smart screen plays fitness videos, the user stands within the camera range of the camera facing the smart screen, and follows the fitness actions demonstrated by the coach displayed on the display screen, text prompts, etc., to perform fitness actions corresponding to the fitness actions, and the user does it for the first time During a certain fitness action, the camera can collect the user's action posture within the camera range at a sampling frequency of 5 times per second and transmit it to the processor. Based on the collected action posture, the processor extracts the fitness action currently displayed on the display from the memory. , And use machine learning (such as LSTM) to identify the user’s bone nodes that match the coach’s bone nodes involved in the fitness action currently played on the display, and calculate the bone nodes of all users involved in the fitness action from the initial position to the end The average moving distance of the location. Take the first fitness action played in the fitness video as "back straight, look forward to the left knee and raise the left knee to the left" as an example. The fitness action is a total of 50 frames, and the bone node involved in this action is the head Head center 1, neck center 2, torso center 15, left hip joint point 9, left knee joint point 10, and left ankle joint point 11.

In some embodiments of the present application, as shown in Figure 3(b), in the three-dimensional coordinate system, when the user performs the action of raising the left knee to the left for the first time, the left knee joint point 10 and the left ankle The displacement direction of node 11 is the positive direction of the Z axis and the positive direction of the Y axis. Assume that the displacement distance of the left knee joint point 10 in the positive direction of the Z axis is 80cm, and the displacement distance of the left knee joint point 10 in the positive direction of the Y axis. The displacement distance of the left ankle joint point 11 in the positive direction of the Z-axis is 60m, the displacement distance of the left ankle joint point 11 in the positive direction of the Y-axis is 40cm, the center of the head 1, the center of the neck 2, and the left hip joint Point 9, the displacement of the torso center 15 is zero, the displacement distance of the left knee joint point 10 is about 90.2cm, and the displacement distance of the ankle joint point 11 is about 72.1cm. The average displacement distance of the left knee joint point 10 and the left ankle joint point 11 of the fitness action "from the front to the left knee raise to the left" is the displacement distance of the left knee joint point 10 and the displacement of the ankle joint point 11. The average value of the distance is about 81.2 cm.

After the user completes the first fitness exercise, he immediately performs the second fitness exercise with the left side of the body bend and the elbow touches the left knee until the user performs the entire fitness exercise indicated by the fitness video.

Step S74: The camera transmits the information of the user's fitness actions collected in real time to the processor and memory of the smart screen. The memory stores the video of each fitness action performed by the user in the order of time recording, and the processor real-time Calculate the speed of each fitness action performed by the user each time and store the speed of each fitness action calculated in real time in the memory.

Step S75: The smart screen extracts from the memory the standard fitness actions demonstrated by the coach corresponding to each fitness action performed by the user, recognizes and matches to determine whether the fitness action performed by the user is standard, and executes it when the fitness action performed by the user is not standard Step S751 stores the fitness action as an error action in the memory. When the fitness action performed by the user is standard, step S752 is executed to define the fitness action as a standard action, and the standard action is optionally stored in the memory.

In some embodiments of the present application, still taking the fitness action of "back straight, looking forward to the left knee to the left side" as an example, the fitness action is a total of 50 frames, then the user completes the "back Straighten up, look forward to the left knee and lift up to the left side." This fitness action, the head center 1, the neck center 2, the left hip joint point 9, and the torso center 15 have zero displacement, and the left knee joint point 10 and the left ankle joint point 11 have an average displacement distance of 81.2cm in the positive direction of the Z axis and the positive direction of the Y axis. The speed during a fitness exercise is

The smart screen stores the fitness action speed 0.62 frames/cm in the memory. When the user starts the fitness action of "back straight, looking forward to the left knee to the left side", the user's right knee is facing When the left side is raised or the right knee is raised to the right side, the camera collects the user's action posture at this time (the right knee is raised on the right side) and transmits it to the processor. The processor recognizes the action posture based on the LSTM network The user’s right knee joint point 13 and right ankle joint point 14 in the LSTM network are used by the processor to identify the fitness action of "back straight, look forward to the left knee to the left side" in the fitness video. The coach’s bone nodes are left knee joint point 10 and left ankle joint point 11. In this fitness exercise, the smart screen will identify the user’s key bone nodes, right knee joint point 13 and right ankle joint point 14 and the coach’s key bone node left Knee joint point 10 and left ankle joint point 11 are compared and matched with the number or name of the bone node. At this time, the number of the bone node of the user who is compared on the smart screen is not consistent with the number of the bone node of the coach in the fitness video, then it is determined The user’s fitness action of "straight back and look forward to the left knee to the left side" is a wrong action. The erroneous exercise action of "raise the left side" is stored in the memory. It is worth noting that the calculation method of the speed at which the user performs the fitness action is the same as the above-mentioned embodiment, and the details are not repeated here in the embodiment of the present application.

Step S76: After the user performs all the fitness actions indicated in the complete fitness video, the smart screen calculates the average speed of each fitness action, and still uses the fitness action as "back straight, look forward to the left knee to the left As an example, the user needs to do 3 times in the whole fitness video with "back straight, look forward to the left knee and raise the left knee to the left". The smart screen extracts the user from the memory for the first time to do Straight, look straight ahead to the left knee and raise the left side to the left side of the fitness movement, the speed is

The smart screen extracts from the memory the speed of the second time the user performs the fitness exercise of "back straight, looking forward to the left knee to the left side", the speed is 0.66 frames/cm, and the smart screen from the memory In extracting the user’s third fitness action of "back straight, looking forward to the left knee lifted to the left", the speed is 0.64 frames/cm, then the user does "back straight , The average speed of the fitness exercise "from the front to the left knee lifted to the left side" is 0.64 frames/cm.

Step S77: After the fitness video on the display screen of the smart screen is completed, the camera on the smart screen stores the recorded user's fitness video in the memory, and the smart screen extracts the non-standard fitness actions that the user has performed wrongly from the memory and the user performs the exercises. The average speed of non-standard fitness actions, the smart screen will display the non-standard fitness actions and the standard fitness actions of the coaches corresponding to the non-standard fitness actions on the same screen. In the case of a standard fitness action, the standard fitness action demonstrated by the coach is played at the average speed of the non-standard fitness action performed by the user. Taking the step S44 as an example, the fitness action of "back straight, look forward to the left knee to the left side" is a non-standard fitness action, and the average speed of this fitness action is 0.64 frames /cm, the smart screen controls the user’s non-standard action videos of "back straight, eyes forward to the left knee raised to the left" stored in the memory and "back straight, eyes straight" in the fitness video. The standard action videos from the front to the left knee raised to the left are all played at a speed of 0.64 frames/cm. When the non-standard action video is played, voice explanations and voice prompts are provided to the user to correct the wrong fitness action of the user.

An adaptive synchronization method for actions disclosed in the embodiments of the present application. The smart screen stores the video of the wrong fitness action performed by the user in the memory. After the entire fitness video on the display screen is played, it extracts the wrong exercise by the user from the memory. The fitness video is compared and explained with the standard fitness video demonstrated by the instructor to correct the user's wrong fitness action, which is beneficial to the user's health and further enhances the user's experience.

In some embodiments of the present application, an electronic device is also provided. The electronic device in the embodiments of the present application will be described below with reference to FIG. 8. FIG. 8 is a schematic structural diagram of an electronic device disclosed in an embodiment of the application.

For at least one embodiment, the controller hub 804 communicates with the processor 801 via a multi-drop bus such as a front side bus (FSB), a point-to-point interface such as a fast path interconnect (QPI), or similar connection. The processor 801 executes instructions that control general types of data processing operations. In one embodiment, the controller hub 804 includes, but is not limited to, a graphics memory controller hub (GMCH) (not shown in the figure) and an input/output hub (IOH) (which may be on a separate chip) ( (Not shown in the figure), where the GMCH includes a memory and a graphics controller and is coupled with the IOH.

The electronic device 800 may also include a coprocessor 806 and a memory 802 coupled to the controller hub 804. Alternatively, one or both of the memory 802 and the GMCH may be integrated in the processor 801 (as described in this application), and the memory 802 and the coprocessor 806 are directly coupled to the processor 801 and the controller hub 804, and control The device hub 804 and the IOH are in a single chip.

In one embodiment, the memory 802 may be, for example, dynamic random access memory (DRAM), phase change memory (PCM), or a combination of the two. The memory 802 may include one or more tangible, non-transitory computer-readable media for storing data and/or instructions. The computer-readable storage medium stores instructions, specifically, temporary and permanent copies of the instructions.

In one embodiment, the coprocessor 806 is a dedicated processor, such as, for example, a high-throughput MIC processor, a network or communication processor, a compression engine, a graphics processor, a GPGPU, or an embedded processor, or the like. The optional nature of the coprocessor 806 is shown in dashed lines in FIG. 8.

In one embodiment, the electronic device 800 may further include a network interface (NIC) 803. The network interface 803 may include a transceiver, which is used to provide a radio interface for the device 800 to communicate with any other suitable devices (such as a front-end module, an antenna, etc.). In various embodiments, the network interface 803 may be integrated with other components of the electronic device 800. The network interface 803 can realize the function of the communication unit in the above-mentioned embodiment.

In one embodiment, as shown in FIG. 8, the electronic device 800 may further include an input/output (I/O) device 805. The input/output (I/O) device 805 may include: a user interface, which is designed to enable users to interact with the electronic device 800; the design of the peripheral component interface enables the peripheral components to also interact with the electronic device 800; and/or the design of sensors To determine environmental conditions and/or location information related to the electronic device 800.

It is worth noting that Figure 8 is only exemplary. That is, although FIG. 8 shows that the electronic device 800 includes multiple devices such as the processor 801, the controller hub 804, and the memory 802, in actual applications, the devices using the methods of the present application may only include the electronic device 800. Some of the devices, for example, may only include the processor 801 and the NIC 803. The properties of optional devices in Fig. 8 are shown by dashed lines.

In some embodiments of the present application, the instructions stored in the computer-readable storage medium of the electronic device 800 may include: when executed by at least one unit in the processor, cause the device to implement such as embodiment 1, embodiment 2, and embodiment 3. The mentioned actions are adaptive synchronization method instructions. When the instructions are executed on the computer, the computer is caused to execute the above-mentioned method for adaptive synchronization of actions as mentioned in Embodiment 1, Embodiment 2 and Embodiment 3.

Referring now to FIG. 9, FIG. 9 is a schematic structural diagram of an SOC disclosed in an embodiment of the present application, and shows a block diagram of an SoC (System on Chip) 1000 according to an embodiment of the present application. In Figure 9, similar parts have the same reference numerals. In addition, the dashed box is an optional feature of the more advanced SoC. The SoC can be used in an electronic device according to an embodiment of the present application, and corresponding functions can be implemented according to the instructions stored in the SoC.

In Figure 9, the SoC 1000 includes: an interconnection unit 1002, which is coupled to a processor 1001; a system agent unit 1006; a bus controller unit 1005; an integrated memory controller unit 1003; a group or one or more coprocessors 1007, which may include integrated graphics logic, image processors, audio processors, and video processors; a static random access memory (SRAM) unit 1008; and a direct memory access (DMA) unit 1004. In one embodiment, the coprocessor 1007 includes a dedicated processor, such as, for example, a network or communication processor, a compression engine, a GPGPU, a high-throughput MIC processor, or an embedded processor, etc.

A static random access memory (SRAM) unit 1008 may include one or more computer-readable media for storing data and/or instructions. The computer-readable storage medium may store instructions, specifically, temporary and permanent copies of the instructions.

When the SoC 1000 is applied to the electronic device according to the present application, the instructions stored in the computer-readable storage medium may include: when executed by at least one unit in the processor, the electronic device is implemented as in embodiment 1, embodiment 2, and The instructions of the action adaptive synchronization method mentioned in the third embodiment. When the instructions are executed on the computer, the computer is caused to execute the above-mentioned method for adaptive synchronization of actions as mentioned in Embodiment 1, Embodiment 2 and Embodiment 3.

In addition, the embodiment of the present application also discloses a computer-readable storage medium, and a processing program is stored on the computer-readable storage medium. The method of adaptive synchronization for the movement.

The computer-readable storage medium may be a read-only memory, a random access memory, a hard disk, or an optical disk.

Claims

An action adaptive synchronization method, characterized in that it is applied to an electronic device, and the action adaptive synchronization method includes:

The electronic device plays fitness videos;

The electronic device collects the user's action posture;

The electronic device matches the fitness action currently played in the fitness video with the action posture of the user;

If the fitness action does not match the user's action posture, the electronic device adjusts the playback speed of the fitness action in the fitness video to synchronize the fitness action with the user's action posture.
The action adaptive synchronization method of claim 1, wherein the electronic device matching the fitness action currently played in the fitness video with the user's action posture comprises:

Acquiring, by the electronic device, one or more bone nodes of at least one fitness coach in the fitness video;

Acquiring, by the electronic device, one or more skeletal nodes of the user;

The electronic device determines key bone nodes from one or more bone nodes of at least one fitness coach and one or more bone nodes of the user in the fitness video.
The method according to claim 2, wherein the determining the key bone node comprises:

The electronic device selects the key bone node of the currently demonstrated fitness action from the bone nodes of the user and the at least one fitness instructor, where the key bone node is the bone node involved in the currently played fitness action.
The action adaptive synchronization method of claim 3, wherein the electronic device matching the fitness action currently played in the fitness video with the user's action posture comprises:

Based on the LSTM, the electronic device extracts the key bone nodes of the fitness action currently demonstrated by at least one fitness trainer in the current fitness video and the action posture of the user to obtain the third posture feature and the posture feature of the currently demonstrated fitness action. For the fourth posture feature of the user, the third posture feature and the fourth posture feature are matched based on the key skeleton node.
The action adaptive synchronization method of claim 2, wherein the electronic device matching the fitness action currently played in the fitness video with the user's action posture comprises:

The electronic device extracts key bone nodes of the fitness action currently demonstrated by at least one fitness trainer in the current fitness video and the action posture of the user based on the LSTM to obtain the first posture feature and the posture feature of the currently demonstrated fitness action. The second posture feature of the user is matched, and the first posture feature and the second posture feature are matched.
The action adaptive synchronization method according to claim 5, wherein the fitness video is video-divided to obtain frame images of consecutive moments of the fitness video.
7. The action adaptive synchronization method of claim 6, wherein the electronic device determines a target frame image corresponding to the second posture feature of the user;

Based on the time relationship between the frame image corresponding to the first posture feature and the target frame image, adjust the playback speed of the fitness video to synchronize the fitness action corresponding to the target frame image with the user's posture .
8. The action adaptive synchronization method of claim 7, wherein the determining the target frame image corresponding to the second posture feature of the user comprises:

The electronic device determines the target frame image based on the total number of frames of the complete fitness action corresponding to the second posture feature and the average moving distance, where the average moving distance is the key bone corresponding to the second posture feature The average moving distance of the node from the initial position to the end position of the complete fitness action.
The action adaptive synchronization method according to claim 8, wherein the average moving distance is that the key bone node corresponding to the second posture feature is in the three-dimensional coordinate system from the initial position to the end position. The average value of the movement distances in the three directions of the X-axis, Y-axis and Z-axis.
The action adaptive synchronization method of claim 7, wherein when the time point corresponding to the target frame image is greater than the time point of the frame image corresponding to the first posture feature, the electronic device adjusts the fitness The playback speed of the video is a target speed, where the target speed is greater than the current playback speed of the fitness video.
The action adaptive synchronization method of claim 7, wherein when the time point corresponding to the target frame image is greater than the time point of the frame image corresponding to the first posture feature, the electronic device controls the fitness The video jumps directly from the currently played fitness action to the target frame image.
The action adaptive synchronization method according to claim 7, wherein when the time point corresponding to the target frame image is less than the time point of the frame image corresponding to the first posture feature, the electronic device controls the fitness The video is played back and jumps directly to the target frame image from the currently played fitness action.
The action adaptive synchronization method according to any one of claims 1 to 5, wherein when the fitness action is not synchronized with the user's action posture, the electronic device provides voice prompts and/or text prompts.
3. The action adaptive synchronization method of claim 1, wherein the electronic device stores the user's action posture;

Determining, by the electronic device, that the standard fitness action does not match the user's action posture standard based on the standard fitness action corresponding to the user's action posture;

Marking the user's action posture as a non-standard action and replaying the standard fitness action corresponding to the non-standard action in the fitness video;

The electronic device replays the standard fitness action in the fitness video at the fitness speed corresponding to the non-standard action performed by the user, and the standard fitness action corresponds to the fitness action corresponding to the non-standard action;

The electronic device provides text and/or voice explanation of the non-standard action.
An electronic device, characterized in that it comprises:

A memory, the memory stores synchronization instructions;

A processor, which implements the steps of the action adaptive synchronization method according to any one of claims 1-14 when the processor executes the synchronization instruction.