WO2019100755A1

WO2019100755A1 - Video generation method and device, and electronic apparatus

Info

Publication number: WO2019100755A1
Application number: PCT/CN2018/098600
Authority: WO
Inventors: 袁冰; 李震; 卞爱娟; 王思佳; 陈海蕾; 徐年强
Original assignee: 乐蜜有限公司; 袁冰
Priority date: 2017-11-23
Filing date: 2018-08-03
Publication date: 2019-05-31
Also published as: CN107952238B; CN107952238A

Abstract

Provided are a video generation method and device, and an electronic apparatus. The method comprises: acquiring a selected audio file and standard movements corresponding to respective time points in the audio file; when the audio file begins to play, starting to collect video picture frames; starting to display, according to a preset advance time, and before the respective time points in the audio file are reached, the corresponding standard movements; recognizing body movements in the video picture frames collected in the process of displaying the standard movements; if the body movements match the standard movements, ending the process of displaying the standard movements; and when playing of the audio file ends, generating a target video according to the collected video picture frames and the audio file. The method is applicable to electronic apparatuses, thereby effectively enhancing applicability thereof. In the invention, a process of displaying a standard movement ends when a body movement matches a standard movement, thereby reducing the number of images demonstrating standard movements in a photographing interface, and allowing a user to view conveniently.

Description

Video generation method, device and electronic device

Cross-reference to related applications

The present application claims priority from Chinese Patent Application No. "201711184350.8" filed on November 23, 2017, entitled "Video Generation Method, Apparatus, and Electronic Apparatus".

Technical field

The present application relates to the field of mobile terminal technologies, and in particular, to a video generation method, apparatus, and electronic device.

Background technique

Somatosensory dance games, through the Internet operating platform, human-computer interaction. The user makes the corresponding body movements according to the prompts of the dancing device according to the somatosensory, so that the user can achieve the fitness function while enjoying the somatosensory interaction experience while dancing.

In the prior art, the somatosensory dance game is mainly applied to fixed devices, such as a somatosensory dance machine, a computer, etc., and the portability is poor. In addition, the judgment of the user's body movement is determined by determining whether the arrow direction of the user's foot is correct or not, and the manner of dancing is relatively simple.

Summary of the invention

The present application aims to solve at least one of the technical problems in the related art to some extent.

To this end, the first object of the present application is to provide a video generation method. Since the standard action is a human body action that the user needs to make, the dance action can be effectively enriched compared to the dance mode of the user's foot arrow in the prior art. To enhance the user experience. The prior art somatosensory dance game is mainly applied to fixed devices, such as a somatosensory dance machine, a computer, etc., and the portability is poor. In addition, the judgment of the user's body movement is a technical problem by which the direction of the arrow of the user's foot is correct or not, and the manner of dancing is relatively simple.

A second object of the present application is to propose a video generating apparatus.

A third object of the present application is to propose an electronic device.

A fourth object of the present application is to propose a non-transitory computer readable storage medium.

A fifth object of the present application is to propose a computer program product.

To achieve the above objective, the first aspect of the present application provides a video generating method, including:

Obtaining selected audio, and standard actions corresponding to each time node in the audio;

When the audio is started to be played, the acquisition of the video picture frame is started;

According to the preset advance time, before the audio is played to each time node, the corresponding standard action is started to be displayed;

Identifying human body motions in a video frame frame acquired during the presentation of the standard action;

Ending the display process of the standard action if the human body action matches the standard action;

When the audio playback ends, a target video is generated based on the captured video frame frame and the audio.

Optionally, as a first possible implementation manner of the first aspect, the displaying, according to the preset advance time, the corresponding standard actions are started before the audio is played to each time node, including:

For each time node, the difference between the time node and the advance time is taken as the starting time;

Starting from the start time, a schematic diagram of the standard action is displayed, and a schematic diagram of controlling the standard action is moved along a preset trajectory.

Optionally, as a second possible implementation manner of the first aspect, after the schematic diagram of controlling the standard action is moved along a preset trajectory, the method further includes:

When the schematic diagram of the standard action moves to the end point of the preset trajectory, if the human body action matching the standard action is not recognized, the schematic diagram showing the standard action is stopped.

Optionally, as a third possible implementation manner of the first aspect, the schematic diagram of controlling the standard action is moved along a preset trajectory, including:

Determining, from a plurality of candidate trajectories, a preset trajectory for displaying a schematic diagram of the standard action; the preset trajectory is different from a trajectory displayed by a schematic diagram corresponding to a standard action of an adjacent time node;

A schematic diagram of controlling the standard action is performed along the preset trajectory at a preset speed and direction.

Optionally, as a fourth possible implementation manner of the first aspect, after determining the preset trajectory for displaying the schematic diagram of the standard action, the method further includes:

If the preset track has a schematic diagram being displayed;

Defining a schematic diagram of the preset trajectory being displayed and a size of the schematic diagram of the standard motion according to the number of the schematics being displayed and the length of the preset trajectory in the preset trajectory, so that the preset trajectory is in phase The distance between the adjacent two schematics is greater than or equal to the threshold distance.

Optionally, as a fifth possible implementation manner of the first aspect, after the acquiring the selected audio, and the standard action corresponding to each time node in the audio, the method further includes:

Show the countdown interface and start counting down;

When the countdown ends, the audio begins to play.

Optionally, as a sixth possible implementation manner of the first aspect, after the recognizing the human body motion in the video frame frame collected during the displaying of the standard action, the method further includes:

And determining whether the human body action matches the standard action according to whether the degree of difference between the human body action and the standard action is greater than a difference threshold.

Optionally, as a seventh possible implementation manner of the first aspect, after the human body action is matched with the standard action, the method further includes:

The motion evaluation information of the human body motion is generated according to the degree of difference between the human body motion and the standard motion.

The video generating method of the embodiment of the present application obtains the selected audio and the standard action corresponding to each time node in the audio, and starts to collect the video picture frame when starting to play the audio; according to the preset advance time, the audio is played to Before each time node, start to display the corresponding standard action; identify the human body motion in the video frame frame collected during the display of the standard action; end the display process of the standard action when the human action matches the standard action; At the end of playback, the target video is generated based on the captured video frame and audio. In this embodiment, since the standard action is a human body action that the user needs to make, compared with the dance mode of the user's foot arrow in the prior art, the dance action can be effectively enriched and the user experience can be improved. In addition, the video generation method can be applied to an electronic device, which can effectively improve the applicability of the method. Moreover, when the human body action is matched with the standard action, the display process of the standard action is ended, and the number of schematic diagrams of the standard actions displayed in the shooting interface can be reduced, which is convenient for the user to watch, further enhance the user experience, and is used to solve the existing body dance. The game is mainly used on fixed devices, such as somatosensory dance machines, computers, etc., and the portability is poor. In addition, the judgment of the user's body movement is a technical problem by which the direction of the arrow of the user's foot is correct or not, and the manner of dancing is relatively simple.

In order to achieve the above objective, the second aspect of the present application provides a video generating apparatus, including:

a selection module for acquiring selected audio and standard actions corresponding to each time node in the audio;

An acquisition module, configured to start collecting video frame frames when starting to play the audio;

a display module, configured to start displaying corresponding standard actions before the audio is played to each time node according to a preset advance time;

An identification module, configured to identify a human body motion in a video frame frame collected during the display of the standard action;

a control module, configured to end a display process of the standard action when the human body action matches the standard action;

And a generating module, configured to generate a target video according to the collected video picture frame and the audio when the audio playing ends.

Optionally, as a first possible implementation manner of the second aspect, the display module includes:

a processing submodule, configured, for each time node, a difference between the time node and the advance time as a starting time;

And a control submodule, configured to display a schematic diagram of the standard action from the start time, and control a schematic diagram of the standard action to move along a preset trajectory.

Optionally, as a second possible implementation manner of the second aspect, the display module further includes:

The stop sub-module is configured to stop displaying the schematic diagram of the standard action if the human body action matching the standard action is not recognized when the schematic diagram of the standard action moves to the end point of the preset track.

Optionally, as a third possible implementation manner of the second aspect, the control submodule is specifically configured to:

Optionally, as a fourth possible implementation manner of the second aspect, the control submodule is further configured to:

When the preset trajectory has a schematic diagram being displayed, according to the number of the schematic diagrams being displayed and the length of the preset trajectory, the schematic diagram of the preset trajectory being displayed and the standard motion are reduced. The size of the schematic is such that the distance between two adjacent schematics in the predetermined trajectory is greater than or equal to a threshold distance.

Optionally, as a fifth possible implementation manner of the second aspect, the device further includes:

The display playing module is configured to display a countdown interface after the selected audio and the standard action corresponding to each time node in the audio, and start counting down, and when the countdown ends, start playing the audio.

Optionally, as a sixth possible implementation manner of the second aspect, the device further includes:

a determining module, configured to determine, according to the human body motion in the video frame frame collected during the displaying of the standard action, whether the difference between the human motion and the standard motion is greater than a difference threshold, It is determined whether the human body action matches the standard action.

Optionally, as a seventh possible implementation manner of the second aspect, the device further includes:

The evaluation information generating module is configured to generate motion evaluation information of the human body motion according to a degree of difference between the human body motion and the standard motion after the human body motion is matched with the standard motion.

The video generating apparatus of the embodiment of the present application acquires the selected audio and the standard action corresponding to each time node in the audio, and starts to collect the video picture frame when starting to play the audio; according to the preset advance time, the audio is played to Before each time node, start to display the corresponding standard action; identify the human body motion in the video frame frame collected during the display of the standard action; end the display process of the standard action when the human action matches the standard action; At the end of playback, the target video is generated based on the captured video frame and audio. In this embodiment, since the standard action is a human body action that the user needs to make, compared with the dance mode of the user's foot arrow in the prior art, the dance action can be effectively enriched and the user experience can be improved. In addition, the video generation method can be applied to an electronic device, which can effectively improve the applicability of the method. Moreover, when the human body action is matched with the standard action, the display process of the standard action is ended, and the number of schematic diagrams of the standard actions displayed in the shooting interface can be reduced, which is convenient for the user to watch, further enhance the user experience, and is used to solve the existing body dance. The game is mainly used on fixed devices, such as somatosensory dance machines, computers, etc., and the portability is poor. In addition, the judgment of the user's body movement is a technical problem by which the direction of the arrow of the user's foot is correct or not, and the manner of dancing is relatively simple.

To achieve the above objective, an embodiment of the third aspect of the present application provides an electronic device including: a housing, a processor, a memory, a circuit board, and a power supply circuit, wherein the circuit board is disposed inside the space enclosed by the housing, and is processed. And a memory disposed on the circuit board; a power supply circuit for powering each circuit or device of the electronic device; a memory for storing executable program code; and the processor operating by reading executable program code stored in the memory The program corresponding to the executable program code is used to execute the video generating method described in the first aspect of the present application.

To achieve the above objective, the fourth aspect of the present application provides a non-transitory computer readable storage medium having stored thereon a computer program, wherein the program is executed by the processor to implement the first aspect of the present application. The video generation method described in the embodiment.

In order to achieve the above object, the fifth aspect of the present application provides a computer program product, where the instructions in the computer program product are executed by a processor, and the video generation method according to the embodiment of the first aspect of the present application is executed. .

The aspects and advantages of the present invention will be set forth in part in the description which follows.

DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings used in the embodiments will be briefly described below. Obviously, the drawings in the following description are some embodiments of the present application. Those skilled in the art can also obtain other drawings based on these drawings without paying any creative work.

FIG. 1 is a schematic flowchart diagram of a first image collection method according to an embodiment of the present application;

2 is a schematic flowchart of a second image collection method according to an embodiment of the present application;

3 is a schematic flowchart of a third image collection method according to an embodiment of the present application;

4 is a schematic structural diagram of an image collection device according to an embodiment of the present application;

FIG. 5 is a schematic structural diagram of another image capturing apparatus according to an embodiment of the present disclosure;

FIG. 6 is a schematic structural diagram of an embodiment of an electronic device according to the present application.

Detailed ways

The embodiments of the present application are described in detail below, and the examples of the embodiments are illustrated in the drawings, wherein the same or similar reference numerals are used to refer to the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are intended to be illustrative, and are not to be construed as limiting.

The existing somatosensory dance game is mainly applied to fixed devices, such as a somatosensory dance machine, a computer, etc., and the portability is poor. In addition, the determination of the user's body motion is to determine whether the direction of the arrow of the user's foot is correct or not, and the manner of dancing is relatively simple. In the embodiment of the present application, the selected audio and the time nodes in the audio are obtained. Corresponding standard action, when starting to play audio, start collecting video picture frames; according to the preset advance time, before the audio is played to each time node, start to display corresponding standard actions; recognition is collected during the display of standard actions The human body motion in the video frame frame; when the human body motion matches the standard motion, the standard motion display process is ended; when the audio playback ends, the target video is generated according to the collected video frame frame and audio. In this embodiment, since the standard action is a human body action that the user needs to make, compared with the dance mode of the user's foot arrow in the prior art, the dance action can be effectively enriched and the user experience can be improved. In addition, the video generation method can be applied to an electronic device, which can effectively improve the applicability of the method. Moreover, when the human body action is matched with the standard action, the display process of the standard action is ended, and the number of schematic diagrams of the standard actions displayed in the shooting interface can be reduced, which is convenient for the user to watch and further enhance the user experience.

The video generation method, apparatus, and electronic device of the embodiments of the present application are described below with reference to the accompanying drawings. The video generation method can be applied to an application of an electronic device, such as a personal computer (PC), a cloud device or a mobile device, a mobile device such as a smart phone, or a tablet computer.

FIG. 1 is a schematic flowchart diagram of a first video generating method according to an embodiment of the present application.

As shown in FIG. 1, the video generation method includes the following steps:

Step 101: Acquire selected audio, and standard actions corresponding to each time node in the audio.

As a possible implementation manner, an application condition of the audio selection may be set on the application of the electronic device. For example, the trigger condition may be an audio selection control, and the user may trigger the selection of audio through the audio selection control. For example, when the user triggers the audio selection control, the song selection interface can be invoked, and then the user can arbitrarily select an audio from the song selection interface as the audio selected by itself. When the user selects the audio, the application can get the audio selected by the user.

As another possible implementation manner, a shooting control may be set on an application of the electronic device, and when the application detects the user's operation for the shooting control, for example, when the user clicks the shooting control, the application interface The song selection interface can be automatically displayed, and then the user can arbitrarily select an audio from the song selection interface as the audio selected by itself. When the user selects the audio, the application can get the audio selected by the user.

In this embodiment, the audio in the song selection interface may be pre-imported into a corresponding standard action. Specifically, each time node in the audio has a corresponding standard action. Therefore, after the application obtains the selected audio, the application The program can obtain standard actions corresponding to each time node from the audio.

Step 102: When the audio is started to be played, the video frame frame is started to be acquired.

Optionally, in the shooting interface, after the user selects the audio, the electronic device can play the audio according to the user's operation. For example, when the electronic device detects that the user clicks the audio, the electronic device can play the audio and simultaneously open the audio. The camera captures each video frame.

Step 103: Start displaying the corresponding standard action before the audio is played to each time node according to the preset advance time.

It can be understood that the user needs to react for a period of time from the time when the user sees the standard action to the human body action. Therefore, in the embodiment of the present application, in order to facilitate the user to make the human body action in time, the audio can be played to each time node. , the advance time preset in advance, showing the corresponding standard action. The preset advance time can be set by the user according to his own needs, or the preset advance time can be preset by the built-in program of the electronic device, which is not limited. It should be understood that the preset advance time should not be set too long, for example, the preset advance time may be 0.2 s.

Specifically, for each time node, the time node is compared with the advance time to obtain a difference, and then the difference is used as a starting time, and then a schematic diagram of the standard action can be displayed from the starting time.

As a possible implementation manner, a schematic diagram of a standard motion may be displayed in any area of the shooting interface. The schematic diagram of the standard motion may be fixed, or the schematic diagram of the standard motion may move along a preset trajectory. limit. The preset track may be preset for the built-in program of the electronic device.

As another possible implementation manner, in order to prevent the user from viewing the content on the screen of the electronic device, the user can watch the standard action. In this embodiment, the semi-transparent mask can be displayed on the shooting interface, wherein the mask has In the hollowed out area of interest, the area of interest displays an image showing the standard action, ie a schematic showing the standard actions in the area of interest. Alternatively, the corresponding standard action can be displayed in the form of a barrage on the shooting interface, which is not limited.

When the schematic diagram of the standard motion moves along the preset trajectory, while the photographing interface displays the schematic diagram of the standard motion, the schematic diagram of the standard motion can be controlled to move along the preset trajectory.

Step 104, identifying a human body motion in a video frame frame collected during the presentation of the standard action.

As a possible implementation manner, the camera for collecting video frame frames may be a camera capable of collecting user depth information, and the acquired depth information may identify human body motions in the video frame. For example, the camera may be a Red-Green-Blue Depth (RGBD), and the depth information of the human body in the video picture frame may be acquired while being imaged, so that the human body motion in the video picture frame can be identified according to the depth information. In addition, the body motion depth information can be acquired by the structured light or the TOF lens, so that the human body motion in the video frame frame can be identified according to the depth information, which is not limited.

As another possible implementation manner, each joint of the human body in the video picture frame can be identified. For example, the face information in the video picture frame and the position information of the human face can be recognized according to the face recognition technology, and then according to the human anatomy. The proportional relationship between the limb and the height can be used to calculate the position information of each joint of the human body. Of course, the position information of each joint of the human body in the video picture frame can also be determined by other algorithms, which is not limited.

After identifying each joint, the two joints adjacent to each joint of the human body can be connected to obtain the connection between the adjacent two joints, and finally according to the actual angle between the connection between the adjacent joints and the preset reference direction. Determine the human motion in the video frame. The preset reference direction may be a horizontal direction or a vertical direction.

Step 105: If the human body action matches the standard action, the display process of the standard action is ended.

In the embodiment of the present application, it may be determined in advance whether the human body motion in the video frame frame matches the standard motion, and when the human motion matches the standard motion, the display process of the standard motion is ended. Thereby, the number of standard motion diagrams displayed in the shooting interface can be reduced, which is convenient for the user to watch and enhance the user experience.

As a possible implementation manner, whether the human motion and the standard motion match can be determined according to whether the difference degree between the human motion and the standard motion is greater than a difference threshold. Specifically, it is possible to determine a standard angle between a line connecting each adjacent two joints and a reference direction when performing a standard action, and compare the corresponding standard angle with the actual line for each adjacent two joints. The difference between the angles. When the difference calculated by the connection between each adjacent two joints is within the error range, it can be determined that the human motion in the video frame is matched with the standard motion, and when there is at least one adjacent two joints When the difference calculated by the connection is not within the error range, it can be determined that the human motion in the video frame does not match the standard motion.

Optionally, when the human body action matches the standard action, it indicates that the degree of difference between the human body action and the corresponding standard action made by the user is small. At this time, the display process of the standard action may be ended to perform the next time node. The corresponding standard action display.

Step 106: When the audio playback ends, the target video is generated according to the collected video picture frame and audio.

In the embodiment of the present application, when the audio playback of the shooting interface ends, the application program may generate the target video according to all the captured video frame frames and audio. Specifically, when the shooting interface displays a standard action corresponding to the last time node of the audio, the camera collects a video frame during the display of the standard action, and if the human action in the video frame matches the standard action, the user may end The presentation process of the standard action, and then the application can generate the target video based on all captured video frame frames and audio.

The video generating method of the embodiment obtains the selected audio, and the standard action corresponding to each time node in the audio, and starts to collect the video picture frame when starting to play the audio; according to the preset advance time, the audio is played to each Before a time node, start to display the corresponding standard action; identify the human body motion in the video frame frame collected during the display of the standard action; end the display process of the standard action when the human action matches the standard action; At the end, the target video is generated based on the captured video frame and audio. In this embodiment, since the standard action is a human body action that the user needs to make, compared with the dance mode of the user's foot arrow in the prior art, the dance action can be effectively enriched and the user experience can be improved. In addition, the video generation method can be applied to an electronic device, which can effectively improve the applicability of the method. Moreover, when the human body action is matched with the standard action, the display process of the standard action is ended, and the number of schematic diagrams of the standard actions displayed in the shooting interface can be reduced, which is convenient for the user to watch, further enhance the user experience, and is used to solve the existing body dance. The game is mainly used on fixed devices, such as somatosensory dance machines, computers, etc., and the portability is poor. In addition, the judgment of the user's body movement is a technical problem by which the direction of the arrow of the user's foot is correct or not, and the manner of dancing is relatively simple.

In order to clearly illustrate the previous embodiment, this embodiment provides another video generation method. FIG. 2 is a schematic flowchart of another video generation method according to an embodiment of the present application.

As shown in FIG. 2, the video generation method may include the following steps:

Step 201: Acquire selected audio, and standard actions corresponding to each time node in the audio.

Step 202: When the audio is started to be played, the video frame frame is started to be collected.

For the execution process of the steps 201 to 202, refer to the execution process of the steps 101 to 102 in the foregoing embodiment, and details are not described herein.

Step 203: For each time node, the difference between the time node and the advance time is taken as the starting time.

For example, when a certain time node is A, the preset advance time of the mark is T, and the start time corresponding to the time node is A-T.

Step 204, starting from the start time, showing a schematic diagram of the standard action.

For the execution process of step 204, refer to the execution process of step 103 in the foregoing embodiment, and details are not described herein.

Step 205: Determine, from among the plurality of to-be-selected tracks, a preset track for displaying a schematic diagram of the standard action.

In the embodiment of the present application, the shooting interface may have multiple to-be-selected tracks, and each track is used to display a schematic diagram of standard actions corresponding to different time nodes, where the preset track is different from the schematic diagram of the corresponding standard action of the adjacent time node. The trajectory, thereby enabling a schematic representation of multiple standard actions simultaneously displayed on the capture interface.

In a specific implementation, a preset trajectory for displaying a schematic diagram of a standard action may be determined from a plurality of candidate trajectories. For example, when there are three candidate tracks in the shooting interface, respectively: track 1, track 2, and track 3, track 1 shows a schematic diagram of the standard action corresponding to time node N, and track 2 shows time node N+1. A schematic diagram of the standard action. If the current time node is the time node N+2, the schematic diagram of the standard action corresponding to the current time node may be displayed on the track 3, or displayed on the track 1, that is, the preset track may be the track 1 or the track 3, Therefore, the preset trajectory may be different from the adjacent time node, that is, the trajectory displayed by the node N+1 corresponding to the schematic diagram of the standard action may be different from the trajectory 2.

Step 206: Determine whether the preset track has a schematic diagram being displayed. If yes, go to step 207. Otherwise, go to step 208.

Step 207: When the distance between two adjacent schematic images in the preset track is less than the threshold distance, according to the number of the schematics being displayed and the length of the preset track in the preset track, the schematic and standard of the preset track being displayed are reduced. The size of the schematic of the action is such that the distance between two adjacent schematics in the preset trajectory is greater than or equal to the threshold distance.

In the embodiment of the present application, in order to facilitate the user to view the schematic diagram of the standard action and ensure the display effect of the standard action, the distance between two adjacent schematic views in the preset track is greater than or equal to the threshold distance, wherein the threshold distance may be built in by the electronic device. The program is preset.

It can be understood that when the preset trajectory on the shooting interface has a schematic diagram being displayed, in order to facilitate the display of the schematic diagram of the standard motion, the number of the schematic diagram being displayed and the length of the preset trajectory may be automatically reduced according to the preset trajectory. The schematic diagram of the preset trajectory and the size of the schematic diagram of the standard motion, so that the distance between two adjacent schematic diagrams in the preset trajectory is greater than or equal to the threshold distance, so that multiple schematic diagrams can be displayed in the preset trajectory at the same time. Avoid the situation where the two adjacent schematics overlap, and ensure the display effect of the standard action.

Step 208: Control a schematic diagram of the standard action to move along the preset trajectory with a preset speed and direction.

In the embodiment of the present application, the preset speed may be preset by a built-in program of the electronic device, for example, the preset speed may be 0.3 pixels per second.

When determining a preset trajectory for displaying a schematic diagram of a standard action, a schematic diagram of the standard action can be controlled to move along the preset trajectory with a preset speed and direction.

In step 209, it is determined whether the human body action matches the standard action. If yes, step 210 is performed; otherwise, step 211 is performed.

In the embodiment of the present application, when the schematic of the standard motion moves along the preset trajectory, whether the human motion and the standard motion match are determined according to whether the difference degree between the human motion and the standard motion is greater than the difference threshold. For details, refer to the description of step 105 in the foregoing embodiment, and details are not described herein.

When the human body action matches the standard motion, step 210 may be triggered, and when the human body motion does not match the standard motion, step 211 may be performed.

Step 210, ending the display process of the standard action.

Optionally, in the process of moving the schematic diagram of the standard action, if the human body action is recognized to match the standard action, indicating the degree of difference between the human body action and the corresponding standard action made by the user, at this time, the standard action can be ended immediately. The presentation process to show the standard actions corresponding to the next time node.

Step 211: When the schematic diagram of the standard action moves to the end point of the preset track, it is determined whether the human body action matching the standard action is recognized. If yes, step 210 is performed; otherwise, step 212 is performed.

In step 212, the schematic diagram showing the standard action is stopped.

Optionally, when the schematic diagram of the standard motion moves to the end point of the preset trajectory, if the human body action matching the standard action is not recognized, at this time, in order to enable the user to make the standard action corresponding to the next time node in time, the final generation is guaranteed. The continuity of the video, in the embodiment of the present application, can stop the schematic diagram showing the standard action.

Step 213, generating motion evaluation information of the human body motion according to the degree of difference between the human body motion and the standard motion.

In the embodiment of the present application, the action evaluation information of the human body action includes a human action action score, which is used to indicate the degree of difference between the human body action and the corresponding standard action. Specifically, the higher the human action action score indicates the human body action and the corresponding The smaller the difference between the standard actions, and the lower the human action score, the greater the difference between the human body action and the corresponding standard action.

Specifically, when the human motion in the video frame does not match the standard motion, the score obtained by the user's human motion may be set to 0, and when the human motion in the video frame matches the standard motion, A line connecting two adjacent joints determines the scoring coefficient of the line according to the corresponding difference and error range. For example, the mark error range is [a, b], and the error is Δ, which can be based on the formula p=1- [2Δ/(ab)], the scoring coefficient p of the connection is calculated, or the scoring coefficient of the connection may be calculated according to other algorithms, which is not limited. After obtaining the scoring coefficient of the connection, the evaluation information of the connection may be generated according to the scoring coefficient of the connection and the score corresponding to the connection. For example, the evaluation information of the connection may be equal to the scoring coefficient of the connection multiplied by the connection. Corresponding score. Finally, the motion evaluation information of the human body motion can be obtained by adding the evaluation information of the links between the adjacent two joints.

Further, the motion evaluation information of the human body motion may further include an animation effect corresponding to the section to which the human motion score belongs. For example, when the human action score is 100, if the human action score belongs to the interval [90, 100], the animation effect can be “perfect or perfect” and match the diamond flash, the interval [80, 90), the animation effect can It is "very good or good" and is matched with flowers.

For example, according to the degree of difference between the standard action of the time node A and the human body motion, the generated human action score is 94 points, and the animation effect generated on the shooting interface is “perfect” and is matched with the diamond flashing. Thereby, the user can be made aware of whether the human body movements made by the user are in a timely manner, thereby improving the user's sense of substitution.

Step 214: When the audio playback ends, the target video is generated according to the audio, each video frame frame, and the motion evaluation information of each human body motion.

In the embodiment of the present application, when the audio playback ends, the action evaluation information of the human body action corresponding to the different time nodes may be acquired, and then the target video is generated according to the audio, the acquired video picture frames, and the motion evaluation information of the corresponding human body motion. .

As a possible implementation manner, motion evaluation information corresponding to the human body motion may be added to each video frame frame according to the human body motion recognized by each video frame frame, and then the motion evaluation information is added according to the audio and the action information. a video frame frame to generate the target video.

As a possible implementation manner, referring to FIG. 3, based on the embodiment shown in FIG. 1 and FIG. 2, after the

step

101 or 201, the video generating method may further include the following steps:

In step 301, the countdown interface is displayed and the countdown is started.

Optionally, in order to leave a preparation time for the user, in the embodiment of the present application, after the user selects the audio, the application of the electronic device can display the countdown interface and start counting down. The countdown time can be set by the user according to his own needs, or the countdown time can be preset by the built-in program of the electronic device, and no limitation is imposed here. For example, the countdown time can be 3s.

Step 302, when the countdown ends, the audio is started to play.

Alternatively, audio can be played when the countdown on the countdown interface ends.

The video generation method of this embodiment displays the countdown interface and starts counting down. When the countdown ends, the audio is started to be played. Thereby, it is possible to realize that during the countdown period, the user can adjust his or her own state, thereby better making a human body action that matches the standard action.

In order to implement the above embodiments, the present application also proposes a video generating apparatus.

FIG. 4 is a schematic structural diagram of a video generating apparatus according to an embodiment of the present disclosure.

As shown in FIG. 4, the video generating apparatus 400 includes a selection module 410, an acquisition module 420, a presentation module 430, an identification module 440, a control module 450, and a generation module 460. among them,

The selection module 410 is configured to acquire selected audio and standard actions corresponding to each time node in the audio.

The collecting module 420 is configured to start collecting video frame frames when starting to play audio.

The display module 430 is configured to start displaying corresponding standard actions before the audio is played to each time node according to the preset advance time.

The identification module 440 is configured to identify a human body motion in a video frame frame collected during the presentation of the standard action.

The control module 450 is configured to end the display process of the standard action when the human body action matches the standard action.

The generating module 460 is configured to generate a target video according to the collected video picture frame and audio when the audio playing ends.

Further, as a possible implementation manner of the embodiment of the present application, referring to FIG. 5, on the basis of the embodiment shown in FIG. 4, the video generating apparatus 400 may further include:

The display play module 470 is configured to display the countdown interface after acquiring the selected audio and the standard action corresponding to each time node in the audio, and start counting down, and when the countdown ends, start playing the audio.

The determining module 480 is configured to determine whether the human body action and the standard action are based on whether the degree of difference between the human body action and the standard action is greater than a difference threshold after identifying a human body motion in the video frame frame collected during the display of the standard action match.

The evaluation information generating module 490 is configured to generate motion evaluation information of the human body motion according to the degree of difference between the human body motion and the standard motion after the human body motion is matched with the standard motion.

In the embodiment of the present application, the display module 430 includes:

The processing sub-module 431 is configured to use, as a starting time, a difference between the time node and the advance time length for each time node.

The control sub-module 432 is configured to display a schematic diagram of the standard action from the start time, and control the schematic diagram of the standard action to move along the preset trajectory.

As a possible implementation manner, the control sub-module 432 is specifically configured to determine, from a plurality of to-be-selected tracks, a preset track for displaying a schematic diagram of the standard action; the preset track is different from the corresponding standard of the adjacent time node. The schematic diagram of the action is displayed; the schematic diagram of the control standard action is moved along the preset trajectory by the preset speed and direction.

Optionally, the control sub-module 432 is further configured to: when the preset trajectory has a schematic diagram being displayed, according to the preset trajectory, the number of the schematic diagram being displayed and the length of the preset trajectory, reducing the schematic diagram that the preset trajectory is being displayed And the size of the schematic diagram of the standard action, such that the distance between two adjacent schematics in the preset trajectory is greater than or equal to the threshold distance.

The stop sub-module 433 is configured to stop the display of the standard action if the human body action matching the standard action is not recognized when the schematic diagram of the standard action moves to the end point of the preset track.

It should be noted that the foregoing description of the video generation method embodiment is also applicable to the video generation apparatus 400 of this embodiment, and details are not described herein again.

The video generating apparatus of the embodiment obtains the selected audio and the standard action corresponding to each time node in the audio, and starts to collect the video picture frame when starting to play the audio; according to the preset advance time, the audio is played to each Before a time node, start to display the corresponding standard action; identify the human body motion in the video frame frame collected during the display of the standard action; end the display process of the standard action when the human action matches the standard action; At the end, the target video is generated based on the captured video frame and audio. In this embodiment, since the standard action is a human body action that the user needs to make, compared with the dance mode of the user's foot arrow in the prior art, the dance action can be effectively enriched and the user experience can be improved. In addition, the video generation method can be applied to an electronic device, which can effectively improve the applicability of the method. Moreover, when the human body action is matched with the standard action, the display process of the standard action is ended, and the number of schematic diagrams of the standard actions displayed in the shooting interface can be reduced, which is convenient for the user to watch, further enhance the user experience, and is used to solve the existing body dance. The game is mainly used on fixed devices, such as somatosensory dance machines, computers, etc., and the portability is poor. In addition, the judgment of the user's body movement is a technical problem by which the direction of the arrow of the user's foot is correct or not, and the manner of dancing is relatively simple.

The embodiment of the present application further provides an electronic device, where the electronic device includes the device described in any of the foregoing embodiments.

FIG. 6 is a schematic structural diagram of an embodiment of an electronic device according to the present application, which may implement the process of the embodiment shown in FIG. 1-5 of the present application. As shown in FIG. 6, the electronic device may include: a housing 61, a processor 62, and a memory. 63, a circuit board 64 and a power supply circuit 65, wherein the circuit board 64 is disposed inside the space surrounded by the housing 61, the processor 62 and the memory 63 are disposed on the circuit board 64; and the power supply circuit 65 is used for the electronic device Each circuit or device is powered; the memory 63 is for storing executable program code; the processor 62 is operative to execute a program corresponding to the executable program code by reading the executable program code stored in the memory 63 for performing any of the foregoing embodiments The video generation method.

The description of the embodiment shown in FIG. 1-5 of the present application is omitted, and details are not described herein.

The electronic device exists in a variety of forms including, but not limited to:

(1) Mobile communication devices: These devices are characterized by mobile communication functions and are mainly aimed at providing voice and data communication. Such terminals include: smart phones (such as iPhone), multimedia phones, functional phones, and low-end phones.

(2) Ultra-mobile personal computer equipment: This type of equipment belongs to the category of personal computers, has computing and processing functions, and generally has mobile Internet access. Such terminals include: PDAs, MIDs, and UMPC devices, such as the iPad.

(3) Portable entertainment devices: These devices can display and play multimedia content. Such devices include: audio, video players (such as iPod), handheld game consoles, e-books, and smart toys and portable car navigation devices.

(4) Server: A device that provides computing services. The server consists of a processor, a hard disk, a memory, a system bus, etc. The server is similar to a general-purpose computer architecture, but because of the need to provide highly reliable services, processing power and stability High reliability in terms of reliability, security, scalability, and manageability.

(5) Other electronic devices with data interaction functions.

One of ordinary skill in the art can understand that all or part of the process of implementing the foregoing embodiments can be completed by a computer program to instruct related hardware, and the program can be stored in a computer readable storage medium. When executed, the flow of an embodiment of the methods as described above may be included. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

The foregoing is only a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto, and any change or replacement that can be easily conceived by those skilled in the art within the technical scope disclosed by the present application is All should be covered by the scope of this application. Therefore, the scope of protection of this application should be determined by the scope of protection of the claims.

In order to implement the above embodiments, the present application further provides a non-transitory computer readable storage medium having stored thereon a computer program, wherein the program is executed by a processor to implement a video generation method as described in the foregoing embodiments. .

In order to implement the above embodiments, the present application also provides a computer program product that, when executed by a processor, executes a video generation method as described in the foregoing embodiments.

In the description of the present specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means a specific feature described in connection with the embodiment or example. A structure, material or feature is included in at least one embodiment or example of the application. In the present specification, the schematic representation of the above terms is not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. In addition, various embodiments or examples described in the specification, as well as features of various embodiments or examples, may be combined and combined.

Moreover, the terms "first" and "second" are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, features defining "first" or "second" may include at least one of the features, either explicitly or implicitly. In the description of the present application, the meaning of "a plurality" is at least two, such as two, three, etc., unless specifically defined otherwise.

Any process or method description in the flowcharts or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logic function or process. And the scope of the preferred embodiments of the present application includes additional implementations, in which the functions may be performed in a substantially simultaneous manner or in the reverse order depending on the functions involved, in accordance with the illustrated or discussed order. It will be understood by those skilled in the art to which the embodiments of the present application pertain.

The logic and/or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer readable medium, Used in conjunction with, or in conjunction with, an instruction execution system, apparatus, or device (eg, a computer-based system, a system including a processor, or other system that can fetch instructions and execute instructions from an instruction execution system, apparatus, or device) Or use with equipment. For the purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport a program for use in an instruction execution system, apparatus, or device, or in conjunction with the instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer readable media include the following: electrical connections (electronic devices) having one or more wires, portable computer disk cartridges (magnetic devices), random access memory (RAM), Read only memory (ROM), erasable editable read only memory (EPROM or flash memory), fiber optic devices, and portable compact disk read only memory (CDROM). In addition, the computer readable medium may even be a paper or other suitable medium on which the program can be printed, as it may be optically scanned, for example by paper or other medium, followed by editing, interpretation or, if appropriate, other suitable The method is processed to obtain the program electronically and then stored in computer memory.

It should be understood that portions of the application can be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware and in another embodiment, it can be implemented by any one or combination of the following techniques well known in the art: discrete with logic gates for implementing logic functions on data signals Logic circuits, application specific integrated circuits with suitable combinational logic gates, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), and the like.

One of ordinary skill in the art can understand that all or part of the steps carried by the method of implementing the above embodiments can be completed by a program to instruct related hardware, and the program can be stored in a computer readable storage medium. When executed, one or a combination of the steps of the method embodiments is included.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. The integrated modules, if implemented in the form of software functional modules and sold or used as stand-alone products, may also be stored in a computer readable storage medium.

The above mentioned storage medium may be a read only memory, a magnetic disk or an optical disk or the like. While the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are illustrative and are not to be construed as limiting the scope of the present application. The embodiments are subject to variations, modifications, substitutions and variations.

Claims

A video generation method, comprising the steps of:

Obtaining selected audio, and standard actions corresponding to each time node in the audio;

When the audio is started to be played, the acquisition of the video picture frame is started;

According to the preset advance time, before the audio is played to each time node, the corresponding standard action is started to be displayed;

Identifying human body motions in a video frame frame acquired during the presentation of the standard action;

Ending the display process of the standard action if the human body action matches the standard action;

When the audio playback ends, a target video is generated based on the captured video frame frame and the audio.
The video generating method according to claim 1, wherein the displaying the corresponding standard action starts before the audio is played to each time node according to the preset advance time length, including:

For each time node, the difference between the time node and the advance time is taken as the starting time;

Starting from the start time, a schematic diagram of the standard action is displayed, and a schematic diagram of controlling the standard action is moved along a preset trajectory.
The video generating method according to claim 2, wherein after the schematic diagram of controlling the standard action moves along a preset trajectory, the method further includes:

When the schematic diagram of the standard action moves to the end point of the preset trajectory, if the human body action matching the standard action is not recognized, the schematic diagram showing the standard action is stopped.
The video generating method according to claim 2 or 3, wherein the controlling the schematic diagram of the standard motion to move along a preset trajectory comprises:

Determining, from a plurality of candidate trajectories, a preset trajectory for displaying a schematic diagram of the standard action; the preset trajectory is different from a trajectory displayed by a schematic diagram corresponding to a standard action of an adjacent time node;

A schematic diagram of controlling the standard action is performed along the preset trajectory at a preset speed and direction.
The video generating method according to claim 4, further comprising: after determining the preset trajectory for displaying the schematic diagram of the standard action, further comprising:

If the preset track has a schematic diagram being displayed;

Defining a schematic diagram of the preset trajectory being displayed and a size of the schematic diagram of the standard motion according to the number of the schematics being displayed and the length of the preset trajectory in the preset trajectory, so that the preset trajectory is in phase The distance between the adjacent two schematics is greater than or equal to the threshold distance.
The video generating method according to any one of claims 1 to 5, wherein after the acquiring the selected audio and the standard action corresponding to each time node in the audio, the method further includes:

Show the countdown interface and start counting down;

When the countdown ends, the audio begins to play.
The video generating method according to any one of claims 1 to 6, wherein the recognizing the human body motion in the video frame frame collected during the display of the standard action further comprises:

And determining whether the human body action matches the standard action according to whether the degree of difference between the human body action and the standard action is greater than a difference threshold.
The video generating method according to any one of claims 1 to 7, wherein if the human body action matches the standard action, the method further includes:

The motion evaluation information of the human body motion is generated according to the degree of difference between the human body motion and the standard motion.
A video generating device, the device comprising:

a selection module for acquiring selected audio and standard actions corresponding to each time node in the audio;

An acquisition module, configured to start collecting video frame frames when starting to play the audio;

a display module, configured to start displaying corresponding standard actions before the audio is played to each time node according to a preset advance time;

An identification module, configured to identify a human body motion in a video frame frame collected during the display of the standard action;

a control module, configured to end a display process of the standard action when the human body action matches the standard action;

And a generating module, configured to generate a target video according to the collected video picture frame and the audio when the audio playing ends.
The video generating apparatus according to claim 9, wherein the display module comprises:

a processing submodule, configured, for each time node, a difference between the time node and the advance time as a starting time;

And a control submodule, configured to display a schematic diagram of the standard action from the start time, and control a schematic diagram of the standard action to move along a preset trajectory.
The video generating apparatus according to claim 10, wherein the display module further comprises:

The stop sub-module is configured to stop displaying the schematic diagram of the standard action if the human body action matching the standard action is not recognized when the schematic diagram of the standard action moves to the end point of the preset track.
The video generating apparatus according to claim 10 or 11, wherein the control submodule is specifically configured to:

Determining, from a plurality of candidate trajectories, a preset trajectory for displaying a schematic diagram of the standard action; the preset trajectory is different from a trajectory displayed by a schematic diagram corresponding to a standard action of an adjacent time node;

A schematic diagram of controlling the standard action is performed along the preset trajectory at a preset speed and direction.
The video generating apparatus according to claim 12, wherein the control submodule is further configured to:

When the preset trajectory has a schematic diagram being displayed, according to the number of the schematic diagrams being displayed and the length of the preset trajectory, the schematic diagram of the preset trajectory being displayed and the standard motion are reduced. The size of the schematic is such that the distance between two adjacent schematics in the predetermined trajectory is greater than or equal to a threshold distance.
The video generating apparatus according to any one of claims 9 to 13, wherein the apparatus further comprises:

The display playing module is configured to display a countdown interface after the selected audio and the standard action corresponding to each time node in the audio, and start counting down, and when the countdown ends, start playing the audio.
The video generating apparatus according to any one of claims 9 to 14, wherein the apparatus further comprises:

a determining module, configured to determine, according to the human body motion in the video frame frame collected during the displaying of the standard action, whether the difference between the human motion and the standard motion is greater than a difference threshold, It is determined whether the human body action matches the standard action.
The video generating apparatus according to any one of claims 9 to 15, wherein the apparatus further comprises:

The evaluation information generating module is configured to generate motion evaluation information of the human body motion according to a degree of difference between the human body motion and the standard motion after the human body motion is matched with the standard motion.
An electronic device, comprising: a housing, a processor, a memory, a circuit board, and a power supply circuit, wherein the circuit board is disposed inside the space enclosed by the housing, and the processor and the memory are disposed on the circuit board; a circuit for powering each circuit or device of the above electronic device; a memory for storing executable program code; the processor running a program corresponding to the executable program code by reading executable program code stored in the memory, for A video generation method according to any one of claims 1-8.
A non-transitory computer readable storage medium having stored thereon a computer program, wherein the program is executed by a processor to implement the video generating method according to any one of claims 1-8.
A computer program product, wherein the video generation method according to any one of claims 1-8 is performed when an instruction in the computer program product is executed by a processor.