WO2022028136A1 - 舞蹈视频的动作提取方法、装置、计算机设备及存储介质 - Google Patents
舞蹈视频的动作提取方法、装置、计算机设备及存储介质 Download PDFInfo
- Publication number
- WO2022028136A1 WO2022028136A1 PCT/CN2021/101384 CN2021101384W WO2022028136A1 WO 2022028136 A1 WO2022028136 A1 WO 2022028136A1 CN 2021101384 W CN2021101384 W CN 2021101384W WO 2022028136 A1 WO2022028136 A1 WO 2022028136A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- target
- node
- skeleton
- skeleton node
- skeletal
- Prior art date
Links
- 238000000605 extraction Methods 0.000 title claims abstract description 35
- 230000033001 locomotion Effects 0.000 title claims abstract description 24
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 30
- 238000000034 method Methods 0.000 claims abstract description 22
- 230000009471 action Effects 0.000 claims description 76
- 238000007621 cluster analysis Methods 0.000 claims description 18
- 230000033764 rhythmic process Effects 0.000 claims description 11
- 230000008859 change Effects 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000004458 analytical method Methods 0.000 abstract description 2
- 230000000875 corresponding effect Effects 0.000 description 51
- 210000000988 bone and bone Anatomy 0.000 description 8
- 239000000284 extract Substances 0.000 description 7
- 230000006870 function Effects 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 230000003993 interaction Effects 0.000 description 4
- 230000036544 posture Effects 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 3
- 230000003238 somatosensory effect Effects 0.000 description 3
- 210000003423 ankle Anatomy 0.000 description 2
- 210000000454 fifth toe Anatomy 0.000 description 2
- 210000001255 hallux Anatomy 0.000 description 2
- 238000003064 k means clustering Methods 0.000 description 2
- 210000003127 knee Anatomy 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 239000000203 mixture Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000010009 beating Methods 0.000 description 1
- 230000001627 detrimental effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/23—Recognition of whole body movements, e.g. for sport training
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
- G06T7/251—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving models
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/20—Input arrangements for video game devices
- A63F13/21—Input arrangements for video game devices characterised by their sensors, purposes or types
- A63F13/213—Input arrangements for video game devices characterised by their sensors, purposes or types comprising photodetecting means, e.g. cameras, photodiodes or infrared cells
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/40—Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
- A63F13/42—Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
- A63F13/428—Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/60—Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor
- A63F13/65—Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor automatically by game devices or servers from real world data, e.g. measurement in live racing competition
- A63F13/655—Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor automatically by game devices or servers from real world data, e.g. measurement in live racing competition by importing photos, e.g. of the player
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/155—Segmentation; Edge detection involving morphological operators
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
- G06T7/248—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving reference images or patches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/762—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using clustering, e.g. of similar faces in social networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/60—Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor
- A63F13/65—Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor automatically by game devices or servers from real world data, e.g. measurement in live racing competition
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/80—Special adaptations for executing a specific game genre or game mode
- A63F13/816—Athletics, e.g. track-and-field sports
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
- A63F2300/60—Methods for processing data by generating or executing the game program
- A63F2300/66—Methods for processing data by generating or executing the game program for rendering three dimensional images
- A63F2300/6607—Methods for processing data by generating or executing the game program for rendering three dimensional images for animating game characters, e.g. skeleton kinematics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20036—Morphological image processing
- G06T2207/20044—Skeletonization; Medial axis transform
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30221—Sports video; Sports image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30241—Trajectory
Definitions
- the present application relates to the technical field of video processing, and in particular, to a method, device, computer equipment and storage medium for motion extraction from dance videos.
- Traditional dance-like somatosensory games are generally produced and updated by the official song. By manually selecting dance movements and creating corresponding dance movement sequences, they are compared with the actual movements of the user, and the interactive content of the game such as scoring and beating can be realized.
- the purpose of this application is to provide a technical solution that can efficiently and accurately convert a dance video uploaded by a user into standard game movements, so as to solve the above problems.
- the application provides a method for extracting movements from a dance video, comprising the following steps:
- the key skeletal node images are output in sequence to obtain a standard action sequence corresponding to each target dancer in the dance video.
- the dance video includes a plurality of dancers, and the target dancer is selected from one of the plurality of dancers.
- the step of extracting the skeletal node image corresponding to the target dancer contained in the video frame based on the skeletal node identification algorithm includes:
- a skeleton node image corresponding to the target dancer is extracted from the video frame according to the determined skeleton node model, where the skeleton node image includes a plurality of skeleton node data.
- the step of extracting the skeletal node image corresponding to the target dancer contained in the video frame based on the skeletal node identification algorithm further includes:
- the step of performing cluster analysis on all the skeletal node images corresponding to each of the target dancers included in the dance video to obtain a plurality of cluster sets includes the following steps: :
- the remaining skeleton node images and the corresponding target skeleton node images with the smallest similarity are classified into the same set to obtain K candidate sets;
- the obtained K candidate sets are used as the cluster set.
- the K target skeleton node images are determined according to the rhythm of the dance video.
- the similarity includes Euclidean distance
- the step of calculating the similarity between each remaining skeleton node image and the target skeleton node image includes:
- the sum of the Euclidean distances of all skeleton nodes is used as the Euclidean distance between the remaining skeleton node images and the target skeleton node image.
- the step of determining a new target skeletal node image according to the identification number of the skeletal node image included in each candidate set includes:
- the skeleton node image corresponding to the target identification number is used as the new target skeleton node image.
- the step of extracting the key skeletal node image serving as the cluster center in each of the cluster sets includes:
- an action extraction device for dance videos including:
- a video receiving module adapted to receive a dance video uploaded by a user that includes one or more dancers, and obtain video frames in the dance video;
- a skeletal node identification module adapted to identify a skeletal node image corresponding to a target dancer contained in the video frame based on a skeletal node identification algorithm; wherein the target dancer is selected from the one or more dancers;
- a clustering module adapted to perform cluster analysis on all the skeletal node images corresponding to each of the target dancers contained in the dance video, to obtain a plurality of cluster sets;
- the key image acquisition module is suitable for acquiring the key skeletal node image as the cluster center in each described clustering set;
- the action output module is adapted to sequentially output the images of the key skeletal nodes, so as to obtain the standard action sequence corresponding to each of the target dancers in the dance video.
- the present application also provides a computer device, comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, and the processor implements the computer-readable instructions when executing the computer-readable instructions.
- a computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, and the processor implements the computer-readable instructions when executing the computer-readable instructions.
- the key skeletal node images are output in sequence to obtain a standard action sequence corresponding to each target dancer in the dance video.
- the present application also provides a computer-readable storage medium on which computer-readable instructions are stored, and when the computer-readable instructions are executed by a processor, the following steps are implemented:
- the key skeletal node images are output in sequence to obtain a standard action sequence corresponding to each target dancer in the dance video.
- the dance video action extraction method, device, computer equipment and computer-readable storage medium can quickly extract standard actions according to the dance videos uploaded by users and output them to the client, so that game players can perform corresponding actions according to the standard actions. Somatosensory operation to realize human-computer interaction or network interaction between multiple users. Specifically, the present application extracts a skeleton node image from each frame of a dance video based on a skeleton node identification algorithm, which can accurately identify different postures of the dancer. Selecting key skeletal node images from all skeletal node images based on the clustering algorithm can not only ensure the integrity of standard actions, but also avoid excessively dense actions that increase the difficulty of the game or affect the score, and ensure that users get a good gaming experience.
- Embodiment 1 is a flowchart of Embodiment 1 of a method for extracting movements from dance videos of the present application;
- FIG. 2 is a schematic flowchart of extracting a skeleton node image in Embodiment 1 of the present application
- FIG. 3 is a schematic diagram of a 25-skeletal node model in Embodiment 1 of an action extraction method for a dance video of the present application;
- FIG. 6 is a schematic flowchart of determining a new target skeleton node image in Embodiment 1 of the present application.
- FIG. 7 is a schematic diagram of selecting a cluster center as a key frame in Embodiment 1 of the present application.
- FIG. 8 is a schematic flowchart of the simplified processing of repeated key skeletal node images in Embodiment 1 of the present application.
- FIG. 9 is a schematic diagram of a program module of Embodiment 1 of an apparatus for extracting movements from dance videos according to the present application;
- FIG. 10 is a schematic diagram of a hardware structure of Embodiment 1 of an apparatus for extracting movements from dance videos of the present application.
- the dance video action extraction method, device, computer equipment and computer-readable storage medium can quickly extract standard actions according to the dance videos uploaded by users and output them to the client, so that game players can perform corresponding actions according to the standard actions. Somatosensory operation to realize human-computer interaction or network interaction between multiple users. Specifically, the present application extracts a skeleton node image from each frame of a dance video based on a skeleton node identification algorithm, which can accurately identify different postures of the dancer. Selecting key skeletal node images from all skeletal node images based on the clustering algorithm can not only ensure the integrity of standard actions, but also avoid excessively dense actions that increase the difficulty of the game or affect the score, and ensure that users get a good gaming experience.
- the present embodiment proposes a method for extracting movements from a dance video, including the following steps:
- S100 Receive a dance video including one or more dancers uploaded by a user, and acquire video frames in the dance video.
- the dance video in this embodiment refers to a video that clearly and completely reflects the dancer completing a set of dance movements.
- the dance video in this embodiment preferably includes one or two dancers.
- a dance video is composed of multiple video frames.
- the duration of a video is 60 seconds and the corresponding frame rate is 20 frames per second, then the video contains a total of 1200 frames.
- the dance video is converted into all video frames.
- S200 Identify, based on a skeletal node identification algorithm, a skeletal node image corresponding to a target dancer contained in the video frame, wherein the target dancer is selected from the one or more dancers.
- the target dancer in this embodiment refers to the dancer who is the object of dance action recognition.
- Target dancers can contain one or more.
- a dancer at a specific position for example, the front side or the left side
- one dancer designated by the user can be used as the target dancer.
- each dancer in the dance video may be used as the target dancer, or dancers at several specific positions may be used as the target dancer, which is not limited in this application.
- the dance movements reflect the different body postures of the dancers at different times, and the body postures can be determined by the positions of the skeletal nodes.
- any existing skeletal node identification algorithm can be used to identify skeletal nodes in each video frame, for example, any one of a 15-skeletal node model, an 18-skeletal node model, or a 25-skeletal node model can be used to identify skeletal nodes.
- the numbers 15, 18, and 25 above represent the number of skeleton nodes, respectively. The more the number of skeleton nodes, the higher the recognition accuracy and the more computing resources that need to be occupied.
- different skeleton node models can be flexibly selected according to the number of dance videos to be processed and the configuration of the server.
- the bone node images extracted in this step are for the same dancer.
- the rhythm of the movements of different dancers is more or less different, and the rhythm of one dancer in children is often consistent.
- this embodiment extracts a group of skeleton node images for the same dancer, and each group of skeleton node images is used to generate a standard action sequence.
- the present application extracts a set of corresponding skeleton node images for each target dancer, thereby generating a standard action sequence corresponding to each target dancer. In this way, different players can make corresponding dance movements according to the standard action sequences of different target dancers according to their preferences, thereby improving the game experience.
- S300 Perform cluster analysis on all the skeletal node images corresponding to each of the target dancers included in the dance video to obtain multiple cluster sets.
- cluster analysis in this step is to divide all skeleton node images into multiple similar sets according to the positional relationship of different skeleton nodes, so that the cluster center of each set can represent a key frame of a standard action.
- Clustering analysis can be performed by any existing clustering algorithm, such as K-Means clustering algorithm, mean-shift clustering algorithm, density-based clustering algorithm, Gaussian mixture model clustering algorithm, agglomerative hierarchical clustering algorithm, etc. The embodiment does not limit this.
- S400 Acquire an image of a key skeletal node serving as a cluster center in each of the cluster sets.
- each key action of a dancer is extracted from a complex dance video and output it sequentially, wherein each key action is represented by a corresponding skeleton node image. Therefore, in this step, on the basis of obtaining multiple cluster sets, one skeleton node image is selected from each cluster set as the key action when outputting the standard action.
- Appropriate skeleton node images can be selected from each cluster set by various methods, such as selecting skeleton node images at fixed positions in each cluster set, or skeleton node images serving as cluster centers, etc.
- the image of the skeleton node selected to output the key action in each cluster set is the image of the key skeleton node.
- S500 Output the key skeletal node images in sequence to obtain a standard action sequence corresponding to each of the target dancers in the dance video.
- each skeleton node image represents the position of each skeleton node of the body at the current moment. Therefore, by sequentially outputting the determined key skeletal node images, the motion positions of different skeletal nodes at different times can be obtained, that is, the standard action sequence corresponding to the dance video.
- the skeleton recognition technology commonly used in the field of computer vision is used to identify the skeleton node image of each frame in the dance video, and by performing cluster analysis on all skeleton node images, the dance video can be quickly determined. keyframes in .
- This embodiment can automatically convert the dance video uploaded by the user into standard game actions, which not only saves huge manpower expenditure, but also greatly improves conversion efficiency, so that the content of songs and levels in the dance game can be quickly expanded, and user experience can be improved.
- Fig. 2 shows a schematic flowchart of extracting the skeleton node image corresponding to the target dancer contained in the video frame based on the skeleton node identification algorithm in step S200.
- step S200 includes:
- S210 Determine a skeleton node model, where the skeleton node model includes any one of 15 skeleton node models, 18 skeleton node models or 25 skeleton node models.
- Figure 3 shows a schematic diagram of the 25 skeleton node model.
- the 25 skeleton nodes here respectively include nose node, neck node, left shoulder node, right shoulder node, left elbow node, right elbow node, left hand node, Right Hand Node, Left Eye Node, Right Eye Node, Left Ear Node, Right Ear Node, Lower Hip Middle Node, Lower Hip Left Node, Lower Hip Right Node, Left Knee Node, Right Knee Node, Left Ankle Node , Right Ankle Node, Left Big Toe Node, Right Big Toe Node, Left Little Toe Node, Right Little Toe Node, Left Heel Node, and Right Heel Node.
- S220 Extract a skeleton node image corresponding to the target dancer from the video frame according to the determined skeleton node model, where the skeleton node image includes a plurality of skeleton node data, and each skeleton node data includes a skeleton Node type, skeletal node abscissa and skeletal node ordinate.
- each skeleton node data may include skeleton node type, skeleton node abscissa and skeleton node ordinate.
- the skeleton node type indicates which node is specific, such as a left shoulder node, a right shoulder node, a left hand node, and the like.
- the abscissa and ordinate represent the position data of the skeleton node in the skeleton node image.
- S230 Determine whether the number of skeleton nodes included in the skeleton node image is within a preset range, and if not, delete the skeleton node image.
- the preset node range here can be based on the maximum number of skeleton nodes that can be recognized by the skeleton node model. To set, for example, the maximum number of bone nodes is 25, then the preset node range can be set to [18,25].
- this embodiment can acquire an accurate and complete skeleton node image, which provides a good foundation for the subsequent output of standard actions.
- FIG. 4 is a schematic flowchart of performing cluster analysis on skeleton node images in Embodiment 1 of the present application. As shown in Figure 4, step S300 includes:
- S310 Determine the total number K of the set according to the rhythm of the dance video, where K is a natural number.
- the purpose of determining the total number of cluster sets in this step is to determine the number of key skeletal node images used as key frames when the standard action is finally output.
- a cluster set finally corresponds to a key skeletal node image.
- this embodiment may also automatically adjust the extraction frequency according to different rhythms, or allow the user to actively intervene in the number of key frames. For example, the user can choose between an average of 0.5 seconds and a key frame of 3 seconds. In this way, for some dance songs with special rhythm, the number of key frame extraction can also be adjusted accordingly to adapt to the rhythm of the video and the song.
- S320 Select K target skeleton node images from all the skeleton node images, and calculate the similarity between each remaining skeleton node image and the target skeleton node image.
- 10 images are arbitrarily selected as the target skeleton node images, assuming F1, F2, ..., F10, and the remaining skeleton node images include F11, F12, ..., F100.
- the target skeleton node images can also be selected according to a specific time interval, for example, a target skeleton node image is extracted every second.
- the similarity between the remaining skeleton node images and the target skeleton node images may include Euclidean distances between skeleton nodes in the remaining skeleton node images and skeleton nodes in the target skeleton node image.
- the specific steps for calculating the similarity are shown in Figure 5, including:
- S322 Use the sum of the Euclidean distances of all skeleton nodes as the Euclidean distance between the remaining skeleton node images and the target skeleton node image.
- the n skeleton nodes contained in FA can be expressed as A1, A2, ..., An
- the n skeletal nodes contained in FB can be expressed as B1, B2, ..., B25
- n can be 15, 18 or 25
- the Euclidean distance Similarity (F11, F1) between F11 and F1 can be expressed as:
- d(Bm, Am) represents the distance between the mth bone node Bm and the mth bone node Am
- (X Am , Y Am ) and (X Bm , Y Bm ) are the horizontal and vertical coordinates of the skeleton node Am and the horizontal and vertical coordinates of the skeleton node Bm, respectively.
- the Euclidean distance Similarity (F11, F2) between F11 and F2 the Euclidean distance Similarity (F11, F3) between F11 and F3, ..., the Euclidean distance Similarity (F11, F10) between F11 and F10 are also according to The above formula is calculated.
- S330 Classify the remaining skeleton node images and the corresponding target skeleton node images with the smallest Euclidean distance into the same set to obtain K candidate sets.
- the Euclidean distance between F11 and each target bone node image select the value with the smallest Euclidean distance, and classify the bone node images corresponding to F11 and the minimum Euclidean distance into the same A collection of clusters. For example, it is determined by calculation that the Euclidean distance between F11 and F5 is the smallest, then F11 is classified as the cluster set where F5 is located.
- S340 Determine a new target skeleton node image according to the identification number of the skeleton node image included in each of the candidate sets, and repeat steps S320 and S330; the identification number is used to uniquely characterize the appearance order of the skeleton node image .
- step S320 the initial target skeleton node images are arbitrarily selected, so these initial target skeleton node images are probably not suitable as an ideal classification reference, so it is necessary to continuously determine new target skeleton node images, so as to obtain the most optimal skeleton node images. good target.
- FIG. 6 is a schematic flowchart of determining a new target skeleton node image in Embodiment 1 of the present application. As shown in Figure 6, step S340 includes:
- each skeleton node image corresponds to a unique identification number, which is used to represent the appearance order of the skeleton node images.
- the skeleton node image is extracted from the corresponding video frame, and each video frame has a unique sequence number in the dance video, so the sequence number of the corresponding video frame can be used as the skeleton node image in this embodiment. sequence number.
- the skeleton node images F1, F2, ..., F100 mentioned above, the numbers 1 to 100 here are the identification numbers corresponding to each skeleton node image.
- the skeleton node images in a candidate set V include F1, F10, F20 and F30, and the identification numbers here are: 1, 10, 20 and 30.
- S342 Average and round the identification numbers of all the skeleton node images included in each data set to obtain the target identification number.
- the known identification numbers include 1, 10, 20, and 30.
- the average of the identification numbers is (1+10+20+30) ⁇ 4. If it is not divisible, the target identification number corresponding to the set V can be obtained as 15 according to the rounding method.
- the corresponding skeleton node image is F15
- F15 is the new target skeleton node image. Since there are K candidate sets in total in this embodiment, K new target skeleton node images can be obtained accordingly.
- FIG. 7 shows a schematic diagram of selecting a cluster center as a key frame in Embodiment 1 of the present application.
- the intermediate frame set 1, the intermediate frame set 2 and the intermediate frame set 3 represent three cluster sets, and the key frame 1, the key frame 2 and the key frame 3 respectively represent the cluster center of each cluster set.
- the target skeleton node image will not change after multiple iterations, thus obtaining a cluster set with stable and accurate classification.
- the iteration can be terminated at this point by setting a limited number of repetitions, such as 100 times. Among them, each time K new target skeletal node images are determined is called completing a repetition.
- step S400 is used to acquire the image of the key skeletal node serving as the cluster center in each of the cluster sets.
- the repeated images of key skeletal nodes are reduced.
- it specifically includes the following steps:
- the identification number n of the skeleton node image corresponds to the sequence number N of the corresponding video frame
- the sequence number N of the video frame can represent the appearance order of the video frame in the dance video.
- the identification number n or sequence number N is arranged according to the order of natural numbers
- the dance video has a fixed frame rate f, that is, several video frames are played per second, so the identification number n divided by the frame rate f is a skeleton
- the time interval between the images of two adjacent key skeletal nodes cannot be too short, otherwise, on the one hand, the user will not have time to respond, and on the other hand, the change of the action will not be enough. obvious.
- the shortest interval between two adjacent key skeleton node images is I
- the frame rates of different videos are different, the interval I can be limited by the total duration of the dance video.
- the shortest response time t for adults to the change between two images is 0.2 seconds. If the frame rate of a dance video is f, then the shortest interval I above is:
- the number 5 is the difference between the identification numbers of two adjacent key skeletal node images.
- the key skeletal node images obtained in the K-means clustering algorithm can be traversed one by one, and when the difference between the identification numbers of the current and last two key skeletal node images is less than the preset difference (for example, 5), the Key bone node images with lower identification numbers are removed from the result set. Repeat the above operations until the intervals of all adjacent key bone node images meet the above conditions. In this way, redundant content in key skeletal node images can be effectively removed, making the extraction of dance movements more representative.
- the preset difference for example, 5
- the apparatus for action extraction 90 may include or be divided into one or more program modules, and one or more program modules are stored in In the storage medium, and executed by one or more processors, the present application is completed, and the above-mentioned action extraction method can be implemented.
- the program module referred to in this application refers to a series of computer-readable instruction segments capable of performing specific functions, and is more suitable for describing the execution process of the action extraction apparatus 90 in the storage medium than the program itself. The following description will specifically introduce the functions of each program module in this embodiment:
- the video receiving module 91 is adapted to receive the dance video uploaded by the user and include the target dancer, and obtain the video frames in the dance video;
- the skeletal node identification module 92 is adapted to identify the skeletal node image corresponding to the target dancer contained in the video frame based on the skeletal node identification algorithm;
- the clustering module 93 is adapted to perform cluster analysis on all the skeletal node images contained in the dance video to obtain a plurality of cluster sets;
- the key image acquisition module 94 is suitable for acquiring the key skeletal node image as the cluster center in each of the cluster sets;
- the action output module 95 is adapted to sequentially output the images of the key skeletal nodes to obtain the standard actions corresponding to the dance video.
- This embodiment also provides a computer device, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server or a cabinet server (including independent servers, or A server cluster composed of multiple servers), etc.
- the computer device 100 in this embodiment at least includes, but is not limited to, a memory 101 and a processor 102 that can be communicatively connected to each other through a system bus, as shown in FIG. 10 .
- FIG. 10 only shows the computer device 100 having components 101-102, but it should be understood that implementation of all of the illustrated components is not required, and more or fewer components may be implemented instead.
- the memory 101 (that is, a readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card-type memory (eg, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Programmable Read Only Memory (PROM), Magnetic Memory, Magnetic Disk, Optical Disk, etc.
- the memory 101 may be an internal storage unit of the computer device 100 , such as a hard disk or a memory of the computer device 100 .
- the memory 101 may also be an external storage device of the computer device 100, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, flash card (Flash Card), etc.
- the memory 101 may also include both the internal storage unit of the computer device 100 and its external storage device.
- the memory 101 is generally used to store the operating system and various application software installed in the computer device 100 , such as the program code of the action extraction apparatus 90 of the first embodiment, and the like.
- the memory 101 can also be used to temporarily store various types of data that have been output or will be output.
- the processor 102 may be a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, a microprocessor, or other data processing chips.
- the processor 102 is typically used to control the overall operation of the computer device 100 .
- the processor 102 is configured to run the program code or process data stored in the memory 101, for example, run the action extraction apparatus 90, so as to implement the action extraction method of the first embodiment.
- This embodiment also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), only Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM), Magnetic Memory, Magnetic Disk, Optical Disc, Server, App Store, etc., on which computer readable memory is stored Instructions, which implement corresponding functions when the program is executed by the processor.
- the computer-readable storage medium of this embodiment is used to store the action extraction apparatus 90, and when executed by the processor, implements the following steps:
- the key skeletal node images are output in sequence to obtain a standard action sequence corresponding to each target dancer in the dance video.
- any description of a process or method in the flowcharts or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing a specified logical function or step of the process , and the scope of the preferred embodiments of the present application includes alternative implementations in which the functions may be performed out of the order shown or discussed, including performing the functions substantially concurrently or in the reverse order depending upon the functions involved, which should It is understood by those skilled in the art to which the embodiments of the present application belong.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Data Mining & Analysis (AREA)
- Human Computer Interaction (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- General Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Image Analysis (AREA)
Abstract
一种舞蹈视频的动作提取方法、装置、计算机设备和存储介质,所述方法包括以下步骤:接收用户上传的包含一个或多个舞者的舞蹈视频,获取所述舞蹈视频中的视频帧(S100);基于骨骼节点识别算法识别所述视频帧中包含的与目标舞者对应的骨骼节点图像(S200);其中,所述目标舞者选自所述一个或多个舞者;对所述舞蹈视频中包含的与每个所述目标舞者对应的全部所述骨骼节点图像进行聚类分析,以得到多个聚类集合(S300);获取每个所述聚类集合中作为聚类中心的关键骨骼节点图像(S400);按顺序输出所述关键骨骼节点图像,以得到所述舞蹈视频中与每个所述目标舞者对应的标准动作序列(S500)。
Description
本申请申明2020年8月6日递交的申请号为202010784431.7、名称为“舞蹈视频的动作提取方法、装置、计算机设备及存储介质”的中国专利申请的优先权,该中国专利申请的整体内容以参考的方式结合在本申请中。
本申请涉及视频处理技术领域,特别涉及一种舞蹈视频的动作提取方法、装置、计算机设备及存储介质。
传统的舞蹈类体感游戏一般由官方制作和更新歌曲,通过人工挑选舞蹈动作并制作对应的舞蹈动作序列来与用户的实际动作进行比对,实现评分、打擂等游戏互动内容。
为了满足用户的个性化需求,有些游戏商允许用户上传自己录制好的舞蹈视频,并根据用户上传的视频制作舞蹈动作序列。由于游戏用户数量众多,并且每个用户可能有多种需求,因此,发明人意识到,单纯地依靠人工挑选舞蹈动作和制作标准动作序列会占用大量的人力资源,并且影响舞蹈视频转制为游戏动作曲的效率。
发明内容
本申请的目的是提供一种能够高效、准确地将用户上传的舞蹈视频转化为游戏标准动作的技术方案,以解决上述问题。
为实现上述目的,本申请提供一种舞蹈视频的动作提取方法,包括以下步骤:
接收用户上传的包含一个或多个舞者的舞蹈视频,获取所述舞蹈视频中的视频帧;
基于骨骼节点识别算法识别所述视频帧中包含的与目标舞者对应的骨骼节点图像;其中,所述目标舞者选自所述一个或多个舞者;
对所述舞蹈视频中包含的与每个所述目标舞者对应的全部所述骨骼节点图像进行聚类分析,以得到多个聚类集合;
获取每个所述聚类集合中作为聚类中心的关键骨骼节点图像;
按顺序输出所述关键骨骼节点图像,以得到所述舞蹈视频中与每个所述目标舞者对应的标准动作序列。
根据本申请提供的舞蹈视频的动作提取方法,所述舞蹈视频中包含多个舞者,所述目标舞者选自所述多个舞者中的一个。
根据本申请提供的动作提取方法,所述基于骨骼节点识别算法提取所述视频帧中包含的与目标舞者对应的骨骼节点图像的步骤包括:
确定骨骼节点模型;
根据确定的所述骨骼节点模型从所述视频帧中提取与所述目标舞者对应的骨骼节点图像,所述骨骼节点图像包括多个骨骼节点数据。
根据本申请提供的动作提取方法,所述基于骨骼节点识别算法提取所述视频帧中包含的与目标舞者对应的骨骼节点图像的步骤还包括:
判断所述骨骼节点图像中包含的骨骼节点的个数是否在预设范围之内,若否,删除所述骨骼节点图像。
根据本申请提供的动作提取方法,所述对所述舞蹈视频中包含的与每个所述目标舞者对应的全部所述骨骼节点图像进行聚类分析,以得到多个聚类集合的步骤包括:
从全部所述骨骼节点图像中任选K个目标骨骼节点图像,计算每个剩余骨骼节点图像与所述目标骨骼节点图像之间的相似度;
将所述剩余骨骼节点图像与对应的相似度最小的目标骨骼节点图像归为同一个集合,以得到K个候选集合;
根据每个所述候选集合中包含的骨骼节点图像的识别编号确定新的目标骨骼节点图像,重复执行根据K个所述新的目标骨骼节点图像计算所述相似度以及所述得到K个候选集合的步骤;所述识别编号用于唯一表征所述骨骼节点图像的出现顺序;
在所述新的目标骨骼节点图像的识别编号不再变化或者达到预设重复次数的情况下,将得到的K个候选集合作为所述聚类集合。
根据本申请提供的动作提取方法,所述K个目标骨骼节点图像是根据所述舞蹈视频的节奏确定的。
根据本申请提供的动作提取方法,所述相似度包括欧氏距离,所述计算每个剩余骨骼节点图像与所述目标骨骼节点图像之间的相似度的步骤包括:
计算所述剩余骨骼节点图像中的每个骨骼节点与所述目标骨骼节点图像中对应骨骼节点之间的欧氏距离;
将所有骨骼节点的欧式距离之和作为所述剩余骨骼节点图像与所述目标骨骼节点图像之间的欧氏距离。
根据本申请提供的动作提取方法,所述根据每个所述候选集合中包含的骨骼节点图像的识别编号确定新的目标骨骼节点图像的步骤包括:
获取每个聚类集合中包含的每个骨骼节点图像的识别编号;
对每个数据集合中包含的所有骨骼节点图像的识别编号求平均并取整,得到目标识别编号;
将与所述目标识别编号对应的骨骼节点图像作为所述新的目标骨骼节点图像。
根据本申请提供的动作提取方法,所述提取每个所述聚类集合中作为聚类中心的关键骨骼节点图像的步骤包括:
计算任意相邻两个关键骨骼节点图像的识别编号之间的差值;
当所述差值小于第一阈值时,将所述识别编号靠后的关键骨骼节点图像删除。
为实现上述目的,本申请还提供一种舞蹈视频的动作提取装置,包括:
视频接收模块,适用于接收用户上传的包含一个或多个舞者的舞蹈视频,获取所述舞蹈视频中的视频帧;
骨骼节点识别模块,适用于基于骨骼节点识别算法识别所述视频帧中包含的与目标舞者对应的骨骼节点图像;其中,所述目标舞者选自所述一个或多个舞者;
聚类模块,适用于对所述舞蹈视频中包含的与每个所述目标舞者对应的全部所述骨骼节点图像进行聚类分析,以得到多个聚类集合;
关键图像获取模块,适用于获取每个所述聚类集合中作为聚类中心的关键骨骼节点图 像;
动作输出模块,适用于按顺序输出所述关键骨骼节点图像,以得到所述舞蹈视频中与每个所述目标舞者对应的标准动作序列。
为实现上述目的,本申请还提供一种计算机设备,包括存储器、处理器以及存储在存储器上并可在处理器上运行的计算机可读指令,所述处理器执行所述计算机可读指令时实现以下步骤:
接收用户上传的包含一个或多个舞者的舞蹈视频,获取所述舞蹈视频中的视频帧;
基于骨骼节点识别算法识别所述视频帧中包含的与目标舞者对应的骨骼节点图像;其中,所述目标舞者选自所述一个或多个舞者;
对所述舞蹈视频中包含的与每个所述目标舞者对应的全部所述骨骼节点图像进行聚类分析,以得到多个聚类集合;
获取每个所述聚类集合中作为聚类中心的关键骨骼节点图像;
按顺序输出所述关键骨骼节点图像,以得到所述舞蹈视频中与每个所述目标舞者对应的标准动作序列。
为实现上述目的,本申请还提供计算机可读存储介质,其上存储有计算机可读指令,所述计算机可读指令被处理器执行时实现以下步骤:
接收用户上传的包含一个或多个舞者的舞蹈视频,获取所述舞蹈视频中的视频帧;
基于骨骼节点识别算法识别所述视频帧中包含的与目标舞者对应的骨骼节点图像;其中,所述目标舞者选自所述一个或多个舞者;
对所述舞蹈视频中包含的与每个所述目标舞者对应的全部所述骨骼节点图像进行聚类分析,以得到多个聚类集合;
获取每个所述聚类集合中作为聚类中心的关键骨骼节点图像;
按顺序输出所述关键骨骼节点图像,以得到所述舞蹈视频中与每个所述目标舞者对应的标准动作序列。
本申请提供的舞蹈视频的动作提取方法、装置、计算机设备和计算机可读存储介质,可以根据用户上传的舞蹈视频快速提取出标准动作并输出到客户端,以供游戏玩家根据标准动作执行对应的体感操作,实现人机交互或者多用户之间的网络交互。具体的,本申请基于骨骼节点识别算法从舞蹈视频的每一帧中提取骨骼节点图像,可以准确识别出舞者的不同姿态。基于聚类算法从全部骨骼节点图像选择关键骨骼节点图像,既可以保证标准动作的完整性,又能够避免动作过于密集而增加游戏难度或影响评分,保证用户得到良好的游戏体验。
图1为本申请的舞蹈视频的动作提取方法实施例一的流程图;
图2为本申请实施例一中提取骨骼节点图像的示意性流程图;
图3为本申请的舞蹈视频的动作提取方法实施例一中25骨骼节点模型的示意图;
图4为本申请实施例一中对骨骼节点图像进行聚类分析的示意性流程图;
图5为本申请实施例一中计算欧氏距离的示意性流程图;
图6为本申请实施例一中确定新的目标骨骼节点图像的示意性流程图;
图7为本申请实施例一中选择聚类中心作为关键帧的示意图;
图8为本申请实施例一中对重复的关键骨骼节点图像进行精简处理的示意性流程图;
图9为本申请的舞蹈视频的动作提取装置实施例一的程序模块示意图;
图10为本申请的舞蹈视频的动作提取装置实施例一的硬件结构示意图。
为了使本申请的目的、技术方案及优点更加清楚明白,以下结合附图及实施例,对本申请进行进一步详细说明。应当理解,此处所描述的具体实施例仅用以解释本申请,并不用于限定本申请。基于本申请中的实施例,本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例,都属于本申请保护的范围。
本申请提供的舞蹈视频的动作提取方法、装置、计算机设备和计算机可读存储介质,可以根据用户上传的舞蹈视频快速提取出标准动作并输出到客户端,以供游戏玩家根据标准动作执行对应的体感操作,实现人机交互或者多用户之间的网络交互。具体的,本申请基于骨骼节点识别算法从舞蹈视频的每一帧中提取骨骼节点图像,可以准确识别出舞者的不同姿态。基于聚类算法从全部骨骼节点图像选择关键骨骼节点图像,既可以保证标准动作的完整性,又能够避免动作过于密集而增加游戏难度或影响评分,保证用户得到良好的游戏体验。
实施例一
请参阅图1,本实施例提出一种舞蹈视频的动作提取方法,包括以下步骤:
S100:接收用户上传的包含一个或多个舞者的舞蹈视频,获取所述舞蹈视频中的视频帧。
本实施例所述舞蹈视频指的是清晰、完整地体现舞者完成一组舞蹈动作地视频。为了降低识别难度,本实施例中的舞蹈视频中优选包含一位或者两位舞者。
可以理解,舞蹈视频是由多个视频帧组成的。例如一段视频的时长60秒,对应帧率为每秒20帧,那么该段视频总共包含1200帧。本步骤在接收舞蹈视频的基础上,将舞蹈视频转化为全部的视频帧。
S200:基于骨骼节点识别算法识别所述视频帧中包含的与目标舞者对应的骨骼节点图像,其中,所述目标舞者选自所述一个或多个舞者。
本实施例中的目标舞者指的是作为舞蹈动作识别对象的舞者。目标舞者可以包含一个或者多个。当目标舞者包含一个时,可以将特定位置(例如前侧或者左侧)的舞者作为目标舞者,或者是将用户指定的一个舞者作为目标舞者。当目标舞者包含多个时,可以将舞蹈视频中的每个舞者都作为目标舞者,也可以将其中几个特定位置处的舞者作为目标舞者,本申请对此不做限制。
可以理解,舞蹈动作反映的是不同时刻舞者不同的身体姿态,其中身体姿态可以通过骨骼节点的位置来确定。本实施例可以利用现有的任何骨骼节点识别算法来识别每个视频帧中的骨骼节点,例如利用15骨骼节点模型、18骨骼节点模型或25骨骼节点模型中的任一种来识别骨骼节点。上文中的数字15、18、25分别表示骨骼节点的个数。骨骼节点的个 数越多,识别精度越高,需要占用的计算资源也就越多。具体实施时,可以根据要处理的舞蹈视频数量以及服务器配置情况而灵活选择不同骨骼节点模型。
需要说明的是,本步骤中提取的骨骼节点图像是针对同一个舞者而言的。对于视频中包含多个舞者的情况,不同舞者的动作节奏或多或少存在差异,儿童一个舞者的节奏往往是前后一致的。为了保证舞蹈动作的准确性,本实施例针对同一个舞者提取一组骨骼节点图像,每组骨骼节点图像用于生成一个标准动作序列。可以理解,当目标舞者包含多个时,本申请对每一个目标舞者都提取一组对应的骨骼节点图像,从而生成与每个目标舞者对应的标准动作序列。这样,不同的玩家可以根据喜好按照不同目标舞者的标准动作序列做出相应的舞蹈动作,从而提升游戏体验。
S300:对所述舞蹈视频中包含的与每个所述目标舞者对应的全部所述骨骼节点图像进行聚类分析,以得到多个聚类集合。
本步骤进行聚类分析的目的,是根据不同骨骼节点的位置关系,将所有骨骼节点图像划分为多个相近的集合,这样每个集合的聚类中心可以代表一个标准动作的关键帧。可以通过现有的任意聚类算法进行聚类分析,例如K-Means聚类算法、均值漂移聚类算法、基于密度的聚类算法、高斯混合模型聚类算法、凝聚层次聚类算法等,本实施例对此不做限制。
S400:获取每个所述聚类集合中作为聚类中心的关键骨骼节点图像。
需要说明的是,本实施例的目的是从复杂的舞蹈视频中提取舞者的每个关键动作并顺序输出,其中每个关键动作是通过对应的骨骼节点图像代表的。因此,本步骤在获得多个聚类集合的基础上,从每个聚类集合中选择其中一张骨骼节点图像作为输出标准动作时的关键动作。可以通过多种方法从每个聚类集合中选择合适的骨骼节点图像,例如选择每个聚类集合中固定位置处的骨骼节点图像,或者是作为聚类中心的骨骼节点图像等。每个聚类集合中被选择用来输出关键动作的骨骼节点图像即为关键骨骼节点图像。
S500:按顺序输出所述关键骨骼节点图像,以得到所述舞蹈视频中与每个所述目标舞者对应的标准动作序列。
如前所述,每一张骨骼节点图像表示当前时刻身体各个骨骼节点的位置。因此,按顺序输出确定的关键骨骼节点图像,就可以获得不同时刻下不同骨骼节点的运动位置,即与舞蹈视频相对应的标准动作序列。
综上所述,本实施例利用计算机视觉领域中较为常见的骨架识别技术识别出舞蹈视频中每一帧的人物骨骼节点图像,并通过对所有的骨骼节点图像进行聚类分析,快速确定舞蹈视频中的关键帧。本实施例可以将用户上传的舞蹈视频自动转换为游戏标准动作,不仅节省了巨大的人力支出,还大大提升转换效率,从而可以快速地扩充舞蹈游戏中的歌曲和关卡内容,提升用户体验。
图2示出了步骤S200中基于骨骼节点识别算法提取所述视频帧中包含的与所述目标舞者对应的骨骼节点图像的示意性流程图。如图2所示,步骤S200包括:
S210:确定骨骼节点模型,所述骨骼节点模型包含15骨骼节点模型、18骨骼节点模型或25骨骼节点模型中的任一种。
图3示出了25骨骼节点模型的示意图,如图2所示,这里的25个骨骼节点分别包括鼻节点、颈节点、左肩节点、右肩节点、左手肘节点、右手肘节点、左手节点、右手节点、左眼节点、右眼节点、左耳节点、右耳节点、下髋部中部节点、下髋部左侧节点、下髋部 右侧节点、左膝盖节点、右膝盖节点、左脚踝节点、右脚踝节点、左大脚趾节点、右大脚趾节点、左小脚趾节点、右小脚趾节点、左脚后跟节点和右脚后跟节点。
S220:根据确定的所述骨骼节点模型从所述视频帧中提取与所述目标舞者对应的骨骼节点图像,所述骨骼节点图像包括多个骨骼节点数据,每个所述骨骼节点数据包括骨骼节点类型、骨骼节点横坐标和骨骼节点纵坐标。
仍以图3示出的25骨骼节点模型为例。从视频帧中提取骨骼节点图像,就是获取视频帧中包含的与上述25个骨骼节点相对应的特定舞者的骨骼节点数据。具体的,每个骨骼节点数据可以包括骨骼节点类型、骨骼节点横坐标和骨骼节点纵坐标。其中,骨骼节点类型表示具体哪个节点,例如左肩节点、右肩节点、左手节点等。横坐标和纵坐标则表示骨骼节点在骨骼节点图像中的位置数据。这样,通过每个骨骼节点的节点类型、节点横坐标和节点纵坐标,就可以确定25个骨骼节点各自的位置,从而通过骨骼节点位置确定舞者动作。
S230:判断所述骨骼节点图像中包含的骨骼节点的个数是否在预设范围之内,若否,删除所述骨骼节点图像。
可以理解,在视频的片头片尾部分或者由于遮挡、拍摄角度等原因,可能会存在视频帧中不包含舞者或者舞者部分身体被遮挡的情况,此时无法获得完整的25个骨骼节点数据,可能只能获得少于25个骨骼节点数据甚至是0个骨骼节点数据。因此为了保证输出动作的清楚完整性,本步骤会将骨骼节点个数不在预设范围之内的骨骼节点图像删除,这里的预设节点范围可以根据骨骼节点模型所能识别的骨骼节点最大个数进行设置,例如骨骼节点最大个数为25,那么可以将预设节点范围设置为[18,25]。
通过上述步骤,本实施例可以获取到准确、完整的骨骼节点图像,为后续输出标准动作提供良好基础。
图4为本申请实施例一中对骨骼节点图像进行聚类分析的示意性流程图。如图4所示,步骤S300包括:
S310:根据所述舞蹈视频的节奏确定所述集合的总数K,K为自然数。
本步骤确定聚类集合总数的目的是为了确定最终输出标准动作时作为关键帧的关键骨骼节点图像的个数。换句话说,一个聚类集合最终对应一个关键骨骼节点图像。
可以理解,如果输出的标准动作中包含的关键帧数量过多,在游戏过程中就会显得十分密集,对于一般玩家来说可能难以及时完成对应的舞蹈动作。相反,如果关键帧数量过少,那么玩家就会经常处于”空闲“的状态,并且本实施例难以在不同的玩家表现之间拉开差距,这些都对于良好的游戏体验是不利的。经过对于多个舞蹈游戏中歌曲的动作序列进行分析,本实施例优选平均1.5秒输出一个关键帧是较为合适的抽取频率。
根据上述抽取频率可以确定关键帧的总数K,也即聚类集合的总数K。例如一段视频的时长60秒,那么K=60÷1.5=40。另外,考虑到不同的舞蹈歌曲节奏各不相同,尤其是对于快歌和慢歌而言,关键的舞蹈动作出现的频率差距会比较大。对此,本实施例还可以根据不同的节奏自动调整抽取频率,或者是允许用户对于关键帧数量进行主动干预。例如用户可以在平均0.5秒到3秒一个关键帧的区间内进行选择。这样一来,对于部分特殊节奏的舞蹈歌曲,关键帧提取数量也可以进行相应的调整以适配视频和歌曲的节奏。
S320:从全部所述骨骼节点图像中任选K个目标骨骼节点图像,计算每个剩余骨骼节点图像与所述目标骨骼节点图像之间的相似度。
假设全部骨骼节点图像的数量共100幅,每幅骨骼节点图像可以分别表示为F1, F2,……,F100,K=10。本步骤首先从中任意选择10幅作为目标骨骼节点图像,假设为F1,F2,……,F10,此时剩余骨骼节点图像包括F11,F12,……,F100。当然,为了避免选择的目标骨骼节点图像过于集中,还可以根据特定的时间间隔选择目标骨骼节点图像,例如每隔一秒抽取一幅目标骨骼节点图像。
在一个示例中,剩余骨骼节点图像与目标骨骼节点图像之间的相似度可以包含剩余骨骼节点图像中的骨骼节点与目标骨骼节点图像中的骨骼节点之间的欧氏距离。在该示例中,计算相似度的具体步骤如图5所示,包括:
S321:计算所述剩余骨骼节点图像中的每个骨骼节点与所述目标骨骼节点图像中对应骨骼节点之间的欧氏距离;
S322:将所有骨骼节点的欧式距离之和作为所述剩余骨骼节点图像与所述目标骨骼节点图像之间的欧氏距离。
计算每个剩余骨骼节点图像与所述目标骨骼节点图像之间的欧氏距离,也就是分别计算F11,F12,……,F100与F1,F2,……,F10的欧式距离。假设FA表示剩余骨骼节点图像F11,F12,……,F100中的任一幅,FB表示目标骨骼节点图像F1,F2,……,F10中的任一幅。其中FA中包含的n个骨骼节点可以表示为A1、A2、……、An,FB中包含的n个骨骼节点可以表示为B1、B2、……、B25,n可以为15、18或25,那么F11与F1的欧氏距离Similarity(F11,F1)可以表示为:
Similarity(FA,FB)=d(B1,A1)+d(B2,A2)+….d(Bm,Am)+….+d(Bn,An)
同理,F11与F2的欧氏距离Similarity(F11,F2)、F11与F3的欧氏距离Similarity(F11,F3)、……、F11与F10的欧氏距离Similarity(F11,F10)也分别按照上面的公式进行计算。
S330:将所述剩余骨骼节点图像与对应的欧氏距离最小的目标骨骼节点图像归为同一个集合,以得到K个候选集合。
如上例所示,在得到F11与每个目标骨骼节点图像的欧氏距离的基础上,从中选出欧氏距离最小的数值,并将F11与最小欧氏距离对应的骨骼节点图像归为同一个聚类集合。例如经过计算确定F11与F5之间的欧氏距离最小,那么就将F11归为F5所在的聚类集合。
S340:根据每个所述候选集合中包含的骨骼节点图像的识别编号确定新的目标骨骼节点图像,重复执行步骤S320和步骤S330;所述识别编号用于唯一表征所述骨骼节点图像的出现顺序。
如步骤S320所述,最初的目标骨骼节点图像是任意选择的,因此这些最初的目标骨骼节点图像很可能并不适合作为理想的分类参照,因此需要不断确定新的目标骨骼节点图像,从而获得最佳目标。图6为本申请实施例一中确定新的目标骨骼节点图像的示意性流程图。如图6所示,步骤S340包括:
S341:获取每个聚类集合中包含的每个骨骼节点图像的识别编号。
本实施例中每个骨骼节点图像对应唯一一个识别编号,用于表征骨骼节点图像的出现顺序。如前所述,骨骼节点图像是从对应的视频帧中提取出来的,每个视频帧在舞蹈视频中具有唯一的顺序编号,因此可以用对应视频帧的顺序编号作为本实施例中骨骼节点图像 的顺序编号。例如上文中提到的骨骼节点图像F1,F2,……,F100,这里的数字1到100就是每个骨骼节点图像对应的识别编号。例如,已知一个候选集合V中的骨骼节点图像包括F1,F10,F20和F30,这里的识别编号为:1、10、20和30。
S342:对每个数据集合中包含的所有骨骼节点图像的识别编号求平均并取整,得到目标识别编号。
以上文中的集合V为例,已知识别编号包括1、10、20和30。对识别编号取平均为(1+10+20+30)÷4,无法整除的,按照四舍五入法取整,可以得到集合V对应的目标识别编号为15。
S343:将与所述目标识别编号对应的骨骼节点图像作为所述新的目标骨骼节点图像。
在确定目标识别编号为15的基础上,对应的骨骼节点图像就是F15,那么F15就是新的目标骨骼节点图像。由于本实施例中共有K个候选集合,相应地可以获得K个新的目标骨骼节点图像。
S350:在所述新的目标骨骼节点图像的识别编号不再变化或者达到预设重复次数的情况下,将得到的K个候选集合作为所述聚类集合。
本步骤规定了聚类分析的迭代终止条件。可以理解,以不同的目标骨骼节点图像作为聚类中心,就会得到不同的聚类集合。图7示出了本申请实施例一中选择聚类中心作为关键帧的示意图。在图7中,中间帧集合1、中间帧集合2和中间帧集合3代表3个聚类集合,关键帧1、关键帧2和关键帧3分别代表每个聚类集合的聚类中心。理想情况下,多次迭代之后目标骨骼节点图像将不再变化,由此获得分类比较稳定准确的聚类集合。当然,在实际操作过程中如果数据量较多,可能很难达到目标骨骼节点图像不发生变化的情况。此时可以通过设置有限的重复次数来终止迭代,例如100次。其中,每当确定K个新的目标骨骼节点图像称为完成一次重复。
通过上述步骤,可以保证通过本实施例得到的K个候选集合具有较高的准确度,能够把比较相近的动作划分到相同集合,把区别较大的动作划分到不同集合。
如前所述,步骤S400用于获取每个所述聚类集合中作为聚类中心的关键骨骼节点图像。在实际操作的过程中,时间间隔较短的相邻关键骨骼节点图像之间可能会存在重复,这种情况往往是由于动作变化节奏较慢而造成的。为了避免给用户造成干扰,在一个示例中对重复的关键骨骼节点图像进行精简处理。如图8所示,具体包括以下步骤:
S410:计算任意相邻两个关键骨骼节点图像的识别编号之间的差值。
如前所述,骨骼节点图像的识别编号n与对应视频帧的顺序编号N相对应,而视频帧的顺序编号N可以表征该视频帧在舞蹈视频中的出现次序。通常,识别编号n或者顺序编号N是根据自然数的顺序依次排列的,同时舞蹈视频具有固定的帧率f,即每秒播放几个视频帧,这样,识别编号n除以帧率f就是一个骨骼节点图像在输出的标准动作视频中的出现时间。例如帧率为20帧/秒,某骨骼节点图像为F50,即识别编号n=50,那么相当于该骨骼节点图像F50出现在50÷20=2.5秒。
为了保证输出的标准动作的变化在正常人反应范围之内,那么相邻两个关键骨骼节点图像之间的时间间隔不能过短,否则一方面用户来不及反应,另一方面动作的变化幅度也不够明显。假设相邻两个关键骨骼节点图像之间的最短间隔为I,由于不同视频的帧率不同,该间隔I可以通过舞蹈视频的总时长进行限制。通常成年人对于两幅图像之间的变化的最短反应时间t为0.2秒,若一段舞蹈视频的帧率为f,那么上述最短间隔I为:
I=f*t
假设t=0.2秒,f为每秒25帧,那么最短间隔即为5帧。此时数字5为相邻两个关键骨骼节点图像的识别编号之间的差值。
S420:当所述差值小于第一阈值时,将所述识别编号靠后的关键骨骼节点图像删除。
本示例中,可以对于K均值聚类算法中所得到的关键骨骼节点图像进行逐一遍历,当前后两个关键骨骼节点图像的识别编号的差值小于预设差值(例如5)时,则将识别编号靠后的关键骨骼节点图像从结果集中删除。重复以上操作,直到所有相邻关键骨骼节点图像的间隔都符合上述条件。通过这样的方式,可以有效地去除关键骨骼节点图像中的冗余内容,使得舞蹈动作的提取更具代表性。
请继续参阅图9,示出了一种舞蹈视频的动作提取装置,在本实施例中,动作提取装置90可以包括或被分割成一个或多个程序模块,一个或者多个程序模块被存储于存储介质中,并由一个或多个处理器所执行,以完成本申请,并可实现上述动作提取方法。本申请所称的程序模块是指能够完成特定功能的一系列计算机可读指令段,比程序本身更适合于描述动作提取装置90在存储介质中的执行过程。以下描述将具体介绍本实施例各程序模块的功能:
视频接收模块91,适用于接收用户上传的包含目标舞者的舞蹈视频,获取所述舞蹈视频中的视频帧;
骨骼节点识别模块92,适用于基于骨骼节点识别算法识别所述视频帧中包含的与所述目标舞者对应的骨骼节点图像;
聚类模块93,适用于对所述舞蹈视频中包含的全部所述骨骼节点图像进行聚类分析,以得到多个聚类集合;
关键图像获取模块94,适用于获取每个所述聚类集合中作为聚类中心的关键骨骼节点图像;
动作输出模块95,适用于按顺序输出所述关键骨骼节点图像,以得到所述舞蹈视频对应的标准动作。
本实施例还提供一种计算机设备,如可以执行程序的智能手机、平板电脑、笔记本电脑、台式计算机、机架式服务器、刀片式服务器、塔式服务器或机柜式服务器(包括独立的服务器,或者多个服务器所组成的服务器集群)等。本实施例的计算机设备100至少包括但不限于:可通过系统总线相互通信连接的存储器101、处理器102,如图10所示。需要指出的是,图10仅示出了具有组件101-102的计算机设备100,但是应理解的是,并不要求实施所有示出的组件,可以替代的实施更多或者更少的组件。
本实施例中,存储器101(即可读存储介质)包括闪存、硬盘、多媒体卡、卡型存储器(例如,SD或DX存储器等)、随机访问存储器(RAM)、静态随机访问存储器(SRAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、可编程只读存储器(PROM)、磁性存储器、磁盘、光盘等。在一些实施例中,存储器101可以是计算机设备100的内部存储单元,例如该计算机设备100的硬盘或内存。在另一些实施例中,存储器101也可以是计算机设备100的外部存储设备,例如该计算机设备100上配备的插接式硬盘,智能存储卡(Smart Media Card,SMC),安全数字(Secure Digital,SD)卡,闪存卡(Flash Card) 等。当然,存储器101还可以既包括计算机设备100的内部存储单元也包括其外部存储设备。本实施例中,存储器101通常用于存储安装于计算机设备100的操作系统和各类应用软件,例如实施例一的动作提取装置90的程序代码等。此外,存储器101还可以用于暂时地存储已经输出或者将要输出的各类数据。
处理器102在一些实施例中可以是中央处理器(Central Processing Unit,CPU)、控制器、微控制器、微处理器、或其他数据处理芯片。该处理器102通常用于控制计算机设备100的总体操作。本实施例中,处理器102用于运行存储器101中存储的程序代码或者处理数据,例如运行动作提取装置90,以实现实施例一的动作提取方法。
本实施例还提供一种计算机可读存储介质,如闪存、硬盘、多媒体卡、卡型存储器(例如,SD或DX存储器等)、随机访问存储器(RAM)、静态随机访问存储器(SRAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、可编程只读存储器(PROM)、磁性存储器、磁盘、光盘、服务器、App应用商城等等,其上存储有计算机可读指令,程序被处理器执行时实现相应功能。本实施例的计算机可读存储介质用于存储动作提取装置90,被处理器执行时实现以下步骤:
接收用户上传的包含一个或多个舞者的舞蹈视频,获取所述舞蹈视频中的视频帧;
基于骨骼节点识别算法识别所述视频帧中包含的与目标舞者对应的骨骼节点图像;其中,所述目标舞者选自所述一个或多个舞者;
对所述舞蹈视频中包含的与每个所述目标舞者对应的全部所述骨骼节点图像进行聚类分析,以得到多个聚类集合;
获取每个所述聚类集合中作为聚类中心的关键骨骼节点图像;
按顺序输出所述关键骨骼节点图像,以得到所述舞蹈视频中与每个所述目标舞者对应的标准动作序列。
上述本申请实施例序号仅仅为了描述,不代表实施例的优劣。
流程图中或在此以其它方式描述的任何过程或方法描述可以被理解为,表示包括一个或更多个用于实现特定逻辑功能或过程的步骤的可执行指令的代码的模块、片段或部分,并且本申请的优选实施方式的范围包括另外的实现,其中可以不按所示出或讨论的顺序,包括根据所涉及的功能按基本同时的方式或按相反的顺序,来执行功能,这应被本申请的实施例所属技术领域的技术人员所理解。
本技术领域的普通技术人员可以理解,实现上述实施例方法携带的全部或部分步骤是可以通过程序来指令相关的硬件完成,所述的程序可以存储于一种计算机可读介质中,该程序在执行时,包括方法实施例的步骤之一或其组合。
在本说明书的描述中,参考术语“一个实施例”、“一些实施例”、“示例”、“具体示例”或“一些示例”等的描述意指结合该实施例或示例描述的具体特征、结构、材料或者特点包含于本申请的至少一个实施例或示例中。在本说明书中,对上述术语的示意性表述不一定指的是相同的实施例或示例。而且,描述的具体特征、结构、材料或者特点可以在任何的一个或多个实施例或示例中以合适的方式结合。
通过以上的实施方式的描述,本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现,当然也可以通过硬件,但很多情况下前者 是更佳的实施方式。
以上仅为本申请的优选实施例,并非因此限制本申请的专利范围,凡是利用本申请说明书及附图内容所作的等效结构或等效流程变换,或直接或间接运用在其他相关的技术领域,均同理包括在本申请的专利保护范围内。
Claims (20)
- 一种舞蹈视频的动作提取方法,包括以下步骤:接收用户上传的包含一个或多个舞者的舞蹈视频,获取所述舞蹈视频中的视频帧;基于骨骼节点识别算法识别所述视频帧中包含的与目标舞者对应的骨骼节点图像;其中,所述目标舞者选自所述一个或多个舞者;对所述舞蹈视频中包含的与每个所述目标舞者对应的全部所述骨骼节点图像进行聚类分析,以得到多个聚类集合;获取每个所述聚类集合中作为聚类中心的关键骨骼节点图像;按顺序输出所述关键骨骼节点图像,以得到所述舞蹈视频中与每个所述目标舞者对应的标准动作序列。
- 根据权利要求1所述的舞蹈视频的动作提取方法,所述舞蹈视频中包含多个舞者,所述目标舞者选自所述多个舞者中的一个。
- 根据权利要求1或2所述的动作提取方法,所述基于骨骼节点识别算法提取所述视频帧中包含的与目标舞者对应的骨骼节点图像的步骤包括:确定骨骼节点模型;根据确定的所述骨骼节点模型从所述视频帧中提取与所述目标舞者对应的骨骼节点图像,所述骨骼节点图像包括多个骨骼节点数据。
- 根据权利要求3所述的动作提取方法,所述基于骨骼节点识别算法提取所述视频帧中包含的与目标舞者对应的骨骼节点图像的步骤还包括:判断所述骨骼节点图像中包含的骨骼节点的个数是否在预设范围之内,若否,删除所述骨骼节点图像。
- 根据权利要求3或4所述的动作提取方法,所述对所述舞蹈视频中包含的与每个所述目标舞者对应的全部所述骨骼节点图像进行聚类分析,以得到多个聚类集合的步骤包括:从全部所述骨骼节点图像中任选K个目标骨骼节点图像,计算每个剩余骨骼节点图像与所述目标骨骼节点图像之间的相似度;将所述剩余骨骼节点图像与对应的相似度最小的目标骨骼节点图像归为同一个集合,以得到K个候选集合;根据每个所述候选集合中包含的骨骼节点图像的识别编号确定新的目标骨骼节点图像,重复执行根据K个所述新的目标骨骼节点图像计算所述相似度以及所述得到K个候选集合的步骤;所述识别编号用于唯一表征所述骨骼节点图像的出现顺序;在所述新的目标骨骼节点图像的识别编号不再变化或者达到预设重复次数的情况下,将得到的K个候选集合作为所述聚类集合。
- 根据权利要求5所述的动作提取方法,所述K个目标骨骼节点图像是根据所述舞蹈视频的节奏确定的。
- 根据权利要求5或6所述的动作提取方法,所述相似度包括欧氏距离,所述计算每个剩余骨骼节点图像与所述目标骨骼节点图像之间的相似度的步骤包括:计算所述剩余骨骼节点图像中的每个骨骼节点与所述目标骨骼节点图像中对应骨骼节点之间的欧氏距离;将所有骨骼节点的欧式距离之和作为所述剩余骨骼节点图像与所述目标骨骼节点图像 之间的欧氏距离。
- 根据权利要求5至7任意一项所述的动作提取方法,所述根据每个所述候选集合中包含的骨骼节点图像的识别编号确定新的目标骨骼节点图像的步骤包括:获取每个聚类集合中包含的每个骨骼节点图像的识别编号;对每个数据集合中包含的所有骨骼节点图像的识别编号求平均并取整,得到目标识别编号;将与所述目标识别编号对应的骨骼节点图像作为所述新的目标骨骼节点图像。
- 根据权利要求1至8任意一项所述的动作提取方法,所述提取每个所述聚类集合中作为聚类中心的关键骨骼节点图像的步骤包括:计算任意相邻两个关键骨骼节点图像的识别编号之间的差值;当所述差值小于第一阈值时,将所述识别编号靠后的关键骨骼节点图像删除。
- 一种舞蹈视频的动作提取装置,包括:视频接收模块,适用于接收用户上传的包含一个或多个舞者的舞蹈视频,获取所述舞蹈视频中的视频帧;骨骼节点识别模块,适用于基于骨骼节点识别算法识别所述视频帧中包含的与目标舞者对应的骨骼节点图像;其中,所述目标舞者选自所述一个或多个舞者;聚类模块,适用于对所述舞蹈视频中包含的与每个所述目标舞者对应的全部所述骨骼节点图像进行聚类分析,以得到多个聚类集合;关键图像获取模块,适用于获取每个所述聚类集合中作为聚类中心的关键骨骼节点图像;动作输出模块,适用于按顺序输出所述关键骨骼节点图像,以得到所述舞蹈视频中与每个所述目标舞者对应的标准动作序列。
- 一种计算机设备,包括存储器、处理器以及存储在存储器上并可在处理器上运行的计算机可读指令,所述处理器执行所述计算机可读指令时实现以下步骤:接收用户上传的包含一个或多个舞者的舞蹈视频,获取所述舞蹈视频中的视频帧;基于骨骼节点识别算法识别所述视频帧中包含的与目标舞者对应的骨骼节点图像;其中,所述目标舞者选自所述一个或多个舞者;对所述舞蹈视频中包含的与每个所述目标舞者对应的全部所述骨骼节点图像进行聚类分析,以得到多个聚类集合;获取每个所述聚类集合中作为聚类中心的关键骨骼节点图像;按顺序输出所述关键骨骼节点图像,以得到所述舞蹈视频中与每个所述目标舞者对应的标准动作序列。
- 根据权利要求11所述的舞蹈视频的计算机设备,所述舞蹈视频中包含多个舞者,所述目标舞者选自所述多个舞者中的一个。
- 根据权利要求11或12所述的计算机设备,所述基于骨骼节点识别算法提取所述视频帧中包含的与目标舞者对应的骨骼节点图像的步骤包括:确定骨骼节点模型;根据确定的所述骨骼节点模型从所述视频帧中提取与所述目标舞者对应的骨骼节点图像,所述骨骼节点图像包括多个骨骼节点数据。
- 根据权利要求13所述的计算机设备,所述基于骨骼节点识别算法提取所述视频帧 中包含的与目标舞者对应的骨骼节点图像的步骤还包括:判断所述骨骼节点图像中包含的骨骼节点的个数是否在预设范围之内,若否,删除所述骨骼节点图像。
- 根据权利要求13或14所述的计算机设备,所述对所述舞蹈视频中包含的与每个所述目标舞者对应的全部所述骨骼节点图像进行聚类分析,以得到多个聚类集合的步骤包括:从全部所述骨骼节点图像中任选K个目标骨骼节点图像,计算每个剩余骨骼节点图像与所述目标骨骼节点图像之间的相似度;将所述剩余骨骼节点图像与对应的相似度最小的目标骨骼节点图像归为同一个集合,以得到K个候选集合;根据每个所述候选集合中包含的骨骼节点图像的识别编号确定新的目标骨骼节点图像,重复执行根据K个所述新的目标骨骼节点图像计算所述相似度以及所述得到K个候选集合的步骤;所述识别编号用于唯一表征所述骨骼节点图像的出现顺序;在所述新的目标骨骼节点图像的识别编号不再变化或者达到预设重复次数的情况下,将得到的K个候选集合作为所述聚类集合。
- 根据权利要求15所述的计算机设备,所述K个目标骨骼节点图像是根据所述舞蹈视频的节奏确定的。
- 根据权利要求15或16所述的计算机设备,所述相似度包括欧氏距离,所述计算每个剩余骨骼节点图像与所述目标骨骼节点图像之间的相似度的步骤包括:计算所述剩余骨骼节点图像中的每个骨骼节点与所述目标骨骼节点图像中对应骨骼节点之间的欧氏距离;将所有骨骼节点的欧式距离之和作为所述剩余骨骼节点图像与所述目标骨骼节点图像之间的欧氏距离。
- 根据权利要求15至17任意一项所述的计算机设备,所述根据每个所述候选集合中包含的骨骼节点图像的识别编号确定新的目标骨骼节点图像的步骤包括:获取每个聚类集合中包含的每个骨骼节点图像的识别编号;对每个数据集合中包含的所有骨骼节点图像的识别编号求平均并取整,得到目标识别编号;将与所述目标识别编号对应的骨骼节点图像作为所述新的目标骨骼节点图像。
- 根据权利要求11至18任意一项所述的计算机设备,所述提取每个所述聚类集合中作为聚类中心的关键骨骼节点图像的步骤包括:计算任意相邻两个关键骨骼节点图像的识别编号之间的差值;当所述差值小于第一阈值时,将所述识别编号靠后的关键骨骼节点图像删除。
- 一种计算机可读存储介质,其上存储有计算机可读指令,所述计算机可读指令被处理器执行时实现权利要求1至9任一项所述方法的步骤。
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US18/018,879 US20230306787A1 (en) | 2020-08-06 | 2021-06-22 | Movement extraction method and apparatus for dance video, computer device, and storage medium |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010784431.7 | 2020-08-06 | ||
CN202010784431.7A CN114093021A (zh) | 2020-08-06 | 2020-08-06 | 舞蹈视频的动作提取方法、装置、计算机设备及存储介质 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2022028136A1 true WO2022028136A1 (zh) | 2022-02-10 |
Family
ID=80116858
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2021/101384 WO2022028136A1 (zh) | 2020-08-06 | 2021-06-22 | 舞蹈视频的动作提取方法、装置、计算机设备及存储介质 |
Country Status (3)
Country | Link |
---|---|
US (1) | US20230306787A1 (zh) |
CN (1) | CN114093021A (zh) |
WO (1) | WO2022028136A1 (zh) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114885210A (zh) * | 2022-04-22 | 2022-08-09 | 海信集团控股股份有限公司 | 教程视频处理方法、服务器及显示设备 |
CN115035221A (zh) * | 2022-06-17 | 2022-09-09 | 广州虎牙科技有限公司 | 一种舞蹈动画合成方法、装置、设备及存储介质 |
CN115273222A (zh) * | 2022-06-23 | 2022-11-01 | 武汉元淳传媒有限公司 | 一种基于人工智能的多媒体互动分析控制管理系统 |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116980717B (zh) * | 2023-09-22 | 2024-01-23 | 北京小糖科技有限责任公司 | 基于视频分解处理的交互方法、装置、设备及存储介质 |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2016019973A1 (en) * | 2014-08-04 | 2016-02-11 | Bitmonlab S.L.U | Method for determining stationary crowds |
CN109508656A (zh) * | 2018-10-29 | 2019-03-22 | 重庆中科云丛科技有限公司 | 一种舞蹈评级自动判别方法、系统及计算机可读存储介质 |
CN110096950A (zh) * | 2019-03-20 | 2019-08-06 | 西北大学 | 一种基于关键帧的多特征融合行为识别方法 |
CN110448870A (zh) * | 2019-08-16 | 2019-11-15 | 深圳特蓝图科技有限公司 | 一种人体姿态训练方法 |
CN110728220A (zh) * | 2019-09-30 | 2020-01-24 | 上海大学 | 基于人体动作骨骼信息的体操辅助训练方法 |
CN111144217A (zh) * | 2019-11-28 | 2020-05-12 | 重庆邮电大学 | 一种基于人体三维关节点检测的动作评价方法 |
Family Cites Families (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109308438B (zh) * | 2017-07-28 | 2020-11-27 | 上海形趣信息科技有限公司 | 动作识别库的建立方法、电子设备、存储介质 |
CN108665492B (zh) * | 2018-03-27 | 2020-09-18 | 北京光年无限科技有限公司 | 一种基于虚拟人的舞蹈教学数据处理方法及系统 |
US10643492B2 (en) * | 2018-06-20 | 2020-05-05 | NEX Team Inc. | Remote multiplayer interactive physical gaming with mobile computing devices |
CN109151501B (zh) * | 2018-10-09 | 2021-06-08 | 北京周同科技有限公司 | 一种视频关键帧提取方法、装置、终端设备及存储介质 |
CN110245638A (zh) * | 2019-06-20 | 2019-09-17 | 北京百度网讯科技有限公司 | 视频生成方法和装置 |
-
2020
- 2020-08-06 CN CN202010784431.7A patent/CN114093021A/zh active Pending
-
2021
- 2021-06-22 US US18/018,879 patent/US20230306787A1/en active Pending
- 2021-06-22 WO PCT/CN2021/101384 patent/WO2022028136A1/zh active Application Filing
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2016019973A1 (en) * | 2014-08-04 | 2016-02-11 | Bitmonlab S.L.U | Method for determining stationary crowds |
CN109508656A (zh) * | 2018-10-29 | 2019-03-22 | 重庆中科云丛科技有限公司 | 一种舞蹈评级自动判别方法、系统及计算机可读存储介质 |
CN110096950A (zh) * | 2019-03-20 | 2019-08-06 | 西北大学 | 一种基于关键帧的多特征融合行为识别方法 |
CN110448870A (zh) * | 2019-08-16 | 2019-11-15 | 深圳特蓝图科技有限公司 | 一种人体姿态训练方法 |
CN110728220A (zh) * | 2019-09-30 | 2020-01-24 | 上海大学 | 基于人体动作骨骼信息的体操辅助训练方法 |
CN111144217A (zh) * | 2019-11-28 | 2020-05-12 | 重庆邮电大学 | 一种基于人体三维关节点检测的动作评价方法 |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114885210A (zh) * | 2022-04-22 | 2022-08-09 | 海信集团控股股份有限公司 | 教程视频处理方法、服务器及显示设备 |
CN114885210B (zh) * | 2022-04-22 | 2023-11-28 | 海信集团控股股份有限公司 | 教程视频处理方法、服务器及显示设备 |
CN115035221A (zh) * | 2022-06-17 | 2022-09-09 | 广州虎牙科技有限公司 | 一种舞蹈动画合成方法、装置、设备及存储介质 |
CN115273222A (zh) * | 2022-06-23 | 2022-11-01 | 武汉元淳传媒有限公司 | 一种基于人工智能的多媒体互动分析控制管理系统 |
CN115273222B (zh) * | 2022-06-23 | 2024-01-26 | 广东园众教育信息化服务有限公司 | 一种基于人工智能的多媒体互动分析控制管理系统 |
Also Published As
Publication number | Publication date |
---|---|
CN114093021A (zh) | 2022-02-25 |
US20230306787A1 (en) | 2023-09-28 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2022028136A1 (zh) | 舞蹈视频的动作提取方法、装置、计算机设备及存储介质 | |
CN110637323B (zh) | 基于部分的跟踪的方法、设备和系统 | |
CN109815826B (zh) | 人脸属性模型的生成方法及装置 | |
CN102693413B (zh) | 运动识别 | |
CN106780681B (zh) | 一种角色动作生成方法和装置 | |
CN108712661A (zh) | 一种直播视频处理方法、装置、设备及存储介质 | |
Raptis et al. | Real-time classification of dance gestures from skeleton animation | |
US20170266491A1 (en) | Method and system for authoring animated human movement examples with scored movements | |
US20230049135A1 (en) | Deep learning-based video editing method, related device, and storage medium | |
CN111179385B (zh) | 一种舞蹈动画的处理方法及装置、电子设备、存储介质 | |
CN109121021A (zh) | 一种视频集锦的生成方法、装置、电子设备及存储介质 | |
US8660306B2 (en) | Estimated pose correction | |
US20130278501A1 (en) | Systems and methods of identifying a gesture using gesture data compressed by principal joint variable analysis | |
WO2022028135A1 (zh) | 在线体感斗舞方法、装置、计算机设备及存储介质 | |
CN111429341B (zh) | 一种视频处理方法、设备及计算机可读存储介质 | |
CN113633983B (zh) | 虚拟角色表情控制的方法、装置、电子设备及介质 | |
CN110772794B (zh) | 智能游戏处理方法、装置、设备及存储介质 | |
Salge et al. | Generative design in Minecraft: Chronicle challenge | |
Xu et al. | Pllava: Parameter-free llava extension from images to videos for video dense captioning | |
CN112560872A (zh) | 一种基于人工智能的多模感知麻将辅助方法 | |
CN113112577A (zh) | 过渡帧预测模型的训练方法以及过渡帧预测方法 | |
Kavasidis et al. | Generation of ground truth for object detection while playing an online game: Productive gaming or recreational working? | |
US11704980B2 (en) | Method, apparatus, and computer storage medium for outputting virtual application object | |
US11895288B2 (en) | Information processing device, proposal device, information processing method, and proposal method | |
CN105224669B (zh) | 一种基于gmm语义特征的运动检索方法 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21854164 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 21854164 Country of ref document: EP Kind code of ref document: A1 |