WO2020082575A1 - 一种音乐生成方法及装置 - Google Patents

一种音乐生成方法及装置 Download PDF

Info

Publication number
WO2020082575A1
WO2020082575A1 PCT/CN2018/123552 CN2018123552W WO2020082575A1 WO 2020082575 A1 WO2020082575 A1 WO 2020082575A1 CN 2018123552 W CN2018123552 W CN 2018123552W WO 2020082575 A1 WO2020082575 A1 WO 2020082575A1
Authority
WO
WIPO (PCT)
Prior art keywords
video
preset
feature point
coordinates
feature points
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2018/123552
Other languages
English (en)
French (fr)
Inventor
王义文
刘奡智
王健宗
肖京
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Technology Shenzhen Co Ltd
Original Assignee
Ping An Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Technology Shenzhen Co Ltd filed Critical Ping An Technology Shenzhen Co Ltd
Publication of WO2020082575A1 publication Critical patent/WO2020082575A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/76—Television signal recording
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/49—Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H1/00—Details of electrophonic musical instruments
    • G10H1/0091—Means for obtaining special acoustic effects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream

Definitions

  • This application relates to the field of artificial intelligence technology, and in particular, to a music generation method and device.
  • the embodiments of the present application provide a music generation method and device to solve the problem of a single composition mode in the prior art.
  • a music generation method comprising: acquiring a video obtained by recording a human action; identifying a plurality of feature points in the video that match an identified object, and Multiple coordinates of each feature point are generated based on a preset coordinate system and video playback sequence, and the recognition object includes limbs and / or hands; the video is divided into multiple frame sequences according to the preset video segmentation principle; from each Among the multiple feature points of each frame sequence, one feature point with the largest coordinate change is selected as the target feature point; multiple coordinates of the target feature point selected from each frame sequence are obtained, and according to the preset music generation rules Generating multiple note codes corresponding to each frame sequence from the multiple coordinates; generating a melody according to the multiple note codes corresponding to each frame sequence and the video playback order.
  • a music generating device includes: an acquiring unit for acquiring a video obtained by recording human actions; an identifying unit for identifying and identifying the video Multiple feature points matching the object, and generating multiple coordinates of each feature point based on a preset coordinate system and video playback order, the recognition object includes a limb and / or a hand; a segmentation unit, which is used according to a preset video The segmentation principle divides the video into multiple frame sequences; a filtering unit is used to select one feature point with the largest coordinate change from the multiple feature points of each frame sequence as the target feature point; the first generating unit is used to Acquiring multiple coordinates of the target feature point selected from each frame sequence, and generating multiple note codes corresponding to each frame sequence from the multiple coordinates according to a preset music generation rule; a second generation unit, It is used to generate a melody according to multiple note codes corresponding to each frame sequence and the video playing order.
  • a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing The steps of the above-mentioned music generation method are realized when the computer program is described.
  • a computer non-volatile storage medium the storage medium includes a stored program, and when the program is running, the device where the storage medium is located is controlled to execute the above-mentioned music The steps of the generation method.
  • FIG. 1 is a flowchart of a music generation method according to an embodiment of the present application
  • FIG. 2 is a schematic diagram of a music generating device according to an embodiment of the present application.
  • FIG. 3 is a distribution diagram of limb recognition feature points according to an embodiment of the present application.
  • FIG. 6 is a schematic diagram of a computer device according to an embodiment of the present application.
  • first, second, third, etc. may be used to describe the terminals in the embodiments of the present application, these terminals should not be limited to these terms. These terms are only used to distinguish the terminals from each other.
  • first division subunit may also be called a second division subunit, and similarly, the second division subunit may also be called a first division subunit.
  • the word “if” as used herein may be interpreted as “when” or “when” or “in response to determination” or “in response to detection”.
  • the phrases “if determined” or “if detected (statement or event stated)” can be interpreted as “when determined” or “in response to determination” or “when detected (statement or event stated) ) “Or” in response to detection (statement or event stated) ".
  • FIG. 1 is a flowchart of a music generation method according to an embodiment of the present application. As shown in FIG. 1, the method includes:
  • Step S101 Obtain a video obtained by recording human movements.
  • Step S102 Identify multiple feature points in the video that match the identified object, and generate multiple coordinates of each feature point based on the preset coordinate system and video playback order.
  • the identified object includes a limb and / or hand.
  • Step S103 the video is divided into a plurality of frame sequences according to the preset video dividing principle.
  • step S104 a feature point with the largest coordinate change is selected from the multiple feature points in each frame sequence as the target feature point.
  • Step S105 Acquire multiple coordinates of the target feature point selected from each frame sequence, and generate multiple notes codes corresponding to each frame sequence from the multiple coordinates according to preset music generation rules.
  • step S106 a melody is generated according to a plurality of note codes corresponding to each frame sequence and video playback order.
  • the video obtained by recording the human motion may be a human dance video or a motion video shot by a camera.
  • multiple feature points of the limb include a nose feature point, a neck feature point, a right shoulder feature point, a right elbow feature point, a right wrist feature point, a left shoulder feature point, Left elbow feature point, left wrist feature point, right hip feature point, right knee feature point, right foot feature point, left hip feature point, left knee feature point, left foot feature point, right eye feature point, left eye feature point, right Ear feature points, left ear feature points.
  • the multiple feature points of the hand include a wrist feature point and four feature points respectively distributed on a corresponding finger. That is, each hand has a total of 21 feature points.
  • identify multiple feature points matching the identified object in the video and generate multiple coordinates of each feature point based on the preset coordinate system and video playback order, including: according to the preset video frame rate and preset sampling rules Select multiple frames of images; identify the coordinates of multiple feature points in each selected frame of images; output the coordinates of multiple feature points in each selected frame of images in the video playback order.
  • the preset video frame rate is 25 frames / second
  • the preset video frame rate is 30 frames per second, and the 10th, 20th, and 30th frame images can be selected.
  • the coordinates of the selected multiple feature points in each frame of the image are identified. Understandably, selecting multiple images at even intervals for recognition can speed up the recognition rate. The more image frames are selected, the shorter the interval is, the more accurate the motion trajectory and the more notes generated.
  • the method of identifying multiple feature points in the video that match the identified object and generating multiple coordinates of each feature point based on the preset coordinate system and video playback sequence can also call the body language recognition model for recognition.
  • the language recognition model can be any one of DensePose, OpenPose, AlphaPose, and DeepPose.
  • the coordinates of feature points include (x, y, score), where x represents the abscissa, y represents the ordinate, and score represents the predicted score.
  • the value of score after normalization is in the range of 0 to 1, the closer it is A value of 1 indicates that the more accurate the coordinate prediction value, the higher the degree of reduction of the feature point corresponding to the coordinate, that is, the higher the degree of posture reduction.
  • the coordinates of the feature points are corrected based on the prediction score, so that the accuracy of the notes generated by the feature points with a high degree of posture reduction is higher. Furthermore, a frame is selected according to the selected multiple feature points in each frame of the image, and the volume of the corresponding generated note is changed according to the size of the area of the frame.
  • the method further includes: normalizing the multiple coordinates so that the values of the abscissa and ordinate of each coordinate are within a preset interval.
  • the y value of each coordinate is in the interval [0, 2]
  • the x value is in the interval [0, 12].
  • the video is divided into multiple frame sequences according to the preset video division principle, and there may be multiple ways, two of which are given below.
  • Method 1 Obtain the sample melody; divide the video according to the tempo of the sample melody to obtain multiple frame sequences. For example, two beat rhythm, four beat rhythm, 3/8 beat rhythm, etc. Thus, the generated melody can be played according to the rhythm of the sample melody.
  • the sample melody can also be a piece of music, such as "Two Tigers".
  • Method 2 Create a video segmentation coordinate system, where the origin of the video segmentation coordinate system is the midpoint of a plurality of preset feature points or a preset feature point; split the video according to the direction of movement of the origin of the video segmentation coordinate system to obtain multiple Frame sequence. By dividing the video into multiple frame sequences, the generated music can be made more rhythmic.
  • the midpoint of multiple preset feature points is used as the origin of the video segmentation coordinate system.
  • the multiple preset feature points include a nose feature point, a neck feature point, a right shoulder feature point, a left shoulder feature point, a right hip feature point, and a left hip feature point. That is, the center position of the chest cavity of the human body is used to divide the origin of the coordinate system of each frame sequence.
  • the origin of the video segmentation coordinate system is the same as when the recognition object is a limb.
  • the recognition object is a hand
  • the coordinates of the wrist feature point are used as the origin of the video segmentation coordinate system.
  • the video is divided to obtain the first frame sequence; then the first frame sequence
  • the end frame of is the start frame, and the above steps are repeated to obtain multiple frame sequences.
  • the movement direction of the human body refers to the change of the set circular area to divide the video. For example, if the human body moves to the left for a certain distance and then moves back, then the node that walked back in the past is used as the node for video segmentation. When the human body jumps up and falls back to the ground, the node that fell back in the past is used as the node for video segmentation.
  • one feature point with the largest coordinate change is selected from the multiple feature points in each frame sequence as the target feature point, including: correcting the coordinates according to the weight ratio of multiple feature points; From the coordinates, a feature point with the largest coordinate change is selected as the target feature point. For example, the movement amplitude of the hand and leg feature points among multiple feature points is definitely higher than that of the nose feature point and eyebrow feature point. Therefore, the weight ratio of the nose feature point and eyebrow feature point is increased, and their coordinates are corrected respectively.
  • the coordinate after point correction selects the feature point with the largest coordinate change as the target feature point; it can make the overall action recognition select the feature point with higher coverage.
  • the preset music generation rule selects two octave notes, namely C4-B5, so that multiple coordinates of the target feature point in each frame sequence can be generated one by one. Note.
  • the melody can be converted into a melody of any timbre among wind instruments, percussion instruments, and string instruments through a midi converter.
  • the melody can be converted into a melody of any timbre among wind instruments, percussion instruments, and string instruments through a midi converter.
  • An embodiment of the present application provides a music generation device, which is used to perform the above-mentioned music generation method.
  • the device includes: an acquisition unit 10, an identification unit 20, a division unit 30, and a screening unit 40.
  • the obtaining unit 10 is used to obtain videos obtained by recording human actions.
  • the recognition unit 20 is configured to recognize a plurality of feature points matching the recognition object in the video, and generate a plurality of coordinates of each feature point based on a preset coordinate system and a video playback sequence.
  • the recognition object includes a limb and / or a hand.
  • the dividing unit 30 is used to divide the video into multiple frame sequences according to the preset video dividing principle.
  • the filtering unit 40 is configured to select one feature point with the largest coordinate change from the multiple feature points in each frame sequence as the target feature point.
  • the first generating unit 50 is configured to acquire multiple coordinates of the target feature point selected from each frame sequence, and generate multiple note codes corresponding to each frame sequence from the multiple coordinates according to preset music generation rules.
  • the second generating unit 60 is configured to generate a melody according to a plurality of note codes corresponding to each frame sequence and video playback order.
  • the identification unit 20 includes a selection subunit, an identification subunit, and an output subunit.
  • a selection subunit is used to select multiple frames of images according to a preset video frame rate and a preset sampling rule; a recognition subunit is used to identify coordinates of multiple feature points in each selected frame of images. The output subunit is used to output the coordinates of multiple feature points in each frame of the selected image in the video playback order.
  • the preset video frame rate is 25 frames / second
  • the preset video frame rate is 30 frames per second, and the 10th, 20th, and 30th frame images can be selected.
  • the coordinates of the selected multiple feature points in each frame of the image are identified. Understandably, selecting multiple images at even intervals for recognition can speed up the recognition rate. The more image frames are selected, the shorter the interval is, the more accurate the motion trajectory and the more notes generated.
  • the recognition unit 20 may also be a body language recognition model, for example, any one of DensePose, OpenPose, AlphaPose, and DeepPose.
  • the coordinates of feature points include (x, y, score), where x represents the abscissa, y represents the ordinate, and score represents the predicted score.
  • the value of score after normalization is in the range of 0 to 1, the closer it is A value of 1 indicates that the more accurate the coordinate prediction value, the higher the degree of reduction of the feature point corresponding to the coordinate, that is, the higher the degree of posture reduction.
  • the coordinates of the feature points are corrected based on the prediction score, so that the accuracy of the notes generated by the feature points with a high degree of posture reduction is higher. Furthermore, a frame is selected according to the selected multiple feature points in each frame of the image, and the volume of the corresponding generated note is changed according to the size of the area of the frame.
  • the device further includes a processing unit for normalizing multiple coordinates so that the value of the abscissa and the value of the ordinate of each coordinate are both within the preset interval.
  • the y value of each coordinate is located in the interval [0, 2]
  • the x value is located in the interval [0, 12].
  • the division unit includes an acquisition subunit and a first division subunit.
  • the obtaining subunit is used to obtain the sample melody; the first dividing subunit is used to divide the video according to the beat of the sample melody to obtain a plurality of frame sequences. For example, two beat rhythm, four beat rhythm, 3/8 beat rhythm, etc.
  • the generated melody can be played according to the rhythm of the sample melody.
  • the sample melody can also be a piece of music, such as "Two Tigers".
  • the dividing unit includes creating a subunit and a second dividing subunit.
  • the second dividing subunit is used to divide the video according to the moving direction of the origin of the video dividing coordinate system to obtain a plurality of frame sequences. By dividing the video into multiple frame sequences, the generated music can be made more rhythmic.
  • the midpoint of multiple preset feature points is used as the origin of the video segmentation coordinate system.
  • the multiple preset feature points include a nose feature point, a neck feature point, a right shoulder feature point, a left shoulder feature point, a right hip feature point, and a left hip feature point. That is, the center position of the chest cavity of the human body is used to divide the origin of the coordinate system of each frame sequence.
  • the origin of the video segmentation coordinate system is the same as when the recognition object is a limb.
  • the recognition object is a hand
  • the coordinates of the wrist feature point are used as the origin of the video segmentation coordinate system.
  • the video is divided to obtain the first frame sequence; then the first frame sequence
  • the end frame of is the start frame, and the above steps are repeated to obtain multiple frame sequences.
  • the video is divided according to the change of the set circular area according to the moving direction of the human body. For example, if the human body moves to the left for a certain distance and then moves back, then the node that walked back in the past is used as the node for video segmentation.
  • the screening unit includes a correction subunit and a screening subunit.
  • the correction subunit is used to correct the coordinates according to the weight ratio of multiple feature points; the screening subunit is used to select the feature point with the largest coordinate change from the coordinates after multiple feature point corrections as the target feature point.
  • the movement amplitude of the hand and leg feature points among multiple feature points is definitely higher than that of the nose feature point and eyebrow feature point. Therefore, the weight ratio of the nose feature point and eyebrow feature point is increased, and their coordinates are corrected respectively.
  • the coordinate after point correction selects the feature point with the largest coordinate change as the target feature point; it can make the overall action recognition select the feature point with higher coverage.
  • the preset music generation rule selects two octave notes, namely C4-B5, so that multiple coordinates of the target feature point in each frame sequence can be generated one by one. Note.
  • the device further includes a conversion unit for converting the melody to any of wind instruments, percussion instruments, and string instruments after generating the melody according to a plurality of note codes corresponding to each frame sequence and the video playback sequence.
  • a melody of timbre Or, take the melody as the main melody and add chords formed by multiple instruments to make the music more pleasant.
  • An embodiment of the present application provides a computer non-volatile storage medium, where the storage medium includes a stored program, wherein, when the program is running, the device where the storage medium is located is controlled to perform the following steps:
  • the video obtained by recording human actions; identify multiple feature points in the video that match the identified object, and generate multiple coordinates of each feature point based on the preset coordinate system and video playback sequence.
  • the identified objects include limbs and / or hands ;
  • the video is divided into multiple frame sequences according to the preset video segmentation principle; the feature point with the largest coordinate change is selected from the multiple feature points of each frame sequence as the target feature point; the acquisition is selected from each frame sequence Multiple coordinates of the target feature point of the target, and generate multiple note codes corresponding to each frame sequence according to the preset music generation rules; generate melody according to multiple note codes corresponding to each frame sequence and video playback sequence .
  • the device that controls the storage medium further performs the following steps: acquiring a sample melody; segmenting the video according to the tempo of the sample melody to obtain multiple frame sequences.
  • the device that controls the storage medium when the program is running, the device that controls the storage medium also performs the following steps: creating a video segmentation coordinate system, where the origin of the video segmentation coordinate system is the midpoint of multiple preset feature points or one preset feature point; press The moving direction of the origin of the video division coordinate system divides the video to obtain multiple frame sequences.
  • the device that controls the storage medium further performs the following steps: selecting multiple frames of images according to a preset video frame rate and preset sampling rules; identifying the coordinates of multiple feature points in each selected frame of images; The video playback sequence outputs the coordinates of multiple feature points in each frame of the selected image.
  • the device that controls the storage medium further performs the following steps: normalizing multiple coordinates so that the values of the abscissa and ordinate of each coordinate are within a preset interval.
  • the device that controls the storage medium when the program is running, the device that controls the storage medium also performs the following steps: correct the coordinates according to the weight ratio of multiple feature points; select the feature point with the largest coordinate change from the modified coordinates of the multiple feature points as the target Feature points.
  • an embodiment of the present application provides a computer device 100, including a memory 102, a processor 101, and a computer program 103 stored in the memory 102 and executable on the processor 101.
  • the processor The following steps are realized when the computer program is executed:
  • the video obtained by recording human actions; identify multiple feature points in the video that match the identified object, and generate multiple coordinates of each feature point based on the preset coordinate system and video playback sequence.
  • the identified objects include limbs and / or hands ;
  • the video is divided into multiple frame sequences according to the preset video segmentation principle; the feature point with the largest coordinate change is selected from the multiple feature points of each frame sequence as the target feature point; the acquisition is selected from each frame sequence Multiple coordinates of the target feature point of the target, and generate multiple note codes corresponding to each frame sequence according to the preset music generation rules; generate melody according to multiple note codes corresponding to each frame sequence and video playback sequence .
  • the processor also implements the following steps when executing the computer program: acquiring a sample melody; segmenting the video according to the tempo of the sample melody to obtain multiple frame sequences.
  • the processor also implements the following steps when executing the computer program: creating a video segmentation coordinate system, where the origin of the video segmentation coordinate system is the midpoint of a plurality of preset feature points or a preset feature point; the coordinates are segmented by video Divide the video in the moving direction of the origin of the system to obtain multiple frame sequences.
  • the processor also implements the following steps when executing the computer program: selecting multiple frames of images according to a preset video frame rate and preset sampling rules; identifying the coordinates of multiple feature points in each selected frame of images; according to the video playback order Output the coordinates of multiple feature points in each selected image.
  • the processor executes the computer program, the following steps are further implemented: the multiple coordinates are normalized, so that the value of the horizontal coordinate and the value of the vertical coordinate of each coordinate are located within a preset interval.
  • the processor executes the computer program, the following steps are further implemented: the coordinates are corrected according to the weight ratio of the multiple feature points; and the feature point with the largest coordinate change is selected from the corrected coordinates of the multiple feature points as the target feature point.
  • terminals involved in the embodiments of the present application may include, but are not limited to, personal computers (Personal Computers, PCs), personal digital assistants (Personal Digital Assistants, PDAs), wireless handheld devices, tablet computers (Tablet Computers), Mobile phones, MP3 players, MP4 players, etc.
  • the application may be an application program (nativeApp) installed on the terminal, or may also be a webpage program (webApp) of a browser on the terminal, which is not limited in this embodiment of the present application.
  • the disclosed system, device, and method may be implemented in other ways.
  • the device embodiments described above are only schematic.
  • the division of units is only a division of logical functions. In actual implementation, there may be other divisions.
  • multiple units or components may be combined or may Integration into another system, or some features can be ignored, or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
  • the above integrated unit can be implemented in the form of hardware, or in the form of hardware plus software functional units.
  • the above integrated unit implemented in the form of a software functional unit may be stored in a computer-readable storage medium.
  • the above software functional unit is stored in a storage medium, and includes several instructions to enable a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the embodiments of the present application .
  • the foregoing storage media include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disks or optical disks and other media that can store program codes .

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Acoustics & Sound (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • Human Computer Interaction (AREA)
  • Processing Or Creating Images (AREA)
  • Auxiliary Devices For Music (AREA)
  • Image Analysis (AREA)

Abstract

本申请实施例提供了一种音乐生成方法及装置，本申请涉及人工智能技术领域，该方法包括：获取录制人体动作得到的视频；识别视频中与识别对象相匹配的多个特征点，并基于预设坐标系及视频播放顺序生成各个特征点的多个坐标；根据预设的视频分割原则将视频分割为多个帧序列；从每个帧序列的多个特征点中筛选出坐标变化最大的一个特征点作为目标特征点；获取从每个帧序列中选取的目标特征点的多个坐标，并根据预设的音乐生成规则将多个坐标生成每个帧序列对应的多个音符代码；根据每个帧序列对应的多个音符代码及视频播放顺序生成旋律。本申请实施例提供的技术方案能够解决现有技术中作曲模式单一的问题。

Description

一种音乐生成方法及装置
本申请要求于2018年10月26日提交中国专利局、申请号为201811257164.7、申请名称为“一种音乐生成方法及装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及人工智能技术领域，尤其涉及一种音乐生成方法及装置。
背景技术
音乐是时间的艺术，现有的音乐创作模式往往是作曲家通过键盘输入形成音乐，作曲家必须具有一定程度和深度的作曲能力、作曲功底、作曲技术、作曲理论。这使得作曲这件事让大多数人觉得高深莫测，难以得到推广。目前，作曲家一般都是通过键盘或乐器进行作曲，作曲模式单一。
申请内容
有鉴于此，本申请实施例提供了一种音乐生成方法及装置，用以解决现有技术中作曲模式单一的问题。
为了实现上述目的，根据本申请的一个方面，提供了一种音乐生成方法，所述方法包括：获取录制人体动作得到的视频；识别所述视频中与识别对象相匹配的多个特征点，并基于预设坐标系及视频播放顺序生成各个特征点的多个坐标，所述识别对象包括肢体和/或手部；根据预设的视频分割原则将所述视频分割为多个帧序列；从每个帧序列的多个特征点中筛选出坐标变化最大的一个特征点作为目标特征点；获取从每个帧序列中选取的所述目标特征点的多个坐标， 并根据预设的音乐生成规则将所述多个坐标生成每个帧序列对应的多个音符代码；根据每个帧序列对应的多个音符代码及所述视频播放顺序生成旋律。
为了实现上述目的，根据本申请的一个方面，提供了一种音乐生成装置，所述装置包括：获取单元，用于获取录制人体动作得到的视频；识别单元，用于识别所述视频中与识别对象相匹配的多个特征点，并基于预设坐标系及视频播放顺序生成各个特征点的多个坐标，所述识别对象包括肢体和/或手部；分割单元，用于根据预设的视频分割原则将所述视频分割为多个帧序列；筛选单元，用于从每个帧序列的多个特征点中筛选出坐标变化最大的一个特征点作为目标特征点；第一生成单元，用于获取从每个帧序列中选取的所述目标特征点的多个坐标，并根据预设的音乐生成规则将所述多个坐标生成每个帧序列对应的多个音符代码；第二生成单元，用于根据每个帧序列对应的多个音符代码及所述视频播放顺序生成旋律。
为了实现上述目的，根据本申请的一个方面，提供了一种计算机设备，包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现上述的音乐生成方法的步骤。
为了实现上述目的，根据本申请的一个方面，提供了一种计算机非易失性存储介质，所述存储介质包括存储的程序，在所述程序运行时控制所述存储介质所在设备执行上述的音乐生成方法的步骤。
在本方案中，通过获取人体动作视频，根据人体动作的幅度变化生成相应的音符，进而得到伴随肢体动作的旋律，从而提供了一种新的音乐生成模式，解决了现有技术中作曲模式单一的问题，丰富了作曲模式。
附图说明
为了更清楚地说明本申请实施例的技术方案，下面将对实施例中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附 图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其它的附图。
图1是根据本申请实施例的一种音乐生成方法的流程图；
图2是根据本申请实施例的一种音乐生成装置的示意图；
图3是根据本申请实施例的肢体识别特征点分布图；
图4是根据本申请实施例的手部识别特征点分布图；
图5是根据本申请实施例的音符生成对照图；
图6是根据本申请实施例的一种计算机设备的示意图。
具体实施方式
为了更好的理解本申请的技术方案，下面结合附图对本申请实施例进行详细描述。
应当明确，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其它实施例，都属于本申请保护的范围。
在本申请实施例中使用的术语是仅仅出于描述特定实施例的目的，而非旨在限制本申请。在本申请实施例和所附权利要求书中所使用的单数形式的“一种”、“所述”和“该”也旨在包括多数形式，除非上下文清楚地表示其他含义。
应当理解，本文中使用的术语“和/或”仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。
应当理解，尽管在本申请实施例中可能采用术语第一、第二、第三等来描述终端，但这些终端不应限于这些术语。这些术语仅用来将终端彼此区分开。例如，在不脱离本申请实施例范围的情况下，第一分割子单元也可以被称为第二分割子单元，类似地，第二分割子 单元也可以被称为第一分割子单元。
取决于语境，如在此所使用的词语“如果”可以被解释成为“在……时”或“当……时”或“响应于确定”或“响应于检测”。类似地，取决于语境，短语“如果确定”或“如果检测(陈述的条件或事件)”可以被解释成为“当确定时”或“响应于确定”或“当检测(陈述的条件或事件)时”或“响应于检测(陈述的条件或事件)”。
图1是根据本申请实施例的一种音乐生成方法的流程图，如图1所示，该方法包括：
步骤S101，获取录制人体动作得到的视频。
步骤S102，识别视频中与识别对象相匹配的多个特征点，并基于预设坐标系及视频播放顺序生成各个特征点的多个坐标，识别对象包括肢体和/或手部。
步骤S103，根据预设的视频分割原则将视频分割为多个帧序列。
步骤S104，从每个帧序列的多个特征点中筛选出坐标变化最大的一个特征点作为目标特征点。
步骤S105，获取从每个帧序列中选取的目标特征点的多个坐标，并根据预设的音乐生成规则将多个坐标生成每个帧序列对应的多个音符代码。
步骤S106，根据每个帧序列对应的多个音符代码及视频播放顺序生成旋律。
在本方案中，通过获取人体动作视频，便能够根据人体动作的幅度变化生成相应的音符，进而形成伴随肢体动作的旋律，提供了一种新的音乐生成模式。
可选地，获取录制人体动作得到的视频，可以是通过摄像机拍摄的一段人体舞蹈视频或动作视频。
可选地，如图3所示，当识别对象为肢体时，肢体的多个特征点包括鼻特征点、颈特征点、右肩特征点、右肘特征点、右腕特征点、左肩特征点、左肘特征点、左腕特征点、右髋特征点、右膝特征点、右脚特征点、左髋特征点、左膝特征点、左脚特征点、右眼特征点、 左眼特征点、右耳特征点、左耳特征点。
可选地，如图4所示，当识别对象为手部时，手部的多个特征点包括一个手腕特征点及分别分布于对应一个手指上的四个特征点。即每只手共21个特征点。
可选地，识别视频中与识别对象相匹配的多个特征点，并基于预设坐标系及视频播放顺序生成各个特征点的多个坐标，包括：按照预设视频帧率和预设采样规则选取多帧图像；识别选取的每帧图像中的多个特征点的坐标；按照视频播放顺序输出选取的每帧图像中的多个特征点的坐标。
比如：预设视频帧率为25帧/秒，可以选取第1、6、11、16、21帧图像。或者预设视频帧率为30帧/秒，可以选取第10、20、30帧图像。进而识别选取的每帧图像中的多个特征点的坐标。可以理解地，选择均匀间隔的多个图像进行识别，可以加快识别速率，选取的图像帧数越多，间隔越短，得到的运动轨迹与越精确，生成的音符也就更多。
可选地，识别视频中与识别对象相匹配的多个特征点，并基于预设坐标系及视频播放顺序生成各个特征点的多个坐标的方式也可以调用肢体语言识别模型来进行识别，肢体语言识别模型可以是DensePose、OpenPose、AlphaPose及DeepPose中的任意一种。特征点的坐标包括(x，y，score)，其中，x代表横坐标，y代表纵坐标，score代表预测评分，score的取值经过归一化处理后范围在0～1之间，越接近1的值表示坐标预测值越准确，坐标对应的特征点的还原度就越高，即姿态还原度越高。
可选地，基于预测评分修正特征点的坐标，从而使得姿态还原度高的特征点所生成的音符的准确性越高。进而，根据选取的每帧图像中的多个特征点进行选框，根据选框面积大小改变对应生成的音符的音量大小。
可选地，在识别视频中与识别对象相匹配的多个特征点，并基于预设坐标系及视频播放顺序生成各个特征点的多个坐标之后，并在 根据预设的视频分割原则将视频分割为多个帧序列之前，方法还包括：将多个坐标进行归一化处理，使得每个坐标的横坐标的值和纵坐标的值均位于预设区间内，在一种实施方式中，使得每个坐标的y值位于[0，2]区间，x值位于[0,12]区间。可选地，根据预设的视频分割原则将视频分割为多个帧序列，具体可以有多种方式，下文给出其中的两种方式。
方式一，获取样本旋律；根据样本旋律的节拍分割视频得到多个帧序列。例如二拍子节奏、四拍子节奏、3/8拍节奏等。从而使得生成的旋律能够根据样本旋律的节拍进行演奏。样本旋律还可以是某段音乐，例如“两只老虎”。
方式二，创建视频分割坐标系，其中，视频分割坐标系的原点为多个预设特征点的中点或一个预设特征点；按视频分割坐标系的原点的移动方向分割视频，得到多个帧序列。通过将视频进行分割成多个帧序列，可以使得生成的音乐更富节奏感。
具体地，当识别对象为肢体时，以多个预设特征点的中点作为视频分割坐标系的原点。多个预设特征点包括鼻特征点、颈特征点、右肩特征点、左肩特征点、右髋特征点、左髋特征点。即人体的胸腔中心位置用于分割每个帧序列的坐标系原点。当识别对象为肢体及手部时，视频分割坐标系的原点与识别对象为肢体时相同。当识别对象为手部时，将手腕特征点的坐标作为视频分割坐标系的原点。
可选地，设定以视频的起始帧的原点为圆心，半径为r的圆形区域，圆形区域包含初始帧上的所有特征点。在视频播放过程中，当原点超出圆形区域后，且原点坐标中的横坐标或竖坐标中有至少一个的值发生反方向运动为止，分割视频得到第一帧序列；再以第一帧序列的结束帧为起始帧，重复上述步骤，从而得到多个帧序列。可以理解地，即人体移动方向参照设定的圆形区域的变化来分割视频。比如，人体往左边走动一段距离后又往回走，则以往回走的那个节点作为视频分割的节点。人体往上跳动又回落至地面，则以往回落的那个节点作为视频分割的节点。
可选地，从每个帧序列的多个特征点中筛选出坐标变化最大的一个特征点作为目标特征点，包括：按多个特征点的权重比例修正坐标；从多个特征点修正后的坐标中筛选出坐标变化最大的一个特征点作为目标特征点。比如：多个特征点中手部、腿部的特征点的动作幅度肯定比鼻特征点、眉特征点，因此提高鼻特征点、眉特征点的权重比例，分别修正其坐标，从多个特征点修正后的坐标筛选出坐标变化最大的一个特征点作为目标特征点；可以使得整体动作识别所能选取特征点的覆盖率更高。
可选地，预设的音乐生成规则，如图5所示，选取两个八度的音符即C4～B5,使得每个帧序列中的目标特征点的多个坐标都能一一生成对应一个音符。
可选地，根据每个帧序列对应的多个音符代码及视频播放顺序生成旋律之后，可以通过midi转换器将旋律转换为管乐器、打击乐器、弦乐器中任意一种音色的旋律。或者，以旋律为主旋律，添加多个乐器形成的和弦，使得音乐的更加动听。
本申请实施例提供了一种音乐生成装置，该音乐生成装置用于执行上述音乐生成方法，如图2所示，该装置包括：获取单元10、识别单元20、分割单元30、筛选单元40、第一生成单元50、第二生成单元60。
获取单元10，用于获取录制人体动作得到的视频。
识别单元20，用于识别视频中与识别对象相匹配的多个特征点，并基于预设坐标系及视频播放顺序生成各个特征点的多个坐标，识别对象包括肢体和/或手部。
分割单元30，用于根据预设的视频分割原则将视频分割为多个帧序列。
筛选单元40，用于从每个帧序列的多个特征点中筛选出坐标变化最大的一个特征点作为目标特征点。
第一生成单元50，用于获取从每个帧序列中选取的目标特征点 的多个坐标，并根据预设的音乐生成规则将多个坐标生成每个帧序列对应的多个音符代码。
第二生成单元60，用于根据每个帧序列对应的多个音符代码及视频播放顺序生成旋律。
在本方案中，通过获取人体动作视频，便能够根据人体动作的幅度变化生成相应的音符，进而形成伴随肢体动作的旋律，提供了一种新的音乐生成模式。
可选地，识别单元20包括选取子单元、识别子单元、输出子单元。
选取子单元，用于按照预设视频帧率和预设采样规则选取多帧图像；识别子单元，用于识别选取的每帧图像中的多个特征点的坐标。输出子单元，用于按照视频播放顺序输出选取的每帧图像中的多个特征点的坐标。
比如：预设视频帧率为25帧/秒，可以选取第1、6、11、16、21帧图像。或者预设视频帧率为30帧/秒，可以选取第10、20、30帧图像。进而识别选取的每帧图像中的多个特征点的坐标。可以理解地，选择均匀间隔的多个图像进行识别，可以加快识别速率，选取的图像帧数越多，间隔越短，得到的运动轨迹与越精确，生成的音符也就更多。
可选地，识别单元20也可以是肢体语言识别模型，例如，可以是DensePose、OpenPose、AlphaPose及DeepPose中的任意一种。特征点的坐标包括(x，y，score)，其中，x代表横坐标，y代表纵坐标，score代表预测评分，score的取值经过归一化处理后范围在0～1之间，越接近1的值表示坐标预测值越准确，坐标对应的特征点的还原度就越高，即姿态还原度越高。
可选地，基于预测评分修正特征点的坐标，从而使得姿态还原度高的特征点所生成的音符的准确性越高。进而，根据选取的每帧图像中的多个特征点进行选框，根据选框面积大小改变对应生成的音符的音量大小。
可选地，装置还包括处理单元，用于将多个坐标进行归一化处理，使得每个坐标的横坐标的值和纵坐标的值均位于预设区间内。在一种实施方式中，使得每个坐标的y值位于[0，2]区间，x值位于[0,12]区间。
可选地，分割单元包括获取子单元、第一分割子单元。
获取子单元，用于获取样本旋律；第一分割子单元，用于根据样本旋律的节拍分割视频得到多个帧序列。例如二拍子节奏、四拍子节奏、3/8拍节奏等。从而使得生成的旋律能够根据样本旋律的节拍进行演奏。样本旋律还可以是某段音乐，例如“两只老虎”。
可选地，分割单元包括创建子单元、第二分割子单元。
创建子单元，用于创建视频分割坐标系，其中，视频分割坐标系的原点为多个预设特征点的中点或一个预设特征点。第二分割子单元，用于按视频分割坐标系的原点的移动方向分割视频，得到多个帧序列。通过将视频进行分割成多个帧序列，可以使得生成的音乐更富节奏感。
具体地，当识别对象为肢体时，以多个预设特征点的中点作为视频分割坐标系的原点。多个预设特征点包括鼻特征点、颈特征点、右肩特征点、左肩特征点、右髋特征点、左髋特征点。即人体的胸腔中心位置用于分割每个帧序列的坐标系原点。当识别对象为肢体及手部时，视频分割坐标系的原点与识别对象为肢体时相同。当识别对象为手部时，将手腕特征点的坐标作为视频分割坐标系的原点。
可选地，设定以视频的起始帧的原点为圆心，半径为r的圆形区域，圆形区域包含初始帧上的所有特征点。在视频播放过程中，当原点超出圆形区域后，且原点坐标中的横坐标或竖坐标中有至少一个的值发生反方向运动为止，分割视频得到第一帧序列；再以第一帧序列的结束帧为起始帧，重复上述步骤，从而得到多个帧序列。可以理解地，即根据人体移动方向参照设定的圆形区域的变化来分割视频。比如，人体往左边走动一段距离后又往回走，则以往回走的那个节点作为视频分割的节点。
可选地，筛选单元包括修正子单元、筛选子单元。
修正子单元，用于按多个特征点的权重比例修正坐标；筛选子单元，用于从多个特征点修正后的坐标中筛选出坐标变化最大的一个特征点作为目标特征点。
比如：多个特征点中手部、腿部的特征点的动作幅度肯定比鼻特征点、眉特征点，因此提高鼻特征点、眉特征点的权重比例，分别修正其坐标，从多个特征点修正后的坐标筛选出坐标变化最大的一个特征点作为目标特征点；可以使得整体动作识别所能选取特征点的覆盖率更高。
可选地，预设的音乐生成规则，如图5所示，选取两个八度的音符即C4～B5,使得每个帧序列中的目标特征点的多个坐标都能一一生成对应一个音符。
可选地，装置还包括转换单元，转换单元用于在根据每个帧序列对应的多个音符代码及视频播放顺序生成旋律之后，通过midi转换器将旋律转换为管乐器、打击乐器、弦乐器中任意一种音色的旋律。或者，以旋律为主旋律，添加多个乐器形成的和弦，使得音乐的更加动听。
本申请实施例提供了一种计算机非易失性存储介质，存储介质包括存储的程序，其中，在程序运行时控制存储介质所在设备执行以下步骤：
获取录制人体动作得到的视频；识别视频中与识别对象相匹配的多个特征点，并基于预设坐标系及视频播放顺序生成各个特征点的多个坐标，识别对象包括肢体和/或手部；根据预设的视频分割原则将视频分割为多个帧序列；从每个帧序列的多个特征点中筛选出坐标变化最大的一个特征点作为目标特征点；获取从每个帧序列中选取的目标特征点的多个坐标，并根据预设的音乐生成规则将多个坐标生成每个帧序列对应的多个音符代码；根据每个帧序列对应的多个音符代码及视频播放顺序生成旋律。
可选地，在程序运行时控制存储介质所在设备还执行以下步骤：获取样本旋律；根据样本旋律的节拍分割视频得到多个帧序列。
可选地，在程序运行时控制存储介质所在设备还执行以下步骤：创建视频分割坐标系，其中，视频分割坐标系的原点为多个预设特征点的中点或一个预设特征点；按视频分割坐标系的原点的移动方向分割视频，得到多个帧序列。
可选地，在程序运行时控制存储介质所在设备还执行以下步骤：按照预设视频帧率和预设采样规则选取多帧图像；识别选取的每帧图像中的多个特征点的坐标；按照视频播放顺序输出选取的每帧图像中的多个特征点的坐标。
可选地，在程序运行时控制存储介质所在设备还执行以下步骤：将多个坐标进行归一化处理，使得每个坐标的横坐标的值和纵坐标的值均位于预设区间内。
可选地，在程序运行时控制存储介质所在设备还执行以下步骤：按多个特征点的权重比例修正坐标；从多个特征点修正后的坐标中筛选出坐标变化最大的一个特征点作为目标特征点。
如图6所示，本申请实施例提供了一种计算机设备100，包括存储器102、处理器101以及存储在所述存储器102中并可在所述处理器101上运行的计算机程序103，处理器执行计算机程序时实现以下步骤：
获取录制人体动作得到的视频；识别视频中与识别对象相匹配的多个特征点，并基于预设坐标系及视频播放顺序生成各个特征点的多个坐标，识别对象包括肢体和/或手部；根据预设的视频分割原则将视频分割为多个帧序列；从每个帧序列的多个特征点中筛选出坐标变化最大的一个特征点作为目标特征点；获取从每个帧序列中选取的目标特征点的多个坐标，并根据预设的音乐生成规则将多个坐标生成每个帧序列对应的多个音符代码；根据每个帧序列对应的多个音符代码及视频播放顺序生成旋律。
可选地，处理器执行计算机程序时还实现以下步骤：获取样本旋律； 根据样本旋律的节拍分割视频得到多个帧序列。
可选地，处理器执行计算机程序时还实现以下步骤：创建视频分割坐标系，其中，视频分割坐标系的原点为多个预设特征点的中点或一个预设特征点；按视频分割坐标系的原点的移动方向分割视频，得到多个帧序列。
可选地，处理器执行计算机程序时还实现以下步骤：按照预设视频帧率和预设采样规则选取多帧图像；识别选取的每帧图像中的多个特征点的坐标；按照视频播放顺序输出选取的每帧图像中的多个特征点的坐标。
可选地，处理器执行计算机程序时还实现以下步骤：将多个坐标进行归一化处理，使得每个坐标的横坐标的值和纵坐标的值均位于预设区间内。
可选地，处理器执行计算机程序时还实现以下步骤：按多个特征点的权重比例修正坐标；从多个特征点修正后的坐标中筛选出坐标变化最大的一个特征点作为目标特征点。
需要说明的是，本申请实施例中所涉及的终端可以包括但不限于个人计算机(Personal Computer，PC)、个人数字助理(Personal Digital Assistant，PDA)、无线手持设备、平板电脑(Tablet Computer)、手机、MP3播放器、MP4播放器等。
可以理解的是，应用可以是安装在终端上的应用程序(nativeApp)，或者还可以是终端上的浏览器的一个网页程序(webApp)，本申请实施例对此不进行限定。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统，装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统，装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，单元的划分，仅仅为一种逻辑功能划 分，实际实现时可以有另外的划分方式，例如，多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用硬件加软件功能单元的形式实现。
上述以软件功能单元的形式实现的集成的单元，可以存储在一个计算机可读取存储介质中。上述软件功能单元存储在一个存储介质中，包括若干指令用以使得一台计算机装置(可以是个人计算机，服务器，或者网络装置等)或处理器(Processor)执行本申请各个实施例方法的部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上仅为本申请的较佳实施例而已，并不用以限制本申请，凡在本申请的精神和原则之内，所做的任何修改、等同替换、改进等，均应包含在本申请保护的范围之内。

Claims (20)

  1. 一种音乐生成方法，其特征在于，所述方法包括：
    获取录制人体动作得到的视频；
    识别所述视频中与识别对象相匹配的多个特征点，并基于预设坐标系及视频播放顺序生成各个特征点的多个坐标，所述识别对象包括肢体和/或手部；
    根据预设的视频分割原则将所述视频分割为多个帧序列；
    从每个帧序列的多个特征点中筛选出坐标变化最大的一个特征点作为目标特征点；
    获取从每个帧序列中选取的所述目标特征点的多个坐标，并根据预设的音乐生成规则将所述多个坐标生成每个帧序列对应的多个音符代码；
    根据每个帧序列对应的多个音符代码及所述视频播放顺序生成旋律。
  2. 根据权利要求1所述的方法，其特征在于，所述根据预设的视频分割原则将所述视频分割为多个帧序列，包括：
    获取样本旋律；
    根据所述样本旋律的节拍分割所述视频得到多个所述帧序列。
  3. 根据权利要求1所述的方法，其特征在于，所述根据预设的视频分割原则将所述视频分割为多个帧序列，包括：
    创建视频分割坐标系，其中，所述视频分割坐标系的原点为多个预设特征点的中点或一个预设特征点；
    按所述视频分割坐标系的原点的移动方向分割所述视频，得到多个所述帧序列。
  4. 根据权利要求1所述的方法，其特征在于，所述识别所述视频中与识别对象相匹配的多个特征点，并基于预设坐标系及所述视频播放顺序生成各个特征点的多个坐标，包括：
    按照预设视频帧率和预设采样规则选取多帧图像；
    识别选取的每帧图像中的所述多个特征点的坐标；
    按照所述视频播放顺序输出所述选取的每帧图像中的所述多个特征点的坐标。
  5. 根据权利要求1-4任意一项所述的方法，其特征在于，在所述识别所述视频中与识别对象相匹配的多个特征点，并基于预设坐标系及所述视频播放顺序生成各个特征点的多个坐标之后，并在所述根据预设的视频分割原则将所述视频分割为多个帧序列之前，所述方法还包括：
    将所述多个坐标进行归一化处理，使得每个坐标的横坐标的值和纵坐标的值均位于预设区间内。
  6. 根据权利要求1所述的方法，其特征在于，所述从每个帧序列的多个特征点中筛选出坐标变化最大的一个特征点作为目标特征点，包括：
    按所述多个特征点的权重比例修正所述坐标；
    从所述多个特征点修正后的坐标中筛选出坐标变化最大的一个特征点作为所述目标特征点。
  7. 一种音乐生成装置，其特征在于，所述装置包括：
    获取单元，用于获取录制人体动作得到的视频；
    识别单元，用于识别所述视频中与识别对象相匹配的多个特征点，并基于预设坐标系及视频播放顺序生成各个特征点的多个坐标，所述识别对象包括肢体和/或手部；
    分割单元，用于根据预设的视频分割原则将所述视频分割为多个帧序列；
    筛选单元，用于从每个帧序列的多个特征点中筛选出坐标变化最大的一个特征点作为目标特征点；
    第一生成单元，用于获取从每个帧序列中选取的所述目标特征点的多个坐标，并根据预设的音乐生成规则将所述多个坐标生成每个帧序列对应的多个音符代码；
    第二生成单元，用于根据每个帧序列对应的多个音符代码及所述视频播放顺序生成旋律。
  8. 根据权利要求7所述的装置，其特征在于，所述装置还包括：
    处理单元，用于将所述多个坐标进行归一化处理，使得每个坐标的横坐标的值和纵坐标的值均位于预设区间内。
  9. 根据权利要求7所述的装置，其特征在于，所述筛选单元包括：
    修正子单元，用于按所述多个特征点的权重比例修正所述坐标；
    筛选子单元，用于从所述多个特征点修正后的坐标中筛选出坐标变化最大的一个特征点作为所述目标特征点。
  10. 根据权利要求7所述的装置，其特征在于，所述分割单元包括：
    获取子单元，用于获取样本旋律；
    第一分割子单元，用于根据所述样本旋律的节拍分割所述视频得到多个所述帧序列。
  11. 根据权利要求7所述的装置，其特征在于，所述分割单元包括：
    创建子单元，用于创建视频分割坐标系，其中，所述视频分割坐标系的原点为多个预设特征点的中点或一个预设特征点；
    第二分割子单元，用于按所述视频分割坐标系的原点的移动方向分割所述视频，得到多个所述帧序列。
  12. 一种计算机设备，包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序，其特征在于，所述处理器执行所述计算机程序时实现以下步骤：
    获取录制人体动作得到的视频；
    识别所述视频中与识别对象相匹配的多个特征点，并基于预设坐标系及视频播放顺序生成各个特征点的多个坐标，所述识别对象包括肢体和/或手部；
    根据预设的视频分割原则将所述视频分割为多个帧序列；
    从每个帧序列的多个特征点中筛选出坐标变化最大的一个特征点作为目标特征点；
    获取从每个帧序列中选取的所述目标特征点的多个坐标，并根据预设的音乐生成规则将所述多个坐标生成每个帧序列对应的多个音符代码；
    根据每个帧序列对应的多个音符代码及所述视频播放顺序生成旋律。
  13. 根据权利要求12所述的计算机设备，其特征在于，所述处理器执行所述计算机程序时还实现以下步骤：
    获取样本旋律；
    根据所述样本旋律的节拍分割所述视频得到多个所述帧序列。
  14. 根据权利要求12所述的计算机设备，其特征在于，所述处理器执行所述计算机程序时还实现以下步骤：
    创建视频分割坐标系，其中，所述视频分割坐标系的原点为多个预设特征点的中点或一个预设特征点；
    按所述视频分割坐标系的原点的移动方向分割所述视频，得到多个所述帧序列。
  15. 根据权利要求12所述的计算机设备，其特征在于，所述处理器执行所述计算机程序时还实现以下步骤：
    按照预设视频帧率和预设采样规则选取多帧图像；
    识别选取的每帧图像中的所述多个特征点的坐标；
    按照所述视频播放顺序输出所述选取的每帧图像中的所述多个特征点的坐标。
  16. 根据权利要求12-15任意一项所述的计算机设备，其特征在于，所述处理器执行所述计算机程序时还实现以下步骤：
    将所述多个坐标进行归一化处理，使得每个坐标的横坐标的值和纵坐标的值均位于预设区间内。
  17. 一种计算机非易失性可读存储介质，所述存储介质包括存储的程序，其特征在于，在所述程序运行时控制所述存储介质所在设备执行以下步骤：
    获取录制人体动作得到的视频；
    识别所述视频中与识别对象相匹配的多个特征点，并基于预设坐标系及视频播放顺序生成各个特征点的多个坐标，所述识别对象包括肢体和/或手部；
    根据预设的视频分割原则将所述视频分割为多个帧序列；
    从每个帧序列的多个特征点中筛选出坐标变化最大的一个特征点作为目标特征点；
    获取从每个帧序列中选取的所述目标特征点的多个坐标，并根据预设的音乐生成规则将所述多个坐标生成每个帧序列对应的多个音符代码；
    根据每个帧序列对应的多个音符代码及所述视频播放顺序生成旋律。
  18. 根据权利要求17所述的计算机非易失性可读存储介质，其特征在于，在所述程序运行时控制所述存储介质所在设备执行以下步骤：
    获取样本旋律；
    根据所述样本旋律的节拍分割所述视频得到多个所述帧序列。
  19. 根据权利要求17所述的计算机非易失性可读存储介质，其特征在于，在所述程序运行时控制所述存储介质所在设备执行以下步骤：
    创建视频分割坐标系，其中，所述视频分割坐标系的原点为多个预设特征点的中点或一个预设特征点；
    按所述视频分割坐标系的原点的移动方向分割所述视频，得到多个所述帧序列。
  20. 根据权利要求17所述的计算机非易失性可读存储介质，其特征在于，在所述程序运行时控制所述存储介质所在设备执行以下步骤：
    按照预设视频帧率和预设采样规则选取多帧图像；
    识别选取的每帧图像中的所述多个特征点的坐标；
    按照所述视频播放顺序输出所述选取的每帧图像中的所述多个特征点的坐标。
PCT/CN2018/123552 2018-10-26 2018-12-25 一种音乐生成方法及装置 Ceased WO2020082575A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201811257164.7 2018-10-26
CN201811257164.7A CN109413351B (zh) 2018-10-26 2018-10-26 一种音乐生成方法及装置

Publications (1)

Publication Number Publication Date
WO2020082575A1 true WO2020082575A1 (zh) 2020-04-30

Family

ID=65470007

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/123552 Ceased WO2020082575A1 (zh) 2018-10-26 2018-12-25 一种音乐生成方法及装置

Country Status (2)

Country Link
CN (1) CN109413351B (zh)
WO (1) WO2020082575A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113221734A (zh) * 2020-11-25 2021-08-06 友达光电股份有限公司 影像识别方法及装置

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110827789B (zh) * 2019-10-12 2023-05-23 平安科技（深圳）有限公司 音乐生成方法、电子装置及计算机可读存储介质
CN115881064A (zh) * 2021-09-28 2023-03-31 北京字跳网络技术有限公司 音乐生成方法、装置、设备、存储介质及程序
US20230197040A1 (en) * 2021-12-20 2023-06-22 Lemon Inc. Interactive movement audio engine
CN117710536A (zh) * 2023-12-15 2024-03-15 咪咕文化科技有限公司 一种动画的生成方法、装置、设备和介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5506370A (en) * 1993-09-13 1996-04-09 Pioneer Electronic Corporation Display controlling apparatus for music accompaniment playing system, and the music accompaniment playing system
CN103885663A (zh) * 2014-03-14 2014-06-25 深圳市东方拓宇科技有限公司 一种生成和播放音乐的方法及其对应终端
CN105786396A (zh) * 2016-04-22 2016-07-20 成都涂鸦科技有限公司 一种划屏与步态相结合的音乐生成方法
CN107995442A (zh) * 2017-12-21 2018-05-04 北京奇虎科技有限公司 视频数据的处理方法、装置及计算设备

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10027883B1 (en) * 2014-06-18 2018-07-17 Amazon Technologies, Inc. Primary user selection for head tracking
US10083521B1 (en) * 2015-12-04 2018-09-25 A9.Com, Inc. Content recommendation based on color match
CN107145326B (zh) * 2017-03-28 2020-07-28 浙江大学 一种基于目标面部表情采集的音乐自动播放系统及方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5506370A (en) * 1993-09-13 1996-04-09 Pioneer Electronic Corporation Display controlling apparatus for music accompaniment playing system, and the music accompaniment playing system
CN103885663A (zh) * 2014-03-14 2014-06-25 深圳市东方拓宇科技有限公司 一种生成和播放音乐的方法及其对应终端
CN105786396A (zh) * 2016-04-22 2016-07-20 成都涂鸦科技有限公司 一种划屏与步态相结合的音乐生成方法
CN107995442A (zh) * 2017-12-21 2018-05-04 北京奇虎科技有限公司 视频数据的处理方法、装置及计算设备

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113221734A (zh) * 2020-11-25 2021-08-06 友达光电股份有限公司 影像识别方法及装置

Also Published As

Publication number Publication date
CN109413351B (zh) 2021-07-13
CN109413351A (zh) 2019-03-01

Similar Documents

Publication Publication Date Title
WO2020082575A1 (zh) 一种音乐生成方法及装置
CN120318388B (zh) 一种基于虚拟数字人的智能化实时交互问答系统
US11972693B2 (en) Method, device, system and apparatus for creating and/or selecting exercises for learning playing a music instrument
US12243437B2 (en) Crowd-based device configuration selection of a music teaching system
KR102139387B1 (ko) 큰 말뭉치에 기초하여 음성 합성을 하기 위한 방법 및 장치
EP3791387A2 (en) Systems and methods for improved speech recognition using neuromuscular information
Kochanski et al. Quantitative measurement of prosodic strength in Mandarin
US11893898B2 (en) Method and apparatus for an adaptive and interactive teaching of playing a musical instrument
CN112309365A (zh) 语音合成模型的训练方法、装置、存储介质以及电子设备
CN115937369B (zh) 一种表情动画生成方法、系统、电子设备及存储介质
CN109346045A (zh) 基于长短时神经网络的多声部音乐生成方法及装置
JP2020046500A (ja) 情報処理装置、情報処理方法および情報処理プログラム
CN115457654B (zh) 一种基于人体关键点的实时视频流手语识别方法
CN115083371B (zh) 驱动虚拟数字形象唱歌的方法及其装置
CN112735371A (zh) 一种基于文本信息生成说话人视频的方法及装置
CN103136780A (zh) 基于关键帧的手语语流音变动画合成方法
CN116980543A (zh) 视频生成方法、装置、存储介质及计算机设备
CN116808586A (zh) 智能体的控制方法、终端设备及存储介质
US10319352B2 (en) Notation for gesture-based composition
JP2004086001A (ja) 会話処理装置、および会話処理方法、並びにコンピュータ・プログラム
Cosentino et al. Human–robot musical interaction
Riad et al. Signsworld; deeping into the silence world and hearing its signs (state of the art)
CN111782858B (zh) 音乐匹配的方法和装置
CN113327312A (zh) 虚拟角色的驱动方法、装置、设备以及存储介质
Emad et al. Note frequency recognition and finger motion capture of a guitarist: a survey and challenges ahead

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18937637

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 19.08.2021)

122 Ep: pct application non-entry in european phase

Ref document number: 18937637

Country of ref document: EP

Kind code of ref document: A1