KR20200065499A - Dancing Robot that learns the relationship between dance and music - Google Patents

Abstract

The present invention relates to a robot capable of learning a correlation between a dance and music to dance. According to the present invention, the robot (1) having a human shape to control a portion corresponding to each joint of a person by a motor (10) to dance comprises: a microphone (100) to receive music; a data storage unit (200) storing data resulting from learning a correlation between a dance and music; a data input unit (300) to communicate with the data storage unit (200) in a wired/wireless manner to input music or dance image data to store the music or the dance image data in the data storage unit (200); and a control unit (400) connected to the motor (10), the microphone (100), and the data storage unit (200) to control to dance in accordance with music received from the microphone (100).

Description

Dancing robot that learns the relationship between music and dance {Dancing Robot that learns the relationship between dance and music}

The present invention relates to a robot that dances by extracting the characteristics of music and dance and learning the correlation therewith.

In recent years, with the development of technology, interest in robots has been focused, and robots for various purposes have been developed. As advancement to an intelligent robot, various developments have been made. For example, a pet dog robot or a humanoid type walking and dancing robot has been developed.

In general, the trend of conventional robot toys, especially robots dancing to music and sound, is focused on simple expression.

Since the conventional robot toy is merely a simple motion in which the body motion moves up and down, the body is fixed, or is shaken by being tied with a string, there is a feeling of lack of vitality due to the lack of emotion expression as a robot toy.

In the case of decorative items of children and adults, which are the demand layer of the robot toys, the level of consciousness is increased, and the conventional robot toys have eliminated the limitations of play effect and the monotony in operation.

In addition, the conventional dancing robot is simply a degree to detect an external stimulus (eg, sound, touch, etc.) to dance with a preset dance action. That is, the dance action is set in advance, and the dance is performed with the preset motion regardless of the sound source.

For example, the memory of a dance sound source is stored in a memory, and a dance sound source is flowed according to a sound source to be built, and the robot toy is combined with the song. will be. However, such a robot toy has a problem in that the user can easily feel the demonstration because it is a repetition of a relatively simple movement, such as shaking or opening or closing a hand or foot.

KR 10-2013-0044430 A KR 10-2014-0071588 A KR 10-2008-0075269 A

Therefore, the present invention is to solve the above problems and to provide a robot that dances by learning the correlation between dance and music that matches the music that is heard by learning the frequency characteristics of music and choreography data in images. have.

The robot for dancing by learning the correlation between music and dance according to the present invention for achieving the above object has a human shape, and controls a part corresponding to each joint of a person with a motor 10 to dance. A possible robot (1), the microphone 100 for receiving music; A data storage unit 200 storing data learning the correlation between music and dance; Data input unit 300, the motor 10, and the microphone 100 to communicate with the data storage unit 200 wirelessly and wirelessly to input music or dance image data to be stored in the data storage unit 200 And a control unit 400 which is connected to the data storage unit 200 and controls to dance according to music received from the microphone 100.

The robot that dances by learning the correlation between music and dance according to the present invention has the effect of learning the correlation between music and physical movement choreography and dancing appropriately even when new music is heard.

1 is a configuration diagram of a robot that dances by learning the correlation between music and dance according to the present invention.
2 is a structure of a deep learning machine learning incorporating reinforcement learning and GAN learning the correlation between music and dance of a dancing robot by learning the correlation between music and dance according to the present invention.
3 is a tempo graph which is one of the extraction components of a robot dancing by learning the correlation between music and dance according to the present invention.
Figure 4 is a tone graph that is one of the extraction components of the dancing robot by learning the correlation between music and dance according to the present invention.
5 is a time signature graph that is one of the extraction components of a robot dancing by learning a correlation between music and dance according to the present invention.
6 is a graph showing designating coordinates according to joints as an example of a robot dancing by learning a correlation between music and dance according to the present invention.

Hereinafter, with reference to the drawings according to an embodiment of the present invention, this is for easier understanding of the present invention, and the scope of the present invention is not limited thereto.

When a part of the specification "includes" a certain component, this means that other components may be further included instead of excluding other components, unless specifically stated to the contrary. In addition, terms such as "... sub" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software. .

Throughout the specification, when a part is "connected" to another part, this includes not only "directly connected" but also "electrically connected" with another element in between. .

In the present specification, when one component'transmits' data or a signal to another component, the component may directly transmit the data or signal to another component, and through at least one other component This means that data or signals can be transmitted to other components.

Prior to the description, a number of aspects and embodiments are described herein, and these are merely illustrative and not limiting.

After reading this specification, skilled artisans will appreciate that other aspects and embodiments are possible without departing from the scope of the invention.

Before addressing the details of the embodiments described below, some terms will be defined or clarified.

GAN stands for Generative Adversarial Network, and is also known as a generative hostile neural network. In addition, it is an artificial intelligence algorithm used for unsupervised learning, and is implemented by two neural network systems competing with each other in a zero-sum game framework.

1 is a configuration diagram of a robot dancing by learning the correlation between music and dance according to the present invention, and FIG. 2 is a diagram showing the relationship between music and dancing according to the present invention and dancing robot It is a structure of deep learning machine learning that combines reinforcement learning to learn correlation and GAN, and FIG. 3 is a tempo graph that is one of extraction components of a robot dancing by learning the correlation between music and dance according to the present invention, Figure 4 is a tone graph that is one of the extraction components of the robot dancing by learning the correlation between music and dance according to the present invention, and Figure 5 is a robot dancing by learning the correlation between music and dance according to the present invention It is a time signature graph which is one of the extraction components of, and FIG. 6 is a graph showing designating coordinates according to joints as an example of a robot dancing by learning a correlation between music and dance according to the present invention.

The robot that dances by learning the correlation between music and dance according to the present invention is a robot (1) that is shaped like a person and controls a part corresponding to each joint of a person with a motor (10) to dance. In, The microphone 100 for receiving music; A data storage unit 200 storing data learning the correlation between music and dance; Data input unit 300, the motor 10, and the microphone 100 to communicate with the data storage unit 200 wirelessly and wirelessly to input music or dance image data to be stored in the data storage unit 200 And a control unit 400 which is connected to the data storage unit 200 and controls to dance according to music received from the microphone 100.

In addition, the microphone 100 and the music input unit further comprises a music feature extraction unit 500 for extracting the characteristics of music from the music input through the input unit 300, the music feature extraction unit 500 of the extracted music Characterized in that the feature is transmitted to the data storage unit 200.

In addition, the control unit 400 is connected to the control unit 400 further includes a camera 700 that can be controlled, and the dance image received through the data input unit 300 or taken through the camera 700 Further comprising a dance feature extraction unit 600 for extracting the characteristics of the dance from the dance of a dancer, the dance feature extraction unit 600 is characterized in that for transmitting the characteristics of the extracted music to the data storage unit 200 do.

Next, with reference to FIGS. 1 to 6 to learn the correlation between music and dance according to the present invention will be described in detail with respect to the dancing robot.

2 is a structure of deep learning machine learning that combines reinforcement learning and GAN to learn the correlation between music and dance, and learning data is streaming through the Internet such as Youtube when inputting a dance choreography video through the data input unit 300. From the dance video of the site, the music feature is extracted from the music feature extraction unit 500 and the dance feature is extracted from the dance feature extraction unit 600. The music is extracted by characterizing the tempo, time signature, and tone.

Next, to describe in detail when the robot dances, the control unit 400 may cause the robot to dance with the movement of the joint. At this time, a number may be assigned to each joint of the robot, and the movement of the joint may be explained through how much the angle of each joint is rotated about the xyz axis. Alternatively, the motion of the joint can be expressed by determining the reference position and where each joint is positioned relative to the reference position.

In addition, the human joint information (skeleton data) extracted using 3D Pose Estimation technique from the image is used as motion data for deep learning training. At this time, an open source tool can be used, and the open source tool also uses a deep learning technique to separate the human body from the image and estimate how each skeleton of the separated human body is located in the 3D space. Output as (vector) data. As shown in FIG. 6, the position values of these joints can be moved by expressing 17 joints with (x,y,z) coordinates.

Next, with reference to FIGS. 3 to 5, the correlation between music and dance according to the present invention will be described to describe in detail the music components extracted by the dancing robot.

It uses three types of extracting ingredients from music. The first is a tempo that tells you how fast to dance in the overall flow, the second is a beat that tells you what type of dance you should dance to, and finally the MFCC (voice), Spectral Centroid (sound of sound) that describes the details of the dance. Brightness), STFT (Fourier Transform at each frame), Chroma (Energy at each note), Constant Q-transform, and Mel-spectrogram.

Although described above with reference to the drawings according to embodiments of the present invention, those skilled in the art to which the present invention pertains will be able to perform various applications and modifications within the scope of the present invention based on the above.

1: Robot
10: motor
100: microphone
200: data storage
300: data input unit
400: control unit
500: Music feature extraction unit
600: Dance feature extraction unit
700: camera

Claims (3)

In the robot (1) that can be made to dance by controlling the part corresponding to each joint of the person by a motor (10) to form a person,
A microphone 100 for receiving music;
A data storage unit 200 storing data learning the correlation between music and dance;
A data input unit 300 for wired and wireless communication with the data storage unit 200 to input music or dance image data to be stored in the data storage unit 200 and
Characterized in that it comprises a control unit (400) connected to the motor (10), the microphone (100) and the data storage unit (200) to control dancing to the music received from the microphone (100). A robot that learns the relationship between music and dance and dances.
The method according to claim 1,
Further comprising a music feature extraction unit 500 for extracting the characteristics of the music from the music received through the microphone 100 and the data input unit 300,
The music feature extraction unit 500 is a robot that learns and dances the correlation between music and dance, characterized in that the characteristics of the extracted music are transmitted to the data storage unit 200.
The method according to claim 1,
It is connected to the control unit 400 further includes a camera 700 that can be controlled by the control unit 400,
Further comprising a dance feature extraction unit 600 for extracting the characteristics of the dance from the dance image received through the camera 700 or the dance image received through the data input unit 300,
The dance feature extracting unit 600 is a robot that learns and dances the correlation between music and dance, characterized in that the characteristics of the extracted music are transmitted to the data storage unit 200.

Images