KR20210063132A - Device and method for providing motion data of action robot optimized for contents - Google Patents

Abstract

An apparatus for providing motion data for an action robot according to an embodiment of the present invention comprises a communication unit, a memory, and a processor for obtaining data of content, which can be output through an action robot from the communication unit or the memory, obtaining interpolation data based on the characteristics of the content, and obtaining interpolated motion data by applying the interpolation data to motion data, which can be output through the action robot. The motion data comprises a plurality of snapshots indicating the pose of the action robot for each of different viewpoints. The interpolation data comprises at least one piece of interpolation characteristic information for setting the motion characteristics of the action robot for each of the sections between the plurality of snapshots. Accordingly, motion speed or pattern of the motion data may be set according to the characteristics of content so that reality on the motion provided by the action robot may be enhanced.

Description

DEVICE AND METHOD FOR PROVIDING MOTION DATA OF ACTION ROBOT OPTIMIZED FOR CONTENTS

The present invention relates to an apparatus and method for providing motion data for an action robot optimized for content to be output.

As robot technology develops, a method of building a robot by modularizing joints or wheels is being used. For example, by electrically and mechanically connecting and assembling a plurality of actuator modules constituting the robot, various types of robots such as dogs, dinosaurs, humans, and spiders can be made.

A robot that can be manufactured by assembling a plurality of actuator modules is generally referred to as a modular robot. Each actuator module constituting the modular robot is provided with a motor inside, so that the motion of the robot is executed according to the rotation of the motor. The robot motion is a concept that collectively refers to the robot's movements, such as motion and dance.

Recently, as entertainment robots stand out, interest in robots for entertainment or people's interest is increasing. For example, technologies are being developed that allow people to dance to music or to take motions or expressions according to stories (fairy tales, etc.).

In this case, the action robot performs the motion by presetting a number of motions suitable for music or fairy tales and executing the preset motions when music or a fairy tale is played from an external device.

SUMMARY OF THE INVENTION An object of the present invention is to provide an apparatus and method for generating optimal motion data according to characteristics of content.

Another object to be solved by the present invention is to provide an apparatus and method for generating motion data for improving the reality of motion output through an action robot.

The apparatus for providing motion data for an action robot according to an embodiment of the present invention obtains interpolation data based on characteristics of content outputable through the action robot, and obtains interpolation data for motion data outputable through the action robot. It can be applied to obtain interpolated motion data.

The motion data may include a plurality of snapshots representing the pose of the action robot for each of different viewpoints.

The interpolation data may include at least one piece of interpolation characteristic information for setting motion characteristics of the action robot for each of the sections between the plurality of snapshots.

According to an embodiment, the device extracts at least one feature point based on a signal characteristic of a signal included in the data of the content, and uses an analysis model learned based on machine learning to extract the characteristic of the content from the at least one feature point can be analyzed.

According to an embodiment, the content may be music content, and the device may analyze at least one of a structure, a mood, and a genre of the music content, and obtain interpolation data based on the analysis result.

According to an embodiment, the content may acquire the interpolated motion data by generating at least one virtual snapshot based on the interpolation data for a section between snapshots included in the motion data.

According to an embodiment, the device may be implemented as an action robot, and the device may provide motion through a figure by controlling at least one motor based on the interpolated motion data while outputting the content through a speaker. can

According to an embodiment of the present invention, an apparatus for providing motion data for an action robot sets a movement speed or pattern between snapshots (poses) of the motion data according to the characteristics of the content, so that the content output through the action robot is It can provide natural motion. Accordingly, the reality of the motion provided by the action robot may be improved.

In addition, the device performs interpolation on motion data according to interpolation characteristics set based on characteristics of content output through the action robot, thereby providing various types of motion using the same motion data to improve user satisfaction. can

1 shows an AI device including a robot according to an embodiment of the present invention.
2 shows an AI server connected to a robot according to an embodiment of the present invention.
3 shows an AI system including a robot according to an embodiment of the present invention.
4 is an exemplary perspective view of an action robot that outputs a motion based on motion data.
Fig. 5 is a block diagram showing an example of a control configuration of the action robot shown in Fig. 4;
6 is a diagram for explaining the structure of motion data according to an embodiment of the present invention.
7 is a flowchart for explaining an embodiment of a method for providing motion data according to an embodiment of the present invention.
8 to 12B are exemplary views related to the method of providing motion data shown in FIG. 7 .

Hereinafter, embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings. The accompanying drawings are only for making it easier to understand the embodiments disclosed in the present specification, and the technical idea disclosed in the present specification is not limited by the accompanying drawings, and all changes and equivalents included in the spirit and scope of the present invention It should be understood to include water or substitutes.

A robot may refer to a machine that automatically processes or operates a task given by its own capabilities. In particular, a robot having a function of recognizing the environment and performing an operation by self-determining may be referred to as an intelligent robot.

Robots can be classified into industrial, medical, household, military, etc. depending on the purpose or field of use.

The robot may be provided with a driving unit including an actuator or a motor to perform various physical operations such as moving a robot joint. In addition, the movable robot includes a wheel, a brake, a propeller, and the like in a driving unit, and can travel on the ground or fly in the air through the driving unit.

Artificial intelligence refers to the field of researching artificial intelligence or the methodology that can create it, and machine learning (Machine Learning) refers to the field of studying methodologies to define and solve various problems dealt with in the field of artificial intelligence. do. Machine learning is also defined as an algorithm that improves the performance of a task through continuous experience.

An artificial neural network (ANN) is a model used in machine learning, and may refer to an overall model with problem-solving capabilities, which is composed of artificial neurons (nodes) that form a network by combining synapses. The artificial neural network may be defined by a connection pattern between neurons of different layers, a learning process for updating model parameters, and an activation function for generating an output value.

The artificial neural network may include an input layer, an output layer, and optionally one or more hidden layers. Each layer includes one or more neurons, and the artificial neural network may include neurons and synapses connecting neurons. In an artificial neural network, each neuron can output a function of an activation function for input signals, weights, and biases input through synapses.

Model parameters refer to parameters determined through learning, and include weights of synaptic connections and biases of neurons. In addition, the hyperparameter refers to a parameter that must be set before learning in a machine learning algorithm, and includes a learning rate, number of iterations, mini-batch size, and initialization function.

The purpose of learning the artificial neural network can be seen as determining the model parameters that minimize the loss function. The loss function can be used as an index to determine an optimal model parameter in the learning process of the artificial neural network.

Machine learning can be classified into supervised learning, unsupervised learning, and reinforcement learning according to the learning method.

Supervised learning refers to a method of training an artificial neural network when a label for training data is given, and a label indicates the correct answer (or result value) that the artificial neural network must infer when training data is input to the artificial neural network. It can mean. Unsupervised learning may mean a method of training an artificial neural network in a state in which a label for training data is not given. Reinforcement learning may mean a learning method in which an agent defined in a certain environment learns to select an action or sequence of actions that maximizes the cumulative reward in each state.

Among artificial neural networks, machine learning implemented as a deep neural network (DNN) including a plurality of hidden layers is sometimes referred to as deep learning (deep learning), and deep learning is a part of machine learning. Hereinafter, machine learning is used in the sense including deep learning.

1 shows an AI device including a robot according to an embodiment of the present invention.

The AI device 100 includes a TV, a projector, a mobile phone, a smartphone, a desktop computer, a notebook computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation system, a tablet PC, a wearable device, and a set-top box (STB). ), a DMB receiver, a radio, a washing machine, a refrigerator, a desktop computer, a digital signage, a robot, a vehicle, and the like.

Referring to FIG. 1 , the AI device 100 includes a communication unit 110 , an input unit 120 , a learning processor 130 , a sensing unit 140 , an output unit 150 , a memory 170 and a processor 180 , etc. may include.

The communication unit 110 may transmit and receive data with external devices such as other AI devices 100a to 100e or the AI server 200 using wired/wireless communication technology. For example, the communication unit 110 may transmit and receive sensor information, a user input, a learning model, and a control signal with external devices.

At this time, communication technologies used by the communication unit 110 include Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), Long Term Evolution (LTE), 5G, Wireless LAN (WLAN), and Wireless-Fidelity (Wi-Fi). ), Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), ZigBee, and Near Field Communication (NFC).

The input unit 120 may acquire various types of data.

In this case, the input unit 120 may include a camera for inputting an image signal, a microphone for receiving an audio signal, and a user input unit for receiving information from a user. Here, by treating a camera or a microphone as a sensor, a signal obtained from the camera or a microphone may be referred to as sensing data or sensor information.

The input unit 120 may acquire training data for model training and input data to be used when acquiring an output by using the training model. The input unit 120 may obtain unprocessed input data, and in this case, the processor 180 or the running processor 130 may extract an input feature as a preprocess for the input data.

The learning processor 130 may train a model composed of an artificial neural network by using the training data. Here, the learned artificial neural network may be referred to as a learning model. The learning model can be used to infer a result value for new input data other than the training data, and the inferred value can be used as a basis for a decision to perform a certain operation.

In this case, the learning processor 130 may perform AI processing together with the learning processor 240 of the AI server 200.

In this case, the learning processor 130 may include a memory integrated or implemented in the AI device 100. Alternatively, the learning processor 130 may be implemented using the memory 170, an external memory directly coupled to the AI device 100, or a memory maintained in an external device.

The sensing unit 140 may acquire at least one of internal information of the AI device 100, information on the surrounding environment of the AI device 100, and user information by using various sensors.

At this time, the sensors included in the sensing unit 140 include a proximity sensor, an illuminance sensor, an acceleration sensor, a magnetic sensor, a gyro sensor, an inertial sensor, an RGB sensor, an IR sensor, a fingerprint recognition sensor, an ultrasonic sensor, an optical sensor, a microphone, and a lidar. , Radar, etc.

The output unit 150 may generate output related to visual, auditory or tactile sensations.

In this case, the output unit 150 may include a display unit outputting visual information, a speaker outputting auditory information, a haptic module outputting tactile information, and the like.

The memory 170 may store data supporting various functions of the AI device 100. For example, the memory 170 may store input data, learning data, a learning model, and a learning history acquired from the input unit 120.

The processor 180 may determine at least one executable operation of the AI device 100 based on information determined or generated using a data analysis algorithm or a machine learning algorithm. In addition, the processor 180 may perform the determined operation by controlling the components of the AI device 100.

To this end, the processor 180 may request, search, receive, or utilize data from the learning processor 130 or the memory 170, and perform a predicted or desirable operation among the at least one executable operation. The components of the AI device 100 can be controlled to run.

In this case, when connection of an external device is required to perform the determined operation, the processor 180 may generate a control signal for controlling the corresponding external device and transmit the generated control signal to the corresponding external device.

The processor 180 may obtain intention information for a user input and determine a user's requirement based on the obtained intention information.

In this case, the processor 180 uses at least one of a Speech To Text (STT) engine for converting a speech input into a character string or a Natural Language Processing (NLP) engine for obtaining intention information of a natural language. Intention information corresponding to the input can be obtained.

At this time, at least one or more of the STT engine and the NLP engine may be composed of an artificial neural network, at least partially trained according to a machine learning algorithm. And, at least one of the STT engine or the NLP engine is learned by the learning processor 130, learning by the learning processor 240 of the AI server 200, or learned by distributed processing thereof. Can be.

The processor 180 collects history information including user feedback on the operation content or operation of the AI device 100 and stores it in the memory 170 or the learning processor 130, or the AI server 200 Can be transferred to an external device. The collected history information can be used to update the learning model.

The processor 180 may control at least some of the components of the AI device 100 in order to drive the application program stored in the memory 170. Further, in order to drive the application program, the processor 180 may operate by combining two or more of the components included in the AI device 100 with each other.

2 shows an AI server 200 connected to a robot according to an embodiment of the present invention.

Referring to FIG. 2, the AI server 200 may refer to a device that trains an artificial neural network using a machine learning algorithm or uses the learned artificial neural network. Here, the AI server 200 may be configured with a plurality of servers to perform distributed processing, or may be defined as a 5G network. In this case, the AI server 200 may be included as a part of the AI device 100 to perform at least a part of AI processing together.

The AI server 200 may include a communication unit 210, a memory 230, a learning processor 240, a processor 260, and the like.

The communication unit 210 may transmit and receive data with an external device such as the AI device 100.

The memory 230 may include a model storage unit 231. The model storage unit 231 may store a model (or artificial neural network, 231a) being trained or trained through the learning processor 240.

The learning processor 240 may train the artificial neural network 231a using the training data. The learning model may be used while being mounted on the AI server 200 of an artificial neural network, or may be mounted on an external device such as the AI device 100 and used.

The learning model can be implemented in hardware, software, or a combination of hardware and software. When part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in the memory 230.

The processor 260 may infer a result value for new input data using the learning model, and generate a response or a control command based on the inferred result value.

3 shows an AI system 1 including a robot according to an embodiment of the present invention.

Referring to FIG. 3, the AI system 1 includes at least one of an AI server 200, a robot 100a, an autonomous vehicle 100b, an XR device 100c, a smartphone 100d, or a home appliance 100e. It is connected with this cloud network 10. Here, the robot 100a to which the AI technology is applied, the autonomous vehicle 100b, the XR device 100c, the smartphone 100d, or the home appliance 100e may be referred to as the AI devices 100a to 100e.

The cloud network 10 may constitute a part of the cloud computing infrastructure or may mean a network that exists in the cloud computing infrastructure. Here, the cloud network 10 may be configured using a 3G network, a 4G or Long Term Evolution (LTE) network, or a 5G network.

That is, the devices 100a to 100e and 200 constituting the AI system 1 may be connected to each other through the cloud network 10. In particular, the devices 100a to 100e and 200 may communicate with each other through a base station, but may directly communicate with each other without through a base station.

The AI server 200 may include a server that performs AI processing and a server that performs an operation on big data.

The AI server 200 includes at least one of a robot 100a, an autonomous vehicle 100b, an XR device 100c, a smartphone 100d, or a home appliance 100e, which are AI devices constituting the AI system 1 It is connected through the cloud network 10 and may help at least part of the AI processing of the connected AI devices 100a to 100e.

In this case, the AI server 200 may train an artificial neural network according to a machine learning algorithm in place of the AI devices 100a to 100e, and may directly store the learning model or transmit it to the AI devices 100a to 100e.

At this time, the AI server 200 receives input data from the AI devices 100a to 100e, infers a result value for the received input data using a learning model, and generates a response or control command based on the inferred result value. It can be generated and transmitted to the AI devices 100a to 100e.

Alternatively, the AI devices 100a to 100e may infer a result value for input data using a direct learning model, and may generate a response or a control command based on the inferred result value.

Hereinafter, various embodiments of the AI devices 100a to 100e to which the above-described technology is applied will be described. Here, the AI devices 100a to 100e illustrated in FIG. 3 may be viewed as a specific example of the AI device 100 illustrated in FIG. 1.

The robot 100a is applied with AI technology and may be implemented as a guide robot, a transport robot, a cleaning robot, a wearable robot, an entertainment robot, a pet robot, an unmanned flying robot, and the like.

The robot 100a may include a robot control module for controlling an operation, and the robot control module may refer to a software module or a chip implementing the same as hardware.

The robot 100a acquires status information of the robot 100a by using sensor information acquired from various types of sensors, detects (recognizes) the surrounding environment and objects, generates map data, or moves and travels. You can decide on a plan, decide on a response to user interaction, or decide on an action.

Here, the robot 100a may use sensor information obtained from at least one sensor among a lidar, a radar, and a camera in order to determine a moving route and a driving plan.

The robot 100a may perform the above-described operations using a learning model composed of at least one artificial neural network. For example, the robot 100a may recognize a surrounding environment and an object using a learning model, and may determine an operation using the recognized surrounding environment information or object information. Here, the learning model may be directly learned by the robot 100a or learned by an external device such as the AI server 200.

At this time, the robot 100a may perform an operation by generating a result using a direct learning model, but it transmits sensor information to an external device such as the AI server 200 and performs the operation by receiving the result generated accordingly. You may.

The robot 100a determines a movement route and a driving plan using at least one of map data, object information detected from sensor information, or object information obtained from an external device, and controls the driving unit to determine the determined movement path and travel plan. Accordingly, the robot 100a can be driven.

The map data may include object identification information on various objects arranged in a space in which the robot 100a moves. For example, the map data may include object identification information on fixed objects such as walls and doors and movable objects such as flower pots and desks. In addition, the object identification information may include a name, type, distance, and location.

In addition, the robot 100a may perform an operation or run by controlling the driving unit based on the user's control/interaction. In this case, the robot 100a may acquire interaction intention information according to a user's motion or voice speech, and determine a response based on the acquired intention information to perform the operation.

4 is an exemplary perspective view of an action robot that outputs a motion based on motion data.

Referring to FIG. 4 , the action robot 400 may include a figure 401 and a base 402 supporting the figure 401 from the lower side.

The figure 401 may have a shape substantially similar to a human body.

The figure 401 may include a head 403 , bodies 404 and 406 , and an arm 405 . The figure 401 may further include a foot 407 and a sub-base 408 .

The head 403 may have a shape corresponding to a human head. The head 403 may be connected to an upper portion of the body 404 .

The bodies 404 and 406 may have a shape corresponding to a human body. The bodies 404 and 406 are fixed and may not move. Inside the body (404, 406), a space in which various parts are built may be formed.

The body may include a first body 404 and a second body 406 .

The inner space of the first body 404 and the inner space of the second body 406 may communicate with each other.

The first body 404 may have a shape corresponding to the upper body of a person. The first body 404 may be referred to as an upper body. An arm 405 may be connected to the first body 404 .

The second body 406 may have a shape corresponding to the lower body of a person. The second body 406 may be referred to as a lower body. The second body 406 may include a pair of legs.

The first body 404 and the second body 406 may be detachably fastened to each other. Accordingly, not only the assembly of the body is simplified, but also parts disposed inside the body can be easily maintained.

Arms 405 may be connected to both sides of the body.

In more detail, the pair of arms 405 may be respectively connected to shoulders located on both sides of the first body 404 . The shoulder may be included in the first body 404 . Shoulders may be located on both sides of the upper portion of the first body (404).

The arm 405 may be rotatable with respect to the first body 404 , more specifically, the shoulder. Accordingly, the arm 405 may be referred to as a movable part.

The pair of arms 405 may include a right arm and a left arm. The right and left arms can move independently of each other.

The foot 407 may be connected to the lower part of the second body 406 , that is, the lower end of the leg. The foot 407 may be supported by a sub-base 408 .

The sub-base 408 may be fastened to at least one of the second body 406 and the foot 407 . The sub-base 408 may be seated and coupled to the base 402 on the upper side of the base 402 .

The sub-base 408 may have a substantially disk shape. The sub-base 408 can rotate relative to the base 402 . Accordingly, the entire figure 401 can rotate with respect to the sub-base 408 .

The base 402 may support the figure 401 from the lower side. In more detail, the base 402 may support the sub-base 408 of the figure 401 from the lower side. The sub-base 408 may be detachably coupled to the base 402 .

Inside the base 402, a processor 480 (see FIG. 5) for controlling the overall operation of the action robot 1, a battery (not shown) for storing power required for the operation of the action robot 1, and a figure ( A figure driving unit 460 (refer to FIG. 5) that operates the 401 may be built-in. In addition, a speaker 452 (refer to FIG. 5 ) for emitting sound may be disposed inside the base 402 . According to an embodiment, an input unit 420 such as a button (refer to FIG. 5 ) or a display 454 (refer to FIG. 5 ) for outputting various information in a visual form may be disposed on one surface of the base 402 .

Fig. 5 is a block diagram showing an example of a control configuration of the action robot shown in Fig. 4;

Referring to FIG. 5 , the action robot 400 may include a communication unit 410 , an input unit 420 , an output unit 450 , a figure driving unit 460 , a memory 470 , and a processor 480 . The configurations shown in FIG. 5 are examples for convenience of description, and the action robot 400 may include more or fewer configurations than those shown in FIG. 5 .

The components shown in FIG. 5 may be provided on the base 402 of the action robot 400 . That is, the base 402 constitutes the main body of the action robot 400 , and the figure 401 is detached from the base 402 , so that the action robot 400 can be implemented as a kind of modular robot.

Meanwhile, the contents related to the AI device 100 of FIGS. 1 to 2 may be similarly applied to the action robot 400 of the present invention. That is, the communication unit 410 , the input unit 420 , the output unit 450 , the memory 470 , and the processor 480 are the communication unit 110 , the input unit 120 , and the output unit 150 shown in FIG. 1 . , the memory 170 , and the processor 180 .

The communication unit 410 may include communication modules for connecting the action robot 400 to a server or a terminal through a network. Each of the communication modules may support any one of the communication technologies described above with reference to FIG. 1 .

For example, the action robot 400 may be connected to a network through an access point such as a router. Accordingly, the action robot 400 may receive various information, data, and contents from a server or a terminal through the network.

The input unit 420 may include at least one input means for acquiring an input or command related to the operation of the action robot 400 or acquiring various types of data. For example, the at least one input means may include a physical input means such as a button 422 or a dial, a touch input unit 424 such as a touch pad or a touch panel, a microphone 426 for receiving a user's voice, and the like. .

Meanwhile, the processor 480 may transmit the user's voice data received through the microphone 426 to the server through the communication unit 410 . The server may analyze the voice data to recognize a startup word, a command, a request, etc. in the voice data, and provide the recognition result to the action robot 400 .

The server may be implemented as the AI server 200 described above in FIG. 2 , and in this case, the server uses a model (artificial neural network 231a) learned through the learning processor 240. Startup words, commands, requests, etc. can be recognized. The processor 480 may process a command or request included in the voice based on the recognition result.

According to an embodiment, the processor 480 may directly recognize a startup word, a command, a request, etc. in the voice data through a model learned by the learning processor in the action robot 400 . That is, the learning processor may use the voice data received through the microphone 426 as training data to train the model composed of the artificial neural network.

Alternatively, the processor 480 may receive data corresponding to the learned model from the server, store it in the memory 470 , and recognize a startup word, a command, a request, etc. in the voice data through the stored data.

The output unit 450 includes various types of information or messages or various types of multimedia content (eg, music, children's stories, education) related to the operation or state of the action robot 400, various services, programs, applications, etc. executed in the action robot 400. content, etc.). For example, the output unit 450 may include a speaker 452 and a display 454 .

The speaker 452 may output the various information, messages, or contents in the form of voice or sound.

The display 454 may output the above-described various information or messages in graphic form. According to an embodiment, the display 454 may be implemented in the form of a touch screen together with the touch input unit 424 . In this case, the display 454 may function not only as an output unit but also as an input unit.

The figure driving unit 460 may provide motion (action) through the figure 401 by operating the figure 401 mounted on the base 402 .

For example, the figure driving unit 460 may include a servo motor or a plurality of motors. As another example, the figure driving unit 460 may include an actuator.

The processor 480 may receive motion data for controlling the figure driving unit 460 through the motion data providing device. The motion data providing device may include various computing devices such as a server, a terminal, and other action robots.

The processor 480 may provide a motion (or action) of the figure 401 corresponding to the motion data by controlling the figure driver 460 based on the received motion data.

The memory 470 performs an operation based on control data for controlling the operation of the components included in the action robot 400 , an input obtained through the input unit 420 or a command or request obtained through the communication unit 410 , etc. Various data, such as data for the purpose, may be stored.

In addition, program data such as a software module or an application executed by at least one processor or controller included in the processor 480 may be stored in the memory 470 .

In addition, at least one multimedia content and at least one motion data may be stored in the memory 470 . According to an embodiment, mapping information for motion data mapped to each multimedia content may be further stored in the memory 470 . In this case, when a request to output a predetermined multimedia content is received, the processor 480 may load motion data mapped to the multimedia content based on the mapping information. The processor 480 may provide a motion corresponding to the motion data by controlling the figure driver 460 based on the loaded motion data. That is, the multimedia content and the mapped motion data may constitute the above-described action robot content.

The memory 470 may include various storage devices such as ROM, RAM, EEPROM, flash drive, and hard drive in terms of hardware.

The processor 480 may control the overall operation of the action robot 400 . For example, the processor 480 may include at least one CPU, an application processor (AP), a microcomputer (or microcomputer), an integrated circuit, an application specific integrated circuit (ASIC), and the like.

The processor 480 may control the output unit 450 to output multimedia content.

In addition, the processor 480 may control the figure driving unit 460 so that the figure 401 takes a predetermined motion or performs an action during the output of the multimedia content or regardless of the output of the multimedia content.

In addition, when the figure 401 is mounted on the base 402, the processor 480 may recognize the mounted figure 401 and transmit identification information of the figure 401 to a terminal or a server. Accordingly, when providing the motion data, the terminal or the server may provide motion data corresponding to the type of the figure 401 recognized based on the identification information.

A structure of motion data according to an example of the present invention will be described below with reference to FIG. 6 .

6 is a diagram for explaining the structure of motion data according to an embodiment of the present invention.

For example, it is assumed that a first pose performed at a first time point and a second pose performed at a second time point when a predetermined time has elapsed from the first time point (different from the first pose) exist. In this case, in order for a person or the like to take the first pose and the second pose, the first pose is taken at the first time point, and then a movement is performed to change from the first pose to the second pose between the first time point and the second time point. By doing so, the second pose can be taken at the second time point. That is, the motion may include a plurality of poses and movement between the plurality of poses.

Based on this, referring to FIG. 6 , the motion data MOTION_DATA may include a plurality of snapshots Snapshot1, Snapshot2, Snapshot3, …. Each of the plurality of snapshots may correspond to any one pose and may be matched to any one output time point.

According to an embodiment, the motion data MOTION_DATA may include a plurality of partial motions Template1, Template2, Template3, ...). The plurality of partial motions may include a plurality of snapshots, and a plurality of some of the plurality of partial motions may be included in motion data MOTION_DATA.

That is, the action robot 400 may provide motion based on the output timing of each of the plurality of snapshots included in the motion data MOTION_DATA. The processor 480 may control the figure driver 460 to provide a snapshot (pose) corresponding to a predetermined output time point when the motion is provided.

Meanwhile, motion characteristics (speed, pattern, etc.) of the action robot 400 for changing between snapshots may vary. However, in the conventional case, since the user has to directly set or change the movement characteristic, it causes inconvenience to the user, and as one movement characteristic (eg, constant velocity movement) is automatically applied in one batch, the sense of realism is reduced and harmony with the content There is a problem that is not done.

To solve this, embodiments to be described later in this specification set the motion characteristics according to the characteristics of content (eg, music, etc.) provided with the motion of the action robot 400 or recommend appropriate motion characteristics to the user, It will be possible to improve the realism of motion and maximize user satisfaction.

Hereinafter, an embodiment of the present invention related thereto will be described with reference to FIGS. 7 to 12B.

7 is a flowchart for explaining an embodiment of a method for providing motion data according to an embodiment of the present invention.

Hereinafter, the embodiments disclosed in this specification may be performed by an apparatus providing motion data (hereinafter, referred to as a 'motion data providing apparatus'). And, the operation of processing the motion data to output the motion and output the content may be performed by the action robot 400 .

The motion data providing device may include a computing device such as a server, a PC, a smart phone, or a tablet PC.

According to an embodiment, the motion data providing apparatus may include an action robot 400 . In this case, all operations according to an embodiment of the present invention may be processed by the action robot 400 .

Meanwhile, the motion data providing device may be implemented as the AI device 100 shown in FIG. 1 or the AI server 200 shown in FIG. 2 . That is, the features related to the AI device 100 or the AI server 200 described above in FIG. 1 or FIG. 2 may be applied to the motion data providing device as well.

Based on this, referring to FIG. 7 , the motion data providing apparatus may acquire content data ( S700 ).

The content is content to be output with motion by the action robot 400 , and may correspond to audio content such as music and a moving picture, and the data may include an audio signal.

The motion data providing device may acquire data of the content (such as a sound source file) from various computing devices, such as a content provider, through a network. Alternatively, the processor 180 , 260 , or 480 of the motion data providing apparatus may load data of the content from the memory 170 , 230 , or 470 .

The motion data providing apparatus may analyze the characteristics of the content from the acquired data (S710). Based on the analysis result of the characteristics of the content, the motion data providing apparatus may obtain interpolation data for the content ( S720 ).

The processor 180 , 260 , or 480 may analyze the characteristic of the content based on the signal characteristic of the audio signal included in the acquired data.

For example, when the content is music content, the processor 180 , 260 , or 480 analyzes the structure (or format) of the music based on the signal characteristics of the audio signal included in the data, and determines the mood and genre of the music. can be analyzed.

The processor 180 , 260 , or 480 may acquire interpolation data based on the analysis result. The interpolation data may include information on interpolation characteristics for each section of the content. At least some of the plurality of snapshots included in the motion data may correspond to the section.

The interpolation characteristic may indicate a motion characteristic (speed and/or pattern) between the snapshots described above with reference to FIG. 6 .

Contents related to steps S710 to S720 will be described in more detail later with reference to FIGS. 8 to 9B .

The processor 180, 260, or 480 outputs the interpolation data through an output unit (eg, a display, etc.) or transmits it to the user's terminal through the communication unit 110, 210, or 410, thereby providing the user with the content. Interpolation characteristics for motion data to be provided can be recommended. The user may obtain interpolated motion data by setting interpolation characteristics for the motion data based on the interpolation data.

According to an embodiment, the motion data providing apparatus may automatically perform interpolation on the motion data using the obtained interpolation data (S730).

As described above, the interpolation data includes information on interpolation characteristics for each section of the content, and at least some of a plurality of snapshots included in the motion data may correspond to the section.

The processor 180 , 260 , or 480 may perform interpolation on each of the sections between the plurality of snapshots based on the interpolation data. For example, when the interpolation characteristic corresponding to the interval between the first snapshot and the second snapshot (the first interval) and the interval between the second snapshot and the third snapshot (the second interval) is the first characteristic, The processor 180 , 260 , or 480 may perform interpolation on the first interval and the second interval according to the first characteristic.

For example, the processor 180 , 260 , or 480 may perform the interpolation by generating a virtual snapshot corresponding to each of at least one time point within a section between snapshots based on the interpolation characteristic. In this case, the interpolated motion data may include a plurality of previously included snapshots and a plurality of virtual snapshots.

When the motion data providing device is a computing device connected to the action robot 400 , the processor 180 or 260 may control the communication unit 110 or 210 to transmit the interpolated motion data to the action robot 400 . According to an embodiment, the motion data providing apparatus may transmit the above-described content data to the action robot 400 in step S710 .

The action robot 400 may provide motion and content by processing the received interpolated motion data, outputting a motion, and outputting the content through the output unit 450 (eg, the speaker 452).

On the other hand, when the motion data providing device is implemented as the action robot 400, the motion data providing device processes the interpolated motion data to output a motion, and outputs the content through the output unit 450 (eg, the speaker 452). can do.

That is, the action robot 400 may provide a natural motion for content output together by outputting a motion based on the motion data interpolated by the motion data providing device.

Hereinafter, the embodiment of FIG. 7 will be described in more detail with reference to FIGS. 8 to 12B.

8 to 12B are exemplary views related to the method of providing motion data shown in FIG. 7 .

Hereinafter, it is assumed that the motion data providing device is the AI device 100 of FIG. 1 .

Referring to FIG. 8 , an apparatus for providing motion data may include a module 500 in which a method for providing motion data is implemented. The module 500 may generate interpolation data IP_DATA based on content and perform interpolation on motion data MOTION_DATA based on the generated interpolation data IP_DATA.

The module 500 may be implemented by hardware, software, or a combination of hardware and software, and the processor 180 of the motion data providing apparatus performs the above-described motion data providing method by controlling the operation of the module 500 . can be done

According to an embodiment, a plurality of modules 500 may exist. In this case, each of the plurality of modules may generate interpolation data for a corresponding type of content and perform interpolation on motion data.

For example, the module 500 corresponding to the music content may include a content structure analyzer 510 , a mood/genre analyzer 520 , and an interpolation data generator 530 .

When the data MUSIC_DATA of the music content is obtained, the processor 180 may analyze the structure (or format) of the music from the data MUSIC_DATA through the content structure analyzer 510 .

The structure (form) of music is defined according to the relationship between the passages included in the music, and there are various types such as AA'BA, ABBA, ABCA, etc., which correspond to already known knowledge, so additional explanations will be omitted. do it with

The processor 180 may analyze whether the music content data (MUSIC_DATA) is similar in units of passages (eg, large passages of 8 bars) through the content structure analyzer 510 .

For example, the content structure analyzer 510 may include hardware or software in which a Mel Frequency Cepstral Coefficient (MFCC) technique is implemented. Accordingly, the processor 180 may extract features from the music content data MUSIC_DATA in units of passages (or units of frames) according to the MFCC technique.

Also, the content structure analyzer 510 may further include hardware or software for determining the similarity of feature points extracted in units of passages. For example, the content structure analyzer 510 may determine the similarity of the feature points through a technique such as a self-similarity matrix.

The processor 180 may acquire information on the structure (format) of the music content based on the determined similarity.

For example, when the degree of similarity between the first section and the second section is equal to or greater than the first reference degree of similarity, the processor 180 may determine that the first section and the second section are the same. Also, when the similarity between the first clause and the second clause is equal to or greater than the second reference similarity that is lower than the first reference similarity, the processor 180 may determine that the first clause and the second clause are similar. On the other hand, when the similarity between the first section and the second section is lower than the second reference degree of similarity, the processor 180 may determine that the first section and the second section are different. The processor 180 may obtain information on the structure of the music content by determining the similarity between passages.

Also, the processor 180 may analyze the music mood and genre from the data MUSIC_DATA through the mood/genre analyzer 520 .

The mood/genre analyzer 520 may include hardware or software (such as an algorithm) for extracting a plurality of feature points (or feature vectors) from the data (MUSIC_DATA) of the music content. For example, the feature points may be extracted according to the above-described MFCC, Root-Mean-Square Error (RMSE), Zero Crossing Rate, Spectral Centroid, Chroma, Spectral Flux, Spectral Roll off point, and the like.

Also, the mood/genre analyzer 520 may further include an analysis model for acquiring information on each mood and genre of the music content from the extracted feature points. The analysis model may include an artificial neural network trained and built based on machine learning (eg, deep learning).

According to an embodiment, the processor 180 divides the data (MUSIC_DATA) into a plurality of partial data (eg, into segments of a section unit), and uses the mood/genre analyzer 520 for each of the divided segments. analysis process can be performed. In this case, the mood and genre information for the music content may include mood information and genre information in units of passages.

The processor 180 may generate interpolation data IP_DATA for the music content through the interpolation data generator 530 . The interpolation data generator 530 may generate the interpolation data IP_DATA based on at least one of information on the structure of music content, mood information, and genre information.

In this regard, referring to FIG. 9A , the processor 180 receives music content data (MUSIC_DATA), information related to the structure of the music content (eg, AA'BA structure), and mood information (eg, Arousing, Angry, Exciting, Powerful). , and genre information (eg, Hip-hop, Electronic) may be acquired.

The analysis model (eg, the first analysis model) included in the mood/genre analyzer 520 is any one included in the mood table MOOD_TABLE shown in FIG. 9A based on the feature points extracted from the data MUSIC_DATA. You can output mood information in units of passages. In addition, the analysis model included in the mood/genre analyzer 520 (eg, a second analysis model that is the same as or different from the first analysis model) includes at least one genre information included in the genre table GENRE_TABLE shown in FIG. 9A . can be printed out.

Referring to FIG. 9B , the processor 180 may obtain interpolation data IP_DATA from the obtained information through the interpolation data generator 530 .

The processor 180 may acquire at least one interpolation characteristic corresponding to genre information of the music content from among the plurality of interpolation characteristics. Information on the interpolation characteristic corresponding to the genre information may be stored in the memory 170 .

For example, when the acquired genre information corresponds to “Hip-hop” and “Electronic”, the processor 180 acquires interpolation characteristics corresponding to “Exponential”, “Constant”, “Logarithmic”, and “Beizer/Continuous” can do. The form of the obtained interpolation characteristics will be described later with reference to FIG. 11 .

The processor 180 may set any one of the obtained interpolation characteristics for each of the passages (segments) of the music content, based on the information on the structure of the music content and the mood information.

Although not shown, information on at least one interpolation characteristic matching each of the structures of music content and various moods may be stored in the memory 170 .

The processor 180 may set an interpolation characteristic for each of the passages based on the interpolation characteristics obtained based on the stored information, the structure information, the mood information, and the genre information.

For example, as shown in FIG. 9B , the processor 180 may obtain the interpolation data IP_DATA in which the interpolation characteristics are set in the order of “Exponential”, “Logarithmic”, “Bezier/Continuous”, and “Constant”.

That is, among the sections of the music content, the “Exponential” property is set to the first section, the “Logarithmic” property is set to the second section, the “Bezier/Continuous” property is set to the third section, and the fourth section is set A “Constant” property may be set.

11 (a) is an example of a graph showing the “Exponential” characteristic. According to the “Exponential” characteristic, the movement speed of the action robot 400 may increase as it approaches the time of the next snapshot (Snapshot n+1) from the time of the current snapshot (Snapshot n).

11 (b) is an example of a graph showing the “Logarithmic” characteristic. According to the “Logarithmic” characteristic, the movement speed of the action robot 400 may become slower as it approaches the time of the next snapshot (Snapshot n+1) from the time of the current snapshot (Snapshot n).

11 (c) is an example of a graph showing the “Bezier/Continuous” characteristic. According to the “Bezier/Continuous” characteristic, the movement speed of the action robot 400 is an intermediate point between the current snapshot n and the next snapshot n+1 from the point of the current snapshot n. , and may become slower from the intermediate time point to the time point of the next snapshot (Snapshot n+1).

11 (d) is an example of a graph showing the “Constant” characteristic. According to the “Constant” characteristic, the action robot 400 maintains the pose of the current snapshot (Snapshot n) until a point close to the time of the next snapshot (Snapshot n+1), and then abruptly maintains the pose of the next snapshot (Snapshot n+1). It can be changed to pose n+1).

The processor 180 may output the obtained interpolation data IP_DATA through the output unit 150 or transmit it to the user's terminal or the action robot 400 through the communication unit 110 .

Referring back to FIG. 8 , according to an embodiment, the module 500 may further include an interpolator 540 that performs interpolation on the motion data MOTION_DATA based on the interpolation data IP_DATA. The processor 180 may acquire interpolated motion data IP_MOTION_DATA according to a result of the interpolation.

In this regard, referring to FIGS. 10 to 11 , the processor 180 (Interpolator 540 ) performs continuous snapshots included in the motion data MOTION_DATA based on the interpolation data IP_DATA, Snapshot n, At least one virtual snapshot (Snapshot x) for a section between snapshots n+1) may be generated.

That is, the processor 180 may generate at least one virtual snapshot (Snapshot x) by reflecting the interpolation characteristic of the interval between snapshots. Accordingly, the interpolated motion data IP_MOTION_DATA obtained may include snapshots of the motion data MOTION_DATA and the generated virtual snapshots.

The action robot 400 may provide a motion in which the interpolation data is reflected by providing a motion based on snapshots and virtual snapshots included in the interpolated motion data IP_MOTION_DATA.

12A to 12B, FIG. 12A shows an example of providing motion using interpolated motion data according to the “Bezier/Continuous” characteristic, and FIG. 12B shows motion using interpolated motion data according to the “Constant” characteristic. shows an example of

When the first snapshot included in the motion data MOTION_DATA corresponds to the first time point T1 and the second snapshot that is the next snapshot of the first snapshot corresponds to the third time point T3, the first time point A virtual snapshot included between the first snapshot and the second snapshot of the interpolated motion data IP_MOTION_DATA may correspond to the second time point T2 between the T1 and the third time point T3 .

The processor 480 of the action robot 400 outputs a pose corresponding to the first snapshot at a first time point T1, a pose corresponding to the virtual snapshot at a second time point T2, and a second time point T2. The figure driver 460 may be controlled so that the pose corresponding to the snapshot is output at the third time point T3.

Referring to FIG. 12A , in the case of interpolation according to the “Bezier/Continuous” characteristic, the pose corresponding to the virtual snapshot may correspond to an intermediate pose between the pose of the first snapshot and the pose of the second snapshot. Specifically, the position of the arm 405a of the figure 401a at the second time point T2 is the position of the arm 405a at the first time point T1 and the arm 405a at the third time point T3. may correspond to an intermediate position of the position of

On the other hand, referring to FIG. 12B , in the case of interpolation according to the “Constant” characteristic, the pose of the virtual snapshot may be the same as the pose of the first snapshot. Specifically, the position of the arm 405a of the figure 401a at the second time point T2 may be the same as the position of the arm 405a at the first time point T1 .

That is, various types of motions using the same motion data MOTION_DATA may be provided by the interpolation characteristic set based on the characteristics of the output content.

The above description is merely illustrative of the technical idea of the present invention, and those of ordinary skill in the art to which the present invention pertains will be able to make various modifications and variations without departing from the essential characteristics of the present invention.

Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but to explain the technical idea, and the scope of the technical idea of the present invention is not limited by these embodiments.

The scope of protection of the present invention should be construed by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

Claims (18)

Communication department;
Memory; and
Obtaining data of content that can be output through the action robot from the communication unit or the memory,
Obtaining interpolation data based on the characteristics of the content,
A processor for obtaining interpolated motion data by applying the interpolation data to motion data outputable through the action robot,
The motion data includes a plurality of snapshots representing the pose of the action robot for each of different viewpoints,
The interpolation data is
An apparatus including at least one interpolation characteristic information for setting the motion characteristic of the action robot for each of the sections between the plurality of snapshots. The method of claim 1,
The content includes audio content,
The processor,
obtaining information about the characteristics of the content from the signal characteristics of an audio signal included in the data of the content;
An apparatus for obtaining the interpolation data based on information on the characteristics of the content. The method of claim 2,
The processor,
extracting at least one feature point based on the signal characteristic of the audio signal,
An apparatus for obtaining information on the characteristics of the content from the extracted at least one feature point using an analysis model learned based on machine learning. The method of claim 2,
The processor,
An apparatus for dividing the audio signal into at least one section, and obtaining the at least one interpolation characteristic information for each of the divided sections. The method of claim 2,
The content includes music content,
The information on the characteristics of the music content includes at least one of a structure, a mood, and a genre of the music content. The method of claim 5,
The memory is
storing matching data for at least one interpolation characteristic information matching at least one of a structure, a mood, and a genre of the music content;
The processor,
An apparatus for obtaining the at least one interpolation characteristic information matching the information on the characteristic of the music content, based on the matching data. The method of claim 1,
The motion characteristic is an apparatus including at least one of a movement speed and a pattern of a joint included in the action robot. The method of claim 1,
The processor,
Based on interpolation characteristic information for a section between the first snapshot and the second snapshot included in the motion data, at least one virtual snapshot included in the section is generated,
An apparatus for acquiring the interpolated motion data including the motion data and the generated at least one virtual snapshot. The method of claim 8,
The processor,
A device for controlling the communication unit to transmit the interpolated motion data to the action robot. The method of claim 8,
the device is the action robot;
The action robot includes a figure including at least one joint, and at least one motor for movement of the at least one joint,
The processor,
output the content through a speaker,
An apparatus for controlling the at least one motor based on the interpolated motion data. A method for providing motion data of an action robot including a figure having at least one joint, the method comprising:
acquiring data of audio content outputable through a speaker of the action robot;
obtaining at least one piece of characteristic information about the audio content based on the data;
obtaining interpolation data based on the obtained at least one piece of characteristic information; and
Comprising the step of obtaining interpolated motion data by applying the interpolation data to the motion data of the motion provided through the figure of the action robot,
The motion data includes a plurality of snapshots representing the pose of the figure for each of different viewpoints,
The interpolation data is
A method including at least one interpolation characteristic information for setting the movement characteristic of the figure for each of the sections between the plurality of snapshots. The method of claim 11,
The acquiring of the at least one characteristic information includes:
A method of obtaining the at least one characteristic information from a signal characteristic of an audio signal included in the data of the audio content. The method of claim 12,
The acquiring of the at least one characteristic information includes:
extracting at least one feature point based on the signal characteristic of the audio signal; and
Using an analysis model learned based on machine learning, the method comprising the step of obtaining the at least one characteristic information from the extracted at least one characteristic point. The method of claim 12,
The acquiring of the at least one characteristic information includes:
dividing the audio signal into at least one section; and
Comprising the step of obtaining at least one piece of characteristic information for each of the divided sections,
Each of the at least one interpolation characteristic information corresponds to any one of the at least one section.
Way. The method of claim 12,
The content includes music content,
The information on the characteristics of the music content includes at least one of a structure, a mood, and a genre of the music content. The method of claim 11,
The movement characteristic of the figure includes at least one of a movement speed and a pattern of a joint. The method of claim 11,
The step of obtaining the interpolated motion data comprises:
generating at least one virtual snapshot included in the section based on interpolation characteristic information for a section between the first snapshot and the second snapshot included in the motion data; and
and acquiring the interpolated motion data including the motion data and the generated at least one virtual snapshot. The method of claim 17,
outputting the audio content through the speaker; and
Based on the interpolated motion data, the method further comprising the step of controlling at least one motor for the movement of the at least one joint.

Images