KR20080057400A - Motion and facial expression composing device and method in human type robot - Google Patents

Abstract

An apparatus and a method for synthesizing facial expressions and operations of a humanoid robot are provided to employ a motor mediator to mediate outputs made by at least one behavior and a behavior-based controller to select and perform desired operations and facial expressions with respect to a sensor input of a robot and an upper-level task output. An apparatus for synthesizing facial expression of a robot includes a sensor module(100), a recognition module(300), a scenario module(400), a task module(500), a behavior-based controller(600), and a motor mediator(700). The sensor module receives information about a real environment to convert the information into a voltage processed by the robot. The recognition module extracts necessary information from various types of information. The scenario module makes user commands into a scenario so that the robot sequentially performs the user commands. The task module processes tasks divided into unit operations. The controller selects and performs an adjustable operation and facial expression. The motor mediator mediates colliding outputs.

Description

Motion and facial expression composing device and method in human type robot}

1 is a block diagram and data flow for expressing the operation and facial expression of a robot according to an embodiment of the present invention.

2A illustrates a detailed structure of an operation container inside an action-based controller according to an embodiment of the present invention.

2B is a diagram illustrating a detailed structure of an expression container inside an action-based controller according to an embodiment of the present invention.

3 is a diagram illustrating a detailed structure of a motor arbiter according to an embodiment of the present invention.

4 is a flow chart for loading an action file of the present invention into an action container.

5 is a flow chart for executing an action in an action container of the present invention.

6 is a flowchart illustrating execution of an operation behavior selected in FIG. 5.

The present invention uses an action-based controller that selects and executes an appropriate action and expression for the sensor input of the robot and the output of the upper task, and the motion of the humanoid robot using a motor arbiter that mediates the output generated by simultaneously performing one or more behaviors. And an apparatus and method for synthesizing facial expressions.

Unlike conventional industrial robots, the range of use of robots has recently been widened to an environment where humans live. In line with this trend, researches on robots that help human life in real environments are actively conducted.

Unlike conventional industrial robots, such robots are required to interact with humans more importantly. Representative examples of the humanoid robots include Kismet developed at the MIT AI Lab and Actroid developed by Professor Ishiguro University of Osaka, Japan.

However, Kismet has various researches on the interaction between robots and humans, but there is a substantial difference from the present invention in that it is not a human face but a face shape for a living creature and no body. On the other hand, Actroid has a body and face similar to humans, but it is different from the present invention in that it understands human speech and expresses emotions with motions and expressions, but discloses only the schematic structure of a robot.

Therefore, in the humanoid robot, research on a method of controlling the motor to express the motion and expression of the robot similar to humans should be preceded.

The present invention devised to solve the above problems is a humanoid robot that can express emotions with facial expressions and body movements, human physiological responses, and lip sync, similar to humans. An object of the present invention is to provide a motion and facial expression synthesizing apparatus and method.

The present invention for achieving the above object, the sensor module for receiving the physical information of the real world from the sensor mounted on the robot to convert the voltage into a voltage that can be processed by the robot; Recognition module for extracting the necessary information from the various information collected from the sensor; A scenario module configured to sequentially configure user commands into scenarios; A task module for processing a task classified into a unit operation; An action-based controller that selects and executes an appropriate action and facial expression for the sensor input of the robot and the output of the upper task; And a motor arbiter for mediating conflicting outputs when one or more behaviors are performed at the same time.

Here, the behavior-based controller includes basic behaviors, physiological response behaviors, motion containers, facial expression containers, lip synchronization behaviors, and eye behaviors.

Preferably, the motion container is loaded with behaviors that can be represented by the body of the robot, one or more motion behaviors can be superimposed and executed by the command of the task, and to mediate the output of each motion behavior Two motor arbiters.

In addition, the facial expression container is loaded with facial expressions that can be represented by the robot's face as a behavior, one or more facial expression behaviors can be executed by the task of the task superimposed, one motor to mediate the output of each facial behavior It includes an arbitrator.

On the other hand, the motor arbiter stores the output value of each behavior in a storage space of the motor interface assigned to each behavior, and the value is

The motor output position is determined by the above equation.

In another aspect, the present invention, the step of loading the motion file to the motion container in order to express the motion and facial expression of the humanoid robot; Instructing execution of an action in the action container; And it provides a method of synthesizing the motion and facial expression of the humanoid robot comprising the execution step of the behavior behavior.

The present invention also provides a method for loading an action file into an action container, comprising: (a) retrieving an action file; (b) determining whether a file is searched, ending the procedure if the file is not found, and determining if the file is a trajectory file; (c) reading the locus file and converting the locus file into an action file if the file is a locus file; (d) if the file searched in step (b) is not a trajectory file, reading the searched operation file; And (e) registering the action file with the action container.

Preferably, step (c) converts the trajectory file into a motion file using cubic spline interpolation.

The present invention also provides a method for instructing execution of an action in an action container, comprising: (a) searching for an action in the action container; (b) determining whether the motion search is successful, and if the motion search fails, the procedure ends, and if the motion search is successful, determining whether to maintain the posture; (c) setting the last action to the default posture if it is not necessary to maintain the posture; And (d) setting run_flag to true to execute the requested operation from the task module.

Here, in step (c), if the flag for determining whether to maintain the posture included in the request of the task module is true, the default behavior is set to the default posture in the basic behavior.

The present invention also provides a method for executing an operation behavior, comprising: (a) determining whether run_flag is true and if the run_flag is not true, ending the procedure; (b) determining a reference position of an operation frame array by calculating an index when the run_flag is true; (c) determining whether the index value is less than zero, and if the index value is less than zero, wait for a while and then return to step (b); (d) determining that the index value is less than fc if the index value is not less than zero; (e) converting run_flag to false if the index is not less than fc; (f) calculating a profile value in the form of a square trapezoid if the index is smaller than fc; And (g) calculating an output position and weight of the axis.

The present invention also provides a computer readable medium executable by the above method.

In order to fully understand the present invention, the advantages of the operability of the present invention, and the objects achieved by the practice of the present invention, reference should be made to the accompanying drawings which illustrate preferred embodiments of the present invention and the contents described in the accompanying drawings.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. Like reference numerals in the drawings denote like elements.

1 is a diagram illustrating a block structure and data flow for expressing an operation and facial expression of a robot according to an exemplary embodiment of the present invention.

Referring to Figure 1, the hardware module for expressing the operation and expression of the robot of the present invention is composed of a sensor module 100 and the motor module 200, the software module for expressing the operation and expression of the robot is a recognition module 300, the scenario module 400, the task module 500, the behavior-based controller 600, and the motor arbiter 700 are configured.

The sensor module 100 receives the physical information of the real world from a sensor mounted on the robot and converts it into a voltage that the robot can process. The sensor module 100 may be a camera, a proximity sensor, a touch sensor, a range finder, or the like.

The recognition module 300 extracts necessary information from various information collected from various sensors and transmits the necessary information to the operation selection module. The extracted information includes the approach and distance of an object, motion detection, motion detection, and face recognition.

The scenario module 400 sequentially configures user commands as scenarios. As an example, the scenario module 400 may be a user command such as 'Get a key from 302'.

The task module 500 processes tasks that are divided into unit operations. The task performs one or more behaviors. For example, the task may be 'move', 'stuff recognition', or 'stuff catching'.

The behavior-based controller 600 is designed by dividing an operation to be executed by the robot into a certain kind and number of behaviors, and immediately selecting and performing the behaviors in each situation to determine the operation of the robot.

In more detail, the behavior-based controller 600 is composed of a basic behavior 610, a physiological response behavior 620, an operation container 630, an expression container 640, a lip synchronization behavior 650, and an eye behavior 660. do. The role of each behavior is as follows:

The basic behavior 610 expresses the basic posture of the robot body and the basic emotional state of the face. The basic posture state of the body continuously represents the state of sleeping, dozing, and waking. In the waking state, the robot can take various postures. For example, it may take a position of holding the left arm or bowing down. The face may show normal, joy, sadness, happiness, or displeasure depending on the emotional state, and several emotions may occur at the same time. For example, in the case of a little joyful, 40% of normal and 60% of joy may be expressed and displayed at the same time.

The physiological response behavior 620 represents a response in which the sympathetic nerve of the human body is activated, such as breathing, pulse, hiccup, yawn, and eye blink. For example, the eyelid may be opened and closed, and an eye may blink in an irregular cycle.

The motion container 630 overlaps one or more intended motions of the body. This is described in detail with reference to FIG. 2A.

The facial expression container 640 overlaps one or more intended facial expressions of a face. This is described in detail with reference to FIG. 2B below.

The lip synchronization behavior 650 synchronizes the shape of the lips with the horse.

The eye behavior 660 represents a direction of eye gaze.

On the other hand, the motor arbiter 700 arbitrates conflicting outputs generated by one or more behaviors simultaneously.

The motor module 200 outputs the output of the motor arbiter to the motor to express the operation and facial expression of the robot.

2A illustrates a detailed structure of an operation container inside an action-based controller according to an embodiment of the present invention.

Referring to FIG. 2A, the operations container 630 may be loaded with behaviors that can be represented by a robot body, and one or more operation behaviors may be superimposed and executed by a command of the task module 500. The operation container 630 includes one motor arbiter to mediate the output of each operation behavior. Examples of behavioral behaviors include greetings, waving hands, and shaking hands.

2B is a diagram illustrating a detailed structure of the facial expression container inside the behavior-based controller according to an embodiment of the present invention.

Referring to FIG. 2B, facial expressions that can be represented by a robot face are loaded in the facial expression container 640 as one or more facial expressions, and one or more facial expression behaviors may be overlapped and executed by a command of the task module 500. The facial expression container 640 includes one motor arbiter to mediate the output of each facial behavior. Examples of facial behaviors include laughter, crying, frowning, and appearance.

3 is a diagram illustrating a detailed structure of a motor arbiter according to an embodiment of the present invention.

Referring to FIG. 3, in the humanoid robot, the motor arbiter 700 is present between the behavior-based controller 600 and the motor module 200 to mediate conflicting outputs generated by one or more behaviors performed simultaneously. . As the behavior is generated, the behavior creates a storage space of a motor interface for storing output information of the motor from the motor arbiter 700 to refer to a pointer. The output value of each behavior is stored in the storage space of the motor interface assigned to each behavior, and this value determines the motor output position according to Equation 1 below.

[Equation 1]

Where n is the number of behaviors registered in the behavior-based controller, priority _i is the weight of the i-th behavior, mask _i is the weight of each axis of the i-th behavior, and posture _i is the output position of each axis of the i-th behavior, Position Means the control position of the motor connected to each axis.

4 is a flowchart for loading an operation file of the present invention into an operation container.

Referring to FIG. 4, as the application program of the humanoid robot is started, a process of searching for a storage medium of the robot and reading a motion and a trajectory file and registering the motion behavior in the motion container is performed. In more detail, the operation file is searched for (S100). It is determined whether the file is searched (S110). If the file is not found, the procedure is terminated. If the file is found, it is determined whether the file is a trajectory file (S120). If the file is a locus file, the locus file is read (S130), and the locus file is converted into an operation file (S140). If the file retrieved in step S120 is not a trajectory file, the operation file is read (S150). When the step of converting the trajectory file into an action file and the reading of the action file are performed through steps S140 and S150, the action file is registered in the action container (S160).

The motion file stores position values for each axis of the robot to pass at discrete time intervals. In the first frame of the motion file, weights for all axes of the robot are stored as values between 0 and 1, and position values for all axes of the robot are stored every second from the next frame. Unlike the motion file, the trajectory file stores position values that each axis of the robot must pass at a specific time. The trajectory file is converted into motion data using cubic spline interpolation and then registered in motion container. After the file is searched, the motion file and the trajectory file can be distinguished by the file extension.

5 is a flowchart for executing an operation in an operation container of the present invention.

Referring to FIG. 5, when the task module 500 requests the action container 630 to perform a specific action, the action container 630 searches for an action behavior with respect to the requested action and succeeds. Set run_flag to true to instruct the execution of the action. The request of the task module 500 includes a flag for deciding whether to maintain a posture. If this flag is true, the basic behavior sets the last action to the default posture.

In order to describe this step by step, first, when the execution of a specific operation is requested through the task module 500, the operation container 630 searches for an operation (S200). It is determined whether the motion search is successful (S210). If the motion search fails, the procedure is terminated. If the motion search is successful, it is determined whether to maintain the posture (S220). If it is not necessary to maintain the posture, the last operation is set as the basic posture (S230). After performing steps S220 and S230, an operation requested by the task module 500 is executed (S240).

6 is a flowchart illustrating execution of an operation behavior selected in FIG. 5.

Referring to FIG. 6, if run_flag of an operation behavior is true, execution continues. The reference position of the motion frame array is determined by calculating the index. st and magnitude are set in the task module 500. If st is greater than ct, index has a value less than 0, and waits for a while without performing the next step. Execution terminates when index is greater than or equal to fc. profile is the weight of motion over time, which is the square of the trapezoidal profile.

If this is described step by step, it is first determined whether run_flag is true (S300). If the run_flag is not true, the procedure ends, and if true, the index is calculated. The expression for calculating the index is index = v * (ct-st) / tr (S310). Where index is an index of a frame stored in an operation file, v is an execution speed of an operation file, ct is a current time, st is a time for starting an operation, and tr is a time resolution. It is determined whether the next index value is smaller than 0 (S320). If the index value is less than 0, the process waits for a while (S330) and returns to step S310. If the index value is not less than zero, it is determined whether the index value is smaller than fc (S340). Here, fc means the total number of frames stored in the action file. If the index is not less than fc, run_flag is switched to false (S350). On the other hand, if the index is less than fc to calculate the profile value. The formula for calculating the profile is equal to profile = min (index / a) ² , (fc-index) ² , 1) (S360). Where a is the acceleration that determines the slope of the increase and decrease of the profile. Calculate the next weight, Mask [axis] and Position [axis]. More specifically, it is obtained by weight = profile * w [axis] and Mask [axis] + = weight. It is found as Position [axis] + = weight * magnitude * p [index] [axis]. Where Mask [axis] is the output weight of the axis referenced by axis, Position [axis] is the output position of the axis referenced by axis, w [axis] is the weight of the axis referenced by axis in the motion file, and magnitude is motion Where p [index] [axis] is the location of the frame referenced by index and the axis referenced by axis in the motion file.

The steps for loading the expression file into the expression container, the step for executing the expression in the expression container, and the execution of the facial expression behavior are performed by different axes as compared to the methods shown in FIGS. 4, 5, and 6. The method is the same, so detailed description is omitted.

Although the present invention has been described with reference to one embodiment shown in the drawings, this is merely exemplary, and those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom. . Therefore, the true technical protection scope of the present invention will be defined by the technical spirit of the appended claims.

The present invention provides an apparatus and method for synthesizing and expressing a humanoid robot capable of expressing emotions with facial expressions and body motions, expressing human physiological responses, and lip syncing similarly to humans. do.

Claims (12)

A sensor module that receives physical information of the real world from a sensor mounted on the robot and converts the physical information into a voltage that the robot can process; Recognition module for extracting the necessary information from the various information collected from the sensor; A scenario module configured to sequentially configure user commands into scenarios; A task module for processing a task classified into a unit operation; An action-based controller that selects and executes an appropriate action and facial expression for the sensor input of the robot and the output of the upper task; And An apparatus for synthesizing motions and facial expressions of a humanoid robot comprising a motor arbiter that mediates conflicting outputs when more than one behavior is performed simultaneously. The method of claim 1, The behavior-based controller may include a basic behavior, a physiological response behavior, an operation container, an expression container, a lip synchronization behavior, and an eye behavior apparatus. The method of claim 2, The motion container is loaded with behaviors that can be expressed by the body of the robot, one or more motion behaviors can be executed by the task command superimposed, An apparatus for synthesizing motions and facial expressions of a humanoid robot including a motor arbiter for mediating the output of each motion behavior. The method of claim 2, The facial expression container is loaded with facial expressions that can be expressed by the face of the robot as a behavior, one or more facial expression behaviors can be executed by the task command superimposed, An apparatus for synthesizing motions and facial expressions of a humanoid robot including a motor arbiter for mediating the output of each facial behavior. The method of claim 1, The motor arbiter stores an output value of each behavior in a storage space of a motor interface assigned to each behavior, and the value is

An apparatus for synthesizing motions and facial expressions of a humanoid robot for determining a motor output position by the above equation. In order to express the motion and expression of the humanoid robot, Loading an action file into an action container; Instructing execution of an action in the action container; And A method of synthesizing motions and facial expressions of a humanoid robot comprising an execution step of a motion behavior. A method for loading an action file into an action container, (a) retrieving an action file; (b) determining whether a file is searched, ending the procedure if the file is not found, and determining if the file is a trajectory file; (c) reading the locus file and converting the locus file into an action file if the file is a locus file; (d) if the file searched in step (b) is not a trajectory file, reading the searched motion file; And (e) registering the action file with an action container. The method of claim 7, wherein In the step (c), the motion file loading method of converting the trajectory file into a motion file using cubic spline interpolation. In a method for instructing execution of an action in an action container, (a) searching for an action in the action container; (b) determining whether the motion search is successful, and if the motion search fails, the procedure ends, and if the motion search is successful, determining whether to maintain the posture; (c) setting the last action to the default posture if it is not necessary to maintain the posture; And (d) setting run_flag to true to execute the requested action from the task module. The method of claim 9, In step (c), The action file execution instruction that sets the last action in the default behavior to the default attitude if the flag that determines whether to keep the attitude contained in the task module's request is true. In the method for executing an operation behavior, (a) determining if run_flag is true and if the run_flag is not true, ending the procedure; (b) determining a reference position of an operation frame array by calculating an index when the run_flag is true; (c) determining whether the index value is less than zero, and if the index value is less than zero, wait for a while and then return to step (b); (d) determining that the index value is less than fc if the index value is not less than zero; (e) converting run_flag to false if the index is not less than fc; (f) calculating a profile value in the form of a square trapezoid if the index is smaller than fc; And (g) calculating an output position and weight of the axis. A computer readable medium executable by the method of claim 6.

Images