JP2011167820A - Head part action control information generating device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a head part action control information generating device for controlling an action of a head part of a robot to establish smoother communication between the robot and a human being. <P>SOLUTION: The head part action generating device 86 generates control information for controlling an action of the head part of a humanoid type robot by synchronizing with voice generated by the robot, and includes a probability model group 100 that regulates probability to execute a plurality of head part actions for each speech function tag attached to every phrase, and a head part action command generating part 104 that selects a probability model among the probability model group 100 based on notes attached to the input phrase, and outputs a head part action command corresponding to the input voice data in the predetermined unit to a control part 90 of the robot with the probability in accordance with the selected probability model. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an improvement in communication with a human by a robot, and more particularly, to a technique for naturally moving the head of a humanoid robot in accordance with the utterance content.

  While speaking, humans naturally move their heads. Sometimes these moves are intentionally made to convey a clear meaning to the other party. For example, whispering represents consent, and shaking the head represents disagreement. In many cases, however, head movements are unconscious. Such movements can sometimes convey more information than the speaker intended. Therefore, in a humanoid robot, if the head can be moved appropriately according to the content of the story, it is expected that communication with the opponent will be smoother.

  As this type of technology, there is one described in Patent Document 1. Japanese Patent Laid-Open No. 2004-133867 adds a signal for determining a timing to speak to a voice signal in advance in a robot that speaks, and detects a timing signal for running from the voice signal. A technique for controlling the neck is disclosed.

JP2009-069789

  As described in Patent Document 1, it is important to move the head according to the utterance of the robot in order to achieve effective communication between the human and the robot. However, no suitable method has yet been proposed as to how the robot's head can be felt naturally by humans and it is easy to understand the meaning of the robot's speech. In the technique described in Patent Document 1, there is only a description relating to pre-programming of the movement of the head, and it is not disclosed how to feel a natural movement if the head is moved.

  SUMMARY OF THE INVENTION Therefore, an object of the present invention is to provide a head motion control information generating device that controls the movement of the head of a robot so that communication between the robot and a human can be performed more smoothly.

  Another object of the present invention is to provide a head motion control information generating device that changes the motion of the head of the robot appropriately according to the content of the utterance and makes it feel naturally.

  A head movement control information generation device according to an aspect of the present invention generates control information for controlling the movement of the head of a humanoid robot in synchronization with the sound generated by the robot. The sound is defined by sound data that has been divided into predetermined units in advance. Each of the predetermined units is provided with an annotation including a discourse function tag indicating a discourse function based on speech uttered by the sound data assigned to the unit. The head movement control information generation device includes a probability model storage unit for storing a probability model that defines a probability of performing a plurality of head movements for each discourse function tag, and a predetermined voice data And a probability model selection means for selecting a probability model determined based on the annotation attached to the unit from among the probability models stored in the probability model storage means, and a probability model selection Head command generation means for generating a head motion command corresponding to the input voice data of a predetermined unit with a probability according to the probability model selected by the means and outputting it to the control unit of the humanoid robot type robot And including.

  Even if the discourse function tag attached to the predetermined unit of the voice data is the same, the head command of the humanoid robot varies depending on the case because the head command is selected by the probability model. Compared to the case where the same movement is always performed in the same situation, the movement that is felt unnatural can be reduced. By appropriately preparing the probability model, the head of the robot can be controlled in an appropriate and natural manner according to the discourse function tag. As a result, a head motion command generation device that controls the movement of the robot's head so that the communication between the robot and the human can be made smoother, and the movement of the robot's head appropriately changes to the content of the utterance. Therefore, it is possible to provide a head motion command generation device that makes the user feel natural.

  Preferably, the speech data is a discourse consisting of any combination of speaker specifying information for specifying a speaker, relationship specifying information for specifying a relationship between a speaker and a conversation partner, and non-linguistic information at the time of speaking. Discourse mode information for specifying the mode is attached. The probability model selection means stores a plurality of probability models prepared in advance according to the combination of the discourse mode information and the discourse function tag. The probability model selecting means includes means for selecting one of a plurality of probability models according to the discourse mode information attached to the voice data and the input predetermined unit.

  Experiments have shown that human interactions differ depending on the speaker, the relationship between the speaker and the conversation partner, and circumstances related to nonverbal information such as the speaker's attitude and emotion at the time of speaking. Therefore, by preparing probabilistic models according to these combinations in advance and selecting head movements according to the probabilistic model adapted to these according to the discourse mode at the time of utterance, the conversation partner is more than just a voice. Information can be given.

  More preferably, the head command generation means determines whether or not to generate a head movement corresponding to the input sound data of a predetermined unit with a probability according to the probability model selected by the probability model selection means. A head motion determining means for performing, a shape information storage means for storing, for each of a plurality of head motions, shape information defining temporal movement of the head corresponding to the head motion, The shape information corresponding to the head motion is stored in the shape information at a timing determined in advance according to the type of the head motion in response to the head motion determining means having determined to generate some head motion. Shape reading means for generating a control command for reading out from the means and controlling the head of the robot and outputting it to the control unit.

  A simple rule can be used to determine the timing for generating head movement. A simple process of storing shape information representing temporal movement for realizing the head movement in advance for each head movement and generating a control command from the shape information at a determined timing. , Can facilitate communication with the conversation partner.

  More preferably, the movement of the robot head is related to the pitch angle among the rotations of the robot head around three axes. The head command generation means further includes an utterance head shape storage means for storing the shape of the head movement prepared in advance, which prescribes the temporal movement of the head when the robot speaks, and a shape reading means. The head shape information storage means at the time of utterance is connected to the control command output from the shape reading means in response to the voice data indicating that the robot is speaking. And means for superimposing the shape of the head movement read from the control unit to the control unit.

  Experiments have shown that, especially with respect to the pitch angle, the speaker has a specific movement when speaking. However, this movement may vary depending on the speaker. Thus, with the configuration in which the shape of a specific head motion prepared in advance is stored in this way, a characteristic head movement during speech can be reproduced by an assumed speaker.

  The movement of the robot head may relate to rotation around any combination of the three axes of the robot head. In this case, the probability model storage means, the probability model selection means, and the head command generation means all generate head control commands independently for each axis constituting an arbitrary combination.

  According to the present invention, with the above-described configuration, it is possible to provide a head movement command generation device that controls the movement of the robot head so that communication between the robot and a human can be performed more smoothly. Furthermore, it is possible to provide a head motion command generation device that changes the motion of the robot head appropriately according to the content of the utterance and makes it feel natural.

It is a figure which shows the setting at the time of the motion capture of the human head performed in order to implement | achieve embodiment of this invention. It is a graph which shows the test subject's head movement obtained as a result of the said experiment, the relationship between each test subject and the conversation partner, and the relationship of the content of the utterance. It is a graph which shows the test subject's head movement obtained as a result of the said experiment, the relationship between each test subject and the conversation partner, and the relationship of the content of the utterance. It is a figure which shows arrangement | positioning of the actuator which implement | achieves the motion of the head of the robot used as the control object in 1st Embodiment. It is a functional block diagram of the head movement generating device concerning a 1st embodiment of the present invention. It is the schematic diagram which showed the program which implement | achieves the head movement position production | generation part shown in FIG. 5 with the block diagram flowchart. It is a graph which shows the time change of the rolling angle (upward is made into positive) for implement | achieving the heading of the robot used in the 1st Embodiment of this invention. It is a graph which shows the time change of the rolling angle for implement | achieving the continuous several rolling operation | movement used in the 1st Embodiment of this invention. It is a graph which shows the time change of the whispering angle of the robot which the robot used in the first embodiment of the present invention is speaking. It is a graph which shows the production | generation process of the temporal change of the beating angle at the time of producing | generating the head movement command from the speech in the 1st Embodiment of this invention. It is the figure which showed the program which implement | achieves the head movement position production | generation part of the head movement production | generation apparatus which concerns on the 2nd Embodiment of this invention with the block diagram flowchart.

  In the following description and drawings, the same parts are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

[Experiment]
In realizing an embodiment of the present invention, which will be described later, an experiment was conducted on the relationship between human speech and head movement, and how the conversation partner understands human head movement. From this result, it was possible to obtain a model having a relatively simple configuration as will be described later, thereby realizing the robot head motion generation device according to the embodiment of the present invention.

<Experimental settings>
─Data─
Seven subjects (4 male speakers and 3 female speakers) were used in the experiment. Table 1 shows a list of these subjects, a partner (discourse partner) who interacted with these subjects, and the relationship between the talk partner and the subject.

これら発話者と、談話相手とのいくつかの組合せごとに、自由な対話による談話（１０分−１５分間）を何セッションか収録した。収録した談話セッションの総数は１９であった。各発話者についてのセッション数は、ＦＭＨ×ＦＫＨ（２）、ＦＫＮ×ＦＫＨ（２）、ＦＭＨ×ＭＨＩ（１）、ＦＫＮ×ＭＨＩ（１）、ＦＭＨ×ＭＳＮ（１）、ＦＫＮ×ＭＳＮ（１）、ＦＭＨ×ＭＩＴ（３）、ＦＫＮ×ＭＩＴ（３）、ＦＭＨ×ＭＳＲ（５）であった。

For each combination of these speakers and the talk partner, several sessions of conversations (10-15 minutes) by free dialogue were recorded. The total number of discourse sessions recorded was 19. The number of sessions for each speaker is FMH × FKH (2), FKN × FKH (2), FMH × MHI (1), FKN × MHI (1), FMH × MSN (1), FKN × MSN (1) FMH × MIT (3), FKN × MIT (3), FMH × MSR (5).

  Audio and motion data were recorded simultaneously for both of the interlocutors. The distance between the participants was set to be as close as possible within the range where both motion captures were possible at the same time. As a result, the distance between the two became 1 m. Directional microphones were used for recording, and these directional microphones were installed for each person who interacted.

  Seven hemispherical passive reflection markers were attached to each speaker's head, and the motion data of the speaker's head was acquired with a commercially available motion capture device using infrared rays. FIG. 1 shows the position of the reflective marker. Referring to FIG. 1, five markers 32, 34, 36, 38 and 40 are mounted around the head of the speaker 30. A marker 42 was further attached to the top of the nose of the speaker 30 and a marker 44 was attached to the chin. The coordinates of the head are obtained by the head and nose markers 32-42, and the movement data and the speech data are associated by the position of the chin marker 44 (relative position with respect to the nose marker 42). .

  The movement of the head is represented by three axes (XYZ axes) shown on the right side of FIG. “Spring”, “swinging”, and “necking” refer to rotating the neck around the X axis, the Z axis, and the Y axis, respectively, in the coordinate axes of FIG. These are the same as what are called pitch, yaw and roll in aeronautical engineering. The rotation angles around these axes are called a pitch angle, a yaw angle, and a roll angle, respectively.

  The rotation angle of the head uses the marker coordinates, MB Stegmann, DD Gomez, “A brief introduction to statistical shape analysis,” http://www2.imm.dtu.dk/pubdb/views/edoc_download.php/403 It was calculated based on the singular value decomposition introduced in /pdf/imm403.pdf, 2002. That is, a combination of the unitary matrix U, the diagonal matrix D, and the unitary matrix V is obtained by singular value decomposition expressed by the following equation (1).

ただし、reference及びtargetはそれぞれ、中立位置及び現在位置の３Ｄマーカ位置を、それらの重心を原点とする新たな座標系に平行移動させた座標を表す。中立位置は、被験者がまっすぐ前方を向いているときに得られたものである。回転を表す行列は次の式（２）により得られる。

Here, reference and target represent coordinates obtained by translating the 3D marker positions of the neutral position and the current position into a new coordinate system with their center of gravity as the origin. The neutral position is obtained when the subject is looking straight ahead. A matrix representing rotation is obtained by the following equation (2).

この回転行列Ｒの要素から、次の式（３）−（５）によって回転角度が算出される。

ただし、ｓｑｒｔ、＾、及びａｔａｎ２はそれぞれ、Ｍａｔｌａｂ（登録商標）の平方関数、べき乗関数、及びアークタンジェント関数である。

From the elements of this rotation matrix R, the rotation angle is calculated by the following equations (3)-(5).

Here, sqrt, ^, and atan2 are a Matlab (registered trademark) square function, power function, and arctangent function, respectively.

  The utterance data thus obtained was manually divided into phrases by Japanese native speakers and transcribed. As a result of the division, a total of 16920 phrases were obtained.

─ Tags for head movement ─
For the part considered to have some meaning during the dialogue in Japanese, the head movement at that time was manually annotated with the following head movement tags.

・ No: No movement ・ nd: Single whisper (lowering face-up movement)
・ Mnd: multiple whispers with phrases ・ fd: face down ・ ud: face up-face down (once)
・ Fu: Face up ・ ti: Fading in the phrase ・ sh: Shaking in the phrase (left-right movement)
A whisper is not always represented by a vertical movement of the head. For example, a slight tilt of the face can be seen as whispering. After annotating in this way, we considered the most significant movement.

  Based on the temporal changes of the three angles and the video image, one test subject annotated the audio information with respect to the head movement and the head movement tag set. This label was checked and corrected by another subject. Both subjects are assistants of the present inventors. Of the annotations made by the first subject, 5% were modified by the second subject.

  Table 2 shows the distribution of annotations using the movement tags of each head. It can be seen that whispering (the sum of nd and mnd) is the most frequent head movement. The “other” column is for phrases related to movements for which the subject was not sure of the classification.

─談話機能タグ─
データセット中の各句について、以下のような談話機能タグの組から選択した談話機能タグを注釈として付した。この際、肯定又は否定の反応、驚き・意外さの表れの感情表現、及び発話交替などについて考察した。

─Discourse function tag─
For each phrase in the data set, a discourse function tag selected from the following set of discourse function tags was annotated. At this time, we considered positive or negative reactions, emotional expressions of surprise and surprise, and speech alternation.

  K (keep): The speaker holds the position (speak order) as the speaker. Short poses or clear pitch resets occur at strong phrase boundaries.

・ K2 (keep): weak phrase boundary in the middle of utterance (no pause between phrases) ・ k3 (keep): When the speaker is thinking, the end of the phrase is extended, but the order is kept (The pause may or may not continue.)

  F (filler): The speaker is preparing to consider the next utterance. For example, "Umm", "Et", or "That".

  F2 (conjunctions): It can be considered in the same way as a filler that is not extended for a long time. For example, “So”, “Ja”, “De”, etc.

  G (give: transfer of talk order): The speaker finishes speaking and passes the speaker's status to the other party of dialogue.

  Q (question): The speaker asks a question or asks for consent from the other party.

  Bc (compilation): The speaker shows the companion (consent response) to the other party. For example, say “Yes” while whispering.

  Su (surprise / unexpected / impressed) The speaker shows a reaction (surprise / unexpected / impressed) with an expression to the other party of the dialogue. For example, “hee”, “lie!”, “Oh”, etc.

  Dn (denial, negation): “No”, “No”, etc. with a swing motion.

  Although the classifications described above are not complete, they are considered sufficient from the perspective of human-robot communication.

  After the first subject added annotations with the discourse function tag for each phrase, the second subject checked and modified the annotations. These subjects are the same subjects that have been annotated with head movement tags. The second subject corrected 5.9% of all annotations.

  Table 3 shows the distribution of annotations by each discourse function tag. In Table 3, “Others” includes a phrase for which the subject could not determine which discourse function tag to attach. This further includes interjections of subcategories of g (transfer of talk order), such as greetings, and interjections other than bc (simple combinations) and su (surprise / unexpected / impressed).

─頭部動作及び談話機能─
図２及び図３に、各談話機能の機能に対する頭部動作の分布を示す。これらグラフは、各発話者と談話相手との関係に応じたグループ（各図の最下部のｘ軸に示す。）に分けて示してある。ｙ軸には各頭部動作タグの生起した確率を示してある。

─Head movement and discourse function─
FIG. 2 and FIG. 3 show the distribution of head movements for each discourse function. These graphs are divided into groups (shown on the x-axis at the bottom of each figure) corresponding to the relationship between each speaker and the conversation partner. The y-axis shows the probability that each head motion tag has occurred.

  Referring to FIGS. 2 and 3, it can be seen that whispering (nd) and multiple whispering (mnd) occur frequently in the combination (bc). Whispering was also frequently observed at strong phrase boundaries (k, g, q). In question (q), it is usually pronounced so that the end of the phrase rises, but whispering was observed more frequently than a face-up-face-down action or a face-up action. This seems to be the reason why the correlation between the pitch (F0 shape) and the head movement is low.

  The observation frequency of whispering is more in the weak phrase boundary (k2) in the middle of utterance and the phrase boundary (k3, f, f2) indicating that the speaker thinks or has not completed the utterance. Lower. In such a discourse function category, the state (no) where there is no movement of the face is most.

  Looking more closely at multiple mund sequences, these movements tend to occur throughout the utterance when the speaker expresses strong agreement, understanding, or interest in the conversation partner's story. I found out that

  In the case of phrases representing surprise, surprise, impression, etc. (su), the frequency of occurrence in the data was lower. There was no head movement (no), face-up movement (fu), and neck-raising (ti).

─Changes between the same speaker and between speakers─
The above experiment also revealed that the frequency of head movements varies from speaker to speaker.

  For example, two out of four male speakers (MSN and MHI) had much less head movement than others. From this fact, it is considered that the social status of the speaker is higher than that of the other party (research assistant). Another possibility is that the age difference between the speaker and the other party may have an effect.

  When considering changes in the same speaker, it seems that the frequency of head movements is affected by the personal relationship with the conversation partner. 2 and 3, the relationship between the speaker and the conversation partner is divided into three stages. That is, “near” (family, boyfriend), “far” (difference in generation and social status), “middle” (friend, friend friend, friend relative). In order to distinguish between “middle” and “far”, we considered not only social status but also the relationship between actual individuals. For example, the factor of whether or not the speaker and the conversation partner are the first to meet each other is also considered.

  The above results further show that the frequency of head movement is much lower when the speaker is closely related to the conversation partner. For example, in the case of speaker FMH, when talking to her mother or her boyfriend (FMH x "close"), there is a high frequency of no head movement (no) in g, q, bc, and k. Occurred and sowing (nd, mnd) occurred only infrequently. Similarly, in the combination (bc), the combination of FMH × “near” was most of the movement (nd), whereas when the conversation with the partner the FMH first saw, the frequency of multiple whispering (mnd) Becomes higher. This fact seems to be explained by the fact that head movements (especially whispering) are used to express attitudes, such as showing that you are interested in the story of the other party. . There are more behaviors that do not care much for the family, resulting in less head movement. As for the utterance change utterance (k), whispering frequently occurs in the combination of FMH × “intermediate”, but the frequency decreases in the combination of FMH × “far”. This fact is that FMH suppresses herself when talking to MSN and MHI ("far" talk partner), while talking to MIT ("middle" talk partner) who is a friend's friend and close in age Sometimes it can be explained by speaking more confidently.

  As for the speaker FKN, when talking to FKH, MHI and MSN, the frequency of multiple strikes (mnd) was higher in bc (combined) than in single strike (nd). It seems that these conversation partners were older than speaker FKN and were the first to meet them. On the other hand, when talking to a friend MIT, a single whisper was the majority. In k and g, there was no particularly dominant movement when talking to FKH, MHI and MSN. This fact is similar to the case of FMH, because FKN did not insist on the first party FKN met (FKH, MHI and MSN), and was more confident when talking to MHI. I can explain.

  In the case of the speaker FKH, when talking to the daughter (FMH), no head movement (no) was more frequent than when talking to the daughter's friend (FKN). It can also be noted that when talking to FKN, multiple whirls (mnd) appeared more frequently.

<Rule-based whispering>
From the above results, it can be seen that the head movement that occurs most frequently during the discourse is whispering, but the frequency of other head movements also has a certain tendency depending on the conversation partner and the discourse function. I understand. Therefore, in the following embodiments, a rule-based head movement generation model and an apparatus for generating a robot head movement command using the rule-based head movement generation model are proposed. In the following description, head movements, particularly whispering, will be mainly described. However, head movement commands can be generated based on the same concept with respect to swinging and necking.

  First, consider a very simple probabilistic model that generates only the timing of whispering from speech data given to the robot. In this model, whispering is generated in the middle of the last syllable of an utterance (k, g, q) and a combination (bc) with strong phrase boundaries.

  Such a model can be obtained as a probabilistic model by statistically processing the experimental results similar to those shown in FIGS. In the above experiment, in order to clarify the relationship between what is considered as a factor for head movement and the actual head movement, what is considered as a factor is narrowed down. Information that seems to be related to non-linguistic information (information transmitted by means other than the spoken language) such as attitudes and emotions to be accompanied when speaking (or the entire utterance) is also labeled in advance for each phrase in the discourse record Similar statistical information can be obtained.

  In this embodiment, for each combination of relationship / attitude / emotion with the speaker / discussion partner (for the sake of simplicity, these combinations are simply referred to as a “discussion mode”, and The label attached to the audio data to specify the mode is called “Discourse mode information”.) Probability that gives the probability of each head movement related to whispering from the discourse function category A model shall be prepared in advance. These probability models can be easily obtained by statistically processing the same experimental results as described above. For each head movement, information indicating at which syllable position in the phrase the movement is performed is attached.

  In the following embodiment using such a probabilistic model, the speech data given to the robot includes speech data, phrase segmentation information from speech, a sequence of phoneme labels separated for each phrase, and assigned to each phrase. And the provided discourse function tag. Each phoneme label is also attached with information for specifying the utterance time in the phrase of the speech corresponding to the phoneme. The utterance data further includes information on the relationship between the speaker and the conversation partner (“near”, “far” or “intermediate”) and discourse mode information indicating the attitude and emotion to be attached to each phrase.

  Those skilled in the art will easily understand that such a probabilistic model can be obtained not only for a whispering motion but also for a swing, a head angling, and the like. In this case, it is necessary to obtain the timing for starting the head movement from the experimental results. For example, the above-described whirling motion is generated in the last syllable, but the swing motion is known to continue over the entire utterance as in the case of multiple whispering. Such information also needs to be obtained in advance through experiments.

  For the head movement, it is determined whether or not a predetermined movement (whipping) is to be generated, and when it is determined at which timing the heading is to be generated, the shape of each movement prepared in advance is controlled at that timing. It may be superimposed on the shape of the control signal for.

  Although not mentioned in the above experiment, it was also found from the experiment that humans raise their faces slightly upwards (facing upwards) during the utterance period. The angle is about 3 degrees (the upward direction is positive). In the following embodiment, this face raising operation at the time of speech is also taken in.

[First Embodiment]
<Configuration>
Referring to FIG. 4, the drive system of head 60 of robot 50 to be controlled includes a motor 62 that rotates head 60 around the y axis, a motor 64 that rotates around x axis, and a z axis. And a motor 66 for rotation. It is only necessary to operate the motor 64 in the whirling operation, the motor 66 in the swinging operation, and the motor 62 in the neck movement.

  Referring to FIG. 5, robot 80 according to the present embodiment stores memory for storing probability model group 100 including probability models obtained by statistically processing the same experimental results as described above. The device 82, the utterance data storage device 88 storing the utterance data to be uttered by the robot 80, and the utterance data given to each phrase of the utterance data are stored in the storage device 82. Using the stored probability model group 100, a head motion generation device 86 for generating a head motion command for controlling the head motion of the robot, speech data stored in the speech data storage device 88, and Based on the head motion command from the head motion generation device 86, the sound reproduction unit 92 for reproducing the sound so that the head motion is synchronized with the sound, and the head motion generation device 86 In response to the head operation command, and a robot controller 90 for controlling the motor 62, 64 and 66 shown in FIG. In FIG. 5, portions of the robot 80 that are not related to head movement are not shown for the sake of clarity.

  The probabilistic model group 100 describes which head motion can be obtained with what probability for a certain discourse function tag for each predetermined combination of relationship / attitude / emotion with the speaker / discussion partner. , 132 included. For example, the probability model 130 describes what kind of head motion is obtained with what probability for the discourse function tag in a combination of “close” and “normal” with the relationship between the speaker A and the conversation partner. is doing. In the present embodiment, for example, in the case of whispering, the discourse function tag is k when the discourse function tag is k, g, q, and bc, and the frequency is low in other cases. , G, q and bc are generated only at any time, and the head movement is not generated for the other discourse function tags. Therefore, the probability model 130 included in the probability model group 100 includes only the probability models for the four discourse function tags described above.

  The head movement generation device 86 reads out the utterance data from the utterance data storage device 88 and determines whether or not to generate a rotation around each xyz axis for each phrase based on the discourse function tag attached to each phrase. The head movement position generation unit 102 for outputting information indicating the timing at which the motion should be superimposed on the robot head movement command for the three axes xyz, Storage devices 108, 110, and 112 for storing a shape model of a whirling motion, a shape model of a swinging motion, and a shape model of a neck-raising motion, which are to be superimposed on the head motion command of the axis, respectively; Based on the information indicating the timing of the three-axis head movement position output from the position generation unit 102, the information is stored in the storage devices 108, 110, and 112, respectively, if necessary. A head motion command generation unit 104 for generating and outputting a head motion command on which the shape specified by the shape model of the whispering, swinging, and neck movement is superimposed, and based on the above experimental results, Among the three-axis head motion commands output from the head motion command generation unit 104, a memory that stores in advance a face-up shape model to be superimposed on a head motion command (pitch angle control signal) related to whispering when speaking The device 114 is connected to receive the three-axis head movement command output from the head movement command generation unit 104 together with the utterance data, and for the discourse section of the utterance data, the face-up shape stored in the storage device 114 A discourse section head motion command generation unit 106 for superimposing the model and giving it to the robot control unit 90.

  Referring to FIG. 6, head movement position generation unit 102 receives utterance data from utterance data storage device 88 and generates a whisper using an appropriate probability model determined by the utterance data in probability model group 100. In the case of generating a whispering, the timing is determined based on the speech data as well as the whirling position generation unit 150 for outputting as information indicating the whirling timing. In the probability model group 100, it is determined whether or not to perform the swing motion based on an appropriate probability model determined by the utterance data, and if so, the timing is determined as information indicating the swing timing. Similar to the swing position generation unit 152, the whirl position generation unit 150, and the swing position generation unit 152 for output, the probability is based on the utterance data. In order to determine whether or not to perform a neck raising operation using an appropriate probability model determined by the utterance data in the Dell group 100, and to determine when to do so and to output information indicating the timing of the neck raising And a neck position generating unit 154. The winding position generation unit 150, the swing position generation unit 152, and the neck position generation unit 154 are substantially realized by computer hardware and a computer program executed by the computer hardware.

  The program routine for realizing the whispering position generation unit 150 determines whether or not the discourse function tag attached to the phrase in the utterance data is a target to be whispered (any one of k, g, q, and bc). Step 170 for diverting the flow of control, and the speaker attached to the utterance data, the relationship between the speaker and the conversation partner, information specifying the attitude / emotion, and the phrase are attached to the phrase from the probability model group 100. A step 172 for selecting a probability model determined according to the value of the discourse function tag, and a probability model selected in step 172 and a random number are used to determine what operation is to be performed as a motion belonging to the head whispering. In accordance with step 174 and the type of action determined in step 174, it is determined at which syllable position of the phrase the utterance is generated, the information indicating the utterance position is output, and the process ends. And a step 176 that. If it is determined in step 170 that the discourse function tag is not the target of the whispering, information indicating that there is no whispering timing is output and the process is terminated.

  The program that realizes the swing position generation unit 152 includes steps 180, 182, and 184 that perform substantially the same processing as the whirling position generation unit 150. The difference in the swing position generation unit 152 is that the discourse function tag determined in step 180 is a tag for swing determination, and the probability model used in step 182 determines the type of swing. It is to be a thing. The discourse function tag used for the determination in step 180 is determined by an experiment similar to that already described for whispering. The same applies to the probability model.

  Similarly, the program that realizes the neck position generation unit 154 includes steps 190, 192, and 194 that perform substantially the same processing as steps 170, 172, and 174 of the neck position generation unit 150, respectively. A difference in the neck position generation unit 154 is that the discourse function tag determined in step 190 is a tag for neck connection determination, and the probability model used in step 192 determines the type of head connection. It is to be a thing. The discourse function tag used for determination in step 190 is also determined by the same experiment as described above for whispering. The same applies to the probability model.

  FIG. 7 shows the shape of a single whispering operation stored in the storage device 108. The shape shown in FIG. 7 is generated as a typical movement from the movement of the face when a human crawls. As shown in FIG. 7, in natural human whispering, there is a time for the face to turn up for a very short time just before turning the face downward, and then it moves down greatly, and then turns to the front. By causing the robot to make such a movement, it is possible to control the movement of the robot so that the robot crawls in a natural manner as seen by humans. In the case of the shape of this whispering operation, it is preferable that the time axis is appropriately expanded and contracted so that the beginning and end of the waveform coincide with the start position and end position of the sentence end at the end of the phrase. However, it is not necessary to perform such expansion and contraction on the time axis. In this way, the position of the phrase at which the beginning and the end of the waveform are matched is stored in advance in the storage device 108 as information accompanying the shape, and the processing in steps 176, 186 and 196 is performed according to that information. Determine the start and end positions of the waveform.

  FIG. 8 shows a typical shape of multiple whispering (mnd) as another example of the shape of the whispering operation. The shape of this operation is also stored in the storage device 108 in FIG. 5, and is used to determine the position to be rolled when a plurality of strokes are selected in step 172 in FIG. As already mentioned, in the case of multiple strokes, head movement is performed throughout the phrase. Therefore, in this case, the timing of winding is determined so that the beginning of the waveform matches the beginning of the phrase and the end of the waveform matches the end of the phrase.

  In addition to this, the storage device 108 stores a plurality of shapes belonging to the shape of the whispering operation. What shape is used may be determined by design. However, in this case, it is necessary to make sure that there is no contradiction between the shape type selected by the probability model and the shape type stored in the storage device 108.

  Similarly, a shape model for swinging is stored in the storage device 110, and a shape model for necking is stored in the storage device 112.

  Referring to FIG. 9, the face-up shape model stored in the storage device 114 rises gently at the beginning of the phrase, then persists during the utterance at a whirling angle of about 3 degrees, and slowly at the last part of the phrase. To fall. As a result of actually observing human utterances, humans often took this kind of behavior during utterances, and the results seemed to be more natural when viewed from the other party. Therefore, in this embodiment, when the robot speaks, such a face-up shape model is superimposed on the whirling head motion command output from the head motion command generation unit 104. The period during which the robot is speaking can be obtained from the speech data.

<Operation>
With reference to FIGS. 5 to 9, the head motion generation device 86 of the robot 80 operates as follows.

  It is assumed that the probabilistic model group 100 shown in FIG. 5 is prepared in advance by conducting the experiment as described above and statistically processing the result. As speech data, the speech data to be uttered, the segmentation information of the phrase, the phoneme label sequence for each phrase, the discourse function tag assigned to each phrase, and the speaker assumed as the robot's voice are identified Information indicating the relationship between the robot and the conversation partner, information indicating the relationship between the speaker and the conversation partner (“near”, “far” or “middle”), and the attitude to be attached to each phrase Information including discourse mode information indicating emotions and the like is given to the head motion generation device 86. Each phoneme label is also attached with information for specifying the utterance time in the phrase of the speech corresponding to the phoneme.

  In particular, referring to FIG. 6, for example, the computer program constituting the whispering position generation unit 150 determines whether or not the discourse function tag assigned to the given phrase is a whirl target (step 170). If the discourse function tag is not to be spoken, information indicating that there is no whisper is given to the head motion command generation unit 104. If the discourse function tag is a target to be spoken, information on the speaker specified by the utterance data from the probability model group 100, information indicating the relationship between the speaker and the conversation partner, attitude, emotion, etc. A probability model is selected based on the displayed discourse mode information and the designated discourse function tag (step 172). Based on the probability model thus selected, a random number generated by a known method is used to select what kind of action is to be performed by the robot as a whispering-related action (step 174). Once the action is determined, the action determines the position within the phrase (the position specified in relation to the syllable) and the position at which the action ends. (Step 176). The timing information obtained in this way is given to the head operation command generation unit 104 (see FIG. 5).

  Similarly, the swing position generation unit 152 generates timing information related to the head swing, and the neck movement position generation unit 154 generates timing information related to the head movement, and supplies the head movement command generation unit 104 with the timing information.

  The head movement command generation unit 104 creates a head movement command in the following manner, for example, regarding whispering. Referring to FIG. 10, for the sake of explanation, it is assumed that the speech of a given utterance includes phrases 210 and 212, which are respectively provided with discourse function flags k and bc. In order to simplify the explanation and the figure, here, information for identifying the speaker, information indicating the relationship between the speaker and the conversation partner, discourse mode information indicating the attitude / feeling, and the like are not shown. Note that the phrase 210 includes the last syllable 222 and the previous syllable sequence 220.

  By the process of determining a whispering action using a random number and a probability model (step 174 in FIG. 6), a single whisper (nd) is made for the phrase 210 of the discourse function flag k, and a plurality of whisper mnd for the phrase 212 of the discourse function flag bc. Are respectively selected.

  The head motion command generation unit 104 first generates a flat motion command 240 as a motion command for the robot. By superimposing the shape of the motion corresponding to the motion assigned to each phrase on this motion command at a predetermined timing within the phrase for each motion command, the final motion is determined. An operation command 246 is obtained.

  For the phrase 210, since the selected whispering action is a single whispering, the shape shown in FIG. 7 is superimposed after the start position of the final syllable 222. On the other hand, for the phrase 212, since the selected whispering operation is a plurality of whispering operations, the plural whirling shapes shown in FIG. As a result of the above processing, the head movement command 242 shown in FIG. 10 is obtained.

  In the present embodiment, the face-up shape model shown in FIG. 9 is superimposed on the head movement command 242 when the robot 80 speaks. As a result, the head movement command (for the speech section) 246 shown in FIG. 10 is obtained. By giving this head movement command to the robot control unit 90 and the voice reproduction unit 92 shown in FIG. 5, the head of the robot 80 is assumed to be the content of the utterance, the relationship assumed between the robot and the conversation partner, the robot Whisks, swings and raises his head with natural movements according to the attitude and feelings he should show when speaking.

  As described above, according to this embodiment, the results obtained by experiments in advance are statistically processed, and depending on the relationship / attitude / emotion between the speaker / speaker and the conversation partner, A probability model for determining the probability of performing such a head movement is prepared in advance. When a robot speaks, a talk function tag (for example, k, g, q, bc, etc. when whispering) that has a high frequency of head movement is used to connect the speaker / speaker assumed by the robot and the conversation partner. Select a probability model according to the relationship / designated attitude / emotion and discourse function tag, determine what kind of head movement to perform by random numbers, and determine the shape of the determined head movement Each time it is superimposed on the head movement command at a preset timing. Such processing is not performed for other discourse tags. By carrying out such processing independently for each of the head swing and the neck movement, the obtained head movement command is sent to the robot control section 90, so that the robot control section 90 moves the head around the three axes. Control. As a result, in a natural way, depending on the assumed speaker and relationship with the conversation partner, the specified attitude and emotion, and the discourse function tag attached to each phrase. The robot's head can be moved. Therefore, the communication between the robot and the conversation partner is smooth, and the conversation partner is the robot's emotion (assumed as what is assumed), the attitude, and the relationship between the robot and the conversation partner (assumed) ) Can be understood naturally.

[Second Embodiment]
In the first embodiment, a predetermined shape is superimposed on the head movement command only when a specific discourse function tag is used. However, head motion commands may be generated for all possible discourse function tags if necessary. The second embodiment has such a configuration.

  Referring to FIG. 11, the head movement position generation unit 280 of the robot according to this embodiment can be used in place of the head movement position generation unit 102 shown in FIGS. The head movement position generation unit 280 can also be realized by the cooperation of the computer hardware and the computer program by a computer program executed on the computer hardware.

  The head movement position generation unit 280 receives the utterance data from the utterance data storage device 88, and generates whether or not to generate a whisper according to an appropriate probability model determined by the utterance data in the probability model group 100. In such a case, when the timing is determined, the whirling position generation unit 290 for outputting as information indicating the whirling timing, and the whispering position generation unit 290, the probability model group 100 is based on the utterance data. The swing position for determining whether or not to perform the swing motion based on an appropriate probability model determined by the utterance data, and determining when to do so and outputting it as information indicating the swing timing Similar to the generation unit 292, the whirling position generation unit 290, and the swing position generation unit 292, the probability model group 100 is based on the utterance data. Decide whether or not to perform neck neck movement using an appropriate probabilistic model defined by the utterance data, and if so, determine when to do so and generate neck neck position for output as information indicating head neck timing Part 294.

  The winding position generation unit 290, the swing position generation unit 292, and the neck position generation unit 294 have similar configurations to each other.

  The program routine that realizes the whispering position generation unit 290 includes information specifying the relationship / attitude and emotion between the speaker / speaker and the conversation partner assumed for the robot, and the utterance data. Based on the discourse function tag attached to each phrase, step 310 for selecting a probability model for determining a whirling action determined by the value of these information from within the probability model group 100 shown in FIG. In accordance with the determined probability model and random numbers, step 174 for determining what kind of motion is to be performed by the robot as the motion belonging to the stroking, and according to the type of motion determined in step 174, the syllable position in the phrase 176 for determining whether or not to generate, outputting information indicating the position to be fired, and ending the processing.

  Similarly, the program routine for realizing the swing position generation unit 292 also includes information for specifying the relationship / attitude and emotion between the speaker / speaker and the conversation partner assumed for the robot, attached to the utterance data. A step 320 of selecting a probability model for determining a swing motion determined by the value of these information from within the probability model group 100 shown in FIG. 5 based on the discourse function tag attached to each phrase of the utterance data; In accordance with the probability model selected in step 320 and the random number, step 184 for determining what kind of motion is to be performed by the robot as the motion belonging to the head swing, and the phrase type according to the type of motion determined in step 184 And step 186 for determining at which syllable position the head swing is generated, outputting information indicating the head swing position, and ending the process.

  Similarly, the program routine for realizing the head position generation unit 294 also includes information specifying the relationship / attitude and emotion between the speaker / speaker assumed to be a robot and the conversation partner attached to the utterance data. A step 330 of selecting a probability model for determining the head-and-head movement determined by the value of the information from within the probability model group 100 shown in FIG. 5 based on the discourse function tag attached to each phrase of the utterance data; In accordance with the probability model selected in step 330 and the random number, step 194 for determining what kind of motion is to be performed by the robot as the motion belonging to the swing, and according to the type of motion determined in step 194 And 196 deciding at which syllable position the neck fold is to be generated, outputting information indicating the neck fold position, and ending the process.

  In this embodiment, the head movement control similar to that described in the first embodiment can be performed for all the discourse function tags. The head movement of the robot becomes more diverse, and the movement close to the actual human head movement can be realized.

  In both of the first and second embodiments, head movement commands for whispering, swinging, and necking are generated independently. A restriction may be provided, such as “no whispering operation is performed” or “multiple whirling and plural swinging are not performed simultaneously”. Furthermore, even if steps 310, 320, and 330 in FIG. 11 are combined into one and three probability models are always selected as a set by the discourse function tag, the same as in the second embodiment. Become. Only one or two of the whirling position generation unit 290, the swing position generation unit 292, and the neck position generation unit 294 are selected and operated by a random number, and the processing units that are not selected are simultaneously operated. Can also be disabled.

  In the above-described embodiment, three types of stroking, swinging and necking are realized, but only one of these or only a combination of any two may be adopted.

  Moreover, in the said embodiment, the face-up shape is superimposed at the time of utterance about the head movement command regarding a whisper. However, such superposition of the face-up shape is not essential, and even if the face-up shape is superimposed, the angle is not limited to 3 degrees, and can be arbitrarily selected within a natural range, and the face-up angle itself Can be changed when the robot is in operation, and as already mentioned as a criterion for that, it is also possible to use the speakers, relationships with the conversation partner, attitudes and emotions, etc. assumed for the robot Needless to say.

  In the above embodiment, the discourse function tag is attached to each phrase. This is because the movement of the head often changes in such units. However, it goes without saying that the unit with the discourse function tag is not limited to phrases. In the above embodiment, it is assumed that a syllable sequence is attached to speech data and information indicating the start time of each syllable is obtained. With such a configuration, it is possible to easily determine the head movement start timing and the like. However, there may be a configuration that does not use information on the start time of phonemes. For example, when head movement is caused in the last syllable of an utterance, the power of the utterance is traced back from the end of the phrase, and after the first peak (the end of the last syllable is found), MFCC (Mel Frequency Cepstrum Coefficient ) And the time when the ΔMFCC value exceeds a certain threshold value is determined as the start position of the last syllable, the same processing as in the above embodiment can be realized. When head movement is performed over the entire phrase, such a problem does not occur because the beginning and end of the phrase are clear. Alternatively, in each phrase, only information related to the start position of the syllable related to the start timing of the head movement may be created in advance by batch processing and attached to the utterance data.

  Furthermore, it goes without saying that the same idea as disclosed above for head movements can be applied not only to head movements but also to limb movements, upper body movements, eye movements, etc. .

  The embodiment disclosed herein is merely an example, and the present invention is not limited to the above-described embodiment. The scope of the present invention is indicated by each claim of the claims after taking into account the description of the detailed description of the invention, and all modifications within the meaning and scope equivalent to the wording described therein are included. Including.

30 Speaker 32, 34, 36, 38, 40, 42, 44 Marker 80 Robot 82, 108, 110, 112, 114 Storage device 86 Head motion generation device 88 Utterance data storage device 90 Robot control unit 92 Voice reproduction unit 100 Probability model group 102 Head movement position generation unit 104 Head movement command generation unit 106 Head movement command generation unit for discourse section 114 Storage device 130, 132 Probability model 150, 290 Stroke position generation unit 152, 292 Swing position generation unit 154,294 Neck position generation unit

Claims (5)

A head motion control information generating device for generating control information for controlling the movement of the head of a humanoid robot in synchronization with the sound generated by the robot,
The speech is defined by speech data previously divided into predetermined units, and each of the predetermined units includes a discourse function tag indicating a discourse function based on speech uttered by the sound data assigned to the unit. Is attached,
Probability model storage means for storing a probability model that defines at what probability a plurality of head movements are executed for each discourse function tag;
A probability model for selecting a probability model determined based on an annotation attached to the unit received from the speech data in the predetermined unit from among the probability models stored in the probability model storage means A selection means;
To generate a head motion command corresponding to the input predetermined unit of speech data with a probability according to the probability model selected by the probability model selection means, and to output it to the control unit of the humanoid robot type robot A head operation control information generating device, comprising: The head movement control information generating device according to claim 1,
The voice data includes a conversation mode comprising any combination of speaker identification information for identifying a speaker, relationship identification information for identifying a relationship between a speaker and a conversation partner, and non-linguistic information at the time of utterance. The specified discourse mode information is attached.
The probability model selection means stores a plurality of probability models prepared in advance according to a combination of the discourse mode information and the discourse function tag,
The probability model selecting means includes means for selecting one of the plurality of probability models according to the discourse mode information attached to the voice data and the input predetermined unit. The head movement control information generation device according to claim 1. The head movement control information generating device according to claim 1 or 2,
The head command generation means includes
A head motion determining means for determining whether or not to generate a head motion corresponding to the inputted predetermined unit of voice data with a probability according to the probability model selected by the probability model selecting means; ,
For each of the plurality of head movements, shape information storage means for storing shape information defining temporal movement of the head corresponding to the head movement;
In response to the fact that the head movement determining means determines to generate any head movement, the shape information corresponding to the head movement is obtained at a timing determined in advance by the type of the head movement. The head movement according to claim 1 or 2, further comprising: a shape reading means for reading out from the shape information storage means and generating a control command for controlling the head of the robot and outputting it to the control unit. Control information generator. The head movement control information generating device according to claim 3,
The movement of the robot head is related to the pitch angle among the rotations about the three axes of the robot head,
The head command generation means further includes:
An utterance head shape storage means for storing a shape of a head movement prepared in advance, which defines temporal movement of the head when the robot speaks;
A control command connected between the shape reading means and the control unit, and in response to the fact that the voice data indicates that the robot speaks is transmitted to the control command output by the shape reading means. The head motion control information generating apparatus according to claim 3, further comprising: means for superimposing a head motion shape read from the head shape storage unit at the time of utterance and giving the head motion shape to the control unit. The head movement control information generating device according to any one of claims 1 to 3,
The movement of the robot head is related to rotation around any combination of the three axes of the robot head,
The probability model storage means, the probability model selection means, and the head command generation means all generate head control commands independently for each axis constituting the arbitrary combination. The head movement control information generation device according to any one of the above.

Images