WO2023228433A1

WO2023228433A1 - Line-of-sight control device and method, non-temporary storage medium, and computer program

Info

Publication number: WO2023228433A1
Application number: PCT/JP2022/038670
Authority: WO
Inventors: カルロストシノリイシイ; 太健新谷
Original assignee: 国立研究開発法人理化学研究所
Priority date: 2022-05-27
Filing date: 2022-10-18
Publication date: 2023-11-30

Abstract

A line-of-sight control device, which provides a device for controlling line of sight such that the line of sight of an agent when in dialogue with a person can be more naturally realized, includes: a line-of-sight movement generation unit 370 for determining a line-of-sight direction of a robot, on the basis of a combination of the role of the robot in a multi-person dialogue and a state of dialogue flow, in response to reaching a timing for determining the line-of-sight direction; and a line-of-sight movement control unit 372 for generating control parameter for controlling the orientation of the face of the robot and the direction of the eyeballs of the robot, in response to the line-of-sight direction of the robot being set by a line-of-sight direction setting means.

Description

Gaze control device and method, non-transitory storage medium, and computer program

This invention relates to a technology for controlling the line of sight of an agent such as a robot when interacting with a person. This application claims priority based on Japanese Application No. 2022-086674 filed on May 27, 2022, and incorporates all the contents described in the said Japanese application.

Various conversational agents, including robots and virtual agents, are entering society. Opportunities for people to interact with conversational agents are beginning to increase. Agents such as robots are increasingly being seen in places where people gather, such as retail stores, hotel lobbies, and train stations.

When an agent such as a robot interacts with a person, not only the linguistic information contained in utterances, but also non-verbal information such as gestures, facial expressions, prosody, and gaze are important for the conversation to proceed smoothly. It is also believed that verbal and non-verbal information serve as a medium through which people consciously or unconsciously express their individuality.

Therefore, when agents such as robots interact with people, not only verbal information but also non-verbal information has great significance. In particular, when an agent interacts with a person, controlling the agent's line of sight is important not only for facilitating the interaction but also for allowing the agent to express its individuality.

Non-Patent Document 1 states that by observing the behavior of two people when they are doing a certain task, extroverts and introverts are able to determine the amount of time they spend looking at the other person and the amount of time they spend on the task. It has been reported that there is a difference in the amount of time the children spend their time focusing on the target. Non-Patent Document 1 reports the results of an experiment regarding the impression that a person who interacts with a robot receives by controlling the robot's movements based on the results.

However, the report in Non-Patent Document 1 is based on human behavior when performing a predetermined task in a specific environment. In general dialogue, guidelines regarding how to control the robot's line of sight cannot be obtained from the disclosure of Non-Patent Document 1.

On the other hand, Patent Document 1 listed below discloses a technology related to controlling the line of sight of a robot when interacting with one or more people. In the technique disclosed in Patent Document 1, the robot's line of sight is directed toward the person who is speaking. However, even if one person speaks while another person speaks, the robot's gaze generally does not turn in the direction of the new speaker. The robot's degree of interest in the utterance of the first speaker is calculated, and if that value is lower than a threshold, the robot directs its gaze in the direction of the new utterance.

According to Non-Patent Document 1, controlling the robot's line of sight in this manner has the effect of preventing the robot from unnaturally moving its line of sight here and there in a short period of time.

Japanese Patent Application Publication No. 2022-057507

During human dialogue, a characteristic movement of the gaze direction can be seen around the time when the right to speak (turn) is transferred from one person to another (when the turn is changed). Not only that, even when a speaker is speaking, other speakers are not necessarily facing the speaker's direction. People may look away from the speaker, or when there are three or more participants, look towards another participant who is not speaking. The same applies to the speaker's line of sight. Furthermore, the movement of the gaze direction also differs depending on a person's personality (individuality). The technology disclosed in Patent Document 1 does not control the robot's line of sight based on such information. Therefore, the resulting movement of the robot's line of sight is not necessarily natural.

Therefore, there is a need for a line-of-sight control device that allows agents such as robots to more naturally control their line-of-sight when interacting with people.

A line-of-sight control device according to a first aspect of the present invention is a line-of-sight control device for controlling the line of sight of a robot in a multi-person interaction, and in response to a timing for determining a line-of-sight direction, A line-of-sight direction setting means for determining the line-of-sight direction of the robot based on a combination of the role of the robot in a multi-person dialogue and the state of the dialogue flow; and control parameter generation means for generating control parameters for controlling the direction of the robot's face and the direction of the eyeballs.

Preferably, the line-of-sight direction setting means calculates the probability that the plurality of participants will each face in a plurality of predetermined directions according to a combination of each role of the plurality of participants in the multiperson dialogue and the state of the dialogue flow. A direction determination model storage means for storing a direction determination model determined for each role, and a direction determination model storage means for storing a direction determination model determined for each role, and a direction determination model that stores the robot's role and the state of the dialogue flow from the direction determination model in response to the timing for determining the line of sight direction. It includes a probability distribution extracting means for extracting a probability distribution according to the combination, and a first sampling means for sampling the line of sight direction of the robot from the probability distribution extracted by the probability distribution extracting means.

More preferably, the plurality of directions of the direction determination model include the directions of the plurality of participants and a gaze aversion direction that is different from any of the directions of the plurality of participants.

More preferably, the line-of-sight direction setting means further includes a line-of-sight direction model for storing a line-of-sight direction model that is a probabilistic model for probabilistically determining the line-of-sight direction according to a combination of the role of the robot and the state of the dialogue flow. a deflection direction model storage means; and a second sampling means for sampling the direction in which the robot's gaze is diverted from the gaze aversion direction model in response to the fact that the gaze direction sampled by the first sampling device is the gaze aversion direction. including.

Preferably, the line-of-sight control device further includes a continuation for calculating the duration of the robot's line-of-sight according to a combination of the role of the robot, the state of the dialogue flow, and the line-of-sight direction determined by the line-of-sight direction setting means. Contains a time calculation section.

More preferably, the timing for determining the line of sight direction is different between when the dialogue flow is in a turn change state and when it is not.

More preferably, the timing for determining the line of sight direction is a predetermined timing during a turn change state when the state of the dialogue flow is a turn change state, and a predetermined timing immediately before the turn change state when the state of the dialogue flow is not a turn change state. This is the timing when the duration calculated by the duration calculation unit expires.

Preferably, the line-of-sight direction setting means allows the plurality of participants to be predetermined according to the combination of the roles of the plurality of participants in the multiperson dialogue, the state of the dialogue flow, and the expected personality of the robot. A direction determination model storage means for storing a direction determination model that determines the probability of facing each of a plurality of directions for each role; Probability distribution extraction means for extracting a probability distribution according to the combination of role, dialogue flow state, and personality, and first sampling for sampling the robot's gaze direction from the probability distribution extracted by the probability distribution extraction means. means.

More preferably, the line-of-sight direction setting means further stores a line-of-sight direction model that is a probabilistic model for probabilistically determining the line-of-sight direction according to a combination of the robot's role, dialogue flow state, and personality. and a second line of sight for sampling the direction in which the robot looks away from the line-of-sight direction model in response to the line-of-sight direction sampled by the first sampling means being the line-of-sight direction. sampling means.

Preferably, the line of sight control device further calculates the duration of the robot's line of sight according to a combination of the robot's role, the state of the dialogue flow, its personality, and the line of sight direction determined by the line of sight direction setting means. Contains a duration calculation section for.

A line-of-sight control method according to a second aspect of the invention is a line-of-sight control method implemented by a computer for controlling the line-of-sight of a robot in a multi-person interaction, wherein the computer determines the timing for determining the line-of-sight direction. In response to this, the computer determines the robot's line of sight direction based on the combination of the robot's role in the multi-person dialogue and the state of the dialogue flow, and the computer determines the robot's line of sight direction in the step of determining the line of sight direction. and generating control parameters for controlling the direction of the robot's face and the direction of the eyes in response to the determination of the robot.

A computer program according to a third aspect of the present invention is a computer program for controlling the line of sight of a robot in a multi-person interaction, and the computer program controls the computer to control the line of sight of a robot in response to timing for determining the line of sight direction. , a line-of-sight direction setting means for determining the line-of-sight direction of the robot based on a combination of the role of the robot in a multi-person dialogue and the state of the dialogue flow, and a response to the line-of-sight direction of the robot being determined by the line-of-sight direction setting means. The controller functions as a control parameter generation means for generating control parameters for controlling the direction of the robot's face and the direction of the eyeballs.

As described above, according to the present invention, it is possible to provide a line-of-sight control device and method, as well as a computer program, which can more naturally realize the line-of-sight of an agent such as a robot when interacting with a person.

FIG. 1 is a diagram schematically showing the settings of a preliminary experiment. FIG. 2 is a schematic diagram for explaining a method for tagging utterances of each participant in a preliminary experiment. FIG. 3 is a schematic diagram for explaining the timing of turn changes in utterances. FIG. 4 is a graph showing the frequency of each speaker's gaze direction during turn changes in a three-way dialogue in a preliminary experiment. FIG. 5 is a graph showing the temporal ratio of each speaker's line of sight direction in a three-way dialogue in a preliminary experiment, except at the time of turn change. FIG. 6 is a diagram showing, in a table format, the proportion of each participant in a dialogue who directs his/her gaze toward each other participant while speaking and the time during which the conversation averts his/her gaze. FIG. 7 is a graph showing the temporal ratio of the line of sight direction of a particular extroverted participant in each of the participating roles in the dialogue in the three-way dialogue of the preliminary experiment. FIG. 8 is a graph showing the temporal ratio of the direction of gaze of a specific introverted participant in each of the participating roles in the dialogue in the three-way dialogue of the preliminary experiment. FIG. 9 is a graph showing the distribution of the times when an extroverted dialogue participant turned his/her gaze toward each participant when the speaker was the speaker. FIG. 10 is a graph showing the distribution of times when an introverted dialogue participant turns his/her gaze toward each participant when the speaker is the speaker. FIG. 11 is a graph showing the frequency of the number of times participants in each role averted their gaze during speech. FIG. 12 is a graph showing a histogram showing the distribution of times when dialogue participants avert their gaze and an approximate curve thereof. FIG. 13 is a diagram showing the ratio of pupil positions when dialogue participants avert their line of sight. FIG. 14 is a diagram showing the ratio of pupil positions when an extroverted conversation participant averts his/her line of sight. FIG. 15 is a diagram showing the ratio of pupil positions when an introverted conversation participant averts his/her line of sight. FIG. 16 is a block diagram showing the hardware configuration of a conversational robot system 150 according to an embodiment of the present invention. FIG. 17 is a diagram showing the outer shape of the robot shown in FIG. 16. FIG. 18 is a block diagram showing the hardware configuration of the computer shown in FIG. 16. FIG. 19 is a block diagram showing the functional configuration of a line-of-sight control device realized by the robot control device shown in FIG. 16. FIG. 20 is a schematic diagram showing the configuration of the line-of-sight direction model shown in FIG. 19. FIG. 21 is a diagram illustrating an example of the gaze direction model for each individual and role during utterance shown in FIG. 20. FIG. 22 is a schematic diagram showing an example of the configuration of the line-of-sight aversion model shown in FIG. 19. FIG. 23 is a schematic diagram showing an example of the individual-by-individual, turn-changing, and gaze-averting gaze direction models shown in FIG. 22. FIG. 24 is a flowchart showing a control structure of a computer program that causes a computer to function as a line-of-sight control device in an embodiment of the present invention. FIG. 25 is a flowchart showing a control structure of a computer program that causes a computer to execute the state sensing step shown in FIG. 24. FIG. 26 is a flowchart showing a control structure of a computer program that causes a computer to execute the step of determining the line-of-sight direction and duration shown in FIG. 24. FIG. 27 is a flowchart showing a control structure of a computer program that causes a computer to execute the line-of-sight direction determining step shown in FIG. 24. FIG. 28 is a flowchart showing a control structure of a computer program that causes a computer to execute the step of determining the gaze duration time shown in FIG. 26. FIG. 29 is a flowchart showing a control structure of a computer program that causes a computer to execute the step of determining the direction of the line of sight when the line of sight is averted, shown in FIG. FIG. 30 is a diagram showing the arrangement of video screens for evaluation experiments. FIG. 31 is a graph showing the results of the evaluation experiment. FIG. 32 is a graph showing the results of the evaluation experiment.

In the following description and drawings, the same parts are given the same reference numerals. Therefore, detailed description thereof will not be repeated. Note that the following embodiment uses a human-shaped robot as an agent. However, the invention is not limited to such embodiments. The robot does not necessarily have to be human-shaped; it can be of any shape as long as it has eyeballs and can talk. This also applies to virtual agents that do not have a three-dimensional entity such as robots, but are expressed as two-dimensional images, or virtual agents that are expressed as three-dimensional images in virtual space. The invention can be applied.

1st preliminary experiment 1. Purpose of data collection In order to make the robot's line of sight movements natural during multiple interactions, it is necessary to investigate the line of sight of actual humans and collect the necessary information. To this end, we conducted the following preliminary experiments. Figure 1 shows the setup of the preliminary experiment.

Referring to FIG. 1, a preliminary experiment was conducted as a three-way dialogue 50. The participants in the three-way dialogue 50 are

participants

60, 62, and 64. In FIG. 1,

participants

60, 62, and 64 are all standing, but in a preliminary experiment, the direction of their line of sight is important, and it is necessary to fix each other's positions. To this end, in a preliminary experiment, we prepared chairs as described below and asked participants to sit on these chairs and have a conversation. In a preliminary experiment, we asked these participants to freely interact in a manner similar to a waterfront meeting, where there is no specific purpose and no clear way to proceed with the discussion, and we collected information about the direction of each participant's line of sight at that time. collected. In such dialogues, there are often no clear rules for how to proceed with the discussion, but the conversation is often lively. In order for agents to advance into society, it is desirable to be able to have a wide range of interactions, from purposeful conversations such as meetings to casual conversations.

By the way, when a three-way dialogue is held like this, who speaks, that is, who has the right to speak, has great significance. Here, such a right will be referred to as a right to speak, and a participant who holds the right to speak will be referred to as a speaker. Other participants will have the role of listeners. This is called a listener here. Speakers will be alternated as appropriate. This exchange of speaking rights is called turn exchange here. Each time there is a turn, each participant in the dialogue becomes a speaker or a listener. The position of each participant who participates in the dialogue is referred to here as a participating role or simply a role.

It is known from past research that participants in dialogue consciously or unconsciously recognize their own roles depending on the situation of the dialogue. That is, each participant always recognizes whether he or she is the speaker, a listener who is primarily being addressed by the speaker, or a listener who is only slightly involved in the conversation. Here, a listener who is mainly addressed by a speaker is called a main listener (ML), and a listener who is only slightly involved in the conversation is called a sublistener (SL).

It is also known from previous research that there is a correlation between the role of each participant in a conversation and their gaze movements. In a preliminary experiment, in accordance with this idea, information was collected regarding how the participants in the three-way dialogue 50 moved their line of sight according to their respective roles.

2. Data analysis (labeling)
A. Dataset Nine people, both male and female, participated in the preliminary experiment. These participants formed 6 groups with 3 people in each group. Within each group, three participants engaged in free dialogue. The content of the dialogue is free talk. The duration of the conversations ranged from 20 to 30 minutes.

Participants in each group spoke while sitting on three chairs placed at the three vertices of a triangle. A camera was installed in front of each participant, and each participant's face and body movements were photographed from directly in front. Each participant wore a headset microphone, and audio data of each participant and video of the entire discussion were collected. After the dialogue was completed, a transcript of the dialogue was created from these recordings and audio data.

Next, the annotator labeled the transcribed data, paying attention to the direction of the line of sight, the direction of the eyeballs, the presence or absence of speaking rights, turn changes, and each participant's role in the dialogue. FIG. 2 shows an example of labeled dialogue data obtained in this way. The following analysis was performed on the labeled dialogue data thus obtained.

B. Data Analysis FIG. 2 shows examples of

gaze label sequences

80, 82, and 84 for three participants A, B, and C. Referring to FIG. 2, for example, the label "B" in the gaze label column 80 of participant A indicates that participant A was looking at participant B during this time period. Similarly, the label "C" in the line of sight label string 80 indicates that participant A was looking at participant C. “Averted gaze” in the line of sight label column 80 indicates that participant A was not looking at either participant B or participant C. In other words, this label indicates that Participant A was looking away from other participants during this period. In this specification, a participant not looking at another participant in this manner is referred to as a "look away", and its duration is referred to as a "look away duration time".

The line-of-sight label strings 82 and 84 were also created in the same way as the line-of-sight label string 80.

C. Direction of Eyeballs When a person looks at a person, there is not a large discrepancy between the label and the position of the face and eyes. On the other hand, when looking away, the line of sight, especially the position of the eyes, is very important. However, considering that there is a limit to the accuracy of determining the direction of the line of sight, in this preliminary experiment, the direction of the eyeball corresponding to the position of the pupil was labeled in nine directions consisting of the top, bottom, left, and right, including the center and diagonal of the center. went.

Below the portion of the line of sight label row 80 in FIG. 2 labeled "averted gaze", a label indicating which direction participant A's gaze was facing at that time is written as additional information.

For example, a period marked "above" indicates that Participant A's line of sight was directed upward, and a period marked "Top right" indicates that Participant A's line of sight was directed upward to the right. . At this time, the line of sight includes not only the angle of the face but also the direction of the eyeballs, and the direction comprehensively determined from these two elements is the direction of the line of sight. Similar labeling was performed for Participant B and Participant C.

D. Presence of speaking right and turn change Figure 3 shows the actual turn change. In this example, participant A initially has the right to speak and is making the utterance 100. This is indicated by turn alternation label 102. Next, speaker C takes the right to speak and makes an utterance 104. This is indicated by turn alternation label 106. The annotator determines the timing of the turn change. For example, the turn change label 106 clearly indicates that the right to speak has been transferred from participant A to participant C.

In this experiment, a total of 2 seconds, 1 second before and after the timing when the right to speak was exchanged, was defined as the turn change 108, and in particular, the movement in the direction of each participant's line of sight during this period was analyzed from the line of sight label.

E. Roles In line-of-sight analysis in three-party dialogue, the role of each participant in the dialogue is important. In this preliminary experiment, we defined the speaker, the main listener, and the sublistener as described above, and analyzed the line of sight for each of them separately at the time of turn change and at other times. Note that the right to speak changes when the turn changes. In order to clarify the role of each participant at the time of turn change, here, the speaker at the time of turn change refers to the person who has taken the right to speak due to turn change. In the case of FIG. 3, participant C is the speaker. The main listener at the time of a turn change is a person who previously had the right to speak and who has given up the right to speak to the speaker due to the turn change. In the case of FIG. 3, participant A is the main listener. A person who is not involved in this exchange of speaking rights is a sublistener. In the case of FIG. 3, participant B is the sublistener.

F. Analysis results a. During turn change Figure 4 shows the changes in the rate of eye movement for each role during turn change. In FIG. 4, the horizontal axis is time (unit: seconds). At time t=0.0, the speaker takes a turn and starts speaking. The vertical axis represents the ratio (0.0-1.0) based on statistics of where the speaker is looking at time t. For example, looking at the values on the vertical axis when the horizontal axis t=0.0 seconds in FIG. 3(A), ML, SL, and gaze aversion are 0.37, 0.19, and 0.44, respectively. This means that out of the total number of the speakers' line of sight when they started speaking, 0.37 were looking at the main listener, 0.19 were looking at the sublistener, and the remaining 0.44 were looking away. represents something. The line of sight direction is calculated every 0.1 seconds, and Figure 3 shows the proportion of the speaker's line of sight in a total of 2 seconds, 1 second before and after the moment when the speaker takes the right to speak and starts speaking. , which shows the transition over time.

In addition, in the embodiment described later, the 2-second interval at the time of turn change is set from -1.0 seconds to -0.3 seconds, and from -0.3 seconds to 0 seconds, with the timing of turn change being the origin (0 seconds). It is divided into three sections: .3 seconds, and from 0.3 seconds to 1 second, and the line-of-sight direction is determined at the beginning of each section. Therefore, in a preliminary experiment, statistics are taken separately for each of these sections and a model is created.

・Speaker (SP)
FIG. 4(A) shows the change in the percentage of the speaker's (SP) gaze during turn changes. One second before taking a turn, the speaker has a high percentage of looking at the main listener (ML), that is, the previous speaker. However, as the speaker takes the right to speak and approaches the point in time (t=0.0 seconds) when he starts speaking, the rate at which he looks away from the main listener increases. The rate at which the speaker looks away reaches a peak around 0.1 seconds after the speaker starts speaking. After that, the rate at which the speaker's line of sight turns toward the main listener and the rate at which the speaker's line of sight averts become almost equally high, completing the line-of-sight transition during the turn change. These results show that people tend to look at the previous speaker at the beginning of a turn change, look away when they start speaking, and then either continue to look away or look at the previous speaker, the main listener. be.

・Main listener (ML)
FIG. 4(B) shows the transition of the gaze ratio of the main listener who had the right to speak until just before the turn change. At the time of turn change, various situations can be considered, such as the main listener passing the right to speak to the speaker, the speaker taking the right to speak, or the speaker taking the right to speak naturally. However, overall, the main listener knows or has decided who the next speaker will be before the next speaker starts speaking, and the main listener knows or has decided who the next speaker will be before the next speaker starts speaking, and the main listener has the ability to listen to the next speaker from 1 second before the turn takes place to 1 second after giving up the right to speak. During a speech change, the main listener tends to keep looking at the speaker.

・Sublistener (SL)
Finally, Fig. 3(C) shows the transition of the gaze ratio of the sublisteners who did not participate in the turn change. In FIG. 4C, when the horizontal axis t=-1.0 seconds, that is, until 1 second before the speaker speaks, the proportion of sub-listeners looking at the main listener and the proportion looking at the speaker are equally high. However, as the sublistener approaches t=1.0 seconds, that is, 1 second has elapsed since the speaker started speaking, the rate at which the sublistener looks at the speaker increases. Therefore, the sublistener guesses the next speaker from 1 second before the turn change until the turn change, and until then, the sublistener looks at the main listener or the next speaker, and then looks at the next speaker. It is thought that there is a tendency to move.

b. Other than when changing turns (speech period)
The period other than the time of turn change is referred to as the utterance period here. It is thought that the beginning and end of the utterance period are affected by turn alternation. Therefore, in the following line-of-sight analysis during the speech period, the two seconds at the beginning and end of the speech period were excluded from the analysis. Figure 5 shows the ratio of each participant's gaze direction during the speaking period. FIG. 6 shows specific numbers of the ratio of line of sight in the utterance section shown in FIG. 5 in a table format.

In FIG. 5, the horizontal axis represents the direction of the line of sight. The vertical axis indicates the proportion of time spent facing each direction relative to the total time.

-Speaker Referring to FIG. 5(A), the rate at which speakers look at the main listener while speaking is slightly high. However, the speaker distributes his/her gaze in a well-balanced manner as a whole.

- Main listener Referring to FIG. 5(B), the main listener faces the speaker for nearly 70% of the speaking period, and the proportion of looking at the sublistener is quite low. The rate at which main listeners look away is significantly higher than the rate at which sublisteners look, but it is significantly lower than the rate at which they look at the speaker.

- Sublistener Referring to FIG. 5(C), the tendency for sublisteners is almost the same as for the main listener. In other words, the sublistener also looks in the direction of the speaker for nearly 70% of the utterance period. The rate of looking away is higher than the rate of looking at the main listener, but lower than the rate of looking away by the main listener.

c. Gaze movements based on individuality Based on the awareness that gaze movements may differ depending on individuality, in addition to the above preliminary experiment (first preliminary experiment), we conducted a separate experiment to analyze gaze movements related to individuality. A preliminary experiment (second preliminary experiment) was conducted. In the second preliminary experiment, a three-way dialogue was conducted with 17 participants, both male and female. A total of 14 sessions were conducted, and each session included 10 to 20 minutes of free conversation. The three speakers sat on three chairs, each placed at the three vertices of an equilateral triangle with sides of about 2 meters, and were equipped with headset microphones and acceleration sensors. Separately, each participant's movements were recorded using an image depth camera.

There is an index called BIG5 that expresses individuality. BIG5 describes human personality characteristics using five dimensions. One of them is "extroversion." In this preliminary experiment, we focused on "extraversion" and aimed to examine how participants' gaze movements differed depending on their extroversion.

In this experiment, we selected two participants with particularly different impressions of extroversion from among the 17 participants, and analyzed their gaze movements based on the data obtained from these two participants. did. In the following description, these two participants will be referred to as speaker A and speaker B, respectively. Speaker A gave the impression of being an extrovert, and speaker B gave the impression of being an introvert. By constructing different models using the analysis results regarding the line of sight of these two people, it is possible that different personalities can be expressed by agents such as robots.

・As a speaker Figure 7 (A) shows the time when speaker A looked at the main listener (ML), the time when he looked at the sublistener (SL), and the time when he looked away (when he was the speaker (SP)). GA). FIG. 8(A) similarly shows the results obtained for speaker B.

From FIG. 7(A), it can be seen that when speaker A is the speaker, the proportion of time spent looking not only at the main listener but also at the sublistener is relatively high. That is, it can be seen that the proportion of time spent looking at the main listener and the sublistener is almost equal to each other, and the proportion of time spent looking away from the main listener is also almost the same. Note that the probability that speaker A looks at the main listener is relatively low compared to FIG. 5A, which is the result of the previous preliminary experiment in which no distinction was made based on personality.

On the other hand, from FIG. 8(A), it can be seen that when speaker B is the speaker, he looks away more often than speaker A, and tends not to look at the sublistener much. In particular, when compared with FIG. 5A, it can be seen that the proportion of time in which speaker B looks away is quite high.

- As the main listener FIG. 7(B) shows the ratio of the time when speaker A looked at the speaker, the time when he looked at the sub-listener, and the time when he looked away when he was the main listener. Figure (B) also shows the results obtained for speaker B.

Referring to FIG. 7(B), it can be seen that when speaker A is the main listener, the listener spends a very high percentage of the time looking at the speaker, and tends to look away from time to time. On the other hand, referring to FIG. 8(B), in the case of speaker B, the proportion of time spent looking at the speaker is high, similar to speaker A, but the proportion of time spent looking away from the speaker is also considerably higher than speaker A. I understand. It can also be seen that the proportion of time spent looking at the sublistener is low for both speakers A and B.

Comparing FIG. 7(B) and FIG. 8(B) with FIG. 5(B), it can be seen that they show the same tendency. However, as shown in FIG. 8(B), it is noteworthy that in the case of speaker B, the proportion of time during which he looks away is high.

- As a sublistener FIG. 7(C) shows the ratio of the time when speaker A looked at the speaker, the time when he looked at the main listener, and the time when he looked away when he was a sublistener. FIG. 8(C) similarly shows the results obtained for speaker B.

From Figure 7(C), when speaker A becomes a sub-listener, the proportion of time spent looking at the speaker is very high, as in the case of the main listener, and the proportion of time spent looking at the main listener or looking away. is quite low. On the other hand, as shown in FIG. 8C, when speaker B is a sublistener, the proportion of time spent looking at the speaker is high, but compared to speaker A, the proportion is low. On the other hand, in the case of speaker B, the proportion of time spent looking away is high and is equal to the proportion of time spent looking at the speaker.

Comparing these with FIG. 5(C), the difference in tendency between speaker A and speaker B becomes clearer. That is, compared to the case where individuality is not taken into account, speaker A spends a higher proportion of the time looking at the speaker and a lower proportion of the time he looks away. Conversely, speaker B spends a lower percentage of time looking at the speaker and a higher percentage of time looking away from the speaker, compared to the case where individuality is not considered. In the case of speaker B, the proportion of time spent looking at the main listener is also quite low.

d. Gaze duration time based on individuality For each of Speaker A and Speaker B, the distribution of gaze duration times during the speaking period (period other than the turn change period) when playing the roles of speaker, main listener, and sublistener. Each was analyzed. The results are shown in FIGS. 9(A) to (C) and FIGS. 10(A) to (C), respectively. These figures show a histogram of the duration of the line of sight under each condition and a distribution curve approximated by the χ ² distribution expressed by the following equation.

In the above equation, n is the degree of freedom of the χ ² distribution, scale is a parameter that determines the magnitude of the amplitude in the vertical axis direction of the distribution curve, and loc is a parameter that determines the bias in the horizontal axis direction of the distribution curve.

For each of Speaker A and Speaker B, we calculated the distribution of three types of duration: the duration of looking at the main listener, the duration of looking at the sub-listener, and the duration of looking away when the speaker is the speaker. FIG. 9 shows the results for speaker A, and FIG. 10 shows the results for speaker B.

Referring to FIG. 9(A), for speaker A, the approximate curve of the distribution of the time spent looking at the main listener is given by n=3.4, loc=0.15, and scale=0.2. The approximate curve of the distribution of the sublistener viewing duration shown in FIG. 9B is given by n=4.34, loc=0.0, and scale=0.18. Therefore, when speaker A is the speaker, the duration of looking at the conversation partner (main listener and sublistener) tends to be longer than when speaker B is the speaker. On the other hand, the approximate curve of the distribution of the duration during which speaker A looks away, shown in FIG. 9C, is n=4.67, loc=0.0, and scale=0.12. From this result, it can be seen that in the case of speaker A, the duration of looking away tends to be shorter.

From the above, it can be seen that when Speaker A becomes the speaker, the duration of looking at the conversation partner tends to be long, and the occasional movement of looking away tends to be short.

On the other hand, for speaker B, referring to FIG. 10(A), the approximate curve of the distribution of the time spent looking at the main listener is given by n=7.9, loc=0.0, scale=0.1. . Further, the approximate curve of the distribution of duration when viewing the sublistener shown in FIG. 10(B) is given by n=4.7, loc=0.12, and scale=0.12. From this result, it can be seen that in the case of speaker B, both the duration of looking at the main listener and the duration of looking at the sub-listener are short. Furthermore, the approximate curve of the distribution of the duration of looking away shown in FIG. 10(C) is given by n=3.28, loc=0.1, scale=0.2). From this result, it can be seen that in the case of speaker B, the duration of looking away tends to be longer.

From the above, it can be seen that when speaker B is the speaker, the duration of gaze aversion is longer, and there is a tendency for the speaker to sometimes look at the main listener and sub-listener for a short period of time.

Note that details regarding the duration distribution of line-of-sight directions when speaker A and speaker B are the main listener and sublistener, respectively, are not shown here. However, a trend in duration distribution similar to that observed when these speakers were the speakers was observed.

e. Gaze aversion/interval of gaze aversion Figure 11 shows a scatter plot of the number of times participants averted their gaze during speech in the first preliminary experiment. The horizontal axis indicates the time during speech. The vertical axis indicates the number of times the person looked away from the person. As mentioned above, the time on the horizontal axis is the time excluding the beginning and end of the utterance. FIG. 11 also shows an approximate straight line for averting the line of sight. The regression coefficient r of the approximate straight line is 0.83, 0.82, and 0.67 for FIGS. 11(A), (B), and (C), respectively. From FIGS. 11 and 6, it can be seen that the interval at which the user looks away does not change regardless of the role.

- Direction of the eyeballs The movement of the gaze when a person looks at another person is very simple. Therefore, when a robot looks at a person, it is only necessary to turn the robot's face toward the other person so that the robot does not feel uncomfortable. However, it turns out that simple movement is not enough when the robot averts its gaze from the person it is interacting with. Looking at data from actual three-way conversations between people, it was found that people only moved their eyeballs and looked away. For this reason, in order to analyze the movement of the human eyeballs, which is necessary when the robot averts its line of sight, we specifically focused on where the human eyeballs are pointing when the robot averts its line of sight. We also analyzed the distribution of how long people look away when they look away.

- Duration of gaze aversion Figure 12 shows a histogram of the duration when a person moves to avert their gaze. This distribution is approximated by the following exponential distribution.

However, μ=0.2 and λ=0.63. Note that we considered samples shorter than 0.2 seconds to be unrealistic when considering implementation in an actual agent, so we analyzed only samples longer than 0.2 seconds. The average duration of the samples analyzed was 0.83 seconds, and the median was 0.55 seconds.

・Pupil distribution when looking away When people look away from the other person in a conversation, they mainly move their pupils (eyeballs) in addition to their faces. We also analyzed how many times people move their eyeballs when they look away in this way, and what patterns they follow when looking away.

From the results of the first preliminary experiment, we classified the direction of each speaker's eyeballs when looking away into nine directions: {forward, up, down, left, right, upper left, upper right, lower left, lower right}. The percentage was calculated. The results are shown in FIG. Referring to FIG. 13, it can be seen that when people avert their gaze, they tend to avert their eyes to two places: the center and the lower center. Furthermore, regarding the other seven locations, it can be seen that, although there is some bias, when a person looks away, the pupils are dispersed at approximately the same rate.

- Pupil distribution according to personality The results of the second preliminary experiment were analyzed in the same way as the first preliminary experiment to analyze the distribution of pupils when looking away. FIG. 14 shows the results for speaker A, and FIG. 15 shows the results for speaker B.

Referring to FIGS. 14 and 15, in the second preliminary experiment, as well as the results in the first preliminary experiment, both speaker A and speaker B looked downward rather than upward when averting their gaze. It can be seen that the frequency of deviation is high. However, speaker B tends to look away downward more frequently. Both speakers have a relatively high frequency of having their eyes facing forward (i.e., their pupils are located in the center). In the three-person dialogue data obtained in the second preliminary experiment, the dialogue participants are located diagonally to the left and right. Therefore, when the speaker's eyes are facing forward, it means that the speaker is looking away from either of the conversation partners.

Note that in FIGS. 14 and 15, the proportion of pupils diverted to the lower right is higher than in FIG. 13, and also higher than in the other six locations that are neither in the center nor in the lower center. The cause of this bias is unknown. However, in view of the results shown in FIG. 13, in the embodiments described later, even in the cases of FIG. 14 and FIG. I implemented it.

・Number of gaze aversions In both the first and second experiments, when conversation participants averted their gaze, they tended to move their eyeballs multiple times rather than looking at one point for a long time. There was a tendency that the longer the gaze averted, the more the number of eye movements. Based on the results of analyzing the distribution of the duration for each number of times the robot looks away, in the embodiment described later, the number of times the robot looks away is once when the duration of the look away is 0.7 seconds or less. , 2 times when the time is 1.4 seconds or less, 3 times when the time is 2.1 seconds or less, and so on, and the number of eyeball movements is increased by one time every 0.7 seconds.

Second embodiment 1. System Configuration Referring to FIG. 16, the configuration of a conversational robot system 150 that employs the line-of-sight control system according to the embodiment of the present invention for controlling the robot 160 will be described below. In order for an agent such as a robot to interact with people, it is necessary to implement a speech recognition function, a speech synthesis function, and a dialogue function using these functions. In this embodiment, it is assumed that existing functions are used. Furthermore, in order to carry out a three-way dialogue, it is necessary to dynamically recognize turn changes and the roles of each dialogue participant. Although it is difficult to recognize turn changes, techniques for recognizing turn change timing, such as those described in the following references, can be used.

Chaoran Liu et al, “Turn-taking Estimation Model Based on Joint Embedding of Lexical and Prosodic Contents”, INTERSPEECH 2017, [Online], August 20-24, 2017, Stockholm, Sweden, [Retrieved May 3, 2020 ], Internet <URL: https://isca-speech.org/archive_v0/Interspeech_2017/pdfs/0965.PDF>
The above reference uses a model that detects turn change timing. In contrast, in this embodiment, it is necessary to detect one second before the turn change timing. However, in this case as well, by setting the timing to be detected as 1 second before the turn change and then training the model using the method described in the above reference, it is possible to detect 1 second before the turn change timing. can.

In addition, in the following embodiments, the recognition of the roles of participants including the robot and the recognition of whether the robot should perform an action to acquire the right to speak are given as given from the outside. shall be. However, for example, the identity of the speaker can be determined by determining the location of the speaker and localizing the sound source as described later. Furthermore, it is possible to estimate who the main listener is based on the facial images of the conversation participants and the proportion of time that the speaker faces each participant. If the robot is the main listener, it may start speaking when a turn change is detected. When a robot is the speaker, a method such as selecting a participant who faces the robot for a longer period of time based on facial images of other participants as the main listener may be considered.

FIG. 16 shows the hardware configuration of the conversational robot system 150 according to this embodiment. Referring to FIG. 16, the conversational robot system 150 includes a probability model storage device 172 that stores a probability model for controlling the gaze of the robot 160, and a probability model stored in the probability model storage device 172. The computer 160 includes an operation control PC (Personal Computer) 162 for controlling line-of-sight-related operations, and a network 176 to which the operation control PC 162 is connected. The robot 160 according to this embodiment is for carrying out a three-way dialogue with two other dialogue participants.

The conversational robot system 150 further includes a human position sensor 178 for detecting the position of a conversation partner, and a human position recognition PC 180 connected to the human position sensor 178 and the network 176. The human position sensor 178 is for detecting the position of the conversation partner and providing a detection signal to the human position recognition PC 180. The human position recognition PC 180 is for calculating the position of the dialogue partner as seen from the robot 160 based on the signal from the human position sensor 178 and transmitting the position to the operation control PC 162 via the network 176.

The conversation robot system 150 further uses a microphone 164 to recognize the utterances of the conversation participants, the turn changes, and the roles of each participant based on the audio signals received from the microphone 164, and sends the results via the network 176. and an audio processing PC 166 for transmitting data to the operation control PC 162. The microphone 164 is a microphone array, and the audio processing PC 166 can specify the position of the sound source based on the output of the microphone 164. The audio processing PC 166 has a function of transmitting to the operation control PC 162 information indicating the position of the person who made the utterance, as well as the utterance text, turn change detection information, and the role of each participant. The operation control PC 162 has a function of determining whether or not it should acquire the right to speak based on the content of the uttered text, the detection result of the turn change timing, and the recognition result of the role by the voice processing PC 166.

The conversation robot system 150 further includes a speech synthesis PC 170 for receiving spoken text and generating an audio signal corresponding to the text, and a speaker 168 for receiving an audio signal from the speech synthesis PC 170 and converting it into speech. The quality of the voice synthesized by the voice synthesis PC 170 is selected to be suitable for the appearance of the robot 160.

The conversation robot system 150 further includes a conversation PC 174 connected to the network 176 and configured to generate and output a response to the utterance in response to receiving utterance text from another PC via the network 176. The operation control PC 162 has a function of sending the text of the speaker's utterance to the dialogue PC 174 when the voice processing PC 166 determines that the robot 160 should acquire the right to speak. The dialogue PC 174 generates an uttered text in response to this input and sends it to the operation control PC 162 and the probability model storage device 172, so that the speech synthesis PC 170 and the speaker 168 generate a voice corresponding to the uttered text, and the operation control The PC 162 can control the movement of the audio processing PC 166 in response to speech.

The audio processing PC 166 performs speech recognition on the audio signal output from the microphone 164, further performs sound source localization based on the output of the microphone 164, and indicates the text obtained as a result of the speech recognition and the position of the speaker. and a voice recognition unit 190 having a function of transmitting information to the operation control PC 162. The voice processing PC 166 further uses the text output from the voice recognition unit 190 and the prosody information including the voice power and fundamental frequency F0 obtained from the voice signal of the microphone 164 to perform a turn according to the method disclosed in the above-mentioned reference document. It includes a turn recognition unit 192 that executes a process of detecting a turn change and a process of recognizing the role of each participant in the turn change, and transmits the results to the operation control PC 162.

FIG. 17 shows the appearance of the robot 160. As shown in FIG. 17, the robot 160 is a rather small robot, and can rotate its upper body and head left and right. Furthermore, a large pupil is drawn on the eyeball of the robot 160, and by controlling the rotation angle of the eyeball, the line of sight direction of the robot 160 can be moved vertically and horizontally.

2. Hardware Configuration FIG. 18 is a hardware block diagram of a computer system that operates as the operation control PC 162 shown in FIG. 16, for example. The voice processing PC 166, voice synthesis PC 170, and human position recognition PC 180 shown in FIG. 16 can also be realized by a computer system having almost the same configuration as the operation control PC 162. Here, only the configuration of the computer system 250 as a representative of these PCs 162 will be described, and details of the configuration of each individual PC will not be described.

Referring to FIG. 18, this computer system 250 includes a computer 270 having a DVD (Digital Versatile Disc) drive 302, a keyboard 274, a mouse 276, and a monitor 272, all connected to the computer 270, for interacting with the user. including. Of course, these are examples of configurations for when user interaction is required, and any general hardware and software that can be used for user interaction (e.g. touch panel, voice input, pointing device in general) can be used. Available.

Referring to FIG. 18, in addition to a DVD drive 302, a computer 270 is connected to a CPU (Central Processing Unit) 290, a GPU (Graphics Processing Unit) 292, and a DVD drive 302. bus 310 and , a ROM (Read-Only Memory) 296 connected to the bus 310 and storing boot-up programs for the computer 270, and a RAM connected to the bus 310 and storing instructions constituting the program, system programs, work data, etc. (Random Access Memory) 298 and an SSD (Solid State Drive) 300 that is a nonvolatile memory connected to a bus 310. The SSD 300 is for storing programs executed by the CPU 290 and GPU 292, data used by the programs executed by the CPU 290 and GPU 292, and the like. The computer 270 further includes a network I/F (Interface) 308 that provides a connection to the network 176 that enables communication with other terminals, and a USB (Universal Serial Bus) memory 284 that is removably connected to the computer. 270.

The computer 270 is further connected to the microphone 164, the speaker 168, and the bus 310, reads audio signals, video signals, and text data generated by the CPU 290 and stored in the RAM 298 or the SSD 300 according to instructions from the CPU 290, and performs analog conversion and amplification processing. It includes an audio I/F 304 having a function of driving a speaker 168, digitizing an analog audio signal from a microphone 164, and storing it in an arbitrary address specified by the CPU 290 in the RAM 298 or SSD 300.

In the above embodiment, programs for realizing the functions of the operation control PC 162, the voice processing PC 166, the voice synthesis PC 170, and the human position recognition PC 180 are all stored in the SSD 300, RAM 298, DVD 278, or USB memory 284 shown in FIG. 18, for example. Alternatively, the information may be stored in a storage medium of an external device (not shown) connected via the network I/F 308 and the network 176. Typically, these data and parameters are written into the SSD 300 from the outside, for example, and loaded into the RAM 298 when the computer 270 is executed.

A computer program for operating this computer system so as to realize the functions of the operation control PC 162, voice processing PC 166, voice synthesis PC 170, and human position recognition PC 180 shown in FIG. 7 and their respective components is installed in the DVD drive 302. The data is stored on the DVD 278 and transferred from the DVD drive 302 to the SSD 300. Alternatively, these programs are stored in the USB memory 284, the USB memory 284 is attached to the USB port 306, and the programs are transferred to the SSD 300. Alternatively, this program may be transmitted to computer 270 via network 176 and stored on SSD 300.

The program is loaded into RAM 298 during execution. Of course, a source program may be input using the keyboard 274, monitor 272, and mouse 276, and the compiled object program may be stored in the SSD 300. In the case of a script language as in the above embodiment, a script input using the keyboard 274 or the like may be stored in the SSD 300. In the case of a program that operates on a virtual machine, it is necessary to install the program that functions as a virtual machine on the computer 270 in advance. Neural networks are used for speech recognition, speech synthesis, etc. A trained neural network may be used, or the training may be performed on the talking robot system 150.

The CPU 290 reads the program from the RAM 298 according to the address indicated by an internal register called a program counter (not shown), interprets the instruction, and stores the data necessary for executing the instruction in the RAM 298 and the SSD 300 according to the address specified by the instruction. Or read it from other devices and execute the process specified by the command. The CPU 290 stores the data of the execution result at an address specified by the program, such as the RAM 298, the SSD 300, or a register within the CPU 290. Depending on the address, the computer outputs a command to the robot's actuator, a voice signal, etc. At this time, the value of the program counter is also updated by the program. Computer programs may be loaded directly into RAM 298 from DVD 278 , from USB memory 284 , or via network 176 . Note that in the program executed by the CPU 290, some tasks (mainly numerical calculations) are dispatched to the GPU 292 according to instructions included in the program or according to an analysis result when the CPU 290 executes the instructions.

A program for realizing the functions of each part according to the above-described embodiment by the computer 270 includes a plurality of instructions written and arranged to cause the computer 270 to operate to realize those functions. Some of the basic functions required to execute this instruction are provided by the operating system (OS) running on the computer 270, third party programs, modules of various toolkits installed on the computer 270, or the program execution environment. In some cases, it may be provided. Therefore, this program does not necessarily include all the functions necessary to implement the system and method of this embodiment. This program includes each of the above-mentioned devices and modules by statically linking or dynamically calling appropriate functions or modules in a controlled manner to obtain the desired results. It is sufficient to include only instructions for executing operations as its constituent elements. The manner in which computer 270 operates for this purpose is well known and will not be repeated here.

Note that the GPU 292 is capable of parallel processing, and can execute a large amount of calculations associated with machine learning simultaneously in parallel or in a pipeline manner. For example, parallel computing elements found in a program when the program is compiled or parallel computing elements discovered when the program is executed are dispatched from the CPU 290 to the GPU 292 and executed, and the results are sent directly or to the RAM 298. is returned to the CPU 290 via a predetermined address, and is substituted into a predetermined variable in the program.

3. Functional Configuration FIG. 19 shows the functional configuration of a line-of-sight control system 350, which is a part related to line-of-sight control in the conversation robot system 150 according to this embodiment. Referring to FIG. 19, the line of sight control system 350 includes a role determination unit 360 for determining and outputting the current roles of the robot 160 and other dialogue participants, and a turn change detection unit 362 for detecting a turn change. including. In this embodiment, turn changes and the roles of dialogue participants are detected each time. However, for example, when a three-party dialogue including a robot is performed according to a scenario prepared in advance, the roles and turn turns of each participant are almost specified by the scenario. Therefore, it is not necessary to provide the role determining section 360 and the turn replacement detecting section 362. In the evaluation experiment described below, line of sight control was performed under such conditions.

The line-of-sight control system 350 further includes a line-of-sight direction model 364 for stochastically determining the line-of-sight direction of the robot 160 during a turn change or while speaking, and a line-of-sight model 364 for determining the line-of-sight direction of the robot 160 in an overefficient manner when the robot 160 averts its line of sight. , and a personality information storage unit 368 for storing information regarding the personality (extrovert, introvert, neutral) of the robot 160. Details of the viewing direction model 364 will be described later with reference to FIGS. 20 and 21. Details of the model 366 will be described later with reference to FIGS. 22 and 23. Note that both the line-of-sight direction model 364 and the line-of-sight aversion model 366 are stored in the probability model storage device 172 shown in FIG. Personality information storage section 368 is also actually realized by probability model storage device 172. The probabilistic model storage device 172 is realized by the SSD 300 in FIG.

The line of sight control system 350 further receives information regarding the roles of the robot 160 and other dialogue participants from the role determination unit 360 in response to receiving a signal from the turn change detection unit 362 indicating that a turn change has been detected. In response, the gaze direction model 364 and the gaze motion generation section 370 create parameters for controlling the gaze motion of the robot 160 using the personality

information storage sections

368 and 368. and a line-of-sight movement control unit 372 for controlling various actuators (not shown) for controlling the line-of-sight of the robot 160 according to the parameters set.

4. Model configuration A. Gaze direction model 364
Referring to FIG. 20, the gaze direction model 364 includes a speech gaze direction model 400 that is prepared in advance according to the combination of personality and role, and a gaze duration model that is prepared in advance according to the combination of personality and role. 402, turn change

gaze direction models

404, 406, and 406 for determining gaze directions in the first, second, and third sections of turn change, respectively, which are prepared in advance according to the combination of personalities and roles. 408.

The configuration of the speech gaze direction model 400 is substantially the same as that shown in FIG. 6. For example, what is shown in FIG. 6 is a model of an extroverted participant. Therefore, the speech gaze direction model 400 includes, in addition to the model shown in FIG. 6, a model for introverted participants and a model for neutral participants.

FIG. 21 shows an example of the configuration of the gaze duration model 402. Referring to FIG. 21, the gaze duration model 402 is based on the combination of the role of the robot 160 (speaker, main listener, and sublistener) and the personality assumed for the robot 160 (extrovert, introvert, neutral). For each gaze destination (roles of other participants excluding one's own role), the degree of freedom (n) of the chi ^- square distribution curve, the bias in the horizontal axis direction (loc), and the amplitude in the vertical axis direction (scale) are calculated. It stores the value. By substituting these values into the equation of the chi ^- square distribution, the distribution is specified, and by randomly sampling values from the distribution, the duration can also be determined when determining the line-of-sight direction.

The turn change

gaze direction models

404, 406, and 408 have the same configuration as the speech gaze direction model 400. The difference is that the stored value is limited to each section at the time of turn change.

B. Averted gaze model 366
FIG. 22 shows the configuration of the line-of-sight aversion model 366 shown in FIG. 19. Referring to FIG. 22, the gaze aversion model 366 includes a gaze aversion direction model 450 during turn change and a gaze aversion direction model 452 during speech. All of these models are provided individually.

FIG. 23 shows an example of the configuration of the direction model 450 for looking away when changing turns. Referring to FIG. 23, a turn change gaze aversion direction model 450 models the gaze aversion direction of the extroverted participant shown in FIG. 14, for example. In this embodiment, the turn change time gaze aversion direction model 450 is realized by an array. The elements are arranged in the order in which the center column and right column in FIG. 14 are moved below the leftmost column. Each element indicates a gaze averting direction (nine directions from the upper left to the lower right) and a rate (probability) of averting the gaze in that direction. In FIG. 14, "L" indicates the left, "C" indicates the center, "R" indicates the right, "U" indicates the top, and "D" indicates the bottom. For example, "LD" represents the lower left, and "CD" represents the lower center.

As will be described later, by referring to the turn change time gaze aversion direction model 450, the program can stochastically determine the direction in which the robot 160 averts its gaze during turn change, depending on the role of the robot 160.

5. Program structure A. Overall Configuration FIG. 24 shows, in a flowchart format, the overall configuration of a program executed by the line of sight control system 350 to control the robot 160 according to this embodiment. Referring to FIG. 24, this program is repeatedly activated at regular time intervals, and at the time of activation, the program senses the state of the dialogue (whether or not it is time to change turns, what the role of each dialogue participant is, etc.) at step 480. and step 482 of branching the flow of control depending on whether or not the current time is the timing to determine the viewing direction. For example, in the case of a turn change, the timing for determining the line of sight direction is -1 second, -0.3 second, and 0.3 second, assuming that the timing at which the turn change occurs is 0 seconds. At times other than when changing turns, that is, when speaking, this is the point in time when the duration of the line of sight determined in the previous line of sight direction determination has expired. If the direction is a line-of-sight direction and the duration thereof is 0.7 seconds or more, the process of re-determining the line-of-sight direction is further performed every 0.7 seconds. Regarding these timings, when each event occurs, the time can be set in a timer, and detection can be made according to whether the timer has expired or not.

This program is further executed when the determination in step 482 is affirmative, and generates a line-of-sight direction model 360 according to the combination of the role currently assigned to the robot 160, its personality, and information on whether it is speaking or changing turns. a step 484 of determining the direction of gaze and its duration according to a model selected from , or further from a model selected from gaze aversion models 366; after step 484, and directly if the determination in step 482 is negative; and step 486, in which each part of the robot 160 is controlled according to the parameters set in step 484, and the execution of the program is ended. Note that in this program, processes for voice recognition, voice synthesis, posture control, etc. for the robot are also executed in other steps not shown. In step 486, control based on such processing is also performed.

In step 486, in accordance with the parameters determined in step 484, the direction of the head of the robot 160, the position of the eyeballs, etc. are controlled in order to control the line of sight of the robot 160 in accordance with the progress of the timer. Since time progresses each time the program is executed, the position of the head and the position of the eyes of the robot 160 change each time the program is executed.

B. Step 480 (condition sensing)
FIG. 25 shows the configuration of a program that implements step 480 in FIG. 24 in a flowchart format. Referring to FIG. 25, step 480 includes a step 510 of reading out information indicating the role of the robot 160 and each participant from the role determination section 360 shown in FIG. and step 514 of reading information indicating the individuality given to the robot 160 from the individuality information storage unit 368 shown in FIG. 19 and ending the execution of this program.

The personality given to the robot 160 can be considered to be fixed. Therefore, information regarding individuality does not necessarily need to be read every time the program is executed. However, if the robot 160 is to be given not only individuality but also states that are expected to change over time, such as emotions, the information is read out each time the program is executed, as shown in FIG. This is desirable.

C. Step 484 (determining gaze direction and duration)
Referring to FIG. 26, step 484 includes step 550 in which a gaze direction model and a gaze duration model corresponding to the role, turn state, and personality assigned to robot 160 are selected and read from gaze direction model 364; A step 552 of selecting and reading out the gaze aversion direction model 450 during turn change or the gaze aversion direction model 452 during utterance from the gaze aversion models 366 according to the role, turn state, and personality assigned to the turn change mode 160 . In step 550, if the turn state is the speaking time, the speaking gaze direction model 400 shown in FIG. 20 is selected. If the turn state is a turn change, one of the turn change line-of-

sight direction models

404, 406, and 408 is selected according to the timing.

The program further continues step 552 with step 554 of sampling a value p in the range [0,1] from a uniform distribution, and using the value p sampled in step 554 to determine the viewing direction selected in step 550. determining 556 a viewing direction from the model. In this specification, determining the line-of-sight direction using randomly sampled values of the line-of-sight direction model in the range [0, 1] is referred to as "sampling of the line-of-sight direction." Details of step 556 will be described later with reference to FIG. 27.

This program further includes a step 558 of sampling a value p in the range [0, 1] from a uniform distribution, similar to step 554, and a process shown in FIG. 25 from the gaze duration model 402 read out in step 550. The parameters n, loc, and scale of the chi ^- square distribution according to the set role of the robot 160 and the personality given to the robot 160 are read out, and the duration of the line of sight is calculated using these values and the value p sampled in step 558. and step 560 of determining. Details of step 560 will be described later with reference to FIG.

This program further includes a step 562 in which the control flow is branched depending on whether the line-of-sight direction determined in step 556 is an averted line-of-sight direction. When the determination at step 562 is negative, execution of this program ends.

The program further includes a step 564 of sampling a value p in the range [0, 1] from a uniform distribution in response to a positive determination in step 562, and a step 564 of sampling a value p in the range [0, 1] from a uniform distribution. and step 566 of determining the viewing direction using the viewing direction model selected in step 556 and terminating the execution of this program. Details of step 566 will be described later with reference to FIG.

FIG. 27 shows details of step 556 in FIG. 26 in flowchart form. In this figure, a variable S _M is used as a variable representing the probability of directing the line of sight to the main listener, S _S is the probability of directing the line of sight to the speaker, and S _B is used as the probability of directing the line of sight to the sublistener. Referring to FIG. 27, step 556 assigns the sum of the value of the variable _SS and the value of the variable _SM to the variable _SM , and the value of the variable _SM whose value has been calculated in this way. The process includes step 600 of assigning the sum of the values of the variable _SB and the value of the variable SB to the variable _SB . By performing this calculation, the probability of facing toward the speaker is stored in the variable _SS . The variable _SM stores the sum of the probability of facing the speaker and the probability of facing the main listener. The variable _SB stores the sum of the probabilities of facing the speaker, the main listener, and the sublistener.

This program further includes a step 602 in which, following step 600, the flow of control is branched depending on whether the value p is smaller than the value of the variable _S. and step 604 in which the direction is determined and the execution of this program is terminated.

This program further includes a step 606 in which, in response to the determination in step 602 being negative, the flow of control is branched depending on whether the value p is smaller than the value of the variable S _M ; In response to the affirmative, determining 608 the viewing direction to be the direction of the main listener and terminating execution of the program.

This program further includes a step 610 in which, in response to the negative determination in step 606, the flow of control is branched depending on whether the value p is smaller than the value of the variable _SB ; At step 612, the line of sight direction is determined to be the direction of the sublistener and the execution of this program is ended. and step 614 of terminating execution of the lever program.

By performing the processing in step 600, such an algorithm can be used to determine the viewing direction according to the value p and the probability represented by the model.

FIG. 28 shows the control structure of the program executed in step 560 of FIG. 26 in a flowchart format. Referring to FIG. 28, the program includes a step 650 of reading the values n, loc, and scale from the gaze duration model 402 selected in step 556 of FIG. Step 652 of sampling p and substituting this value into the variable x, and substituting the values of n, loc, scale, and x into the equation of the chi ^- square distribution for calculating the gaze duration, which has already been described, calculates the gaze duration. and step 654 of calculating the value and terminating the execution of this program.

FIG. 29 shows the control structure of the program that implements step 566 in FIG. 26 in a flowchart format. Referring to FIG. 29, this program sets 0 to the repetition variable i for loop processing in the program, and 0 to the variable S for accumulating each probability shown in FIG. 23 in order from array number 0. , respectively, and following step 680, a step 682 of branching the flow of control depending on whether the value of variable i is less than 8. Here, "8" is the maximum value of the subscript in the array shown in FIG.

This program further includes a step 684 of adding the probability P(i) of the array shown in FIG. 23 to the variable S when the determination in step 682 is affirmative; Step 686 branches the flow of control according to whether or not and returning control to step 682. If the determination in step 686 is affirmative, step 690 is included in which the line-of-sight averting direction is determined to be the line-of-sight averting direction D(i) in the array of FIG. 23 and the execution of the program is ended.

This program further includes a step 692 in which, in response to the negative determination in step 682, the direction of averting the line of sight is determined to be the direction of averting the line of sight D(8) and the execution of the program ends.

By executing this program, it is possible to determine the direction of gaze aversion according to the probability distribution shown in FIG. 14, for example.

6. Operation The conversational robot system 150 according to this embodiment operates as follows. Note that the following explanation will mainly be made with reference to FIGS. 24 to 29 only regarding the line of sight control system 350.

When the line of sight control system 350 is activated, step 480 in FIG. 24 is executed. In this example, it is assumed that each role and turn state of the dialogue participants including the robot 160 is set as a predetermined initial value as role=speaker, and the turn state is set as a speaking period (other than the turn change period). Further, it is assumed that the personality information storage unit 368 shown in FIG. 19 stores information indicating the personality to be given to the robot 160. Here, it is assumed that the robot 160 is given the personality of "extrovert." In step 480, such information is loaded into RAM 298 of FIG.

Subsequently, in step 482, it is determined whether it is time to determine the line of sight direction. Assuming that the utterance period is set as the initial value of the turn state and the gaze duration timer is cleared, the determination in step 482 of FIG. 24 is affirmative, and the control proceeds to step 484.

Referring to FIG. 26, in step 550, the speaking gaze direction model 400 corresponding to role=speaker, turn state=speech period, personality=extraversion is read out, and in step 552, role=speaker, personality= Each extrovertedly corresponding gaze duration model 402 is selected. Furthermore, the value p is sampled in step 554 and step 556 is executed. In step 556, the process shown in FIG. 27 is executed to determine the viewing direction. Here, since the role of the robot 160 is the speaker, one of main listener, sublistener, and averting the line of sight is selected as the line of sight direction.

Returning to FIG. 26, the value p is sampled again in step 558. In the following step 560, the gaze duration time is determined as follows.

Referring to FIG. 28, in step 650, from the gaze duration model 402 read out in step 552 of ^FIG . , loc and scale are read. In step 652, the value p sampled in step 558 is assigned to variable x. In the following step 654, the duration of the line of sight is determined by substituting the parameters n, loc, and scale and the value of the variable x into a formula for duration calculation to calculate the value.

Returning to FIG. 26 again, in the following step 560, it is determined whether the line of sight direction determined in step 556 is an averted line of sight. Here, it is assumed that the direction of the line of sight is not averted. Then, the process shown in FIG. 26 (step 484 in FIG. 24) ends immediately. Returning to FIG. 24, in step 486, control of the robot 160 according to the line-of-sight direction and line-of-sight duration determined in step 484 is started. Note that in step 486, control of the robot 160 is also executed in addition to the line-of-sight control. If the line-of-sight direction determined in step 556 is a line-of-sight direction, steps 564 and 566 in FIG. 26 are executed to determine the line-of-sight direction of the robot 160. In step 486, the line of sight of the robot 160 is controlled so as to be in a direction that is neither the main listener nor the sublistener.

Next, it is assumed that the process shown in FIG. 24 is started. Step 480 is the same as the first process. In step 482, since the duration time of the line of sight direction has not yet expired, the process in step 484 is not executed, and only the process in step 486 is executed. Therefore, if the change in the line-of-sight direction of the robot 160 has not yet been completed, the change in the line-of-sight direction is continued; if it has been completed, the change in the line-of-sight direction is maintained.

It is assumed that a turn change is detected after the process in FIG. 24 has been executed multiple times in this way. Here, one second before the actual turn change timing is detected and notified to the robot 160 in step 480 of FIG. 24. One second before the turn change timing is the timing for determining the line of sight direction. Therefore, the process of step 484 described above is executed again, using the gaze direction model and gaze continuation model selected according to the new role of the robot 160, the turn state = turn change period, and the personality of the robot 160. A new gaze direction and a new gaze duration are determined. Thereafter, in step 486 of FIG. 24, control of the line of sight of the robot 160 is started according to the new parameters.

Hereafter, the line of sight control system 350 repeats the above-described process. As already mentioned, the robot 160's line of sight changes when the previously determined line of sight duration time expires, or when the turn change timing is -1 second, -0.3 seconds, and 0.3 seconds. A direction determination is made.

Furthermore, in actual three-way dialogue, when looking away, it was observed that the direction of the line of sight changed midway through, rather than looking at one point for a long time. There was a tendency for the number of these changes to increase as the time spent averting gaze increased. Therefore, based on the analysis results of the distribution of duration for each number of times the line of sight was averted, we found that 1 time when the duration was 0.7 seconds or less, 2 times when the duration was 1.4 seconds or less, and 3 times when the duration was 2.1 seconds or less. In addition, we decided to increase the number of times the gaze direction was changed by once every 0.7 seconds.

In this case as well, the viewing direction was basically changed according to the distributions shown in FIGS. 13 to 15. However, when the line-of-sight direction is further changed while the line-of-sight is averted, the distributions shown in FIGS. 13 to 15 are applied on the premise that the line-of-sight direction is set at the center and the line-of-sight direction is changed to a position other than the center. Furthermore, when the line of sight direction is changed more than once while looking away, actual observation results show that there is a tendency for the line of sight to return to the immediately previous direction, so in implementation, the line of sight is returned to the immediately previous direction.

Third evaluation experiment 1. Evaluation experiment 1
Regarding the robot 160 according to the embodiment described above, an evaluation experiment was conducted to examine how a change in the line of sight direction is perceived by an observer when no personality is specified.

A. Model In this embodiment, role-specific gaze patterns and pupil movements when looking away are important elements in the interaction. Therefore, in the evaluation experiment, it was decided to compare the evaluation of the movement of the robot 160 under the following four conditions.

a. Baseline (same proportion - head model)
As a baseline, we used a model in which the robot 160 turns its gaze at the same rate in the direction of two interlocutors when its role is as a speaker. Regarding the ratio of looking at the two interlocutors and the others, the ratio of looking at the first interlocutor, the second interlocutor, and the others was set to be approximately 3:3:4. At this time, the robot looks at the person's face or at a location obtained by applying a two-dimensional Gaussian distribution with an angle of 4 degrees distributed between the two interlocutors. . When acting as a listener, I made the robot face the direction of the speaker so as not to make the listener feel uncomfortable.

b. Comparison model (same ratio - head - eyeball model)
Eye movements are important for robot gaze control. Therefore, in the same-proportion model described above, we added not only the head but also the eyeballs and moved them together.

c. Evaluation model 1 (head-eyeball model according to embodiment)
This is a model that implements line-of-sight control and line-of-sight aversion based on the above embodiment.

d. Evaluation model 2 (head model according to embodiment)
Among the evaluation models described above, this model eliminates eyeball movements and performs only head movements.

B. Impression Evaluation The most desired effect in this embodiment is whether the robot's behavior seems natural. Therefore, we created a question for the evaluators: ``Overall, did the robot's behavior feel natural?''

C. Experimental Settings In this evaluation experiment, a video was created for each of the above-mentioned models in which the robot was loaded with the audio of one of the conversations that were actually recorded between the three people. As a result, four videos were obtained. Evaluators were asked to watch these videos and evaluate the way their eyes moved during the video. Actually, each evaluator was asked to watch a video as shown in FIG. 30.

Thirty men and women (average age 32.1 years, variance 10.3) were asked to watch four videos and rate their impressions of each. FIG. 30 shows an example of a video image 720 used in this experiment. The experiment is strictly related to changes in the line of sight direction of the robot 160. Therefore, in this evaluation experiment, a three-party dialogue between actual humans was used for the utterances, and the robot 160 made the utterances of one of them. At that time, the line of sight direction of the robot 160 was controlled to move in the same manner as in the above embodiment according to the role, personality, and turn state of the robot 160. Since the actual three-way dialogue is labeled, the role determination unit 360 and turn change detection unit 362 shown in FIG. I decided to give it to

In the video image 720, as shown in FIG. 30, a moving image of the robot 160 is placed in the center, and

simple character images

730 and 732 showing the other two speakers are placed on the left and right sides. The

character images

730 and 732 are used to indicate a three-way dialogue, but for example, when a speaker on the right side of the robot 160 (left side in FIG. 30) is speaking, the characters around the character image 730 are A sign will now be displayed to indicate that it is present. The same applies to the speaker on the left side of the robot 160 (on the right side when facing the robot 160). FIG. 30 shows a situation where the speaker on the right side is speaking.

In an evaluation experiment, 30 men and women watched these four videos and collected impression ratings. When viewing this video, the utterances of the speaker on the right will be heard in the evaluator's right ear, the utterances of the speaker on the left will be heard in the evaluator's left ear, and the utterances of the robot will be heard in the evaluator's both ears. I made it audible. In this way, we asked participants to evaluate their impressions of each of the four models mentioned above.

The experimental procedure is as follows. First, the presentation order of the four videos was randomized to reduce order effects. Next, the evaluators were asked to rate their impressions of the videos of the four methods individually on a 7-point scale (1: very unnatural, 4: neither, 7: very natural). The above was considered as one set, and impressions were evaluated for a total of 12 videos for three dialogue sections (each dialogue section was about 1 minute in length), and the results were tabulated.

D. Evaluation Results Among the three dialogue sections mentioned above, no significant differences were observed between the conditions in two. In the remaining one dialogue section, a significant difference was observed between the conditions as shown in FIG. 31. FIG. 31 shows the evaluation results for the question "Overall, did the robot's behavior feel natural?" For each model, the results of multiple comparisons were calculated based on the mean value, standard error, and Ryan method of the results.

As a result of the Ryan method, the p value was 0.020 (≦.05) between the baseline model and evaluation model 1, the p value was 0.001 (≦.05) between the comparison model and evaluation model 1, and the p value was 0.001 (≦.05) between the comparison model and evaluation model 1. There was a significant difference between evaluation model 1 and p value 0.008 (≦.05).

From the above results, it was shown that among the above four models, the robot behavior according to evaluation model 3 based on the embodiment is the most natural.

2. Evaluation experiment 2
In addition to the settings of Evaluation Experiment 1 above, we also took into account individuality in the robot's gaze control, and conducted an experiment to evaluate the impression of individuality (extroversion) based on gaze movements.

A. Experimental Method In order to conduct an experiment to evaluate eye gaze movements, we first extracted dialogue sections in which speaker A or speaker B participated from the three-party dialogue data. Furthermore, the robot's gaze movements were generated using the conversational voices of each speaker, and a video was created. In the experiment, participants were asked to evaluate the gaze movements generated under multiple conditions. In order to investigate the difference in the impression of personality expression due to gaze movements, we used a model created from speaker A's data (PA-M) and a model created from speaker B's data (PB-M) to determine the robot's gaze. generated the action. For comparison purposes, we also prepared one that reproduced the speaker's gaze movements (PA-R or PB-R) and one that presented only voice without movements (PA-S or PB-S).

Regarding speaker A and speaker B, conversation sections of around 40 seconds each were extracted, videos were recorded under the above four conditions, and used in the subject experiment. The content of the video is similar to that shown in FIG. 30. The way the utterances were heard by the evaluator was also the same as in Evaluation Experiment 1. In order to make it clear that there are interlocutors on the left and right (diagonally in front), we draw character images representing the interlocutors on both sides of the screen, as shown in Figure 7(b), so that when a voice is uttered, a radiation line is drawn. I made it. The voices of the interlocutors were also audible to the left and right ears, respectively, and the voices of speaker A or B were audible to both ears. In the audio-only video, the target speaker also drew a character image without movement instead of a robot.

In each interaction section, the experiment participants first watched an audio-only video (SM), and then watched three videos that included the robot's gaze movements. The presentation order of the three videos was randomized to reduce order effects. Immediately after watching each video, participants were asked to rate their impressions on a 7-point scale (1: very introverted, 4: neutral, 7: very extroverted). Furthermore, the participants were not informed that the experiment involved controlling gaze movements.

B. Experimental Results 41 men and women (mean age 37.5 years, standard deviation 14.1 years) participated in the experiment. Figure 32 shows the results of impression evaluation.

a. Voice of Speaker A From FIG. 32(A), it can be seen that the impression of extroversion when listening only to the voice (PA-S) is toward extroversion (4 or more). When the gaze behavior is controlled through the robot, the gaze behavior generation model (PA-M) constructed from speaker A's data produces the same impression of extroversion as when the gaze behavior of speaker A is reproduced (PA-R). Based on this, a model (PB-M) constructed from the data of speaker B, who leans towards introversion, shows that the impression of extroversion decreases by a significant difference.

b. Voice of Speaker B From FIG. 32(B), it can be seen that when only the voice is heard (PB-S), the person is more introverted (4 or less). When the gaze behavior is controlled through the robot, the gaze behavior generation model (PB-M) constructed from speaker B's data produces the same impression of extroversion as when the gaze behavior of speaker B is reproduced (PB-R). The model (PA-M) constructed from the data of speaker A, who is more extroverted, has been shown to significantly increase the impression of extroversion.

C. Discussion The above results show that by using different gaze generation models for speakers with different personalities (extraversion), it is possible to change the impression of personality expressed even in the same voice. These results also suggest that impressions of extraversion can be continuously controlled by interpolating or extrapolating model parameters.

Fourth Modified Example In the above embodiment, the model stores the probabilities of directing the line of sight in each direction, and when actually determining the line of sight direction, these probabilities are added together and then the line of sight direction is sampled using random numbers. However, the invention is not limited to such embodiments. For example, the model may store in advance a value obtained by adding up each probability (cumulative probability). In the above embodiment, there are nine directions when looking away. However, the invention is not limited to such embodiments. Directions may be classified more finely.

Further, in the above embodiment, a line-of-sight direction model and a line-of-sight continuation model are prepared and used based on a combination of the interaction role, turn state, and personality of an agent such as a robot. However, the invention is not limited to such embodiments. A model may be prepared without any of these. Further, criteria different from these may be used for selecting the line-of-sight direction model. For example, age, social positions such as teacher and student, parent and child, etc. may be used as criteria.

The above embodiment relates to three-party dialogue. However, as is clear from the description of the above embodiments, the present invention can be applied to conversations involving four or more people as long as each speaker can be distinguished and their roles can be classified.

The embodiments disclosed herein are merely examples, and the present invention is not limited to the above-described embodiments. The scope of the present invention is indicated by each claim, with reference to the detailed description of the invention, and includes all changes within the scope and meanings equivalent to the words stated therein. .

50 Three-

way dialogue

60, 62, 64

Participants

80, 82, 84

Gaze label string

100, 104

Utterances

102, 106 Turn change label 108 Turn change 150 Conversation robot system 160 Robot 162 Operation control PC
164 Microphone 166 Audio processing PC
168 Speaker 170 Voice synthesis PC
172 Probabilistic model storage device 174 PC for interaction
176 Network 178 Person position sensor 180 Person position recognition PC
190 Voice recognition unit 192 Turn recognition unit 250 Computer system 270 Computer 272 Monitor 274 Keyboard 276 Mouse 278 DVD
284 USB memory 290 CPU
292 GPUs
296 ROM
298 RAM
300 SSD
302 DVD drive 304 Audio I/F
306 USB port 308 Network I/F
310 Bus 350 Gaze control system 360 Role determination section 362 Turn change detection section 364 Gaze direction model 366 Model 368 Personality information storage section 370 Gaze motion generation section 372 Gaze motion control section 400 Gaze direction model during speech 402

Gaze duration model

404, 406 , 408 View direction model during

turn change

450, 452 Direction model

Claims

A line-of-sight control device for controlling the line-of-sight of a robot in multi-person dialogue,
Gaze direction setting for determining the gaze direction of the robot based on a combination of the role of the robot in the multi-person dialogue and the state of the dialogue flow in response to the timing for determining the gaze direction. means and
control parameter generation means for generating control parameters for controlling the direction of the robot's face and the direction of the eyeballs in response to the gaze direction of the robot being determined by the gaze direction setting means; , line of sight control device.
The line-of-sight direction setting means
Direction determination that determines, for each role, the probability that the plurality of participants will face in a plurality of predetermined directions, according to a combination of each role of the plurality of participants in the multi-person dialogue and the state of the dialogue flow. a direction determining model storage means for storing the model;
Probability distribution extraction means for extracting a probability distribution according to the combination of the role of the robot and the state of the dialogue flow from the direction determination model in response to the timing for determining the line of sight direction. and,
The line-of-sight control device according to claim 1, further comprising: first sampling means for sampling the line-of-sight direction of the robot from the probability distribution extracted by the probability distribution extraction means.
The line-of-sight control device according to claim 2, wherein the plurality of directions of the direction determination model include directions of the plurality of participants and a line-of-sight aversion direction that is different from any of the directions of the plurality of participants.
The viewing direction setting means further includes:
a gaze aversion direction model storage means for storing a gaze aversion direction model comprising a probabilistic model for probabilistically determining the gaze aversion direction according to a combination of the role of the robot and the state of the dialogue flow;
and second sampling means for sampling a direction in which the robot's line of sight is averted from the line of sight direction model in response to the line of sight direction sampled by the first sampling unit being the line of sight averting direction. The line of sight control device according to claim 3.
Furthermore, a duration calculation unit for calculating the duration of the line of sight of the robot according to a combination of the role of the robot, the state of the dialogue flow, and the line of sight direction determined by the line of sight direction setting means. The line of sight control device according to any one of claims 1 to 4, comprising:
The line-of-sight control device according to claim 5, wherein the timing for determining the line-of-sight direction is different when the dialog flow state is a turn change state and at other times.
The timing for determining the line of sight direction is a predetermined timing in a turn change state when the state of the dialogue flow is a turn change state; The line of sight control device according to claim 6, wherein the timing is when the duration calculated by the duration calculation unit expires.
The line-of-sight direction setting means is configured to be set in advance by the plurality of participants in accordance with a combination of the roles of the plurality of participants in the multi-person dialogue, the state of the dialogue flow, and the expected personality of the robot. direction determination model storage means for storing a direction determination model that determines the probability of facing each of the plurality of directions for each role;
A probability for extracting a probability distribution according to a combination of the role of the robot, the state of the dialogue flow, and the personality from the direction determination model in response to the timing for determining the line of sight direction. a distribution extraction means;
The line-of-sight control device according to claim 1, further comprising: first sampling means for sampling the line-of-sight direction of the robot from the probability distribution extracted by the probability distribution extraction means.
The line-of-sight control device according to claim 8, wherein the plurality of directions of the direction determination model include directions of the plurality of participants and a line-of-sight aversion direction that is different from any of the directions of the plurality of participants.
The viewing direction setting means further includes:
Gaze aversion direction model storage means for storing a gaze aversion direction model consisting of a probabilistic model for probabilistically determining the gaze aversion direction according to a combination of the role of the robot, the state of the dialogue flow, and the individuality. and,
and second sampling means for sampling a direction in which the robot's line of sight is averted from the line of sight direction model in response to the line of sight direction sampled by the first sampling unit being the line of sight averting direction. The line of sight control device according to claim 9.
Further, a continuation for calculating the duration of the line of sight of the robot according to a combination of the role of the robot, the state of the dialogue flow, the personality, and the line of sight direction determined by the line of sight direction setting means. The line of sight control device according to any one of claims 8 to 10, comprising a time calculation section.
The line-of-sight control device according to claim 11, wherein the timing for determining the line-of-sight direction differs between when the dialog flow state is a turn change state and at other times.
The timing for determining the line of sight direction is a predetermined timing in the turn change state when the state of the dialogue flow is a turn change state, and a predetermined timing in the turn change state when the state of the dialogue flow is not a turn change state. The line of sight control device according to claim 11, wherein the timing is when the duration calculated by the duration calculation unit expires.
A computer-implemented gaze control method for controlling the gaze of a robot in multi-person dialogue, the method comprising:
a step in which the computer determines the line of sight direction of the robot based on a combination of the role of the robot in the multi-person interaction and the state of the dialog flow in response to the timing for determining the line of sight direction; ,
the computer generating control parameters for controlling the direction of the face and the direction of the eyes of the robot in response to the direction of the robot's line of sight being determined in the step of determining the direction of the line of sight; Gaze control method.
A computer program for controlling the line of sight of a robot in multi-person dialogue, the computer program
Gaze direction setting for determining the gaze direction of the robot based on a combination of the role of the robot in the multi-person dialogue and the state of the dialogue flow in response to the timing for determining the gaze direction. means and
Functions as a control parameter generation means for generating control parameters for controlling the direction of the robot's face and the direction of the eyeballs in response to the gaze direction of the robot being determined by the gaze direction setting means. , computer program.
A line-of-sight control device realized by a computer with a processor for controlling the line-of-sight of a robot in multi-person dialogue,
The processor includes:
In response to the timing for determining the line of sight direction of the robot, determining the line of sight direction of the robot based on a combination of the role of the robot in the multi-person dialogue and the state of the dialogue flow;
A line-of-sight control device configured to generate control parameters for controlling a face direction and an eyeball direction of the robot in response to determining the line-of-sight direction of the robot.
A non-transitory, computer-readable storage medium storing a computer program for operating a computer to function as the line-of-sight control device according to claim 16.