JP2013119155A - Device and method for creating scenario - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device for creating a scenario capable of automatizing the creation of the scenario of a combination of a speech and a gesture.SOLUTION: Based on text cut pieces with respect to input text data stored in a storage part 42 and the information of the gesture, an SGAE service part 50 extracts a feature amount for each of combinations of them to be used as an input value. The likelihood of a combination of a voice and the gesture of a robot is calculated by each likelihood evaluation module 54 registered therein, and the scenario is created by a creation part 56.

Description

  The present invention relates to a scenario generation apparatus and a scenario generation method for generating a scenario in which an utterance to be controlled by a computer agent is associated with a gesture.

  Regarding the role of gestures in human conversation, the role, type, and generation process have been clarified through observation of conversation. The role of gestures ranges from expressing the content of communication to meta-control of communication and the creation of emotional “Kizuna”. Human utterances and gestures co-occur from the smallest psychological units called "growth points" and the process of composing conversations while complementing each other has been clarified from previous studies.

  The utterance of robots and agents and the generation of gestures have been developed with the goal of reproducing the functions of these people's utterances and gestures (for example, Non-Patent Document 1 and Non-Patent Document 2).

  Further, for example, in Patent Document 1, when any part of a plurality of movable parts or voice output parts is not used, a mobile type capable of performing an action that induces an utterance of a conversation target by effectively utilizing the part. A robot is disclosed.

  Cassell and colleagues, on the other hand, have developed research on natural language processing and developed a system that generates agent utterances and gestures to explain things in virtual space using ECA (Embodied Conversational Agent). Non-patent document 3, Non-patent document 4). For humanoid robots, Victor et al. From HRI (Honda Research Institute) proposed a model that automatically synchronizes ASIMO speech and gestures based on input text (Non-Patent Document 5). The system developed by Victor et al. Observes the gestures that people actually express, and uses the probabilistic model based on the patterns to make the main gestures clarified in previous studies such as emblems, representations, and repetitions. And the audio can be synchronized. In these related researches, research is being carried out mainly to observe and reproduce the conversations between people by synchronizing the gestures and voices of robots.

JP 2008-279529 A

B. Hartmann, M. Mancini, and C. Pelachaud. “Implementing expressive gesture synthesis for embodied conversational agents.” In In Gesture in Human-Computer Interaction and Simulation, volume 3881, pages 188-199. Springer, 2006. S. Kopp and I. Wachsmuth. “Synthesizing multi-modal utterances for conversational agents.” Comp. Anim. Virtual Worlds, 15 (1): 3952, 2004. J. Cassell, H. Ho gni Vilhja lmsson, and T. Bickmore. “Beat: the behavior expression animation toolkit.” In SIGGRAPH 2001: Proceedings of ACM SIGGRAPH, pages 477486, New York, NY, USA, 2001. K. Striegnitz, P. Tepper, A. Lovett, J. Cassell, “Knowledge Representation for Generating Locating Gestures in Route Directions”. In Proceedings of Workshop on Spatial Language and Dialogue (5th Workshop on Language and Space). October 23-25 , Delmenhorst, Germany, 2005. V. Ng-Thow-Hing, P. Luo and S. Okita, “Synchronized Gesture and Speech Production for Humanoid Robots,” The 2010 IEEE / RSJ International Conference on Intelligent Robots and Systems October 18- 22, Taipei, Taiwan, 2010.

  However, communication robots that have been developed so far have been developed to mimic a part of human body functions. Conversely, with the exception of some robots such as Android, these robots are unable to express all gestures that humans can express. For example, a robot without a finger cannot express an emblem gesture such as an OK sign.

  Therefore, when creating a gesture for a communication robot, the developer edits the gesture to be expressed so that the intention is transmitted according to the restrictions of each robot. As a result, these gestures often move differently than the modeled human gestures.

  In addition, since the conversation adjustment function such as shaking the body or blinking cannot be reproduced as it is, a change to add another movement to replace them is also added. Inevitably, the combination of robot voices and gestures will be different from humans. In addition to reproducing human gestures, it is difficult to build a psychological model that edits speech and gestures for each individual robot.

  There is also a debate about whether these assignment methods are universal. As the number of people who use robots increases and as time goes by, various methods will be devised for assigning and creating voices and gestures. It is required that the system evolves along with the learning and development of those users. However, there has never been such an evolutionary framework.

  The present invention has been made to solve the above-described problems, and based on a pattern extraction result when a person assigns an utterance and a gesture to a control target such as a communication robot, To provide a scenario generation device or a scenario providing method capable of automating the creation of a gesture combination scenario.

  Another object of the present invention is to provide a scenario generation apparatus or a scenario providing method capable of developing a system for automatically creating a scenario in an expansive manner.

  According to one aspect of the present invention, a scenario generation device for creating a scenario in which a gesture is assigned to an utterance to be controlled, which stores text data corresponding to the utterance and information for controlling the gesture Storage means for dividing the text data of a predetermined length among the text data corresponding to the utterance to be controlled, into a plurality of text segment candidates based on a predetermined termination pattern, and a plurality of text segments For each candidate combination of a candidate and a plurality of predetermined gestures, the first likelihood that the text segment candidate is delimited by a predetermined termination pattern and the ratio of the reproduction time of the text segment candidate or the reproduction time and the gesture time Based on the second likelihood based on at least one of the combination candidates, the combination candidate having the highest likelihood is selected from the combination candidates. The selection means for selecting as a combination in Rio, and the text data of a predetermined length following the text segment corresponding to the selected combination of the text data, by the dividing means and the selection means to the end of the text data Scenario creation means for repeatedly selecting combinations and creating scenarios.

  Preferably, the second likelihood is a product of the likelihood based on the reproduction time of the text segment candidate and the likelihood based on the ratio between the reproduction time and the gesture time.

  Preferably, the selection unit calculates the likelihood by multiplying a third likelihood based on a keyword existing in the text segment candidate in addition to the first and second likelihoods.

  Preferably, the scenario generation device is a server device, and further includes means for registering information for controlling the gesture with respect to the storage means via the network.

  Preferably, the first to third likelihoods are calculated by the corresponding likelihood evaluation modules, and further include means for registering the likelihood evaluation modules with respect to the selection means.

  According to another aspect of the present invention, a scenario generation method for creating a scenario in which a gesture is assigned to an utterance to be controlled, which stores text data corresponding to the utterance and information for controlling the gesture The arithmetic unit, based on the information in the storage device, divides the text data of a predetermined length among the text data corresponding to the utterance to be controlled into a plurality of text segment candidates based on a predetermined termination pattern The arithmetic unit, for each combination candidate of a plurality of text segment candidates and a plurality of predetermined gestures, a first likelihood that the text segment candidates are delimited by a predetermined termination pattern, and reproduction of the text segment candidates Based on the second likelihood based on at least one of the time or the ratio of the playback time and the gesture time, The step of selecting the most likely combination candidate as a combination in the scenario, and the arithmetic unit calculates text for a predetermined length of text data following the text segment corresponding to the selected combination of the text data. A step of creating a scenario by repeating the division process and the combination selection process until the end of the data.

  According to the present invention, based on a pattern extraction result when a person assigns an utterance and a gesture to a control target such as a communication robot, the creation of a combination scenario of an utterance and a gesture that is natural for a person viewing the gesture is automated. Is possible.

  Alternatively, according to the present invention, a system for automatically creating a scenario is developed by allowing a plurality of people to participate in creating a gesture and creating a method for determining the likelihood of a combination of an utterance and a gesture. It is possible to go.

It is a figure which shows the flow of the extraction of the utterance and gesture assignment pattern for a communication robot, and its progressive development. 2 is a block diagram of a computer system of a server device 2000. FIG. It is a flowchart for demonstrating the gesture assignment process to speech. It is a conceptual diagram which shows the example of the scenario produced. It is a conceptual diagram for demonstrating the structure of the prototype of a scenario production | generation apparatus. It is a figure which shows the external appearance of robovie (trademark) mR2. It is a figure which shows the user interface in UI part 10 of the system of a scenario production | generation. It is a figure which shows Johnson SU distribution in various parameter values. It is a figure which shows an experimental environment. Ngi obtained by normalizing each evaluation value obtained by Expression (3) by Expression (4) is plotted. It is a figure which shows the histogram of the ratio of the reproduction time of the audio | voice and gesture in each command of all the trials obtained by experiment. It is a figure which shows the likelihood evaluation model approximated curve by normal distribution and Johnson SU distribution. It is a figure which shows the likelihood evaluation model approximated curve by normal distribution and Johnson SU distribution. The calculated score of each scenario was evaluated using ANOVA (one-factor intra-subject analysis). It is a figure which shows an evaluation result. The calculated score of each scenario was evaluated using ANOVA (one-factor intra-subject analysis). It is a figure which shows an evaluation result.

  Hereinafter, the configuration of the scenario generation system according to the embodiment of the present invention will be described with reference to the drawings. In the following embodiments, components and processing steps given the same reference numerals are the same or equivalent, and the description thereof will not be repeated unless necessary.

  As will be described below, in this embodiment, instead of reproducing a human psychological model, data of a combination of created robot voices and gestures is collected, and the pattern is extracted, so that Generate a combination of voice and gesture. Develop a more practical scenario generation system that matches the target communication robot by combining utterances and gestures based on past history data without introducing existing concepts of gestures and utterances in conversations between people can do.

  In the present embodiment, a robot voice and gesture assignment system using SOA (Service-Oriented Architecture) is constructed when using historical data.

  In recent years, systems that use service-oriented architecture (SOA), in which software equivalent to one process is regarded as a service in the business and those services are linked on the network to build the entire system, have been announced one after another. ing. These systems have two major advantages.

  The first advantage is that less computer resources are required by the user using the system. Since most of the services required by these systems are executed by a server on the network, the user can use the services by simply opening a browser.

  A second advantage is that the service provider can collect user usage histories. By analyzing how the service is used, it is possible to provide a more advanced service.

  These two advantages are exactly what robotic systems need. In order to realize a service in response to a user request, various processes such as speech recognition, speech synthesis, face recognition, and position acquisition must be performed.

  Introducing all of these in each user's local environment requires considerable computer resources and is a major barrier to the spread of services. In addition, in order to determine what actions a robot should take and develop its interface continuously, it is essential to collect and analyze a huge amount of usage history of users and operators. It is done.

  Therefore, it is desirable to collect a user's editing history and to develop a robot voice and gesture assignment system in an advanced manner by operating a technique as described below in an SOA-based system.

  In the following, the system configuration and flow of these SOA-based robot services will be described for the purpose of constructing a model for generating robot behavior (combination of voice and gesture) from the editing history of the user.

  However, the present invention is not necessarily limited to such a SOA-based system, and any other system configuration may be used as long as it can realize a function equivalent to each service in the SOA-based system. Also good.

  In the following description, “computer agent” refers to software for causing a robot as a physical entity to execute a corresponding utterance and gesture according to a scenario in which the utterance and gesture are combined. It can be a program. However, the “computer agent” may more generally be a software program for causing a control target to execute a corresponding utterance and gesture according to a scenario. In this case, the “control target” has less freedom of speech or gesture than humans, and is not limited to a visual target as a physical entity, for example, a character image displayed on a display. Such a visual object may be used.

(Extraction of speech and gesture assignment patterns for communication robots)
FIG. 1 is a diagram showing the flow of extraction and utterance development of speech and gesture assignment patterns for a communication robot.

  As shown in FIG. 1, the scenario generation system according to the present embodiment includes a user interface unit 10 executed on the client device side and a robot instruction generation (hereinafter referred to as “Robot Instruction Generation”) executed on the server device 2000 side. RIG) service unit 30, database and data storage (DBDS) service unit 40, and voice-gesture assignment evaluation (SGAE). ) It consists of four components with the service unit 50.

  The user 2 can create a robot command (a combination of voice and gesture) using the RIG service unit 30 from the interface 10 on the browser of the client device. The resource generated by the RIG service 30 and the combination information of the voice and gesture are registered by the DBDS service unit 40 in a storage unit 42 configured by a storage device such as a database or a storage server.

  Although not particularly limited, REST API (Representational State Transfer Application Programming Interface) can be used to transfer information from the user interface unit 10 to the RIG service unit 30.

  The SGAE service unit 50 extracts feature values for each of the combinations based on the text intercepts and gesture information for the input text data stored in the storage unit 42, and uses them as input values. Then, the likelihood of the combination of the robot voice and gesture is calculated by each likelihood evaluation module 54 registered inside, and the scenario is generated by the generation unit 56 according to the procedure described later. The calculated likelihood and scenario are transmitted from the generation unit 56 to the user interface unit 10.

  In this system, in addition to a user of an interface for generating robot instructions, a gesture creator 4 and a likelihood evaluation service analyst (developer) 6 are interposed.

  The gesture creator 4 literally refers to a person who creates a robot gesture with a dedicated tool and registers it in the DBDS service unit 40. The gesture creator 4 may be a user who uses the interface. However, the creation requires specialized knowledge such as grasping the axis arrangement of the robot, and in order to clarify the role, it is defined here as another existence. Although not particularly limited, the gesture creator 4 can also register information for controlling the gesture in the DBDS service unit 40 via a network using a dedicated interface.

  The user 2 who creates the robot command uses the gesture created by the gesture creator 4 to create the robot command by assigning the gesture and voice using the scenario generation system.

  Although not particularly limited, in this embodiment, in order to be applicable to a wider range of robots without being limited to a specific robot, robot gestures cannot be automatically generated and are registered. I will explain from the standpoint.

  On the other hand, the analyst 6 refers to a person who analyzes accumulated history data and defines and registers each likelihood evaluation module in order to realize the SGAE service unit 50. It is thought that various elements are involved in the assignment of robot voices and gestures, and the system will not be completed in a single implementation, but will be developed progressively. For such an advanced development, the SGAE service unit 50 becomes a framework that continuously learns and regularly reflects the results. The analyst 6 installs the likelihood evaluation module 54 and its parameter calculation module (not shown) in the SGAE service unit 50 and incorporates them in the system. The parameter calculation module is not particularly limited, but may be configured based on, for example, the least square method. The system periodically updates the parameter based on the parameter calculation module from the history data, and reflects the result in the likelihood evaluation module 54. By repeating these steps, the overall service can be automated, and further development by adding the likelihood evaluation module 54 and updating the parameters becomes possible.

(Hardware configuration)
FIG. 2 is a block diagram of a computer system of the server apparatus 2000.

  2, in addition to the memory drive 2020 and the disk drive 2030, the computer main body 2010 of the server apparatus 2000 stores a CPU 2040, a bus 2050 connected to the disk drive 2030 and the memory drive 2020, and a program such as a bootup program. A RAM 2070 for temporarily storing application program instructions and providing a temporary storage space, and a hard disk (HDD) 2080 for storing application programs, system programs, and data. And a communication interface 2090 for communicating with an external device such as a storage server via a network or the like.

  The functions of the RIG service unit 30, the DBDS service unit 40, and the SGAE service unit 50 described above are realized by arithmetic processing executed by the CPU 2040 based on the program.

  A program that causes the server apparatus 2000 to execute the functions of the information processing apparatus and the like of the above-described embodiment is stored in the CD-ROM 2200 or the memory medium 2210, inserted into the disk drive 2030 or the memory drive 2020, and further the hard disk 2080. May be transferred to. Alternatively, the program may be transmitted to the computer main body 2010 via a network (not shown) and stored in the hard disk 2080. The program is loaded into the RAM 2070 at the time of execution.

  The server apparatus 2000 further includes a keyboard 2100 and a mouse 2110 as input devices, and a display 2120 as an output device.

  The above-described program for functioning as a server does not necessarily include an operating system (OS) that causes the computer main body 2010 to execute functions such as an information processing apparatus. The program only needs to include an instruction portion that calls an appropriate function (module) in a controlled manner and obtains a desired result. How the server apparatus 2000 operates is well known, and detailed description thereof is omitted.

  Further, the computer that executes the program may be singular or plural. That is, centralized processing may be performed, or distributed processing may be performed.

The basic hardware configuration of the client device that executes the user interface unit 10 is the same as that of FIG.
(Assigning gestures to utterances)
Hereinafter, processing for assigning a gesture to an utterance from text data will be described.

  FIG. 3 is a flowchart for explaining a process for assigning a gesture to an utterance.

  In the following, the following is taken as an example of text. That is, it is assumed that the following text data is input from the user interface unit 10.

“This research aims to construct a system that extracts and automates patterns when people assign speech and gestures to communication robots, and proposes a method for developing the system progressively. The utterance and gesture assignment method in Japanese communication robots has been implemented in robots by analyzing the roles of utterances and gestures in human conversations and modeling their roles, but… ”
Referring to FIG. 3, when the process of assigning a gesture to an utterance is started (S100), RIG service unit 30 first extracts data for a predetermined length from the beginning of the text data described above. Here, for example, data for two sentences is taken out. Of course, how many sentences of data are extracted is not particularly limited.

  Next, the RIG service unit 30 divides the text data from the text data for two sentences based on the termination pattern (for example, a punctuation mark or punctuation mark in Japanese, or a comma or period in English). Is created (S102).

  In the example of the text described above, for example, it is as follows.

Section 1: “This study is”
Section 2: “This study extracts patterns when people assign speech and gestures for communication robots.”
Section 3: “This research aims to build a system that extracts and automates patterns when people assign speech and gestures of communication robots.”
Section 4: "This research aims to build a system that extracts and automates patterns when people assign speech and gestures of communication robots. And"
Section 5: “This research aims to build a system that extracts and automates patterns when people assign speech and gestures of communication robots. And, it proposes a method to develop the system progressively. ”
As will be described later, the term “terminal pattern” means a specific pattern in text that tends to divide text with a certain probability as a computer agent utterance. It is not necessarily limited to punctuation marks, commas, or periods.

  For explanation, it is assumed that i text sections are obtained.

  Further, the RIG service unit 30 generates (i × j) combination candidates of j gestures registered in advance and the i text slices, and the DBDS service unit stores the candidate data as data. Store in the storage (S102).

  Subsequently, the SGAE service unit 50 multiplies all likelihoods of likelihood evaluation modules 1 to m (m is an integer of 2 or more) corresponding to each candidate data, as will be described in detail later. The likelihood for each candidate data is calculated (S104).

  As the likelihood evaluation module, the following can be used.

1) Likelihood evaluation module 1 for calculating the likelihood of dividing a text by “termination pattern” obtained experimentally and empirically.
In addition to the likelihood evaluation module 1, at least one of the following likelihood evaluation modules may be configured based on the likelihood obtained in advance experimentally and empirically, and combined with the likelihood evaluation module 1. it can.

2) Likelihood evaluation module 2 for calculating the likelihood of assigning the corresponding gesture to the keyword in the text
3) Likelihood evaluation module 3 for calculating the likelihood of selecting a gesture reproduction time
4) Likelihood evaluation module 4 for calculating the likelihood of combining speech and gesture based on the ratio of the speech playback time and the gesture playback time of the text segment.
As the likelihood evaluation module, if the likelihood based on other factors is taken into consideration for the text segment and the gesture, another likelihood evaluation module can be added as necessary.

  Subsequently, the SGAE service unit 50 selects a candidate with the highest likelihood (S106).

  The SGAE service unit 50 determines whether the text segment selected as a candidate has reached the final end of the input text data (S108). If not, the process returns to step S102.

  On the other hand, if the final end has been reached, for example, in the server device 2000, the generation unit 56 generates a scenario by sequentially connecting candidates with the highest likelihood.

  For example, when section 2 is first selected, the processes from step S102 to step S106 are repeated for the text after section 2, and again for the text for two sentences.

  FIG. 4 is a conceptual diagram showing an example of a scenario to be created.

  As shown in FIG. 4, the scenario is such that a combination of a text section having a high likelihood and a gesture ID for specifying a gesture is sequentially arranged for the input text in the order of text reproduction.

(Configuration of scenario generator prototype)
In FIG. 1, in the scenario generation device, the user interface unit 10 functions on the client device side, and the RIG service unit 30, the DBDS service unit 40, and the SGAE service unit 50 are configured as an SOA-based system on the server device side. It was supposed to be.

  In the following, the configuration of a prototype for examining the feasibility of the function of the scenario generation device shown in FIG. 1 will be described. However, in the prototype configuration as described below, the scenario generation device can also be realized as a robot instruction generation function that has the scenario automatic generation function of the SGAE service unit 50 described in FIG. .

  FIG. 5 is a conceptual diagram for explaining a configuration of a prototype of the scenario generation device.

  The configuration shown in FIG. 5 is a simplified implementation of the configuration of the scenario generation device shown in FIG. In the following description, the experimental results based on the configuration of the prototype shown in FIG. 5 will be mainly described.

  Referring to FIG. 5, the system includes a user interface (UI) application unit (hereinafter, UI unit) 10, a robot command generation (RIG) server 30, a database / data storage (DBDS) unit 40, and a robot. It consists of four modules of the renderer unit 3000.

  Although not particularly limited, the robot command generation (RIG) server 30, the DBDS unit 40, and the robot renderer unit 3000 can each operate on different server devices. Alternatively, these may be configured to operate on the same server device.

  Further, for example, the function of the robot renderer unit 3000 may be executed on the client device side. That is, there is no particular limitation on how the functions of each unit illustrated in FIG. 3 are distributed and processed between the server device and the client device. Alternatively, all functions may be executed on one computer device.

  The system for generating a scenario receives the input to the UI unit 10 and performs the following two processes.

  (1) Upon receiving an input to the UI unit 10, a robot command is generated. This process is performed using the UI unit 10 and the robot command generation server 30, and the result is stored in the DBDS unit 40. These processing steps are represented by solid arrows in FIG.

  (2) Upon receiving an input to the UI unit 10, the robot operation is controlled. This process is performed using the UI unit 10, the DBDS unit 40, and the robot renderer unit 3000. These processing steps are represented by dotted arrows in FIG.

  The generation of the robot command is performed according to the following procedure.

  First, the UI unit 10 generates a robot voice based on the input text, and registers the synthesized voice information in the database of the DBDS unit 40. At the same time, the synthesized voice is stored in the data storage server of the DBDS unit 40. Next, based on the gesture ID selected by the UI unit 10, the gesture information registered on the database is acquired. Finally, a robot command (scenario) is generated by combining the synthesized voice and gesture motion, and registered in the database of the DBDS unit 40. Also here, it is assumed that the information for controlling the motion of the gesture is created in advance by the gesture creator and stored in the DBDS unit 40 in association with the gesture ID.

  On the other hand, the robot behavior control is performed in the following procedure.

  First, the UI unit 10 instructs a Web socket server on the robot renderer 3000 to reproduce a robot command specified by a certain ID using a Web socket client. Upon receiving the instruction, the Web socket server 3002 on the robot renderer 3000 sends the robot instruction ID to the instruction analysis module 3004. The command analysis module 3004 inquires the command serialization module 38 of the robot command generation server 30 about the command content using the received robot command ID as a key, and downloads the content. The instruction analysis module 3004 analyzes the instruction contents, extracts the necessary voice and gesture file URI information, instructs the resource manager 3008 to download the resources from the data storage server of the DBDS unit 40. Finally, the command analysis module 3004 instructs the actuator controller 3006 to execute the command. The actuator controller 3006 uses the resources secured by the resource manager 3008 to send a motion command to the robot 1000 and reproduce the voice and gesture. Note that the voice and gesture resources once acquired by the resource manager 3008 are not downloaded twice.

  By performing the above processing, the mounted prototype system can generate both robot commands and store the commands and execute the commands.

(robot)
In the present embodiment, it is assumed that robotie (registered trademark) mR2 is used as a robot controlled by the system.

  FIG. 6 is a diagram showing the appearance of robovie (registered trademark) mR2. 6A shows a front view, FIG. 6B shows a side view, and FIG. 6C shows the degree of freedom of the robot.

  This robot is designed on the assumption that it is placed on a desk and has a height of 30.0 cm, a radius of 15.0 cm, and a weight of 2.0 kg. Designed based on the upper body of the human body, it has 3 DOF on the head, 2 DOF on the eyes, 2 DOF on the eyelid, 4 DOF on the arm, and 1 DOF on the waist. As a feature of the robot, there is a space for ipod touch / iphone (registered trademark) connection in the abdomen. By storing the ipod touch / iphone (registered trademark) in the abdomen, the robot can be connected to the robot via a serial cable. There is a point that can be controlled. By executing the control software on the mobile terminal side, the robot can be controlled without using a personal computer. Due to the above features, robovie (registered trademark) mR2 has excellent portability and can be easily introduced into the home environment. In the experiment described in this embodiment, aiming at an environment where the user can easily create the playback content (sound and motion combination) of the robot, the playback content of robovie (registered trademark) mR2 is created using the system of FIG. Created.

(User interface design)
FIG. 7 is a diagram showing a user interface in the UI unit 10 of the scenario generation prototype system.

  The assignment of text and gestures in the system is performed based on the instruction generation window. The window includes an instruction number, a composition button, a text area for inputting text, and a check box for selecting a gesture. The command number indicates the order in which the combined voice and gesture are played. The user enters text in the text area and selects a gesture from the check box.

  After that, the robot voice and gesture can be assigned by pressing the synthesis button. When the synthesis is completed, the system displays a confirmation screen and notifies the user that the synthesis is completed. Also, the label name of the composition button is changed from “Composition” to “Play”. The user can confirm whether the combination of the assigned voice and gesture is valid by pressing the play button. Gestures and voices can be assigned as many times as necessary, and the user can adjust the length of the synthesized voice and the length of the assigned gestures until the user is satisfied. The instruction generation window can be added by an “add” button at the top of the UI unit 10. By creating in the order of the command generation windows, a combination of voice and gesture of the entire sentence to be reproduced finally is created.

(Evaluation of voice and gesture assignment (SGAE))
The service design voice and gesture assignment evaluation (SGAE) service shown in FIG. 1 is calculated by a plurality of registered likelihood evaluation modules based on the robot voice and gesture history registered in the DBDS service unit 40. The combination of speech and gesture is evaluated from the total likelihood. Here, a design method of the likelihood evaluation module in the present embodiment will be described.

  When the likelihood of the likelihood evaluation module i is Li, the likelihood L finally calculated by the SGAE service 50 is given by the following equation.

Although various methods can be considered as to how the likelihood evaluation module calculates the likelihood, in this embodiment, the likelihood is calculated and determined by one of the following two methods.

1) When discontinuous data is given: Rule-based likelihood determination 2) When continuous data is given: Likelihood calculation based on probability density function Rule-based likelihood determination is discontinuous Used for data analysis. In this embodiment, it is mainly used when analyzing the text that is the basis of the assigned speech. In the existing research so far, grammatical analysis or morphological analysis of text has been adopted, and a method for determining a gesture to be assigned based on keywords and parts of speech included therein has been adopted. Also in the present embodiment, a morphological analysis is performed on the text, and a likelihood evaluation module is defined based on the obtained gesture assignment pattern.

  On the other hand, likelihood calculation using a probability density function is used when analyzing consecutive values. For patterns that can be analyzed numerically, such as voice playback time, gesture playback time, or their ratio, the pattern is approximated by a probability density function.

  The probability distribution most used for such analysis is a Gaussian distribution (normal distribution), but the data obtained by analysis does not necessarily show a symmetrical distribution. Rather, the distribution is often asymmetric in the left-right direction. In order to satisfy such a requirement, the Johnson SU distribution is used as an approximate expression in the present embodiment. The Johnson SU distribution has the characteristic that the shape of the normal distribution can be manipulated fairly freely by appropriately giving skewness (distribution asymmetry) and kurtosis (hem thickness), and its probability density function is It is given by equation (2).

The probability density function (2) can determine the center, the spread of the tail, the skewness, and the kurtosis by the four variables γ, δ, λ, and ε.

  FIG. 8 is a diagram showing the Johnson SU distribution at various parameter values.

  As shown in FIG. 8, various distributions can be defined by the Johnson SU distribution.

  In this embodiment, the likelihood evaluation module of the SGAE service unit 50 is designed by using any one of these rule bases and an approximate expression using a probability density function.

(Data collection for construction of SGAE service department 50)
The robot voice and gesture assignment evaluation (SGAE) proposed in the present embodiment, the service unit 50 evaluates the assigned combination from robot commands (combination of voice and gesture) constructed in the past. Therefore, in order to construct the SGAE service unit 50, a robot command generation history in advance is required.

  An experiment was conducted to collect history data necessary for the likelihood evaluation module of the SGAE service unit 50. In the experiment, the subject divided the sentence of the taught document using the prototype system described with reference to FIG. 5, and assigned a gesture to each divided sentence. The divided sentences and the gestures assigned to the sentences are stored in a database in the system. After the experiment, a likelihood evaluation module was defined based on the information stored in the database.

(Experiment overview)
FIG. 9 is a diagram showing an experimental environment.

  In the experiment, the subject sits in front of the table and controls the laptop. On the laptop, the UI unit 10 of the prototype system described with reference to FIG. 5 is set up, and the utterance and gesture of robovie (registered trademark) mR2 can be assigned.

  In FIG. 9, an image necessary for explanation is drawn on the right display (whether or not an image is drawn on the display depends on experimental conditions). In this experiment, 27 male and female university students (13 males and 14 females) participated in the utterance of robots and assigning gestures, and then responded to a questionnaire.

(Experimental procedure)
The experimental procedure is as follows.

  (1) First, the experimenter hands over a sentence that the robot will explain to the subject.

  (2) The subject first reads aloud a sentence to be explained by the robot and grasps the content.

  (3) The experimenter explains how to use the user interface of the prototype system implemented on the subject while actually creating the first two sentences (the contents created by these experimenters are excluded from the analysis target). Accept.

  (4) The subject prepares a robot command (combination of voice and gesture) for each sentence so that the robot can explain the given sentence.

  (5) When all the commands have been created and playback confirmation is complete, write them in the questionnaire.

  In the present embodiment, attention is paid to the length of the reproduction time as well as the type of gesture as the gesture pattern. Therefore, voice and gesture assignments were performed under two conditions: Condition 1 for combining gestures and sentences according to time series and Condition 2 for combining different types of motion and sentences. The details of each condition are summarized in Table 1.

Under each condition, the sentence that is the source of the robot's command is to analyze the pattern of text division according to the sentence structure, and to explain the specialized content that tends to become a complex sentence structure. The sentence obtained from the Wikipedia page Created based on. When each sentence is synthesized by speech synthesis software Ximera, the length is about 60.0 seconds, and the error is within 0.5 seconds.

  Ximera is disclosed in the following document.

Literature: H. Kawai, T. Toda, J. Ni, M. Tsuzaki, and K. Tokuda, “Ximera: A New Tts from ATR Based on Corpus-Based Technologies,” ISCA Speech Synthesis Workshop, pp. 179-184, 2004.
On the other hand, two patterns were prepared for gestures. In condition 1, a total of 10 types of repeated gestures were prepared every 2.0 seconds from 2.0 seconds to 20.0 seconds for the purpose of analyzing the user's selection tendency due to the difference in playback time. In order not to change the assignment depending on the type of gesture, the creator of the gesture was careful not to be a symbolic gesture that expresses the image of the speaker, and it was created as a repeated gesture that moves parts of the body alternately. .

  On the other hand, in condition 2, emblems ("byebye"), symbolic (direct (pointing), and descriptive gestures) and repeated gestures were assigned. For these gestures, two types of options of 5.0 seconds and 10.0 seconds were prepared so as to correspond to the length of each voice. Since the direct gesture requires an object to be pointed, in the condition 2, a diagram explaining complications due to diabetes is displayed on the display (the content of the diagram corresponds to the content of the sentence).

(Configuration of likelihood determination module of SGAE service unit 50 using experimental data)
In the following, the likelihood evaluation module of the SGAE service unit 50 is defined based on the voice and gesture assignment history collected in the experiment. In the present embodiment, two types of rule-based evaluation and two types of evaluation based on probability density are performed, and a total of four analyzes are performed. The contents of each are shown below.

1) Analysis of text termination pattern based on sentence structure analysis 2) Analysis of gesture and keyword assignment pattern 3) Analysis of assignment pattern based on gesture playback time 4) Assignment based on the ratio of voice and gesture playback time Pattern Analysis The combination of speech and gesture is evaluated from the rule obtained by the analysis or the total likelihood (equation (1)) determined from the approximate expression.

(Analysis by text termination pattern based on sentence structure analysis)
When analyzing the data obtained in the experiment, the effect of sentence structure, especially the influence of punctuation marks, was noticeable in the text segmentation pattern that was the source of speech. In this section, a rule for determining likelihood is derived from the end pattern of the input text. The analysis was conducted for all combinations of voices and gestures in Experimental Conditions 1 and 2.

  Looking at the end of the text that the subject split, most of them were split with punctuation marks.

  In particular, the punctuation mark “.” Is divided with a probability of almost 100%, which is a remarkable pattern. Next, there are many divisions by the reading “,”. On the contrary, the case where it was divided in places other than punctuation marks was rare. The division by punctuation was biased in the division ratio due to the sentence structure that preceded it. The pattern of the sentence structure before the punctuation was various, but in this embodiment, the division ratio is high, the combination of verb and preposition, punctuation division by verb phrase (or verb clause), and other The percentage was calculated by dividing it into reading divisions. Table 2 shows the percentages of division by phrases, punctuation (other than verb phrases and verb phrases), and others. As shown in Table 2, when the ratios of adjacent patterns are compared, it can be seen that there is an opening of about 2.0 times.

  The likelihood evaluation module based on the sentence structure uses the patterns and ratios shown in Table 2 to determine the likelihood of a given robot voice and gesture combination.

(Verification of differences in text termination patterns between human explanations and robot behavior generation)
In order to show the effectiveness of the method proposed by the present embodiment, the parameters of the likelihood function are completely different when the proposed method is used to model with human-person and human-robot data. In addition, it is necessary to show that the proposed method gives better results when controlling the robot used for modeling.

  With regard to gestures, it is difficult to set up a control experiment combining human and robot gestures, so this module is an example of a model that assigns a person's own behavior model and the creator's voice and gesture. Indicate that the parameters are different and verify their effectiveness.

  In order to investigate the difference between a person's own behavior model and the model for assigning robot speech and gestures, we analyzed the speech that the subject read the scenario when teaching the experiment. In the analysis, two analysts were prepared and instructed that the scenario used for the reading was to "hear the subject's reading and put a diagonal line in the part that he thought was separating the sentence". The two subjects both interpret the sentence as “breaking the sentence” as the end of the text, and in the text, as in “analysis by the text end pattern based on the sentence structure analysis” It was classified into verb phrase delimiters), punctuation marks (other than verb phrase delimiters), and others, and the average value and ratio were obtained. Table 3 shows a comparison between the reading analysis result and the assignment analysis result data.

In Table 3, the average value indicates the average value of the corresponding termination pattern entered by the subject for one scenario. The ratio indicates the ratio of the corresponding termination pattern in all termination patterns (the ratio in the assignment indicates the total number of corresponding termination patterns existing in the scenario and the ratio of the number actually divided by the termination pattern, The parameter is different). As shown in Table 3, the average value of each punctuation point is almost the same, but for other termination patterns, it decreases from the punctuation point average value at the time of assignment, while it increases at the time of reading, Inevitably, the ratio will change.

  This difference is that when a person reads by himself, the sentence can be separated by intonation and gradual, whereas in the robot, the speech synthesis function (in this embodiment, the speech synthesis software Ximera) It is thought that due to the limitations, such expression is difficult. In other words, in an environment where diversity of expressions is guaranteed, people try to cut the sentences into short sentences, but in an environment where there are limited ways to express the separators, It is thought that the policy has been changed.

From the above results, it was shown that the patterns differ in how the sentences are separated when a person speaks by himself or when the robot expresses them. We will further examine the effects of such a change in the policy on sentence separation for listeners. In order to verify the above hypothesis, among the scenarios created by the subjects, the average length of the combination of voice and gesture is a) the longest scenario, and b) the shortest scenario, the nine who have experimented (4 males, 5 females) subjects were compared and asked to evaluate which was preferred. As a result, all the subjects answered that the scenario of a) was better. From the above results, it can be said that the change of the sentence break policy in the robot voice and gesture assignment has the effect of giving a favorable impression to the listener.
(Analysis of gesture and keyword assignment patterns)
Consider whether the gesture assigned to the voice has a random probability. In related research so far, there are many systems that employ algorithms that apply gestures such as emblems and representations when certain keywords are included in speech text. It is self-evident that certain types of gestures tend to be assigned by certain keywords or their combinations in the speech text.

  Again, a likelihood evaluation module that analyzes the keyword and gesture assignment patterns included in the speech text and determines the likelihood based on the rule base is defined. So far, we do not use the types of gestures classified in the existing research, but adopt the policy of determining the likelihood based only on the relationship between registered individual gestures and keywords.

  In this embodiment, the morphological analysis software Chasen is used to perform morphological analysis on the assigned text, and for the six types of parts of speech, nouns, verbs, adjectives, adverbs, and conjunctions, the gestures assigned to each word and its Percentages were analyzed to create a matrix.

  Chasen is disclosed below.

References: Yuji Matsumoto, Kei Kitauchi, Tatsuo Yamashita, Yoshitaka Hirano, Hiroshi Matsuda, Kazuma Takaoka, Masayuki Asahara, “Japanese Morphological Analysis System“ Chaya ”version 2.2.1 Instruction Manual,” Dec, 2000.
Tables 4 and 5 show the labels of the assigned gestures and their ratios with respect to the keywords “bye-bye” and “figure” that are particularly characteristic among the analyzed results.

Among the assigned gestures, a gesture including “byebye” in the label is an emblem gesture in which the robot shakes its hand. The short / long in the second half indicates whether the length is 5 seconds or 10 seconds. On the other hand, a gesture including “point” is a direct gesture in which the robot points a finger while looking at the display, and the definition regarding the length is the same as that of “byebye”. Finally, a gesture including “high” is a drawing gesture for showing that the size and degree of the gesture are large, in which both hands are extended to the height of the head and lowered. From these examples, we can confirm that the keyword and gesture assignments defined in this section are working.

  When analyzing text, a plurality of keywords (nouns, verbs, adjectives, adverbs, conjunctions) may be extracted. In this case, the matrix of each keyword is analyzed, and the one with the fewest candidate gestures (the fact that there are few candidate gestures means that a certain gesture is selectively assigned) is adopted. If there is no corresponding matrix, 1.0 is returned.

The likelihood evaluation module based on text and gesture assignment patterns determines the likelihood of a given robot voice and gesture combination using the relationship matrix group represented by the patterns and ratios shown in Tables 4 and 5. To do.
(Analysis of assignment pattern based on gesture playback time)
In this embodiment, the gesture of the robot is not automatically generated, but is created by the gesture creator and registered in the DBDS service unit 40.

  It has not yet been clarified how long a motion is favored by the user, assuming a robot gesture is made. Here, a likelihood evaluation module that evaluates a subject's selection tendency with respect to the length of a gesture is defined.

  First, in the condition 1 of the experiment, the selected number of times was obtained for all the allocated gestures. Due to the characteristics of the task, you can make more robot commands by dividing sentences and assigning short gestures. Therefore, the number of times a short gesture is selected tends to increase compared to a long gesture. In order to eliminate this imbalance, the selection tendency was evaluated by using the value Egi, which is obtained by multiplying the number of times each gesture was selected by the ratio of the gesture to the whole sentence. Egi represents the evaluation value of the i-th gesture gi and is given by the following equation.

Where cgi is the number of gestures adopted through all trials, dgi is the length of the corresponding gesture, and the denominator is the sum of the speech length dsj, that is, the length of the sentence (depending on the experimental conditions, 60 (The average time of the divided voice is 7.0 seconds).

  FIG. 10 is a plot of Ngi obtained by normalizing each evaluation value obtained by Expression (3) by Expression (4).

The Johnson SU distribution parameters γ, δ, λ, and ε that approximate these distributions were obtained by the following equation (5) using the least square method.

The transition of the Johnson SU distribution according to the parameters obtained in this way is shown in FIG. The approximate curve has a lower position than the plotted value because the sum of the plotted points is 1.0 while the approximate expression has an area of 1.0. This is because it is a function.

  The likelihood evaluation module based on the reproduction time of the gesture determines the likelihood of a given combination of voice and gesture of the robot based on the probability density function defined by Equation (2) and Parameter Equation (5).

  Although this likelihood evaluation module is based on the reproduction time of a gesture, the meaning of the distribution shown in FIG. 10 is to seek a longer combination if the length of the combination of voice and gesture (reproduction time) is too short. On the other hand, if it is too long, it is thought that it is a person's selection tendency to seek a shorter combination.

In this experiment, it was shown that the boundary between the long and short judgments is around 10.0 seconds. Therefore, by introducing this likelihood evaluation module, the system looks for a longer combination when the speech becomes 10.0 seconds or less, and conversely, when the speech becomes 10.0 seconds or more, the system uses a shorter combination. Operates to seek.
(Assignment pattern analysis based on the ratio of voice and gesture playback time)
When assigning the voice and gesture of the robot, one of the factors considered to be important from the viewpoint of the user is the ratio of the reproduction time of the voice and the gesture. It is considered that the user decides the combination so that the lengths of the voice and the gesture match as much as possible. In this section, we define a likelihood evaluation module based on the playback time of speech and gestures.

  FIG. 11 is a diagram showing a histogram of the ratio of the reproduction time of the voice and the gesture in each command for all trials (including condition 1 and condition 2) obtained in the experiment.

  The ratio ri in the instruction i is calculated by the following equation (6).

Here, si represents the playback time of the voice in command i, and gi represents the playback time of the gesture in command i.

  Referring to FIG. 11, an increase corresponding to the normal distribution is shown up to the ratio 1.0, but when the value exceeds 1.0, the value tends to decrease rapidly. The parameters γ, δ, λ, and ε of the Johnson SU distribution that approximates the distribution of this histogram were obtained by the following equation (7) using the least square method.

The distribution of probability density approximated by the above parameters is shown in FIG.

The likelihood evaluation module based on the ratio of the reproduction time of the gesture and the voice calculates the likelihood of the given robot voice and gesture combination based on the probability density function defined by the equation (2) and the parameter equation (7). decide.
(Verification of effectiveness of approximation by Johnson SU distribution)
In the present embodiment, even when the history data pattern of the combination of speech and gesture obtained by the prototype system has different kurtosis, distortion, or left-right asymmetry compared to approximation by Gaussian distribution. Approximation was performed using the Johnson SU distribution so that the distribution could be approximated more accurately. This section examines its effectiveness.

  12 and 13 are diagrams showing likelihood evaluation model approximate curves based on the normal distribution and the Johnson SU distribution.

  In the figure, a curve indicated by a dotted line is an approximate curve based on a Gaussian distribution. On the other hand, the curve indicated by the solid line is an approximate curve based on the Johnson SU distribution.

  As can be seen from FIG. 12, regarding the approximation of the selection tendency of the reproduction time of the gesture, the two approximations have not changed much (the Johnson SU distribution has a slightly smaller residual).

In contrast, FIG. 13 shows that the Johnson SU distribution captures the characteristics of the ratio distribution of the reproduction time of the gesture and the voice. The distribution of the ratio between the gesture and audio playback time tends to rapidly decrease (distort) the probability of occurrence when the ratio exceeds 1.0. It is considered that the approximation by the Johnson SU distribution proposed in this embodiment has captured the feature.
(SGAE service unit 50 simulation)
Exactly verifying whether the total sum of likelihoods output from the likelihood evaluation module of the SGAE service 50 configured based on the above experimental data can actually present an appropriate combination of speech and gesture is very It is a difficult problem.

  The total number of voice and gesture combinations reaches a considerable number, and it is difficult to verify all of them.

  On the other hand, evaluating each module individually does not lead to an evaluation of the entire service. For example, even if a gesture is selected only by a keyword, if the ratio of the length of the voice and the gesture is extremely far from 1.0, the evaluation for the combination is considered to be low. On the other hand, if the ratio of speech to gesture is kept constant, it becomes difficult to divide the text by reflecting the sentence structure.

  Here, for simple verification of the SGAE service unit 50, another subject evaluates a combination of voices and gestures of three subjects who performed simulation and created a characteristic combination, and a combination with high evaluation, Confirm that the combinations with the highest likelihood of calculation match.

(Simulation procedure)
In the simulation, the likelihood of the combination of gestures and voices made by the subjects for each of two types of scenarios, the condition 1 scenario and the condition 2 scenario, was obtained, and the average was obtained. The following three types were selected for each scenario.

Min. Scenario with the lowest average likelihood calculated by the likelihood evaluation module defined above
Mid. A scenario in which the mean of likelihoods calculated by the likelihood evaluation module defined above is an intermediate value
Max. The scenario with the highest average likelihood calculated by the likelihood evaluation module defined above, and 9 male and female subjects (including 3 males and 6 females) evaluated and listened to 3 scenarios. They were sorted using inequality / equal signs in the order they thought it was easy. Table 6 shows the inequality signs used and their scores.

The evaluation method for each scenario is as follows.

(1) The score of the scenario with the lowest evaluation is set to 1. (2) The score is added in order based on the addition score from the lowest scenario. For example, Max.> Min. >> Mid. When the evaluation is performed, the evaluation of scenario Mid. Is 1.0, the evaluation of Min. Is 3.0, and the evaluation of Max. Is 4.0.
(simulation result)
14 and 15, the score of each calculated scenario was evaluated using ANOVA (one-factor intra-subject analysis). It is a figure which shows an evaluation result.

  As shown in FIG. 14, in the evaluation for the scenario of condition 1, a significant difference was confirmed with respect to the evaluation score (p <.01; F = 10.77). In a multiple comparison test (LSD), the scenario with the highest likelihood average is significantly higher in evaluation than the scenario with the lowest likelihood average (p <.05), and the likelihood average is intermediate. It was confirmed that the scenario with the value is significantly higher in evaluation than the scenario with the lowest average value (p <.05). From the above results, it was shown that the scenario with the intermediate value and the maximum value is more highly evaluated than the scenario with the average likelihood. In the scenario of condition 1, a significant difference could not be confirmed between the scenario with the intermediate likelihood and the scenario with the maximum value.

  On the other hand, as shown in FIG. 15, even in the evaluation for the scenario of condition 2, a significant difference was confirmed with respect to the score (p <.05; F = 4.90).

  In the multiple comparison test, it was shown that the scenario with the highest likelihood average was significantly higher in evaluation than the scenario with the average likelihood average (p <.05). There was no significant difference between the other conditions.

  From the above, it can be seen that at least the scenario with the highest likelihood score is evaluated as favorable for the user.

  Therefore, by dividing the input sentence and evaluating the robot command (combination of voice and gesture) that can be created using the SGAE service unit 50 and selecting the combination with the highest likelihood, It is possible to automate robot command generation.

  According to the scenario generation device, for example, it is possible to create a robot service that explains blog text while gesturing. Eventually, a huge amount of html content can be taken into the robot service.

  Moreover, since the gesture creator 4 and the analyst 6 are independent of the robot content creator 2, the system can be developed progressively by using the scenario generation device. .

  Embodiment disclosed this time is an illustration of the structure for implementing this invention concretely, Comprising: The technical scope of this invention is not restrict | limited. The technical scope of the present invention is shown not by the description of the embodiment but by the scope of the claims, and includes modifications within the wording and equivalent meanings of the scope of the claims. Is intended.

  2 users, 4 gesture creators, 6 analysts, 10 user interface units, 30 RIG service units, 40 DBDS service units, 42 storage units, 50 SGAE service units, 54 likelihood evaluation modules, 56 generation units, 2000 server devices.

Claims (6)

A scenario generation device for creating a scenario in which a gesture is assigned to an utterance to be controlled,
Storage means for storing text data corresponding to the utterance and information for controlling the gesture;
Dividing means for dividing text data of a predetermined length among text data corresponding to the utterance to be controlled into a plurality of text segment candidates based on a predetermined termination pattern;
For each combination candidate of the plurality of text segment candidates and a plurality of predetermined gestures, a first likelihood that the text segment candidates are delimited by the predetermined termination pattern, and a reproduction time of the text segment candidates or Selection means for selecting, as a combination in the scenario, a combination candidate having the highest likelihood among the combination candidates based on a second likelihood based on at least one of the ratios of the reproduction time and the gesture time. When,
Of the text data, the selection of the combination by the dividing unit and the selection unit is repeated until the end of the text data with respect to the text data of the predetermined length following the text segment corresponding to the selected combination. A scenario generation device comprising scenario creation means for creating the scenario.   The scenario generation device according to claim 1, wherein the second likelihood is a product of a likelihood based on a reproduction time of the text segment candidate and a likelihood based on a ratio between the reproduction time and the gesture time.   The said selection means calculates likelihood by the multiplication of the 3rd likelihood based on the keyword which exists in the said text intercept candidate in addition to the said 1st and said 2nd likelihood. The scenario generator described.   The said scenario production | generation apparatus is a server apparatus, Comprising: The means for registering the information for controlling the said gesture with respect to the said memory | storage means via a network is further provided. The scenario generation device described in 1.   The said 1st thru | or 3rd likelihood is calculated by the corresponding likelihood evaluation module, and further comprises the means for registering the said likelihood evaluation module with respect to the said selection means. Scenario generator. A scenario generation method for creating a scenario in which a gesture is assigned to an utterance to be controlled,
Based on the information in the storage device that stores the text data corresponding to the utterance and the information for controlling the gesture, the arithmetic unit has a predetermined length of text data corresponding to the utterance to be controlled. Dividing the text data into a plurality of text segment candidates based on a predetermined termination pattern;
The arithmetic unit, for each combination candidate of the plurality of text segment candidates and a plurality of predetermined gestures, a first likelihood that the text segment candidate is divided by the predetermined termination pattern, and the text segment candidates Or the second likelihood based on at least one of the ratios of the reproduction time and the gesture time, and the combination candidate having the highest likelihood is selected as the combination in the scenario. A step to choose;
An arithmetic unit selects the selection of the division process and the combination of the text data of the predetermined length following the text segment corresponding to the selected combination of the text data up to the end of the text data. A scenario generation method comprising: repeating the process to create the scenario.