JP6768597B2 - Dialogue system, control method of dialogue system, and device - Google Patents

Description

It relates to a dialogue system that realizes dialogue with a user through a dialogue interface.

In recent years, the development of a dialogue interface for continuous dialogue with humans using robots and CG has been progressing.

In the development of a dialogue interface, a technique has been proposed in which a dialogue feature is calculated from dialogue data indicating actions related to dialogue such as human facial expression, voice, and body movement, and the dialogue interface is controlled using the dialogue feature. .. By controlling the dialogue interface based on the dialogue features, the facial expression of the robot or CG is changed, and the voice is reproduced. Such control can induce human empathy. As the above-mentioned techniques, the techniques described in Patent Document 1 and Patent Document 2 are known.

According to Patent Document 1, "the interaction state quantity calculation unit detects the interaction state quantity including the speaker change latency, the pitch, power, or the mora of the speech section in the speech of the user P. The amount of communication synchronization deviation between the user and the robot system is calculated, and the amount of communication synchronization deviation is minimized by continuous pull-in control by the state equation representing the synchronization model. The state quantity is brought closer to that of the robot system 10, or the interaction state quantity of the robot system 10 is brought closer to that of the user P while the interaction state quantity of the user P is brought closer to that of the robot system 10. " Are listed.

In Patent Document 2, "a step of inputting a user utterance, a step of extracting the input prosodic characteristics of the user utterance, and generating a prosody that responds to the user utterance based on the extracted prosodic characteristics. A voice dialogue method is described which comprises a step and, upon generating the apocalypse, adjusting the prosodic characteristics of the abutment so that the prosodic characteristics of the abutment match the prosodic characteristics of the user utterance.

Japanese Unexamined Patent Publication No. 2012-181697 Japanese Unexamined Patent Publication No. 2016-38501

By combining the techniques described in Patent Document 1 and Patent Document 2, it is possible to control an interactive interface that mimics human action. At this time, the control timing of the dialogue interface that imitates the human action can be arbitrarily set. For example, the control timing at which the robot or CG performs the aizuchi can be arbitrarily set for the human aizuchi.

In order to induce human empathy, it is necessary to set the control timing according to the action to be imitated. However, neither Patent Document 1 nor Patent Document 2 is supposed to have the above-mentioned problems. Therefore, in the invention in which Patent Document 1 and Patent Document 2 are combined, the control timing of the aizuchi can be set arbitrarily, and the action of the dialogue interface is not diverse. Therefore, the effect of inducing human empathy is low.

An object of the present invention is to provide a technique for realizing a dialogue interface capable of strongly inducing human empathy.

A typical example of the invention disclosed in the present application is as follows. That is, a dialogue system that provides a dialogue interface for interacting with a user, wherein the dialogue system includes a computing device, a storage device connected to the computing device, and a computer having an interface connected to the computing device. The storage device includes a measuring device for measuring a signal related to the user's speech, and the storage device adjusts the type of response action performed by the dialogue interface to the user and the control timing of the response action. A setting range indicating a settable range of the control time and a control time definition information including an entry composed of the control time are held, and the calculation device is a user in the speech of the user based on a signal measured by the measurement device. calculating a first feature quantity indicating the feature of the action, the first based on the feature amount, calculates a second characteristic quantity which is a control value of the response actions, by referring to the control time definition information, said response action It is characterized in that a random control time is calculated based on the set range corresponding to the type of the above, and the dialogue interface is controlled based on the second feature amount and the control time.

According to the present invention, by controlling the dialogue interface that executes the response action based on the delay time, it is possible to induce the user's empathy and realize a continuous dialogue and a multi-information dialogue. Issues, configurations and effects other than those described above will be clarified by the description of the following embodiments.

It is a figure which shows the configuration example of the system of Example 1. FIG. It is a figure explaining an example of the operation flow at the time of having a dialogue using the system of Example 1. It is a figure explaining the configuration example of the computer of Example 1. FIG. It is a figure which shows an example of the data structure of the dialogue signal information of Example 1. It is a figure which shows an example of the data structure of the imitation information of Example 1. It is a figure which shows an example of the data structure of the delay time definition information of Example 1. FIG. It is a figure which shows an example of the data structure of the language signal information of Example 1. It is a figure which shows an example of the data structure of the utterance information of Example 1. FIG. It is a figure which shows an example of the data structure of the output information of Example 1. FIG. It is a figure which shows an example of the data structure of the output condition information of Example 1. FIG. It is a figure which shows an example of the data structure of the output history information of Example 1. FIG. It is a flowchart explaining the process executed by the imitation information calculation module of Example 1. It is a flowchart explaining the process executed by the language information calculation module of Example 1. It is a flowchart explaining the process executed by the output information generation module of Example 1. It is a figure which shows an example of the data structure of the delay time definition information of Example 2. It is a flowchart explaining the process which the learning module of Example 2 executes. It is a figure which shows the flow of the learning process of the delay time executed by the learning module of Example 2. It is a figure which shows the configuration example of the system of Example 3. It is a figure which shows an example of the structure of the biological signal information held by the computer of Example 3. It is a figure which shows an example of the data structure of the output information of Example 3. It is a flowchart explaining the process which the imitation information calculation module of Example 3 executes. It is a flowchart explaining the process which the imitation information calculation module of Example 3 executes. It is a figure which shows an example of GUI for making the setting for the computer of Example 5. It is a figure which shows an example of GUI for making the setting for the computer of Example 5.

Hereinafter, examples will be described with reference to the drawings. In the configurations of the inventions described below, the same or similar configurations or functions are designated by the same reference numerals, and duplicate description will be omitted.

In this embodiment, a computer that provides a dialogue interface that realizes a dialogue with a user calculates a control feature amount of a response action to be executed by using a dialogue signal, and further, a response action is performed according to the type of the response action. Set the control time (delay time) to adjust the control timing of the interactive interface corresponding to. In addition, the computer identifies the utterance content of the user based on the measured language signal, and selects the utterance content to be output for the utterance content. The computer generates output information based on the control feature amount, the control time, and the utterance content. Here, the terms in the present specification will be described.

The dialogue signal is a signal acquired from the user in the dialogue, particularly a signal including information on the user's action such as a line of sight, a face orientation, and a body movement. The dialogue signal is used to calculate the dialogue data. For example, an image of the user, acceleration of a part of the user's body, and the like are acquired as dialogue signals.

The dialogue data is generated by using the dialogue signal, and is data for calculating the dialogue feature amount.

The dialogue feature quantity is a value that characterizes the user's action. For example, the direction of the user's line of sight, the direction of the face, the moving direction of the body, the moving distance of the body, and the like are calculated as the dialogue features.

The language signal is a language emitted from the user, that is, a signal for evaluating the user's utterance. For example, the user's voice or the like is acquired as a language signal.

The response action represents the movement of a person image displayed on the display device constituting the dialogue interface, the movement of the robot constituting the dialogue interface, and the like.

The control feature represents the control value of the dialogue interface for realizing the response action.

In this embodiment, a dialogue interface that imitates to induce empathy of the user will be described as an example. The following processing is executed in the system that realizes the interface.

The computer calculates three types of dialogue data, the user's eyeball image, face image, and body image, from the dialogue signal including the user's image, and calculates the dialogue features related to the user's line of sight, face orientation, and body movement. To do. The computer calculates the control features of the response action that mimics the user's line of sight, face orientation, and body movements based on the dialogue features. The computer sets a control time for each of the three types of response actions. In addition, the computer selects the utterance content to be output to the user based on the acquired language signal.

The computer generates output information based on the control feature amount, the control time, and the utterance content, and controls the dialogue interface based on the output information to execute the response action and output the utterance content.

The dialogue data is not limited to the eyeball image, the face image, and the body image. The dialogue data may include various signals during dialogue such as an image of a part of the body such as a hand or the surrounding environment, human voice, and a pupil diameter measured by near-infrared light. The dialogue features are not limited to the user's line of sight, face orientation, and body movement. The dialogue feature amount may include various feature amounts that can be extracted from the dialogue signal and the language signal, such as the facial expression change extracted from the face image and the voice pitch extracted from the voice. The output information may be generated only from the control feature amount, or may be generated only from the utterance content.

In this embodiment, the user's movement and the user's utterance are described as user actions. Further, the response action and the output of the utterance content (reproduction of voice) are described as the output action. Further, the response action that imitates the movement of the user is described as the imitation action, and the control feature amount of the imitation action is described as the imitation feature amount.

FIG. 1 is a diagram showing a configuration example of the system of the first embodiment.

The system includes a computer 100, an interactive signal measuring device 110, a language signal measuring device 120, a display device 130, and an interactive output device 140. Further, an input device 150 such as a mouse, a keyboard, and a touch panel is connected to the computer 100.

The computer 100 generates output information 316 (see FIG. 3) using the dialogue signal and the language signal, and outputs information and the like from the display device 130 and the dialogue output device 140 to the user based on the output information 316. In this embodiment, the display device 130 and the dialogue output device 140 function as a dialogue interface. On the display device 130 of this embodiment, a person image that performs an imitation action that imitates the movement of the user is displayed.

The dialogue signal measuring device 110 acquires a dialogue signal from the user. The dialogue signal measuring device 110 of this embodiment acquires an image as a dialogue signal. An angle sensor and an acceleration sensor worn on the user's head may be used as the dialogue signal measuring device 110. The values measured by the sensor can be used to calculate the orientation of the face and the movement of the body.

The language signal measuring device 120 acquires a language signal from the user. The language signal measuring device 120 of this embodiment acquires voice as a language signal.

The input device 150 is a device for inputting an external signal. Here, the external signal represents a signal for controlling the operation of the computer 100. In this specification, in particular, signals excluding dialogue signals and language signals are treated as external signals.

Here, an outline of the software configuration of the computer 100 will be described. The computer 100 includes an imitation information calculation module 101, a language information calculation module 102, an output information generation module 103, and a learning module 104, and also holds an information group 105.

The imitation information calculation module 101 calculates dialogue data from the dialogue signal, and calculates the dialogue feature amount based on the dialogue data. The imitation information calculation module 101 calculates the imitation feature amount based on the dialogue feature amount. It is assumed that the algorithm for calculating the imitation feature based on the dialogue feature is set in advance.

The present invention is not limited to an algorithm for calculating imitation features. An algorithm that calculates one type of imitation feature based on one type of dialogue feature, an algorithm that calculates one type of imitation feature based on a plurality of types of dialogue feature, or one type An algorithm may be used to calculate a plurality of types of imitation features based on the dialogue features of.

In the above algorithm, when the input dialogue feature amount is a vector amount, an operation of converting the direction of the vector in the opposite direction, an operation of correcting the component value based on the size, shape, position, etc. of the person image are executed. Will be done. When the user's line of sight is directed to an object existing at an arbitrary coordinate, the imitation feature amount for reproducing the movement of the eyeball and the change in the movement such that the direction of the eyeball of the person image is adjusted to the coordinate is calculated.

The language information calculation module 102 evaluates the utterance content of the user by analyzing the language signal. The language information calculation module 102 selects the utterance content to be output from the dialogue interface based on the utterance content of the user. It is assumed that the algorithm for selecting the utterance content to be output from the dialogue interface is set in advance.

The present invention is not limited to the algorithm for selecting the utterance content output from the dialogue interface.

The output information generation module 103 generates output information 316 based on the imitation feature amount and the utterance content, and controls the dialogue interface based on the output information 316.

The learning module 104 learns the history of the output information 316 (see FIG. 3). The process executed by the learning module 104 will be described in the second embodiment.

In addition, although one computer 100 has each module in FIG. 1, each module may be arranged in a plurality of computers. Further, in FIG. 1, a measuring device such as the interactive signal measuring device 110 and the computer 100 are described as separate devices, but the computer 100 may include the measuring device.

For each module included in the computer 100, two or more modules may be combined into one module, or one module may be divided into a plurality of modules for each function.

FIG. 2 is a diagram illustrating an example of an operation flow when a dialogue is performed using the system of the first embodiment.

When a user action is detected by measuring an interactive signal or a language signal, the computer 100 calculates the imitation feature amount of the imitation action and sets a delay time as a control time of each imitation action. By executing the imitation action with a delay time by the dialogue interface, it is possible to sympathize with the user and to have a dialogue corresponding to the content of the user's utterance.

At input 201, the computer 100 acquires the dialogue signal and the language signal from the dialogue signal measuring device 110 and the language signal measuring device 120. The computer 100 calculates the dialogue data based on the dialogue signal, and calculates the dialogue feature amount based on the dialogue data. The computer 100 evaluates the utterance content based on the language signal. The computer 100 executes operations 202 to 205 in parallel based on the dialogue feature amount and the utterance content of the user.

In the calculation 202, the computer 100 calculates the imitation feature amount for performing the movement of the eyeball of the person image that imitates the movement of the user's line of sight, based on the dialogue feature amount related to the user's line of sight. Further, the computer 100 sets a delay time corresponding to the movement of the eyeball of the person image.

In the calculation 203, the computer 100 calculates the imitation feature amount for performing the face movement of the person image that imitates the face movement of the user based on the dialogue feature amount regarding the movement of the user's face. Further, the computer 100 sets a delay time corresponding to the movement of the face of the person image.

In the calculation 204, the computer 100 calculates the imitation feature amount for performing the body movement of the person image that imitates the body movement of the user based on the dialogue feature amount regarding the movement of the user's body. In addition, the computer 100 sets a delay time corresponding to the movement of the body of the person image.

In the calculation 205, the computer 100 selects the utterance content according to the utterance content of the user. In this embodiment, the delay time is not set in particular for the output of the utterance content, but the delay time can also be set for the output of the utterance content.

At output 206, the computer causes the dialogue interface to execute the imitation action with a delay time, and outputs the utterance content to the dialogue interface.

FIG. 3 is a diagram illustrating a configuration example of the computer 100 of the first embodiment.

As shown in FIG. 3, the computer 100 has an arithmetic unit 301, a storage device 302, an input interface 303, and an output interface 304.

The arithmetic unit 301 is hardware such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), and executes a program stored in the storage device 302. The arithmetic unit 301 functions as a predetermined module by operating according to the program. In this embodiment, the arithmetic unit 301 functions as the imitation information calculation module 101, the language information calculation module 102, and the output information generation module 103.

The storage device 302 is hardware such as a memory, and stores a program executed by the arithmetic unit 301 and information used by the program. The storage device 302 also includes a work area. The programs and information stored in the storage device 302 will be described later. The computer 100 may have a storage device such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive).

The input interface 303 is hardware that accepts inputs of dialogue signals, language signals, and external signals. When connected to a device that inputs a signal via a USB (Universal Social Bus) standard connection line, the interface having a USB terminal becomes the input interface 303, and is connected to a device that inputs a signal via a network. The interface having the port is the input interface 303.

The output interface 304 is hardware that outputs various information. If the device that outputs information is a display, the interface that has a terminal such as a VGA terminal or HDMI terminal (HDMI is a registered trademark) is the output interface 304, and if the device that outputs information is a speaker, the interface that has a USB terminal outputs. It becomes the interface 304.

Here, the program and information stored in the storage device 302 will be described.

The storage device 302 stores a program that realizes the imitation information calculation module 101, the language information calculation module 102, the output information generation module 103, and the learning module 104. Further, the storage device 302 stores dialogue signal information 311, imitation information 312, delay time definition information 313, language signal information 314, utterance information 315, output information 316, output condition information 317, and output history information 318.

The dialogue signal information 311 is information for managing the dialogue signal. An example of the data structure of the dialogue signal information 311 will be described with reference to FIG. The imitation information 312 is information for managing the imitation feature amount of the imitation action. An example of the data structure of the counterfeit information 312 will be described with reference to FIG. The delay time definition information 313 is information that defines the delay time corresponding to the imitation action. An example of the data structure of the delay time definition information 313 will be described with reference to FIG.

The language signal information 314 is information for managing the language signal. An example of the data structure of the language signal information 314 will be described with reference to FIG. The utterance information 315 is information for managing the utterance content of the user and the selected utterance content. An example of the data structure of the utterance information 315 will be described with reference to FIG.

The output information 316 is information for performing an imitation action and controlling an interactive interface that outputs the utterance content. An example of the data structure of the output information 316 will be described with reference to FIG.

The output condition information 317 is information for managing the condition for executing the response action. An example of the data structure of the output condition information 317 will be described with reference to FIG. The process using the output condition information 317 will be described in the fourth embodiment.

The output history information 318 is information for managing the history of the output information 316. An example of the data structure of the output history information 318 will be described with reference to FIG.

The imitation information calculation module 101 generates imitation information 312 by calculating the imitation feature amount of each imitation action from the dialogue signal and setting the delay time corresponding to each imitation action.

The language information calculation module 102 generates utterance information 315 by specifying the utterance content of the user from the language signal and selecting the utterance content for the utterance content of the user.

The output information generation module 103 generates output information 316 using imitation information 312 and utterance information 315.

FIG. 4 is a diagram showing an example of the data structure of the dialogue signal information 311 of the first embodiment.

The dialogue signal information 311 includes an entry composed of an ID 401, a signal name 402, an acquisition time 403, and a pointer 404. One entry corresponds to one dialogue signal.

The ID 401 is a field for storing identification information for uniquely identifying the entry of the dialogue signal information 311.

The signal name 402 is a field for storing information for identifying the type of the dialogue signal and the like. In this embodiment, the name of the dialogue signal measuring device 110 that measures the dialogue signal is stored in the signal name 402. The signal name 402 may store the type of data included in the signal such as "image".

The acquisition time 403 is a field for storing the acquisition time of the dialogue signal. In the acquisition time 403, for example, a time based on the start time of measurement of the dialogue signal is stored. A field may be provided to store a value indicating the relationship between the measurement start times of the dialogue signal and the language signal.

The pointer 404 is a field that stores a pointer of a storage area in which the acquired dialogue signal is stored. In addition, instead of the pointer 404, a field for storing the acquired dialogue signal may be provided. For example, a plurality of combinations of coordinates and RGB values are stored in the field.

FIG. 5 is a diagram showing an example of the data structure of the imitation information 312 of the first embodiment.

The imitation information 312 is an entry composed of ID 501, reference ID 502, imitation feature type 503, imitation feature 504, delay time 505, dialogue data name 506, dialogue data 507, dialogue feature type 508, and dialogue feature 509. including. One entry corresponds to the imitation feature of one imitation action.

ID 501 is a field for storing identification information for uniquely identifying the entry of imitation information 312. In this embodiment, three types of imitation features are calculated for one dialogue signal.

The reference ID 502 is a field for storing the identification information of the dialogue signal used when calculating the imitation feature amount. The value of ID 401 is stored in the reference ID 502.

The imitation feature quantity type 503 is a field for storing information indicating the type of the imitation action. In the imitation feature quantity type 503 of this embodiment, any one of "line of sight", "face movement", and "body movement" is stored.

The imitation feature amount 504 is a field for storing the calculated imitation feature amount. When the imitation feature amount type 503 is "line of sight", the imitation feature amount 504 stores the angle of the eyeball of the person image, the moving speed, etc. The amount 504 stores the face angle, movement speed, etc. of the person image, and when the imitation feature amount type 503 is "body movement", the imitation feature amount 504 contains the movement direction, movement amount, etc. Is stored.

The delay time 505 is a field for storing the delay time corresponding to the imitation action.

The dialogue data name 506 is a field for storing the identification information of the dialogue data used for calculating the dialogue feature amount.

The dialogue data 507 is a field for storing dialogue data. In this embodiment, since the dialogue data is an image, the dialogue data 507 includes a plurality of combinations of coordinates and RGB values. In addition, instead of the dialogue data 507, a field for storing a pointer indicating a storage area in which the dialogue data is stored may be provided.

The dialogue feature type 508 is a field for storing the identification information of the dialogue feature. In the dialogue feature quantity type 508 of this embodiment, any one of "line of sight", "face orientation", and "body movement" is stored.

The dialogue feature amount 509 is a field for storing the calculated dialogue feature amount. When the dialogue feature type 508 is "line of sight", the coordinates of the target of the user's line of sight are stored in the dialogue feature 509, and when the dialogue feature type 508 is "face orientation", the dialogue feature 509 is stored. The tilt angle of the user's face and the like are stored in, and when the dialogue feature amount type 508 is "body movement", the dialogue feature amount 509 stores the movement amount of the user's body and the like.

The coordinates of the target of the user's line of sight may be the coordinates on the plane of the display which is the dialogue output device 140, or may be the coordinates of the three-dimensional space with the dialogue output device 140 as the origin. The tilt angle of the face is calculated based on the change in the distance between the feature points calculated from the tip of the nose and the contour of the face, the laterality, and the like. The amount of movement of the body per unit time is calculated based on the sum of the movement distances of the feature points calculated from the contour of the body.

In this embodiment, one type of mimicking feature is calculated using one type of dialogue feature. However, one type of imitation feature may be calculated using a plurality of types of dialogue features. In this case, one entry includes a plurality of lines having the dialogue data name 506, the dialogue data 507, the dialogue feature amount type 508, and the dialogue feature amount 509 as one set.

FIG. 6 is a diagram showing an example of the data structure of the delay time definition information 313 of the first embodiment.

The delay time definition information 313 includes an entry composed of the ID 601, the imitation feature amount type 602, and the delay time 603. One entry corresponds to one type of imitation action. In this embodiment, since there are three types of imitation actions, the delay time definition information 313 includes three entries.

ID601 is a field for storing identification information for uniquely identifying the entry of the delay time definition information 313.

The imitation feature amount type 602 is the same as the imitation feature amount type 503.

The delay time 603 is a field for storing the delay time corresponding to the imitation action. The initial delay time shall be set in advance by an expert or the like. In this embodiment, the delay time starting from the time when the output information 316 is generated is set. The present invention is not limited to the setting standard of the delay time.

The delay time may be set for the combination of the type of the imitation feature amount and the range of the value of the imitation feature amount. In this case, the entry contains a field that specifies the range of mimic features.

FIG. 7 is a diagram showing an example of the data structure of the language signal information 314 of the first embodiment.

The language signal information 314 includes an entry composed of an ID 701, a signal name 702, an acquisition time 703, and a pointer 704. One entry corresponds to one language signal.

The ID 701 is a field for storing identification information for uniquely identifying the entry of the language signal information 314.

The signal name 702 is a field for storing information for identifying the type of the language signal and the like. In this embodiment, the name of the language signal measuring device 120 that measures the language signal is stored in the signal name 702. The signal name 702 may store the type of data included in the language signal such as "voice".

A field for storing a value indicating the relationship between the measurement start times of the dialogue signal and the language signal and a field for storing the measurement start time of the dialogue signal may be provided.

The acquisition time 703 is a field for storing the acquisition time of the language signal. In the acquisition time 703, for example, a time based on the start time of measurement of the language signal is stored.

The pointer 704 is a field that stores a pointer of a storage area in which the acquired language signal is stored. The language signal is stored as an A / D converted value. In addition, instead of the pointer 704, a field for storing the acquired language signal may be provided.

FIG. 8 is a diagram showing an example of the data structure of the utterance information 315 of the first embodiment.

The utterance information 315 includes an entry composed of an input utterance content 801, a reference ID 802, and an output utterance content 803. One entry corresponds to the user's utterance.

The input utterance content 801 is a field for storing the utterance content of the user identified by analyzing the language signal.

The reference ID 802 is a field for storing the identification information of the language signal used for specifying the utterance content. The value of ID 701 is stored in the reference ID 802.

The output utterance content 803 is a field for storing the utterance content selected based on the utterance content of the user. For example, the utterance content that prompts the user's utterance or the utterance content that responds to the user's utterance is selected. As a method of selecting the utterance content, a method using a language database or the like, a method using a history of past utterance contents, or the like can be considered.

When setting the delay time for the output of the utterance content, the utterance information 315 may be provided with a field for storing the delay time.

FIG. 9 is a diagram showing an example of the data structure of the output information 316 of the first embodiment.

The output information 316 includes an entry composed of an output device 901, an output value 902, and a delay time 903. One entry indicates one output action.

The output device 901 is a field for storing identification information of a device that controls a device corresponding to an output action. In this embodiment, the person image displayed on the display is controlled based on the imitation feature amount and the delay time of the imitation action, and the speaker is controlled based on the utterance content.

The output value 902 stores a value for controlling the interactive interface that performs the output action. When the output action is an imitation action, the type of the imitation feature amount and the imitation feature amount are stored in the output value 902, and when the output action is the output of the utterance content, the utterance content is stored in the output value 902.

FIG. 10 is a diagram showing an example of the data structure of the output condition information 317 of the first embodiment.

The output condition information 317 includes an entry composed of ID 1001, action type 1002, and condition 1003. One entry indicates one condition.

ID1001 is a field for storing identification information for uniquely identifying the entry of the output condition information 317.

The action type 1002 is a field for storing the type of output action. The action type 1002 stores "utterance", "line of sight", "face movement", and "body movement".

The condition 1003 is a field for storing a condition that defines the timing for executing the output action corresponding to the action type 1002. The condition 1003 includes a value that specifies a specific condition, and a value that indicates the delay time and the priority of the condition.

For example, the condition can be set when the user's utterance content is "yes" or "no" and the output utterance content is "yes" or "no". In addition, it can be set on condition that the user starts speaking. Further, the conditions may be set by using external information such as the temperature other than the relationship between the user and the computer 100.

Note that multiple conditions can be set for one type of output action. In this case, the output condition information 317 includes a plurality of entries having the same action type 1002.

The priority does not necessarily have to be set. In this case, the control may be performed when the delay time elapses and the condition is satisfied.

FIG. 11 is a diagram showing an example of the data structure of the output history information 318 of the first embodiment.

The output history information 318 includes an entry composed of ID 1101, sympathy 1102, output value 1103, and delay time 1104. One entry indicates one output information 316.

The ID 1101 is a field for storing identification information for uniquely identifying the entry of the output history information 318.

The empathy 1102 is a field for storing the empathy indicating the degree of empathy of the user who uses the interactive interface. In this embodiment, empathy is used as an index indicating the effectiveness of the output action performed by the dialogue interface. An index other than empathy may be used.

Empathy can be calculated using, for example, the duration of the dialogue. Here, the duration of the dialogue indicates the time from the start of the utterance of the user or the dialogue interface to the end of the utterance of the user or the dialogue interface. It may be determined that the start of the utterance is when the voice is reproduced from the speaker or when the language signal is detected, and the end of the utterance is when a certain period of time has passed when the utterance is not performed. In addition, the duration may be measured by the user pressing a button notifying the start and end of the dialogue.

The sympathy may be calculated based on other information such as the amount of speech of the user. In addition, the empathy may be calculated based on a questionnaire filled out by the user after the end of the dialogue.

The output value 1103 and the delay time 1104 are the same as the output value 902 and the delay time 903.

FIG. 12 is a flowchart illustrating a process executed by the imitation information calculation module 101 of the first embodiment.

The imitation information calculation module 101 determines whether or not an event has occurred (step S101). For example, it is determined that an event has occurred when a user's movement is detected or when a user's utterance is detected. Further, when the user instructs the start of the process by using the input device 150, the imitation information calculation module 101 determines that an event has occurred.

If it is determined that no event has occurred, the imitation information calculation module 101 shifts to the waiting state until the event occurs.

When it is determined that the event has occurred, the imitation information calculation module 101 acquires the dialogue signal from the dialogue signal measuring device 110 (step S102).

At this time, the imitation information calculation module 101 stores the acquired dialogue signal in the storage area. The imitation information calculation module 101 initializes the dialogue signal information 311 and adds entries to the dialogue signal information 311 by the number of acquired dialogue signals. The imitation information calculation module 101 sets a value in each field of the added entry.

The imitation information calculation module 101 may repeatedly execute the process of step S102 until all the necessary dialogue signals are acquired. Further, when the timer is set, the imitation information calculation module 101 may continue to acquire the dialogue signal until the timer elapses.

Next, the imitation information calculation module 101 initializes the variable i (step S103).

Specifically, the imitation information calculation module 101 sets the variable i to "1". Here, the variable i corresponds to the number of acquired dialogue signals and also corresponds to the ID 401 of the dialogue signal information 311.

Next, the imitation information calculation module 101 calculates dialogue data based on the entry whose ID 401 matches the variable i (step S104).

For example, the imitation information calculation module 101 calculates an eyeball image, an entire face image, and a whole body image as dialogue data from one image. At this time, the imitation information calculation module 101 associates the variable i with the dialogue data and stores it in the storage area.

The eyeball image and the face image can be obtained by calculating the feature amounts related to the eyeball and the face from the image, statistical analysis using the feature amount of the image, or a method of acquiring the face image based on machine learning. Conceivable.

Next, the imitation information calculation module 101 calculates the dialogue feature amount (step S105).

At this time, the imitation information calculation module 101 associates the dialogue feature quantity with the type of the dialogue feature quantity and stores it in the storage area.

Next, the imitation information calculation module 101 initializes the variable j (step S106).

Specifically, the imitation information calculation module 101 sets the variable j to "1". At this time, the imitation information calculation module 101 initializes the imitation information 312.

Here, the variable j corresponds to the type of imitation action. In this embodiment, when the variable j is "1", it represents "eye movement", when the variable j is "2", it represents "face movement", and when the variable j is "3", it represents "body". Represents "movement of".

Next, the imitation information calculation module 101 calculates the imitation feature amount of the imitation action corresponding to the variable j based on the calculated dialogue feature amount (step S107). It is assumed that a mathematical formula for calculating the imitation feature amount from the dialogue feature amount is given in advance.

At this time, the imitation information calculation module 101 adds an entry to the imitation information 312, and sets the identification information in the ID 501 of the added entry. The imitation information calculation module 101 sets the value of the variable i in the reference ID 502 of the added entry, sets the identification information corresponding to the variable j in the imitation feature amount type 503, and sets the imitation feature calculated in the imitation feature amount 504. Set the amount. Further, the imitation information calculation module 101 adds as many rows as the number of dialogue features used when calculating the counterfeit features to the added entries, and the dialogue data name 506, dialogue data 507, dialogue feature type 508 of each row, And the identification information and the value are set in the dialogue feature amount 509.

Next, the imitation information calculation module 101 refers to the delay time definition information 313 and sets the delay time of the imitation action corresponding to the variable j (step S108).

Specifically, the imitation information calculation module 101 refers to the delay time definition information 313 and searches for an entry in which the imitation feature amount type 602 matches the type corresponding to the variable j. The imitation information calculation module 101 acquires a value from the delay time 603 of the searched entry, and sets the acquired value in the delay time 505 of the entry of the imitation information 312 added in step S107.

Next, the imitation information calculation module 101 determines whether or not all the imitation feature amounts have been calculated (step S109).

When it is determined that all the imitation feature amounts have not been calculated, the imitation information calculation module 101 adds "1" to the variable j (step S110), and then returns to step S107.

When it is determined that all the imitation feature quantities have been calculated, the imitation information calculation module 101 determines whether or not the processing of all the dialogue signals is completed (step S111).

When it is determined that the processing of all the dialogue signals is not completed, the imitation information calculation module 101 adds "1" to the variable i (step S112), and then returns to step S104.

When it is determined that the processing of all the dialogue signals is not completed, the imitation information calculation module 101 outputs the imitation information 312 to the output information generation module 103, and ends the processing.

In this embodiment, the delay time of the imitation action is set based on the delay time definition information 313, but the following setting method may also be used. The imitation information calculation module 101 refers to the output history information 318 and searches for the latest history (entry). The imitation information calculation module 101 acquires the value of the imitation action delay time 1104 corresponding to the variable j from the searched entry, and sets the acquired value in the delay time 505.

When a plurality of dialogue signals are acquired by using the plurality of dialogue signal measuring devices 110 or a plurality of measurement channels, the imitation information calculation module 101 uses the average value of the dialogue features calculated from each dialogue signal. The imitation feature amount may be calculated.

In addition to the values measured by the dialogue signal measuring device 110, the user may input the dialogue signal or dialogue data using the input device 150.

FIG. 13 is a flowchart illustrating a process executed by the language information calculation module 102 of the first embodiment.

The language information calculation module 102 determines whether or not an event has occurred (step S201). The determination method in step S201 may be the same as the determination method in step S101, or may be different.

If it is determined that no event has occurred, the language information calculation module 102 shifts to the waiting state until the event occurs.

When it is determined that the event has occurred, the language information calculation module 102 acquires the language signal from the language signal measuring device 120 (step S202).

At this time, the language information calculation module 102 stores the acquired language signal in the storage area. The language signal information 314 is initialized, and entries are added to the language signal information 314 as many as the number of acquired language signals. The language information calculation module 102 sets a value in each field of the added entry. Further, the language information calculation module 102 initializes the utterance information 315.

Next, the language information calculation module 102 initializes the variable i (step S203).

Specifically, the language information calculation module 102 sets the variable i to "1". Here, the variable i corresponds to the number of acquired language signals and also corresponds to the ID 701 of the language signal information 314.

At this time, the language information calculation module 102 generates as many lines as the number of acquired language signals in the reference ID 802 of the utterance information 315.

The language information calculation module 102 analyzes the language signal corresponding to the entry whose ID 701 matches the variable i (step S204). Since a known method may be used as the method for analyzing the language signal, detailed description thereof will be omitted.

Next, the language information calculation module 102 updates the utterance content based on the analysis result (step S205).

Specifically, the language information calculation module 102 updates the input utterance content 801 of the utterance information 315 based on the analysis result, and sets the variable i in the empty line of the reference ID 802.

Next, the language information calculation module 102 determines whether or not the processing of all the language signals has been completed (step S206).

When it is determined that the processing of all the language signals is not completed, the language information calculation module 102 adds "1" to the variable i (step S207), and then returns to step S204.

When it is determined that the processing of all the language signals is completed, the language information calculation module 102 selects the output utterance content based on the input utterance content and sets it in the output utterance content 803 of the utterance information 315 (step S208). ). After that, the language information calculation module 102 outputs the utterance information 315 to the output information generation module 103, and ends the process. Since a known method may be used for selecting the output utterance content, detailed description thereof will be omitted.

In this embodiment, only the content of the user's utterance is specified, but the speed and rhythm of the utterance may be calculated as the language feature amount. The feature amount can be used to control the output speed and rhythm of the utterance content of the dialogue interface.

FIG. 14 is a flowchart illustrating a process executed by the output information generation module 103 of the first embodiment.

The output information generation module 103 determines whether or not the information has been received (step S301).

Specifically, the output information generation module 103 determines whether or not at least one of the imitation information 312 and the utterance information 315 has been received. Since the generation timings of the imitation information 312 and the utterance information 315 do not always match, the output information generation module 103 may accept input of information for a certain period of time after receiving any of the information.

If it is determined that the information is not accepted, the output information generation module 103 continues to wait until the information is accepted.

When it is determined that the information has been received, the output information generation module 103 generates the output information 316 (step S302). Specifically, the following processing is executed.

When the output information generation module 103 receives the imitation information 312, the output information generation module 103 reads one entry from the imitation information 312. The output information generation module 103 adds an entry to the output information 316 and sets a "display" in the output device 901 of the added entry. The output information generation module 103 sets the values of the imitation feature amount type 503 and the imitation feature amount 504 of the selected entry to the output value 902 of the added entry. Further, the output information generation module 103 sets the value of the delay time 505 of the selected entry to the delay time 903 of the added entry. The output information generation module 103 executes the same processing for each entry of the imitation information 312.

When the output information generation module 103 receives the utterance information 315, the output information generation module 103 reads one entry from the utterance information 315. The output information generation module 103 adds an entry to the output information 316 and sets a "speaker" in the output device 901 of the added entry. The output information generation module 103 sets the value of the output utterance content 803 of the selected entry to the output value 902 of the added entry. Further, the output information generation module 103 sets "0" to the delay time 903 of the added entry. The above is the description of the process of step S302.

Next, the output information generation module 103 sets a timer (step S303).

Next, the output information generation module 103 determines whether or not there is an output action that satisfies the condition (step S304).

Specifically, the output information generation module 103 searches the output information 316 for entries whose delay time 903 is equal to or greater than the value of the timer. When there is an entry whose delay time 903 is equal to or greater than the value of the timer, the output information generation module 103 determines that an output action satisfying the condition exists.

If it is determined that there is no output action that satisfies the condition, the output information generation module 103 returns to step S304 and continues updating the timer.

When it is determined that an output action satisfying the condition exists, the output information generation module 103 executes the output action corresponding to the entry searched in step S304 (step S305).

Specifically, the output information generation module 103 controls the interactive interface based on the output value 902 of the searched entry. By this control, the dialogue interface outputs the utterance content, and the dialogue interface executes the response action.

Next, the output information generation module 103 updates the output history information 318 (step S306). Specifically, the following processing is executed.

The output information generation module 103 adds an entry to the output history information 318, and sets the identification information in the ID 1101 of the added entry.

The output information generation module 103 generates the same number of rows as the number of entries searched in step S304 in the added entries. The output information generation module 103 sets the output value 902 and the delay time 903 of the entry searched in step S304 in the output value 1103 and the delay time 1104 of the generated line. In the first embodiment, after the control of the dialogue interface based on the output information 316 is completed, the sympathy calculated based on the duration of the dialogue, the amount of speech of the user, the questionnaire, or the like is set to the sympathy 1102. ..

Further, the output information generation module 103 deletes the entry selected in step S304 from the output information 316. The above is the description of the process of step S306.

Next, the output information generation module 103 determines whether or not all the output actions have been executed (step S307).

Specifically, the output information generation module 103 determines whether or not the output information 316 is empty. If the output information 316 is empty, the output information generation module 103 determines that all output actions have been executed.

If it is determined that all the output actions have not been executed, the output information generation module 103 returns to step S304 and executes the same process.

When it is determined that all the output actions have been executed, the output information generation module ends the process.

According to the first embodiment, by controlling the dialogue interface that executes the output action based on the delay time, it is possible to induce the user's empathy and realize a continuous dialogue and a multi-information dialogue.

In the second embodiment, the computer 100 updates the delay time definition information 313 based on the analysis result of the output history information 318, and also updates the utterance content update or the utterance content selection algorithm. The second embodiment will be described with a focus on the differences from the first embodiment.

The system configuration of the second embodiment is the same as the system configuration of the first embodiment. The configuration of the computer 100 of the second embodiment is the same as the configuration of the computer 100 of the first embodiment. As for the information held by the computer 100 of the second embodiment, the delay time definition information 313 is different from the delay time definition information 313 of the first embodiment. Other information is the same as the information held by the computer 100 of the first embodiment.

FIG. 15 is a diagram showing an example of the data structure of the delay time definition information 313 of the second embodiment.

The delay time definition information 313 includes an entry composed of ID 1501, imitation feature quantity type 1502, and delay time range 1503. One entry corresponds to one type of imitation action.

ID 1501 and imitation feature quantity type 1502 are the same fields as ID 601 and mimic feature quantity type 602.

The delay time range 1503 is a field that stores the delay time range corresponding to the imitation action. The delay time range 1503 is a field composed of a minimum value and a maximum value.

If the delay time is always constant, the user may detect the execution of the imitation action and the empathy may decrease. Therefore, in the second embodiment, the computer 100 randomly sets the delay time within a predetermined delay time range. This can prevent the execution of the imitation action from being detected.

The range of delay times during which empathy increases varies with the passage of time and changes in circumstances. Therefore, the computer 100 of the second embodiment updates the range of the delay time by analyzing the history of the output information 316.

FIG. 16 is a flowchart illustrating a process executed by the learning module 104 of the second embodiment. FIG. 17 is a diagram showing a flow of learning processing of the delay time executed by the learning module 104 of the second embodiment.

The learning module 104 executes the process described below after the output information 316 is output or when an instruction is received from the user. The learning module 104 may periodically execute the process.

The learning module 104 initializes the variable i (step S401).

Specifically, the learning module 104 sets the variable i to "1". Here, the variable i corresponds to the type of imitation action. In this embodiment, when the variable i is "1", it represents "eye movement", when the variable i is "2", it represents "face movement", and when the variable i is "3", it represents "body". Represents "movement of".

Next, the learning module 104 executes a learning process of the delay time of the imitation action corresponding to the variable i (step S402). Specifically, the following processing is executed.

(First step) The learning module 104 reads a predetermined number of entries from the output history information 318. Here, a predetermined number of entries are read in ascending order, that is, in descending order of ID1101. The number of entries to be read is set in advance and can be changed as appropriate.

(Second step) The learning module 104 selects one entry from the read entries, and identifies the imitation action corresponding to the variable i in the output value 1103 from the rows included in the selected entry. Search for the line where the information is set. The learning module 104 plots the sympathy 1102 of the selected entry and the delay time 1104 of the retrieved row in a space centered on the sympathy and delay time.

When there are a plurality of rows in which the identification information of the imitation action corresponding to the variable i is set, the learning module 104 plots the statistical value of the delay time 1104 and the sympathy 1102 of each row in the above-mentioned space. The statistical value may be a maximum value, a minimum value, an average value, or the like.

The learning module 104 executes the process (second step) for all the read entries. As a result, graph 1701 is generated.

(Third step) The learning module 104 calculates the optimum delay time based on the processing result of (second step). The following methods can be considered as the calculation method.

(Method 1) The learning module 104 calculates from the graph 1701 the delay time at which the sympathy is maximized as the optimum delay time.

(Method 2) The learning module 104 sets the sympathy as y, the delay time as x, sets a relational expression including an undecided coefficient, and determines the coefficient based on the least squares method. The learning module 104 calculates x having the maximum value of y using the relational expression. For the relational expression, for example, y = ax ² + bx + c is set. If the delay time at which the sympathy is maximized is equal to the maximum or minimum value of all delay times, set y = ax + b as the relational expression and calculate x at which y is maximum within a certain range. Just do it.

(Fourth step) The learning module 104 calculates the delay time range using the optimum delay time. For example, the learning module 104 calculates the optimum delay time multiplied by 0.95 as the minimum value of the range, and the optimum delay time multiplied by 1.05 as the maximum value of the range. Alternatively, the learning module 104 calculates a range in which the empathy is greater than the threshold based on graph 1701.

The above-mentioned process is an example and is not limited to this. For example, multivariate analysis such as principal component analysis using the imitation features included in the entry may be performed.

The learning process of Example 2 can also be applied to Example 1. In this case, (first step), (second step), and (third step) are executed. The learning module 104 sets the optimum delay time to the delay time 603. The above is the description of the process of step S402.

Next, the learning module 104 updates the delay time definition information 313 based on the result of the learning process (step S403).

Specifically, the learning module 104 sets the calculated maximum and minimum values in the delay time range 1503 of the entry of the imitation action corresponding to the variable i.

Next, the learning module 104 determines whether or not the processing is completed for all the imitation actions (step S404).

When it is determined that the processing is not completed for all the imitation actions, the learning module 104 adds "1" to the variable i (step S405), and then returns to step S402.

When it is determined that the processing is completed for all the imitation actions, the learning module 104 executes the learning process of the utterance content (step S406).

Specifically, the following processing is executed.

(First step) The learning module 104 reads a predetermined number of entries from the output history information 318. Here, a predetermined number of entries are read in ascending order, that is, in descending order of ID1101. The number of entries to be read is set in advance and can be changed as appropriate.

(Second step) The learning module 104 searches for a line in which the utterance content is set in the output value 1103 from the lines included in each of the read entries. The learning module 104 acquires the sympathy 1102 and the utterance content of the searched line.

At this time, the learning module 104 may acquire the utterance content set in the searched line as it is, may acquire the utterance content for a predetermined number of characters, or may acquire the utterance content including a specific term. You may get the contents. When the utterance content set in the searched line is acquired as it is, the learning module 104 may divide each phrase and treat it as data in which the phrase and the sympathy are associated with each other.

(Third step) The learning module 104 calculates the relationship between the empathy and the utterance content. The learning module 104 stores utterance contents with high empathy in a language database. At this time, only the clauses including a specific term or expression may be stored in the language database. Further, the learning module 104 may modify the algorithm based on the above-mentioned relationship. The above is the description of the process of step S406.

In the second embodiment, the processing executed by the imitation information calculation module 101 is partially different. Specifically, in step S108, the imitation information calculation module 101 randomly determines the delay time within the range shown in the delay time range 1503. The imitation information calculation module 101 sets the determined delay time in the delay time 505 of the entry of the added imitation information 312. Other processing is the same as the processing of Example 1.

In the second embodiment, the processing executed by the language information calculation module 102 is partially different. Specifically, in step S208, the language information calculation module 102 selects the utterance content stored in the updated language database. When there are a plurality of applicable utterance contents, the selection may be made based on the similarity of the utterance contents and the like. Other processing is the same as the processing of Example 1.

The process executed by the output information generation module 103 of the second embodiment is the same as the process of the first embodiment.

The history of the output information 316 may be managed on a user-by-user basis. In this case, the output history information 318 may be provided with a field for storing the user's identification information. Thereby, the learning process using only the output information 316 related to the target user and the learning process using the output information 316 related to the user other than the target user can be executed. By the learning process, the delay time of the imitation action can be set for each user.

According to the second embodiment, the dialogue that induces higher empathy can be realized by updating the delay time range or the delay time by the learning process. In addition, by randomly selecting the delay time from the delay time width, it is possible to execute various output actions naturally, which induces higher empathy and realizes continuous dialogue and multi-information dialogue. it can.

In the third embodiment, the computer 100 acquires a biological signal for calculating the sympathy and calculates the sympathy in real time. Further, in the third embodiment, the computer 100 calculates the delay time based on the sympathy. The third embodiment will be described with a focus on the differences from the first embodiment.

FIG. 18 is a diagram showing a configuration example of the system of the third embodiment.

The system of the third embodiment includes a measuring device that acquires a biological signal. Specifically, the system of the third embodiment includes an electroencephalogram activity measuring device 1801, a heart rate measuring device 1802, and a sweating amount measuring device 1803.

The electroencephalogram activity measuring device 1801 measures biological signals indicating brain waves, electroencephalogram blood flow, and the like. The heart rate measuring device 1802 measures a biological signal indicating a heart rate or the like. The sweating amount measuring device 1803 measures a biological signal indicating the sweating amount. The heart rate measuring device 1802 and the sweating amount measuring device 1803 are examples of devices for measuring an index used for evaluating autonomic nerve activity, and are not limited thereto.

In addition, the information group 105 of Example 3 includes biological signal information 1900. FIG. 19 is a diagram showing an example of the data structure of the biological signal information 1900 held by the computer 100 of the third embodiment. The biological signal information 1900 includes an entry composed of an ID 1901, a signal name 1902, an acquisition time 1903, and a pointer 1904. One entry corresponds to one biological signal.

ID1901 is a field for storing identification information for uniquely identifying the entry of the biological signal information 1900.

The signal name 1902 is a field for storing information for identifying the type of biological signal and the like. In this embodiment, the name of the measuring device that measured the biological signal is stored in the signal name 1902. The type of data included in the biological signal may be stored.

The acquisition time 1903 is a field for storing the acquisition time of the biological signal. In the acquisition time 1903, for example, a time based on the start time of measurement of the biological signal is stored.

The pointer 1904 is a field for storing a pointer of a storage area in which the acquired biological signal is stored. In addition, instead of the pointer 1904, a field for storing the acquired biological signal may be provided.

In the third embodiment, the output information 316 is partially different from the information included in the information group 105. FIG. 20 is a diagram showing an example of the data structure of the output information 316 of the third embodiment.

The entries included in the output information 316 of Example 3 include sympathy 2001. The sympathy 2001 is a field for storing the sympathy calculated using the biological signal.

In the third embodiment, the processing executed by the imitation information calculation module 101 is partially different. 21A and 21B are flowcharts illustrating the processing executed by the imitation information calculation module 101 of the third embodiment.

When it is determined that the event has occurred, the imitation information calculation module 101 acquires a biological signal together with the dialogue signal (step S151).

At this time, the imitation information calculation module 101 stores the acquired biological signal in the storage area. The imitation information calculation module 101 initializes the biological signal information 1900, and adds entries to the biological signal information 1900 by the number of acquired biological signals. The imitation information calculation module 101 sets a value in each field of the added entry.

Next, the imitation information calculation module 101 calculates the sympathy using the biological signal (step S152).

For example, the imitation information calculation module 101 calculates the amount of sweating per unit time when a biological signal indicating the amount of sweating is acquired from the sweating amount measuring device 1803. When the amount of sweating is large, it indicates that the sympathetic nerve activity is enhanced among the autonomic nerve activities, and when the amount of sweating is small, it indicates that the sympathetic nerve activity is suppressed. Therefore, the imitation information calculation module 101 calculates the sympathy based on a mathematical formula in which the amount of sweating is a variable. When sympathetic nerve activity is suppressed, the empathy is high because it indicates that the user is in a relaxed state.

When a plurality of types of biological signals are acquired, the imitation information calculation module 101 calculates an index for calculating sympathy from each of the different types of biological signals, and calculates the sympathy based on a mathematical formula using the index as a variable. calculate. The index may be a value standardized using a biological signal at rest. For example, when a biological signal indicating a heartbeat interval and a sweating amount is acquired, each index is calculated, the rate of change from the reference value is calculated, and the average value of the rate of change is calculated as sympathy. As the reference value, it is conceivable to use the average value of the biological signal values per unit time calculated using the biological signal acquired at rest.

When there are a plurality of measuring devices that acquire the same type of biological signal, the average value of the indexes calculated from each type of biological signal may be used.

After the process of step S107 is executed, the imitation information calculation module 101 sets the delay time (step S153). Specifically, the following processing is executed.

The imitation information calculation module 101 refers to the delay time definition information 313 and searches for an entry in which the imitation feature amount type 602 matches the type corresponding to the variable j. The counterfeit information calculation module 101 acquires a value from the delay time 603 of the searched entry.

The imitation information calculation module 101 calculates the delay time to be set by using a mathematical formula in which the sympathy and the delay time are variables.

The imitation information calculation module 101 sets the calculated delay time in the delay time 505 of the entry of the imitation information 312 added in step S107. Further, the imitation information calculation module 101 sets the delay time calculated in the delay time 603 of the entry searched from the delay time definition information 313.

The imitation information calculation module 101 refers to the output history information 318, searches for the entry having the highest sympathy from the entries in which the value of the sympathy 1102 is larger than the calculated sympathy, and finds the searched entry. The value of the delay time 1104 may be set as the delay time 505.

An interactive signal may be used when calculating the sympathy. In this case, the process of step S152 is executed after the process of step S107. The above is the description of the process of step S153.

The imitation information calculation module 101 may acquire biological signals and dialogue signals for a certain period of time, and may instruct the learning module 104 to execute the learning process. In this case, the learning process with a short time interval is executed. The imitation information calculation module 101 sets the optimum delay time calculated by the learning module 104.

The processing from step S101 to step S107 and the processing from step S109 to step S112 are the same as the processing described in the first embodiment.

In this embodiment, the sympathy is calculated using the biological signal, but the sympathy may be calculated using the dialogue signal. For example, the imitation information calculation module 101 can calculate a similar index by analyzing a face image, calculating feature amounts of the user's eyebrows, eyes, mouth, etc., and evaluating the user's facial expression from the feature amounts.

The process executed by the language information calculation module 102 is the same as the process of the first embodiment. The processing executed by the output information generation module 103 is partially different from the processing in step S302 and the processing in step S306. The processing of the other steps is the same as the processing of the first embodiment.

In step S302, the output information generation module 103 adds an entry to the output information 316, and then sets the sympathy 2001 of the added entry to the sympathy calculated in step S152. Other processes are the same as those in Example 1.

In step S306, the output information generation module 103 sets the sympathy 1102 of the entry added to the output history information 318 to the sympathy calculated in step S152. Other processes are the same as those in Example 1.

According to the third embodiment, by setting the delay time based on the empathy calculated in real time, it is possible to realize a dialogue that induces higher empathy.

In the fourth embodiment, the computer 100 uses the delay time and the output condition information 317 as the information for controlling the interactive interface for executing the output action. The fourth embodiment will be described with a focus on the differences from the first embodiment.

The system configuration of the fourth embodiment is the same as the system configuration of the first embodiment. The configuration of the computer 100 of the fourth embodiment is the same as the configuration of the computer 100 of the first embodiment. The information held by the computer 100 of the fourth embodiment is the same as the information held by the computer 100 of the first embodiment.

The processing executed by the imitation information calculation module 101 and the language information calculation module 102 of the fourth embodiment is the same as the processing of the first embodiment.

In the fourth embodiment, the processing executed by the output information generation module 103 is partially different. Specifically, the processes of step S304 and step S305 are different.

The output information generation module 103 searches the output information 316 for entries whose delay time 903 is equal to or greater than the value of the timer. Further, the output information generation module 103 refers to the output condition information 317 and searches for an entry corresponding to the current state under the condition 1003 based on the imitation feature amount, the utterance content, and the like.

When an entry corresponding to at least one of the two search processes exists, the output information generation module 103 determines that an output action satisfying the condition exists.

In step S305, the output information generation module 103 adjusts the control timing of the interactive interface based on the delay time 903 and the condition 1003 acquired by the two search processes. When an entry is searched from each of the two search processes, the interactive interface is controlled based on the priority.

According to the fourth embodiment, the computer 100 can execute various output actions by controlling the execution of the output actions based on the delay time and the output conditions. Therefore, it is possible to induce user empathy and realize continuous dialogue and multi-information dialogue.

In the fifth embodiment, a GUI (Graphical User Interface) for making settings for the computer 100 will be described in each embodiment.

22 and 23 are diagrams showing an example of a GUI for making settings for the computer 100 of the fifth embodiment.

The GUI 2200 shown in FIG. 22 is a GUI for setting information, and includes an interactive feature amount setting field 2210, a mimicking feature amount setting field 2220, a condition setting field 2230, a learning process setting field 2240, a biological signal setting field 2250, and empathy. It includes a degree evaluation setting field 2260, a delay time setting button 2270, and a save button 2275.

The dialogue feature amount setting field 2210 is a field for setting the dialogue feature amount, and includes a set number display field 2211, a radio button 2212, and a setting field group 2213. By inputting a value in the dialogue feature amount setting field 2210, the dialogue data to be calculated and the dialogue feature amount can be set in the computer 100.

The set number display field 2211 is a field for displaying the number of types of set dialogue features.

The radio button 2212 is a button operated when setting the dialogue feature amount. By operating the radio button 2212, it is possible to input to the setting field group 2213.

The setting field group 2213 includes a field for setting the type of the dialogue signal, the type of the dialogue data, and the type of the dialogue feature amount. The setting field group 2213 may include other fields.

The imitation feature amount setting field 2220 is a column for setting the imitation feature amount, and includes a set number display field 2221, a radio button 2222, and a setting field group 2223. By inputting a value in the imitation feature amount setting field 2220, the imitation feature amount to be calculated and the delay time corresponding to the imitation action can be set.

The set number display column 2221 is a column for displaying the number of types of set imitation feature quantities.

The radio button 2222 is a button operated when setting the dialogue feature amount. By operating the radio button 2222, it is possible to input to the setting field group 2223.

The setting field group 2223 includes fields for setting the type of imitation feature amount, the type of dialogue feature amount, and the delay time. The setting field group 2223 may include other fields. The delay time included in the setting field group 2223 is set in the delay time definition information 313.

The optimum delay time or delay time range calculated by using GUI2300, which will be described later, is set in the delay time field included in the setting field group 2223. An expert may manually set a value in the delay time field.

The condition setting field 2230 is a field for setting the condition of the imitation feature amount, and includes the set number display field 2231, the radio button 2232, and the setting field group 2233. The output condition information 317 can be set by inputting a value in the condition setting field 2230.

The set number display field 2231 is a field for displaying the number of set conditions.

The radio button 2232 is a button operated when setting a condition. By operating the radio button 2232, it is possible to input to the setting field group 2233.

The setting field group 2233 includes the imitation feature amount type and the condition. The setting field group 2233 may include other fields.

The learning process setting field 2240 is a field for setting a learning method for the delay time, and includes a setting display field 2241 and a set radio button group. The learning method can be set by inputting a value in the learning process setting field 2240.

In this embodiment, a radio button for selecting the presence or absence of learning data and a radio button for selecting the learning data to be used are included. The operation result of the setting radio button group is displayed in the setting display field 2241.

The biological signal setting field 2250 is a field for setting a biological signal and an index, and includes a set number display field 2251, a radio button 2252, and a setting field group 2253. By inputting a value in the biological signal setting field 2250, the biological signal to be measured and the index to be calculated can be set.

The set number display field 2251 is a field for displaying the number of set types of biological signals.

The radio button 2252 is a button operated when setting a biological signal. By operating the radio button 2252, it is possible to input to the setting field group 2253.

The setting field group 2253 includes fields for setting the biological signal type, activity, and index. The activity is a field for storing identification information of the activity to be evaluated based on the biological signal corresponding to the biological signal type. An index is a field that sets the relationship between an activity and an index. The setting field group 2253 may include other fields.

The sympathy evaluation setting field 2260 is a field for setting an index used when calculating the sympathy, and includes a setting display field 2261 and a set radio button group. By inputting a value in the sympathy evaluation setting field 2260, it is possible to set whether or not to calculate the sympathy in real time.

In this embodiment, a radio button for selecting whether or not to evaluate the sympathy in real time and a radio button for selecting a biological signal to be used at the time of evaluation are included. The operation result of the setting radio button group is displayed in the setting display field 2261.

The delay time setting button 2270 is a button for instructing the execution of the learning process. When the button is operated, the learning module 104 executes the process shown in FIG. 16 and displays the process result as the GUI 2300 shown in FIG.

The save button 2275 is a button operated when saving the input of each setting field.

The GUI 2200 may include a button for switching between a display mode and an edit mode.

The GUI 2300 shown in FIG. 23 is a GUI for setting a delay time or a delay time width based on a learning process, and is a mimicry feature amount type selection field 2301, an optimum delay time display field 2302, a delay time range display field 2303, and a graph. Display field 2304, setting display field 2305, fitting method setting field 2306, setting display field 2307, learning data selection field 2308, setting display field 2309, calculation method setting field 2310, setting display field 2311, delay time range calculation method setting field 2312 , And a return button 2313.

The imitation feature amount type selection column 2301 is a column for selecting the imitation feature amount corresponding to the imitation action to be learned.

The optimum delay time display column 2302 is a column for displaying the optimum delay time calculated by the learning process.

The delay time range display column 2303 is a column for displaying the delay time range calculated by the learning process.

The graph display column 2304 is a column for displaying a graph showing the relationship between the delay time and the sympathy.

The setting display field 2305 is a field for displaying the operation result of the fitting method setting field 2306.

The fitting method setting field 2306 is a field for setting the fitting method. The fitting method setting field 2306 includes a group of radio buttons for selecting a fitting method.

The setting display field 2307 is a field for displaying the operation result of the learning data selection field 2308.

The learning data selection field 2308 is a field for selecting data to be used for learning. The training data selection field 2308 includes a radio button for selecting either personal data or group data.

The setting display field 2309 is a field for displaying the operation result of the calculation method setting field 2310.

The calculation method setting column 2310 is a column for setting the calculation method of the optimum delay time. The calculation method setting field 2310 includes a radio button for selecting whether or not to calculate the optimum delay time, and a radio button for selecting the calculation standard for the optimum delay time.

The setting display field 2311 is a field for displaying the operation result of the delay time range calculation method setting field 2312. The maximum value and the minimum value are displayed in the calculation standard.

The delay time range calculation method setting column 2312 is a column for setting the delay time range calculation method. The delay time range calculation method setting field 2312 includes a radio button for selecting whether or not to calculate the delay time range, and a radio button for selecting a delay time calculation standard. Percentages and thresholds are displayed in the calculation criteria. The values set for the ratio and the threshold value can be changed.

The return button 2313 is a button for switching to the GUI 2200.

By using GUI2200 and GUI2300, it is possible to easily set the information necessary for setting the delay time such as the dialogue feature amount, the imitation feature amount, and the biological signal, and it is possible to easily confirm and change the set value. ..

The present invention is not limited to the above-mentioned examples, and includes various modifications. Further, for example, the above-described embodiment describes the configuration in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to the one including all the described configurations. Further, it is possible to add, delete, or replace a part of the configuration of the embodiment with another configuration.

Further, each of the above configurations, functions, processing units, processing means and the like may be realized by hardware by designing a part or all of them by, for example, an integrated circuit. The present invention can also be realized by a program code of software that realizes the functions of the examples. In this case, a storage medium in which the program code is recorded is provided to the computer, and the CPU included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the function of the above-described embodiment, and the program code itself and the storage medium storing the program code itself constitute the present invention. Examples of the storage medium for supplying such a program code include a flexible disk, a CD-ROM, a DVD-ROM, a hard disk, an SSD (Solid State Drive), an optical disk, a magneto-optical disk, a CD-R, and a magnetic tape. Non-volatile memory cards, ROMs, etc. are used.

In addition, the program code that realizes the functions described in this embodiment can be implemented in a wide range of programs or script languages such as assembler, C / C ++, perl, Shell, PHP, and Java (registered trademark).

Further, by distributing the program code of the software that realizes the functions of the examples via the network, it is stored in a storage means such as a hard disk or memory of a computer or a storage medium such as a CD-RW or a CD-R. The CPU provided in the computer may read and execute the program code stored in the storage means or the storage medium.

In the above-described embodiment, the control lines and information lines show what is considered necessary for explanation, and do not necessarily indicate all the control lines and information lines in the product. All configurations may be interconnected.

100 Computer 101 Counterfeit information calculation module 102 Language information calculation module 103 Output information generation module 104 Learning module 110 Dialogue signal measurement device 120 Language signal measurement device 130 Display device 140 Dialogue output device 150 Input device 154 Delay time 301 Calculation device 302 Storage device 303 Input interface 304 Output interface 311 Dialogue signal information 312 Imitation information 313 Delay time definition information 314 Language signal information 315 Speech information 316 Output information 317 Output condition information 318 Output history information 1801 Brain wave activity measuring device 1802 Heart rate measuring device 1803 Sweat amount measuring device 1900 Biosignal information

Claims (13)

An interactive system that provides an interactive interface for interacting with users.
The dialogue system includes an arithmetic unit, a storage device connected to the arithmetic unit, a computer having an interface connected to the arithmetic unit, and a measuring device for measuring a signal related to the utterance of the user.
The storage device contains an entry composed of a type of response action performed by the dialogue interface to the user and a setting range indicating a settable range of a control time for adjusting the control timing of the response action. Holds control time definition information including
The arithmetic unit
Based on the signal measured by the measuring device, the first feature amount indicating the feature of the user action in the utterance of the user is calculated.
Based on the first feature amount, it calculates a second characteristic quantity which is a control value of the response actions,
With reference to the control time definition information , a random control time is calculated based on the setting range corresponding to the type of the response action.
A dialogue system characterized in that the dialogue interface is controlled based on the second feature amount and the control time. The dialogue system according to claim 1.
The storage device holds condition information that manages execution conditions of the response action.
The arithmetic unit is a dialogue system characterized in that the dialogue interface is controlled based on the second feature quantity, the control time, and the condition information. An interactive system that provides an interactive interface for interacting with users.
The dialogue system includes an arithmetic unit, a storage device connected to the arithmetic unit, a computer having an interface connected to the arithmetic unit, and a measuring device for measuring a signal related to the utterance of the user.
The storage device is
Control time definition information for managing the control time for adjusting the control timing of the response action, which is set for each type of the response action performed by the dialogue interface for the user, and
History information that manages the history of the response actions performed by the dialogue interface, and
Hold,
The historical information includes an index indicating the effectiveness of the response action.
The arithmetic unit
Based on the signal measured by the measuring device, the first feature amount indicating the feature of the user action in the utterance of the user is calculated.
Based on the first feature amount, the second feature amount, which is a control value of the response action, is calculated.
Based on the control time definition information, the control time according to the type of the response action is calculated.
The dialogue interface is controlled based on the second feature amount and the control time.
Further, the arithmetic unit is
With reference to the history information, the relationship between the index and the control time is analyzed.
An interactive system characterized in that the control time definition information is updated based on the result of the analysis. The dialogue system according to claim 3.
The storage device holds condition information that manages execution conditions of the response action.
The arithmetic unit is a dialogue system characterized in that the dialogue interface is controlled based on the second feature quantity, the control time, and the condition information. An interactive system that provides an interactive interface for interacting with users.
The dialogue system includes an arithmetic unit, a storage device connected to the arithmetic unit, a computer having an interface connected to the arithmetic unit, and a measuring device for measuring a signal related to the utterance of the user.
The arithmetic unit
Based on the signal measured by the measuring device, the first feature amount indicating the feature of the user action in the utterance of the user is calculated.
Based on the first feature amount, the second feature amount, which is a control value of the response action performed by the dialogue interface to the user, is calculated.
Based on the signal measured by the measuring device, an index indicating the effectiveness of the response action is calculated.
Based on the first feature amount and the index, the control time for adjusting the control timing of the response action is calculated.
A dialogue system characterized in that the dialogue interface is controlled based on the second feature amount and the control time. An interactive system that provides an interactive interface for interacting with users.
The dialogue system includes an arithmetic unit, a storage device connected to the arithmetic unit, a computer having an interface connected to the arithmetic unit, and a measuring device for measuring a signal related to the utterance of the user.
The arithmetic unit
Based on the signal measured by the measuring device, the first feature amount indicating the feature of the user action in the utterance of the user is calculated.
Based on the first feature amount, the second feature amount, which is a control value of the response action performed by the dialogue interface to the user, is calculated.
A control time for adjusting the control timing of the response action is calculated according to the type of the response action.
The dialogue interface is controlled based on the second feature amount and the control time.
The response action is an action that imitates the action of the user.
The dialogue system, wherein the control time is a delay time. A method of controlling a dialogue system that provides a dialogue interface for interacting with a user.
The dialogue system includes an arithmetic unit, a storage device connected to the arithmetic unit, a computer having an interface connected to the arithmetic unit, and a measuring device for measuring a signal related to the utterance of the user.
The storage device contains an entry composed of a type of response action performed by the dialogue interface to the user and a setting range indicating a settable range of a control time for adjusting the control timing of the response action. Holds control time definition information including
The control method of the dialogue system is
The first step in which the arithmetic unit calculates a first feature amount indicating the characteristics of the user action in the utterance of the user based on the signal measured by the measuring device.
A second step in which the arithmetic unit calculates a second feature amount, which is a control value of the response action, based on the first feature amount.
A third step in which the arithmetic unit calculates the control time according to the type of the response action, and
The arithmetic unit includes a fourth step of controlling the interactive interface based on the second feature amount and the control time.
The third step is characterized in that the arithmetic unit includes a step of calculating a random control time based on the setting range corresponding to the type of the response action by referring to the control time definition information. How to control the interactive system. The control method of the dialogue system according to claim 7.
The storage device holds condition information that manages execution conditions of the response action.
The fourth step is a control method of a dialogue system, wherein the arithmetic unit includes a step of controlling the dialogue interface based on the second feature amount, the control time, and the condition information. A method of controlling a dialogue system that provides a dialogue interface for interacting with a user.
The dialogue system includes an arithmetic unit, a storage device connected to the arithmetic unit, a computer having an interface connected to the arithmetic unit, and a measuring device for measuring a signal related to the utterance of the user.
The storage device is
Control time definition information for managing the control time for adjusting the control timing of the response action, which is set for each type of the response action performed by the dialogue interface for the user, and
History information that manages the history of the response actions performed by the dialogue interface, and
Hold,
The historical information includes an index indicating the effectiveness of the response action.
The control method of the dialogue system is
The first step in which the arithmetic unit calculates a first feature amount indicating the characteristics of the user action in the utterance of the user based on the signal measured by the measuring device.
A second step in which the arithmetic unit calculates a second feature amount, which is a control value of the response action, based on the first feature amount.
A third step in which the arithmetic unit calculates the control time according to the type of the response action based on the control time definition information.
A fourth step in which the arithmetic unit controls the interactive interface based on the second feature amount and the control time.
A fifth step in which the arithmetic unit analyzes the relationship between the index and the control time with reference to the history information, and
A sixth step in which the arithmetic unit updates the control time definition information based on the result of the analysis.
A method of controlling an interactive system, characterized in that it includes. The control method of the dialogue system according to claim 9.
The storage device holds condition information that manages execution conditions of the response action.
The fourth step is a control method of a dialogue system, wherein the arithmetic unit includes a step of controlling the dialogue interface based on the second feature amount, the control time, and the condition information. A method of controlling a dialogue system that provides a dialogue interface for interacting with a user.
The dialogue system includes an arithmetic unit, a storage device connected to the arithmetic unit, a computer having an interface connected to the arithmetic unit, and a measuring device for measuring a signal related to the utterance of the user.
The control method of the dialogue system is
The first step in which the arithmetic unit calculates a first feature amount indicating the characteristics of the user action in the utterance of the user based on the signal measured by the measuring device.
A second step in which the arithmetic unit calculates a second feature amount, which is a control value of a response action performed by the dialogue interface to the user, based on the first feature amount.
A third step in which the arithmetic unit calculates an index indicating the effectiveness of the response action based on the signal measured by the measuring device.
A fourth step in which the arithmetic unit calculates a control time for adjusting the control timing of the response action based on the first feature amount and the index.
A control method for a dialogue system, wherein the arithmetic unit includes a fifth step of controlling the dialogue interface based on the second feature amount and the control time. A method of controlling a dialogue system that provides a dialogue interface for interacting with a user.
The dialogue system includes an arithmetic unit, a storage device connected to the arithmetic unit, a computer having an interface connected to the arithmetic unit, and a measuring device for measuring a signal related to the utterance of the user.
The control method of the dialogue system is
The first step in which the arithmetic unit calculates a first feature amount indicating the characteristics of the user action in the utterance of the user based on the signal measured by the measuring device.
A second step in which the arithmetic unit calculates a second feature amount, which is a control value of a response action performed by the dialogue interface to the user, based on the first feature amount.
A third step in which the arithmetic unit calculates a control time for adjusting the control timing of the response action according to the type of the response action.
The arithmetic unit includes a fourth step of controlling the interactive interface based on the second feature amount and the control time.
The response action is an action that mimics the action of the user.
A control method for an interactive system, wherein the control time is a delay time. A device that provides an interactive interface for interacting with a user.
The device includes an arithmetic unit, a storage device connected to the arithmetic unit, an interface connected to the arithmetic unit, and a measuring instrument for measuring a signal related to the utterance of the user.
The storage device provides delay time definition information for managing a delay time for adjusting the control timing of the response action for each type of response action that imitates the user's action, which is performed on the user by the dialogue interface. Hold and
The arithmetic unit
Based on the signal measured by the measuring instrument, the first feature amount indicating the feature of the user action in the utterance of the user is calculated.
Based on the first feature amount, the second feature amount, which is a control value of the response action, is calculated.
Based on the delay time definition information, the delay time according to the type of the response action is calculated.
An apparatus characterized in that the dialogue interface is controlled based on the second feature amount and the delay time.

Images