JP7317403B2 - Processing device, processing method, and program - Google Patents

Description

The present invention relates to technology for supporting communication between a person and an interactive device.

Techniques for realizing pseudo-communication (hereinafter simply referred to as "communication") between a person and an interactive device have been conventionally proposed. US Pat. No. 5,300,000 discloses an interactive toy. Also, an interactive device has been proposed that simulates the blinking of an eye. Patent Literature 2 discloses opening and closing the eyes of a CG character at preset intervals between blinks. Patent Literature 3 discloses a robot that starts from the timing of a nodding motion of the head and performs a blinking motion at timings that are exponentially distributed over time. Regarding the blinking of the speaker and the listener, the following items are disclosed in Non-Patent Document 1 and Non-Patent Document 2 by the inventors of this case. Non-Patent Document 1 discloses that the blinks of the speaker and the listener are synchronized with a time delay. In addition, Non-Patent Document 1 discloses that a speaker's blinking increases near the end of a talk or between utterances. Non-Patent Document 2 discloses the time-delayed synchronization of blinks between a robot speaker and a human listener.

Japanese Patent Publication No. 2010-525848 Patent No. 5639440 JP-A-2000-349920

Tamami Nakano and Shigeru Kitazawa, "Eyeblink entrainment at breakpoints of speech", Experimental Brain Research, 205(4), p.577-81, [online], [searched March 12, 2018], Internet <URL: https ://www.ncbi.nlm.nih.gov/pubmed/20700731>, (2010) Kyohei Tatsukawa, Tamami Nakano, Hiroshi Ishiguro and Yuichiro Yoshikawa, "Eyeblink Synchrony in Multimodal Human-Android Interaction", Scientific Reports, 6:39718, [online], [searched March 12, 2018], Internet <URL: https ://www.nature.com/articles/srep39718>, (2016)

Regardless of the communication between the user and the interactive device, even if the interactive device blinks, it may be difficult to contribute to improving the quality of communication.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to support communication between an interactive device and a user by using blinking motions.

An embodiment of the present invention comprises an environment information acquisition unit that acquires environment information indicating the environment around an interaction device, and a blinking action control unit that controls the frequency of blinking actions performed by the interaction device based on the environment information. A processing apparatus is provided having:
In the above processing device, the blinking motion control unit learns the relationship between the surrounding environment of the interactive device and the blinking performed by a human under the environment, and causes the interactive device to perform the blinking motion based on the relationship. may
In the above processing device, the environment information may be dialogue information from a user to the dialogue device.
In the above processing device, the blinking motion control section may increase the frequency of blinking motions performed by the interactive device when the environment information satisfies a predetermined condition.
In the above processing device, the blinking motion control unit may increase the frequency of blinking motions when determining that the dialogue information has become a highly unexpected topic based on the learning result.
An embodiment of the present invention is a processing method executed by a computer, in which environmental information indicating the environment around an interactive device is obtained, and the frequency of blinking performed by the interactive device is controlled based on the environmental information. Provide processing methods.
An embodiment of the present invention is a program for causing a computer to obtain environmental information indicating the environment around an interactive device and to control the frequency of blinking actions performed by the interactive device based on the environmental information. I will provide a.
An embodiment of the present invention comprises: a first acquisition unit that acquires the number of first blinking actions performed by an interactive device during a specific period or scene; A processing device is provided that includes a second acquisition unit and a processing unit that performs interactive processing according to the number of first blinking operations and the number of second blinking operations in a predetermined period.
In the above processing device, the processing unit calculates an evaluation value based on the number of times of the first blinking action and the number of times of the second blinking action in the specific period or scene, and determines the relationship between the evaluation value and a predetermined threshold value. You may perform interactive processing based on.
In the above processing device, the processing unit may change the current topic when the evaluation value is less than a predetermined threshold.
In the above processing device, the processing unit may end the interactive processing when the evaluation value is less than a predetermined threshold.
In the above processing device, the processing unit may continue the current topic when the evaluation value is equal to or greater than a predetermined threshold.
An embodiment of the present invention is a computer-implemented processing method, which obtains the number of first blinking actions by an interactive device in a specific period or scene, Provided is a processing method that acquires the number of operations and performs interactive processing according to the number of times of the first blinking operation and the number of times of the second blinking operation in a predetermined period.
According to an embodiment of the present invention, a computer acquires a first number of blinking actions by an interactive device in a specific period or scene, acquires a second number of blinking actions by a user of the interactive device in the specific period, and obtains a predetermined number of blinking actions. A program is provided for realizing interactive processing according to the number of times of the first blinking action and the number of times of the second blinking action in the period of .
An embodiment of the present invention comprises: a first acquisition unit that acquires a blinking timing of a dialogue device; a second acquisition unit that acquires a blinking timing of a user of the dialogue device; and a processing unit that performs processing according to the difference from the timing of blinking.

In the above processing device, the processing unit may perform processing according to the index value based on the difference.

In the above processing device, the processing unit may perform processing according to the degree to which the blinking timing of the user is included within a predetermined period corresponding to the timing of the blinking motion.

In the above processing device, the predetermined period of time may include a time point of 500 milliseconds or less from the timing of the blinking action.

In the above processing device, the processing unit stores the degree in first data in which the user's blink timing and the blink action timing are arranged in chronological order on a predetermined time axis, the user's blink timing and the blink. The processing may be performed according to the degree in the second data in which the order of at least one of the operation timings is changed.

In the above processing device, the processing unit may cause the dialogue device to perform dialogue processing according to the difference.

In the above processing device, the processing unit may output evaluation data corresponding to the difference in association with an identifier of the interactive device.

In the above processing device, an environment information acquisition unit that acquires environment information indicating an environment around the interaction device; a blinking operation control unit that causes the interaction device to perform a blinking operation at a first timing according to the environment information; may have

In the above processing device, a storage control unit is provided for accumulating in a storage device data that associates the timing at which the user blinks with the environment, and the blinking operation control unit stores the first timing in the storage device. The timing may be determined according to the data accumulated in the storage and the environmental information.

In the processing device described above, the blinking motion control section may cause the interactive device to perform a blinking motion at a second timing different from the first timing.

The processing device described above includes an eyelid portion corresponding to an eyelid, and a blinking motion control portion that controls a blinking motion of the eyelid portion by opening and closing the eyelid portion, wherein the first obtaining portion comprises: The timing of the blinking motion of the eyelid may be acquired.

The above processing device has a display unit and a blinking action control unit that controls a blinking action of an object displayed on the display unit, and the first acquisition unit acquires the timing of the blinking action of the object. may

An embodiment of the present invention acquires the timing of a blinking action of an interactive device and the timing of a blinking of a user of the interactive device, and performs processing according to the difference between the timing of the blinking action and the timing of the user's blinking. to do, to provide a treatment method.

In one embodiment of the present invention, a computer acquires the timing of a blinking action of an interactive device and the timing of a blinking of a user of the interactive device, and according to the difference between the timing of the blinking action and the timing of the user's blinking, Provide a program for realizing the function to perform the processing.

According to the present invention, it is possible to support communication between the interactive device and the user by using blinking motions.

It is a figure showing an example of the appearance composition of the interactive device which is a 1st embodiment of the present invention. FIG. 4 is a diagram illustrating a blinking action performed by the interactive device according to the first embodiment of the present invention; 1 is a block diagram showing the hardware configuration of a dialogue device that is a first embodiment of the present invention; FIG. 1 is a block diagram showing the functional configuration of a dialogue device that is a first embodiment of the present invention; FIG. 4 is a flow chart showing processing executed by the interactive device according to the first embodiment of the present invention; 4 is a flowchart showing index calculation processing according to the first embodiment of the present invention; It is a figure explaining the calculation method of the timing difference of 1st Embodiment of this invention. 4 shows a graph showing the distribution of appearance frequencies of timing differences according to the first embodiment of the present invention; It is a figure which shows an example of the random data of 1st Embodiment of this invention. It is a figure which shows an example of the evaluation value of 1st Embodiment of this invention. 5 is a graph showing the frequency distribution of listener's blinks according to the verification of the first embodiment of the present invention. FIG. 10 is a graph showing the frequency distribution of the listener's blinks for each answer result related to the verification of the first embodiment of the present invention; FIG. It is the graph which showed the evaluation value which concerns on the verification of 1st Embodiment of this invention according to sex and goods. FIG. 10 is a graph showing the degree of interest in products according to the verification of the first embodiment of the present invention by sex and product; FIG. FIG. 4 is a block diagram showing the functional configuration of a dialogue device that is a second embodiment of the present invention; 9 is a flow chart showing learning processing executed by the interactive device according to the second embodiment of the present invention; 9 is a flow chart showing processing related to a blinking action executed by the interactive device according to the second embodiment of the present invention; FIG. 11 is a block diagram showing the hardware configuration of a dialogue device that is a third embodiment of the present invention; FIG. 11 is a block diagram showing the functional configuration of a dialogue device that is a third embodiment of the present invention; FIG. 11 is a flow chart showing processing executed by a dialogue device according to a third embodiment of the present invention; FIG. It is a block diagram showing a functional configuration of a processing device of one embodiment of the present invention.

An embodiment of the present invention will be described in detail below with reference to the drawings. The embodiments shown below are examples of embodiments of the present invention, and the present invention is not limited to these embodiments. In the drawings referred to in this embodiment, the same parts or parts having similar functions are denoted by the same reference numerals or similar reference numerals (reference numerals followed by A, B, etc.). may be omitted.

The inventors have found that the difference in blink timing between the speaker and the listener can be used to evaluate the quality of communication performed by the speaker and the listener. For example, when the blink timings of the speaker and the listener match to a high degree, it can be inferred that the listener shows a high degree of interest in the speaker's utterance. Conversely, if the degree of agreement is low, it can be inferred that the listener does not show much interest in the speaker's utterance. The verification that led to such findings will be described later. An embodiment in which such knowledge is applied to a technique for realizing communication between an interactive device and its user will be described below.

[First embodiment]
FIG. 1 is a diagram showing an example of the external configuration of a dialogue device 10 that is the first embodiment of the present invention. The interaction device 10 is a processing device that interacts with the user U. FIG. A user U is a user of the interactive device 10 . A user U faces the dialogue device 10 and communicates with the dialogue device 10 through dialogue.

The interactive device 10 is a robot having an appearance imitating a living thing. The interactive device 10 has, for example, an appearance that resembles a human or other animal (eg, dog or cat), or a fictional character (eg, animation character). The appearance of the interactive device 10 does not matter.

The interactive device 10 has a face portion 101 and an eyelid portion 102 . A facial portion 101 is a portion corresponding to the face, and an eyelid portion 102 is arranged on the facial portion 101 and is a portion corresponding to eyelids. By opening and closing the eyelid part 102, the eyelid part 102 performs an action simulating the blinking of the eye (hereinafter referred to as "blinking action"). In this embodiment, the two eyelids 102 perform the same action.

FIG. 2 is a diagram for explaining the blinking action of the interactive device 10. As shown in FIG. The interactive device 10 normally opens the eyelid portion 102 . The interactive device 10 transitions the eyelid 102 from the open state to the closed state (arrow A1) and from the closed state to the open state (arrow A2) at the timing of blinking. The timing of transition from the open state to the closed state and from the closed state to the open state of the eyelid portion 102 is set in advance so that the eyelid portion 102 behaves like a blink of an eye.

The interactive device 10 may have a mouth, nose and other parts on the face 101 . The interactive device 10 may also move these parts arranged on the face 101, but the description is omitted in this embodiment.

The installation location and use of the interactive device 10 are not particularly limited. The interactive device 10 is installed, for example, in commercial facilities (for example, shops), public facilities, or other facilities. In this case, user U is a user of the facility. The interactive device 10 may be used in medical applications, toys, or other applications.

FIG. 3 is a block diagram showing the hardware configuration of the interactive device 10. As shown in FIG. The interactive device 10 has a control unit 11 , a voice input unit 12 , a voice output unit 13 , a storage unit 14 , an imaging unit 15 and an eyelid unit 102 . The control unit 11 controls each unit of the interactive device 10 . The control unit 11 includes, for example, an arithmetic processing unit exemplified by a CPU, and a memory. The memory includes, for example, a RAM used as a work area by the arithmetic processing unit, and a ROM for storing control programs.

The voice input unit 12 receives input of voice. The voice input unit 12 converts the received voice into a voice signal and supplies the voice signal to the control unit 11 . The voice input unit 12 includes, for example, a microphone, an A (Analog)/D (Digital) conversion circuit, and a filter.

The audio output unit 13 outputs audio. The audio output unit 13 outputs sound converted from the audio signal supplied from the control unit 11 . The audio output unit 13 includes, for example, a D/A conversion circuit and a speaker.

The storage unit 14 stores data. The storage unit 14 stores, for example, a program 141 and dialogue data 142 . The program 141 is a program for causing the control unit 11 to implement a predetermined function.

The dialogue data 142 is data for the dialogue device 10 to dialogue with the user U. FIG. The dialog data 142 stores, for example, a plurality of data in which input data and output data are associated with each other. The input data is data in which the content of an utterance assumed to be uttered by the user U is represented by a character string. The output voice is data representing the content of the response to the utterance in a character string. For example, when the input data is "What is your name?", the output data associated with the input data is "My name is XX" ("XX" is the name of the interactive device 10).

Dialogue data 142 may include identifiers that identify topics in association with input data and output data. For example, the first identifier "ID001" is associated with input data and output data used for utterances related to soccer. The second identifier “ID002” is associated with the input data and output data used for utterances regarding meals.

It should be noted that the dialogue data 142 may be data in another format. The storage unit 14 can include any type of recording medium (storage device) exemplified by, for example, an optical recording medium, a magnetic recording medium, and a semiconductor recording medium.

The imaging unit 15 captures an image of a subject and generates imaging data representing the captured image. The subject is the user U. The imaging unit 15 includes, for example, an imaging element exemplified by a CCD (Charge Coupled Device) image sensor, and a lens. The lens of the imaging unit 15 is provided, for example, near the eyelid 102 on the face 101 , but may be provided at another position on the face 101 or at a position other than the face 101 .

The eyelid part 102 opens and closes under the control of the control part 11 . The eyelid 102 includes, for example, an opening/closing mechanism (eg diaphragm and cylinder) and a drive circuit for driving the opening/closing mechanism. Various known techniques can be applied to the mechanism for realizing the blinking action.

FIG. 4 is a block diagram showing the functional configuration of the interactive device 10. As shown in FIG. By executing the program 141, the control unit 11 of the interactive device 10 corresponds to the blinking operation control unit 111, the first acquisition unit 112, the blink detection unit 113, the second acquisition unit 114, and the processing unit 115. Realize the function to

The blinking motion control unit 111 controls the blinking motion of the interactive device 10 by opening and closing the eyelid part 102 . The blinking motion control unit 111 outputs, for example, blinking control data for performing a blinking motion to the eyelid unit 102 . The eyelid part 102 opens and closes according to the blink control data.

The first acquisition unit 112 acquires the timing of the blinking motion of the interactive device 10 (eyelid unit 102). The first acquisition unit 112 acquires, for example, data indicating the timing of the blinking motion from the blinking motion control unit 111 .

The blink detection unit 113 detects user U's blink. Specifically, the blink detection unit 113 detects blinks of the user U based on the imaging data generated by the imaging unit 15 .

The second acquisition unit 114 acquires the timing of user U's blinking. In the present embodiment, the second acquisition unit 114 acquires data indicating timing of blinking of the user U (hereinafter referred to as “blink data”). Blink data is data (first data) in which times when blinks are detected are arranged in order of time. The second acquisition unit 114 generates blink data, for example, based on the result of blink detection by the blink detection unit 113 .

The processing unit 115 performs processing according to the difference between the timing of the blinking action of the interactive device 10 and the timing of the user U's blinking. The processing unit 115 performs interactive processing in this embodiment. Dialogue processing is processing for having a dialogue using the dialogue data 142 . Specifically, the dialogue processing includes processing for recognizing speech input via the speech input unit 12 and converting it into input data. Further, the dialogue processing includes processing for converting output data associated with the input data into speech and outputting the speech via the speech input unit 12 .

Next, the operation of the interactive device 10 will be described. FIG. 5 is a flow chart showing the processing executed by the interactive device 10. As shown in FIG.

The processing unit 115 starts interactive processing (step S1). It does not matter what the trigger is for starting the interactive process. The processing unit 115 may recognize the presence of the user U at the timing. The user of the interactive device 10 is, for example, a person recognized from the image indicated by the imaging data, a person positioned in the imaging direction of the imaging unit 15 , or a person at a position corresponding to the position of the imaging unit 15 . A user of the interactive device 10 may be a person who has logged into the interactive device 10 . Further, when the processing unit 115 recognizes a predetermined voice (for example, a greeting voice) from the voice input via the voice input unit 12 or receives a predetermined operation, the processing unit 115 starts dialogue processing. may

Next, the processing unit 115 causes the imaging unit 15 to start imaging (step S2). When imaging starts, the interactive device 10 performs the processing described below.

First, the processing related to the blinking motion will be described. The blinking motion control section 111 determines whether or not to perform a blinking motion (step S11). The blinking motion control unit 111 determines to perform a blinking motion, for example, during speech of the interactive device 10 or at the timing when the speech ends. The timing at which the speech ends is, for example, the timing at which the conversation ends. The timing of the blinking action may include random timing.

If it is determined as "YES" in step S11, the blinking motion control section 111 causes the interactive device 10 to perform a blinking motion (step S12). If it is determined "NO" in step S11, the blinking motion control section 111 does not perform the blinking motion. Then, the processing of the interactive device 10 proceeds to step S3.

Next, the process for evaluating the quality of communication between the interactive device 10 and the user will be described.

The first acquisition unit 112 acquires the timing of the blinking motion of the interactive device 10 (step S21). Next, the blink detection unit 113 detects blinks of the user U based on the imaging data supplied from the imaging unit 15 (step S22). Various known techniques may be applied to the blink detection algorithm. The blink detection unit 113 extracts, for example, a plurality of feature points along the periphery of the user's U eyes from the image indicated by the imaging data. The blink detection unit 113 extracts feature points based on Haar-like, for example. The blink detection unit 113 detects whether or not the user U blinks by specifying the direction of movement of the extracted feature points and the temporal change in the speed thereof based on the imaging data of a plurality of frames. For example, due to human blinking, there is a sudden change in velocity of feature points between approximately 0 and 300 milliseconds. Therefore, the blink detection unit 113 detects that the user U has blinked when the change in speed within a predetermined period is greater than or equal to the threshold.

Next, the second acquisition unit 114 acquires blink data indicating the blink timing of the user U based on the blink detection result (step S23).

Next, the processing unit 115 performs index calculation processing (step S24). The index calculation process is a process of calculating an index of the quality of communication between the interactive device 10 and the user U. FIG.

FIG. 6 is a flowchart showing index calculation processing. The index calculation process will be described below with specific examples.

First, the processing unit 115 calculates the difference between the blinking timing of the user U and the blinking motion timing of the interactive device 10 (step S241). For example, the processing unit 115 calculates differences (hereinafter referred to as “timing differences”) for all combinations of the blink timings of the user U and the blinking motion timings of the eyelids 102 in a predetermined period. The predetermined period is, for example, 30 seconds, but may be less than 30 seconds or longer than 30 seconds. The predetermined period is, for example, the whole or a part of the period before a predetermined period of time from the timing when the speech of the dialogue device 10 ends.

FIG. 7 is a diagram explaining a method of calculating the timing difference. The timing chart shown in FIG. 7 shows the timing of blinking of the user U and the timing of the blinking motion of the interactive device 10 . As shown in FIG. 7, the blink timings of the user U are expressed as t11, t12, . . . t1B in order of time. The timings of the blinking motions of the interactive device 10 are expressed as t21, t22, . . . t27 in order of time. In this case, the processing unit 115 calculates the difference between each of t11, t12, . . . t1B and each of t21, t22, . The timing difference between the blinking timing t1i and the blinking timing t2j is hereinafter referred to as "Δtij". In this case, the processing unit 115 calculates timing differences TD {Δt11, Δt12, . . . , Δt17, Δt21, Δt22, .

FIG. 8 shows a graph DG showing the distribution of the appearance frequency of timing differences. In the graph DG of FIG. 8, the horizontal axis corresponds to the timing difference, and the vertical axis corresponds to the degree of appearance of each timing difference (that is, appearance frequency). In the example shown in FIG. 8, within a certain time range T, the appearance frequency is high.

By the way, the appearance frequency distribution shown by the graph DG is not only the communication between the user U and the dialogue device 10, but also the characteristics of the blinking of the user U (for example, the number and frequency) and the characteristics of the blinking action of the dialogue device 10 ( for example, number and frequency). For example, when the frequency of blinking of the user U and the frequency of the blinking action of the interactive device 10 are high, the frequency of occurrence of small timing differences tends to be high. Therefore, it is necessary to clarify how much the timing difference TD is caused by the communication between the interactive device 10 and the user U. Therefore, the processing unit 115 analyzes the appearance frequency distribution based on the surrogate data method.

That is, the processing unit 115 generates random data (second data) (step S242). The random data includes data obtained by randomly changing the order of blinking motion intervals of the interactive device 10 on the time axis.

FIG. 9 is a diagram showing an example of random data R1-RK. The processing unit 115 generates K types (for example, 1000 types) of random data R1 to RK. In the random data R1 to RK shown in FIG. 9, the order of the blinking intervals of the interactive device 10 is changed, and the order of the blinking intervals of the user U is not changed. Note that the timing "t2j" shown in FIG. 7 corresponds to the timing "t2ja" shown in FIG.

Next, the processing unit 115 calculates a timing difference, which is the difference between the blinking timing of the user U and the blinking motion timing of the interactive device 10, for each of the generated random data (step S243). The timing difference calculation method may be the same as in step S241. The timing difference, which is the difference between the blinking timing t1ia and the blinking timing t2ja, is hereinafter referred to as "Δtija". In the case shown in FIG. 9, the processing unit 115 calculates timing differences TR {Δt11a, Δt15a, . ·, ΔtB3a} are calculated. The appearance frequency of the timing difference in random data is indicated by graph RG in FIG. 8, for example. Graph RG indicates the average frequency of appearance of the timing differences of random data R1 to RK.

Next, processing unit 115 calculates an evaluation value corresponding to the timing difference based on blink data and the timing difference based on random data (step S244). The evaluation value is an index value that indicates the quality of communication between the interactive device 10 and the user U. FIG. Random data is data obtained by randomly changing intervals between blinking motions of the interactive device 10 . For this reason, the random data can be said to be data in which time-series information has been disrupted while maintaining the number and interval of blinking motions of the interactive device 10 . Therefore, by comparing the distribution of the timing difference TD with the distribution of the random data R1 to RK, it can be determined that the appearance distribution of the timing difference indicated by the blink data of the user U is caused by the communication between the interactive device 10 and the user U. The degree of appearance can be grasped.

The processing unit 115 calculates the evaluation value using the Z value. That is, the processing unit 115 calculates each timing difference TD {Δt11, Δt12, . . . , Δt17, Δt21, Δt22, . , the average value of the timing differences in the random data R1 to RK is subtracted, and the obtained value is divided by the standard deviation of the timing differences in the random data R1 to RK to calculate the evaluation value. For example, when the distribution of the timing difference TD is the same as the distribution of random data, the evaluation value is "0". In this case, it can be inferred that user U's blinking is not affected by communication between interactive device 10 and user U. On the other hand, when the evaluation value is large and the difference between the distribution of the timing difference TD and the distribution of the random data is large, it can be inferred that the blink of the user U is affected by the communication between the interactive device 10 and the user U. can. To explain using FIG. 8, the greater the difference Δ in appearance frequency, the greater the influence of communication, and the greater the evaluation value.

FIG. 10 is a graph showing an example of evaluation values. In the graph shown in FIG. 10, the horizontal axis corresponds to the timing difference, and the vertical axis corresponds to the evaluation value. If the timing difference is a positive value, it means that the blinking timing of the user U is behind the blinking motion timing of the interactive device 10 . If the timing difference is a negative value, it means that the user U blinks earlier than the interactive device 10 blinks. Also, the timing difference is expressed here in increments of 250 milliseconds. In FIG. 10, a timing difference of "0" milliseconds indicates that the timing difference is 0 milliseconds or more and less than 250 milliseconds. A timing difference of "+250" milliseconds and "-250" milliseconds indicates that the timing difference is greater than or equal to 250 seconds and less than 500 milliseconds. In this embodiment, the processing unit 115 determines the quality of communication between the dialogue device 10 and the user U, based on the degree to which the timing of the blinking of the user U is included within a predetermined period after the timing of the blinking action of the dialogue device 10. Calculate the evaluation value of Specifically, the processing unit 115 calculates the evaluation value corresponding to the timing difference of “+250” milliseconds as the evaluation value of the quality of communication between the interactive device 10 and the user U. FIG. That is, the processing unit 115, based on the occurrence frequency of blinking, where the blinking timing of the user U is later than the blinking operation timing of the interactive device 10 and the timing difference is 250 milliseconds or more and less than 500 milliseconds, Calculate the evaluation value. In the example shown in FIG. 10, the evaluation value is "0.4". The above is the description of the index calculation processing.

The random data may be data in which the order of the blinking intervals of the user U is changed without changing the order of the blinking motion intervals of the interactive device 10 . The random data may be data in which the order of the blinking intervals of the interactive device 10 and the order of the blinking intervals of the user U are changed. When the index calculation process ends, the process of the interactive device 10 proceeds to step S3.

Next, referring back to FIG. 5, processing related to interactive processing will be described. The processing unit 115 determines whether the evaluation value calculated in the index calculation process of step S24 is equal to or greater than the threshold (step S31). The evaluation value here is the evaluation value corresponding to the most recent period. The threshold is a value that serves as an index for determining whether the user U is interested in interacting with the interactive device 10 . The threshold is, for example, a predetermined value.

If "YES" is determined in step S31, the processing unit 115 performs first interactive processing (step S32). If "NO" is determined in step S32, the processing unit 115 performs a second dialogue process different from the first dialogue process (step S33). That is, the processing unit 115 performs different interactive processing depending on whether the evaluation value is equal to or greater than the threshold. If the evaluation value is equal to or greater than the threshold, it can be inferred that the user U has a high degree of interest in communication with the interactive device 10 . Therefore, the processing unit 115 performs, for example, the first interactive process without changing the current topic. For example, when the processing unit 115 is having a dialogue about soccer with the user U, the processing unit 115 continues to have a dialogue about soccer. At this time, the processing unit 115 performs the first dialogue process based on the input data and output data associated with the identifier "ID001" included in the dialogue data.

On the other hand, if the evaluation value is less than the threshold, it can be inferred that the user U has a low degree of interest in communication with the interactive device 10 . Therefore, the processing unit 115 performs the second dialogue process with the current topic changed. For example, when the processing unit 115 is having a dialogue with the user U about soccer, the processing unit 115 changes the dialogue to a dialogue about today's lunch. At this time, the processing unit 115 performs the second dialogue process based on the input data and output data associated with the identifier “ID002” included in the dialogue data.

As described above, the processing unit 115 performs processing according to an index value (evaluation value in the present embodiment) based on the difference between the blinking timing of the interactive device 10 and the blinking timing of the user U. However, the above-described first dialogue processing and second dialogue processing are examples, and various modifications are possible. The processing unit 115 may continue the first dialogue process instead of immediately performing the second dialogue process in which the topic of dialogue is changed when the evaluation value becomes less than the threshold value. In this case, when the period in which the evaluation value is less than the threshold continues for a predetermined period of time, or when the number of times the evaluation value is less than the threshold exceeds a predetermined number of times, the processing unit 115 performs You can change to interactive processing. Then, the processing of the interactive device 10 proceeds to step S3.

At step S3, the processing unit 115 determines whether or not to end the interactive process. For example, when the presence of the user U is no longer recognized based on the imaging data supplied from the imaging unit 15, the processing unit 115 determines to end the dialogue processing. When the processing unit 115 recognizes a predetermined voice (for example, a voice indicating a farewell greeting) from the voice input via the voice input unit 12 or receives a predetermined operation, the processing unit 115 terminates the dialogue processing. You can judge.

If it is determined to continue the dialogue processing (step S3; NO), the processing of the dialogue device 10 is returned to steps S11, S21 and S31. If it is determined to end the dialogue processing (step S3; YES), the processing unit 115 ends the dialogue processing.

According to the dialogue device 10, the degree of interest of the user U in communication with the dialogue device 10 can be quantitatively evaluated according to the difference between the timing of the blinking of the user U and the timing of the blinking action of the dialogue device 10. can. Further, the dialogue device 10 can support communication between the dialogue device 10 and the user U by reflecting this evaluation in dialogue processing. Further, the interactive device 10 can evaluate the quality of communication based on the natural action of the user U blinking. Therefore, according to the interactive device 10, the evaluation can be performed without requesting the user U to perform an action necessary for the evaluation.

Here, the reason why the difference between the blinking timing of the user U and the blinking motion of the interactive device 10 can be used to evaluate the quality of communication will be described. The inventors conducted verification by the method described below to confirm that the difference in blink timing between the speaker and the listener serves as an indicator of the degree of interest in communication between the speaker and the listener.

The speaker is a demonstrator whose business is demonstrating sales of goods. The demonstrating sales person introduced four products, namely, "dryer for women", "cosmetics for women", "watch for men", and "electric shaver for men". The introduction time for each product is about 3 minutes. The interviewees were a total of 38 university students, 18 men and 20 women. Thirty-eight listeners, after watching the video of the product introduction by the speaker, answered whether they found each product introduction interesting. The difference in blink timing between the speaker and the listener was detected from moving images of the speaker and the listener, and the listener's blink timing before and after the speaker's blink timing was analyzed.

FIG. 11 is a graph showing the frequency distribution of blinks of 38 listeners. In the graph shown in FIG. 11, the horizontal axis corresponds to time and the vertical axis corresponds to evaluation value (Z value). For time, the timing of the blink of the speaker is set to "0", the timing of the blink of the listener earlier than that is indicated by a negative value, and the timing of the listener's blink later is indicated by a positive value. That is, the horizontal axis of the graph in FIG. 11 indicates the timing difference. In FIG. 11 as well, the timing difference is expressed in increments of 250 seconds as in FIG. In this verification, in calculating the evaluation value, random data is used by a method in which the order of blink intervals of the speaker is changed and the order of blink intervals of the listener is not changed. As shown in FIG. 11, the listener's blink increased with a delay of +250 milliseconds or more and less than +500 milliseconds from the speaker's blink, and the evaluation value increased. Note that the p-value is 0.000001.

FIG. 12 is a graph showing the distribution of listener blink frequency for each answer result. In the graph shown in FIG. 12, as in FIG. 11, the horizontal axis corresponds to the time and the vertical axis corresponds to the evaluation value. The solid line graph is the evaluation value of listeners who answered that the product introduction was interesting. The dashed line graph is the evaluation value for listeners who answered that the product introduction was boring.

As shown in FIG. 12, the listeners who answered that the product introduction was interesting had a delay of +250 milliseconds or more and less than +500 milliseconds from the speaker's blinks, and their blinks increased and the evaluation value increased. On the other hand, for the listeners who answered that the product introduction was boring, such an increase in blinking and an increase in the evaluation value could not be confirmed. Note that the p-value is 0.004.

FIG. 13 is a graph showing evaluation values according to listener's gender and product. As shown in FIG. 13, regarding "cosmetics for women", female listeners showed high evaluation values, while male listeners showed low evaluation values. Regarding "watches for men" and "electric shavers for men", male listeners showed high evaluation values, while female listeners showed low evaluation values.

FIG. 14 is a graph showing the degree of interest in products according to the sex of listeners and products. The degree of interest in the product shown in FIG. 14 is "1" when each subject answers "extremely bored", "2" when "slightly bored", and "a little interesting". The level of interest is "3" when the answer is yes, and the level of interest is "4" when the answer is "very interesting". A larger value of the degree of interest means that the listener showed a higher interest in the product introduction. By comparing FIG. 13 and FIG. 14, it was confirmed that there is a correlation between the evaluation value and the degree of interest in product introduction for each product.

From the above verification, the inventors have obtained the knowledge that the difference in blink timing between the speaker and the listener correlates with the listener's degree of interest in the speaker's dialogue.

[Second embodiment]
The second embodiment controls the timing of the blinking action of the interactive device 10 based on the environment around the interactive device 10 . In the following description, elements that are the same as those of the first embodiment described above are denoted by the same reference numerals. The hardware configuration of the interactive device 10 of this embodiment may be the same as that of the first embodiment described above.

FIG. 15 is a block diagram showing the functional configuration of the interactive device 10 of this embodiment. By executing the program 141, the control unit 11 of the interactive device 10 controls the blink operation control unit 111, the first acquisition unit 112, the blink detection unit 113, the second acquisition unit 114, the processing unit 115, the environment Functions corresponding to the information acquisition unit 116 and the storage control unit 117 are realized.

The environment information acquisition unit 116 acquires environment information indicating the environment around the interactive device 10 . The environmental information here indicates the environment when the interactive device 10 is used by the user U, in other words, the environment when the interactive device 10 is interacting with the user U. FIG. Environmental information includes, for example, one or more of audio information, sound pressure information, prosody, motion information, and ambient information. The audio information includes information indicating audio input via the audio input unit 12, information indicating audio output via the audio output unit 13, or both. The sound pressure information indicates the sound pressure in a predetermined frequency band (for example, audible range) of the audio information. Prosody refers to phonetic properties that appear in speech, such as intonation. The movement information indicates body movement of the user U (for example, movement of the face, body, or facial expression). The peripheral information indicates the environment around the user U (for example, the brightness of the space where the user U is present). The audio information, the sound pressure information, and the prosody are specified based on the audio input via the audio input section 12 and the audio signal supplied to the audio output section 13 . Motion information and peripheral information are specified using the imaging unit 15 . Environmental information may be obtained using another measurement device. Moreover, the environment information may further include the assumed age, gender, occupation, and information about the user U of the interactive device 10 .

The storage control unit 117 causes the storage unit 14 to accumulate, as learning data 143, data indicating the association between the timing at which the user U blinks and the environment in which the interactive device 10 is used. That is, the learning data 143 is data indicating the result of learning the relationship between the surrounding environment of the interactive device 10 and the timing at which the user U actually blinks. Note that the storage control unit 117 may store the learning data 143 in a storage device other than the storage unit 14, such as a storage device related to the cloud storage service.

The blinking motion control unit 111 causes the interactive device 10 to perform a blinking motion at a timing (first timing) according to the environment information acquired by the environment information acquiring unit 116 . Specifically, blinking control unit 111 causes interactive device 10 to blink at timing corresponding to learning data 143 stored in storage unit 14 and environment information acquired by environment information acquiring unit 116 . The blinking motion control unit 111 controls, for example, the timing at which the interactive device 10 performs the blinking motion, the frequency of the blinking motion, and both of these.

Next, the operation of the interactive device 10 will be described. FIG. 16 is a flow chart showing the learning process executed by the interactive device 10. As shown in FIG. Learning processing is performed in parallel with interactive processing.

The environment information acquisition unit 116 acquires environment information (step S41). Next, the second acquisition unit 114 acquires the blink timing of the user U (step S42). Next, the storage control unit 117 causes the storage unit 14 to accumulate data indicating the environment indicated by the environment information acquired in step S41 and the blink timing acquired in step S42 as learning data 143 (step S43). The above is the description of the learning process.

FIG. 17 is a flow chart showing a process related to a blinking action executed by the interactive device 10. As shown in FIG. The processing of FIG. 17 is executed in place of the processing of steps S11 and S12 described with reference to FIG.

The environment information acquisition unit 116 acquires environment information (step S51). Next, the blinking motion control section 111 determines whether or not to perform a blinking motion (step S52). Here, based on the environmental information acquired in step S51 and the learning data 143, the blinking motion control unit 111 determines whether or not to perform the blinking motion. The motion control unit 111 determines the presence or absence of the blinking motion and the timing of the blinking motion by machine learning, for example. The machine learning algorithm is, for example, a neural network, but other algorithms may be used. Until a predetermined amount of learning data 143 is accumulated, the blinking motion control section 111 may determine whether to perform the blinking motion in the same manner as in step S11 of the first embodiment described above.

When the blinking motion control unit 111 determines that the blinking motion is to be performed (step S52; YES), the blinking motion control unit 111 causes the interactive device 10 to perform the blinking motion (step S53). When the blinking motion control unit 111 determines not to perform the blinking motion (step S52; NO), the interactive device 10 does not perform the blinking motion. The blink processing unit 115 learns the relationship between the surrounding environment of the interactive device 10 and the blinking performed by humans in that environment, and causes the interactive device 10 to perform the blinking action according to the relationship. For example, the blinking motion control unit 111 may increase the frequency of the blinking motion when determining that the topic has become highly unexpected. The above is the description of the processing related to the blinking action.

According to the interactive device 10 of the present embodiment, in addition to the effects equivalent to those of the above-described first embodiment, learning data 143 can be used to perform a blinking motion at a more natural timing. As a result, an improvement in the quality of communication between the interactive device 10 and the user U can be expected.

The blinking motion control unit 111 may cause the interactive device 10 to perform the blinking motion at another timing (second timing) in addition to the timing according to the environment information. For example, at random timing, the blinking motion control unit 111 causes the interactive device 10 to blink a predetermined number of times (for example, 20 times per minute) within a predetermined period of time. The blinking motion control unit 111 may cause the interactive device 10 to perform a blinking motion at timing according to a predetermined rule. As a result, the interactive device 10 can be expected to have the effect of performing the blinking motion at a more natural timing.

The storage control unit 117 may accumulate the learning data 143 during the period when the evaluation value is equal to or greater than the threshold. As a result, the interactive device 10 can perform a blinking action in accordance with the blinking performed by a person during high-quality communication.

Note that if the learning data 143 is stored in the storage unit 14 in advance, the interactive device 10 does not have to have the function of accumulating the learning data (that is, the storage control unit 117).

[Third Embodiment]
The third embodiment differs from the first embodiment described above in that the dialogue device functions as a display device. In the following description, elements that are the same as the elements of the first embodiment described above are denoted by the same reference numerals, and elements that correspond to the elements of the first embodiment described above are denoted by the same reference numerals with an "A" at the end. to represent

FIG. 18 is a diagram showing an example of the external configuration of a dialogue device 10A that is the third embodiment of the present invention. The interactive device 10A has a display area 161. FIG. A display area 161 is an area where an image is displayed. A display area 161 displays the object 20 . The object 20 is an image similar to the appearance of the interactive device 10 described in the first embodiment. The object 20 has a face 201 and eyelids 202 . The facial part 201 is a part corresponding to the face. The eyelid part 202 is a part that is arranged on the face part 201 and corresponds to the eyelid. The eyelid part 202 performs a blinking action by opening and closing. The blinking action of this embodiment is different from that of the above-described first embodiment in that an image is displayed on the display area 161 . Note that the object 20 shown in FIG. 18 is merely an example, and may include at least an image representing a blinking motion. For example, object 20 includes at least one eyelid. A display section may be provided in place of the eyelid section 102 of the interactive device 10 having the appearance shown in FIG. 1, and an object corresponding to the eyelid section 202 may be displayed on the display section.

A user U is a person who communicates with the interactive device 10A. The user U faces the interactive device 10A, observes the object 20 displayed in the display area 161, and interacts with it.

FIG. 19 is a block diagram showing the hardware configuration of the interactive device 10A. The dialogue device 10 has a control unit 11 , an audio input unit 12 , an audio output unit 13 , a storage unit 14 , an imaging unit 15 and a display unit 16 . The storage unit 14 stores a program 141A for causing the control unit 11 to perform a predetermined function. The display unit 16 has a display area 161 that displays an image. The display unit 16 is, for example, a liquid crystal display, an organic EL display, or another display device.

FIG. 20 is a block diagram showing the functional configuration of the interactive device 10A. By executing the program 141A, the control unit 11 of the interactive device 10 corresponds to the blink operation control unit 111A, the first acquisition unit 112, the blink detection unit 113, the second acquisition unit 114, and the processing unit 115. Realize the function to The blinking motion control unit 111A causes the object 20 displayed in the display area 161 of the display unit 16 to perform a blinking motion. The blinking control unit 111A supplies, for example, blinking control data for blinking to the display unit 16 . The blink control data is data for controlling display on the display unit 16 . The display unit 16 causes the object 20 to blink in accordance with the blink control data. The first acquisition unit 112 acquires the timing of the blinking motion of the object 20 (eyelid part 202).

The operation of the interactive device 10A is the same as that of the first embodiment described above, except that the blinking operation is performed under the control of the display unit 16. FIG.

The configuration of this embodiment can also be applied to the interactive device 10 of the second embodiment described above.

[Modification]
The present invention is not limited to the above embodiments, and can be modified as appropriate without departing from the scope of the invention. A modification of the dialogue device 10 of the first embodiment will be described below, but the dialogue device 10 of the second embodiment and the dialogue device 10A of the third embodiment are also applicable.

The calculation method of the evaluation value of the processing unit 115 of the embodiment described above is merely an example. For example, the processing unit 115 counts the number of blinks of the user U and the number of blinking actions of the interactive device 10 in time series, and calculates an evaluation value based on the number of blinks and blinking actions in a specific period (scene). You may In this case, the processing unit 115 calculates an evaluation value indicating that the degree of interest of the user U in communication with the interactive device 10 is higher in a period in which the number of times of blinking and blinking is greater. This is because the timing difference between the timing of blinking of the user U and the timing of the blinking motion of the interactive device 10 is considered to be smaller than in other periods during periods when there are many blinks and blinking motions. Alternatively, the processing unit 115 may perform processing according to the difference between the timing of the blinking motion and the timing of the user's blinking without calculating the evaluation value.

The process according to the difference between the timing of blinking of the user U and the timing of the interactive device 10 is not limited to speech processing. The processing unit 115 may perform processing for evaluating the interactive device 10, for example. In this case, the processing unit 115 outputs evaluation data in association with the identifier of the interactive device 10 . The evaluation data indicates the evaluation of the interactive device 10. FIG. The evaluation data may be data indicating an evaluation value, or may be data generated using the evaluation value. Output of the evaluation data is performed, for example, by transmission, printing, display, or other methods. According to this modification, the quality of communication performed by the interactive device 10 can be evaluated.

Blink detection may be performed by a method other than the method using imaging data. As the method, there is a method using a non-contact sensor such as a radio wave sensor (for example, a 4 GHz radio wave sensor module), an infrared sensor, or a Doppler sensor. There is also a method using a sensor that detects a blink based on the movement of facial muscles.

Some or all of the functions realized by the control unit 11 described in the above embodiment may be provided by a processing device external to the interactive device. In this case, the processing device, for example, controls the dialogue device by communication (for example, communication via a public communication line). This control may include one or both of blinking control and interactive processing control. The processing device may control multiple interactive devices. In short, as shown in FIG. 21 , the processing device according to the present disclosure includes a first acquisition unit 301 that acquires the timing of the blinking action of the dialogue device, and a second acquisition unit that acquires the timing of the blinking of the user of the dialogue device. 302, and a processing unit 303 that performs processing according to the difference between the timing of the blinking motion and the timing of the blinking of the user.

Some of the configurations and operations of the above-described embodiments may be omitted. Configurations and operations that were described in the above embodiments may be added. Also, the execution order of the processes described in the above embodiment is merely an example, and may be changed as appropriate.

The functions realized by the control unit 11 can be realized by combining a plurality of programs or by linking a plurality of hardware resources. When the functions of the control unit 11 are realized using a program, the programs 141 and 141A for realizing this function are stored in computer-readable media such as various magnetic recording media, optical recording media, magneto-optical recording media, and semiconductor memories. may be provided in a state stored in a suitable recording medium. Also, this program may be distributed via a network. The present invention can also be grasped as a processing method.

10, 10A: interactive device, 11: control unit, 12: voice input unit, 13: voice output unit, 14: storage unit, 15: imaging unit, 16: display unit, 20: object, 101: face part, 102: Eyelid part 111, 111A: blinking operation control unit 112: first acquisition unit 113: blink detection unit 114: second acquisition unit 115: processing unit 116: environment information acquisition unit 117: storage control unit 141, 141A: Program, 142: Dialogue Data, 143: Learning Data, 161: Display Area, 201: Face, 202: Eyelid, 301: Blink Operation Control Unit, 302: Acquisition Unit, 303: Processing Unit

Claims (6)

an environment information acquisition unit that acquires environment information indicating the environment around the interactive device;
a blinking motion control unit that controls the frequency of blinking motions performed by the interactive device based on the environment information;
The blinking motion control unit learns the relationship between the surrounding environment of the interactive device and the blinking performed by a human under the environment, and causes the interactive device to perform the blinking motion based on the relationship.
processing equipment. The environment information is dialogue information from a user to the dialogue device,
2. The processing apparatus of claim 1 . The blinking motion control unit increases the frequency of blinking motions performed by the interactive device when the environmental information satisfies a predetermined condition.
3. A processing apparatus according to claim 2 . The blinking motion control unit increases the frequency of the blinking motion when determining that the dialogue information has become a highly unexpected topic based on the learning result.
4. A processing apparatus according to claim 3 . A computer-implemented processing method comprising:
Acquiring environmental information indicating the surrounding environment of the interactive device,
controlling the frequency of blinking actions performed by the interactive device based on the environmental information;
learning the relationship between the surrounding environment of the dialogue device and the blinking performed by a human under the environment, and causing the dialogue device to perform a blinking action based on the relationship;
Processing method. to the computer,
Acquiring environmental information indicating the surrounding environment of the interactive device,
controlling the frequency of blinking actions performed by the interactive device based on the environmental information;
A program for learning the relationship between the surrounding environment of the dialogue device and the blinking performed by a person under the environment, and causing the dialogue device to perform the blinking motion based on the relationship.

Images