JP7323098B2 - Dialogue support device, dialogue support system, and dialogue support program - Google Patents

Description

The present invention relates to a dialogue support device, a dialogue support system, and a dialogue support program.

In daily conversations, it is not uncommon for people to not be able to grasp the other person's emotions or to convey their intentions accurately. In particular, in the case of teleconferencing systems, remote conversations using Skype (registered trademark), etc., the transmitted information is degraded video and audio, and the problem becomes even more pronounced.

On the other hand, in a situation such as a discussion with a plurality of people, it is difficult to appropriately control the dialogue while overlooking the whole. For example, there are controls to ensure that statements are not biased, that people with opinions are given proper opportunities to speak, and that discussions are not overly heated.

As a technology that can be applied to solve such problems, Patent Document 1 discloses facial expression analysis means for analyzing a facial expression from a moving image of a person, and a simple image of a person's face from the moving image of a person. simplified image creation means for creating and storing in a storage device, simplified image selection means for selecting a corresponding simplified image from the simplified images stored in the storage device according to the analysis result of the facial expression, and for the selected simplified image and a special effect processing means for applying a special effect according to the facial expression analyzed by the facial expression analysis means.

Further, Patent Document 2 discloses a network system in which first and second devices are connected, in which acquiring means for acquiring user information and image data in the first device, and An extraction means for extracting a face area, an identification means for identifying the facial expression of the face area extracted by the extraction means, and for each facial expression, expression identification information indicating the facial expression and display information are stored in association with each other. A storage means, an identification means for identifying display information stored in association with expression identification information corresponding to the facial expression identified by the identification means from among the display information, and the identified display information and the acquisition means for obtaining the display information. A technique is disclosed that includes display means for displaying on a display unit of a second device in association with the user information obtained.

Japanese Patent Application Laid-Open No. 2004-64102 JP 2015-165407 A

However, the techniques described in Patent Literature 1 and Patent Literature 2 do not consider the overall situation of the dialogue participants in the dialogue, so the dialogue cannot necessarily be effectively activated. It wasn't.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a dialogue support device, a dialogue support system, and a dialogue support program capable of effectively activating dialogue.

According to claim 1, the dialogue support apparatus according to the present invention comprises an acquisition unit that acquires a physical quantity from which a situation in the dialogue of a dialogue participant can be derived, and using the physical quantity acquired by the acquisition unit, the performing at least one of a derivation unit for deriving a situation in the dialogue of the participant, a display process for displaying situation information corresponding to the situation derived by the derivation unit, and a storage process for storing the situation information a processing unit, wherein the situation information is information representing an emotion of the participant; the information representing the emotion is image information representing the emotion; This is the photographed face image information obtained by photographing the face of the corresponding participant in the case of .

According to the dialogue support device according to the present invention, the dialogue can be effectively activated by performing at least one of displaying and storing the situation information corresponding to the situation in the dialogue. .

In addition, according to the dialogue support device of the present invention, the situation information is information representing the emotions of the participants, so that activation of the dialogue can be promoted more effectively.

In addition, according to the dialogue support device of the present invention, the emotion-expressing image information is used as the emotion-expressing image information, so that the participant's emotion can be grasped more intuitively.

Further, according to the dialogue support device of the present invention as set forth in claim 1 , the image information is a photographed face image obtained by photographing the face of the corresponding participant when the degree of emotion is maximized. By using information, it is possible to grasp the emotions of the participants more effectively.

According to claim 2 , the dialogue support apparatus according to the present invention is the dialogue support apparatus according to claim 1 , wherein the image information includes information exaggerating the emotion in addition to the photographed face image information. image information that can be

According to the dialogue support device according to the present invention described in claim 2 , the image information is image information that includes information that exaggerates emotions in addition to face-shot image information. can understand the emotions of

According to claim 3 , the dialogue support apparatus according to the present invention is the dialogue support apparatus according to claim 1 , wherein the physical quantity is an image obtained by photographing the participant and an image obtained by photographing the participant. At least one of the voices indicating

According to the dialogue support device according to the present invention described in claim 3 , the physical quantity is at least one of an image obtained by photographing the participant and a voice indicating the speech of the participant, so that the special device Activation of dialogue can be promoted without using

According to claim 4 , the dialogue support apparatus according to the present invention is the dialogue support apparatus according to claim 3 , wherein the situation includes a physical quantity representing the degree of emotion of the participant and a motion of the participant. At least one of the physical quantities represented.

According to the dialogue support device according to the present invention as recited in claim 4 , the situation is at least one of a physical quantity representing the degree of emotion of the participant and a physical quantity representing the action of the participant. It can encourage activation of dialogue.

The dialogue support device according to the present invention described in claim 5 is the dialogue support device according to claim 4 , wherein the situation is a physical quantity indicating the frequency of nodding of the participant obtained from the image, and a physical quantity indicating the degree of speech of the participant obtained from at least one of the image and the voice.

According to the dialogue support device according to the present invention as recited in claim 5 , the situation is represented by a physical quantity indicating the frequency of nodding of the participant obtained from the image, a physical quantity indicating the degree of expression of the participant obtained from the image, the image and the By using at least one of the physical quantity obtained from at least one of the voices and indicating the degree of speech of the participant, activation of the dialogue can be prompted more easily.
According to claim 6, there is provided the dialogue support apparatus according to claim 5, wherein the degree of utterance is a degree corresponding to the utterance speed of the most recent predetermined period. It is.
According to claim 7, the dialogue support apparatus according to the present invention is the dialogue support apparatus according to claim 5, wherein the physical quantity indicating frequency of nodding is information indicating a decrease rate of the frequency. .
According to an eighth aspect of the present invention, there is provided a dialogue support device according to claim 1, which learns the information representing the emotion.
According to a ninth aspect of the present invention, there is provided a dialogue support device according to the first aspect, wherein the situation information is information representing emotions of all participants in the dialogue.

According to claim 10 , there is provided a dialogue support system according to the present invention, comprising: the dialogue support device according to any one of claims 1 to 9 ; a terminal provided with a transmission unit that transmits possible physical quantities, and a display unit that is a display target of the display processing when the processing unit of the dialogue support device performs the display processing.

According to the dialogue support system according to the tenth aspect of the present invention, the dialogue can be effectively activated by performing at least one of the processing of displaying and storing the situation information corresponding to the situation in the dialogue. .

According to claim 11 , the dialogue support program acquires a physical quantity from which a situation in the dialogue of a participant in a dialogue can be derived, and uses the acquired physical quantity to estimate the situation in the dialogue of the participant. A dialog support program for executing at least one of a display process for deriving and displaying situation information corresponding to the derived situation and a storage process for storing the situation information, wherein the situation is The information is information representing the emotion of the participant, the information representing the emotion is image information representing the emotion, and the image information is the corresponding image of the participant when the degree of the emotion is maximum. This is face photographed image information obtained by photographing the face of the participant .

According to the dialogue support program according to the eleventh aspect of the present invention, the dialogue can be effectively activated by at least one of displaying and storing the situation information corresponding to the situation in the dialogue. .

As described above, according to the present invention, dialogue can be effectively activated.

It is a block diagram showing an example of hardware constitutions of a dialogue support system concerning an embodiment. It is a block diagram showing an example of functional composition of a dialogue support system concerning an embodiment. 5 is a time chart for explaining the degree of speech according to the embodiment; FIG. 3 is a schematic diagram showing an example of the configuration of a situation correspondence information database according to the embodiment; It is a schematic diagram with which it uses for description of the learning method of the correspondence information which concerns on embodiment. 4 is a schematic diagram showing an example of the configuration of a dialogue information database according to the embodiment; FIG. FIG. 3 is a schematic diagram showing an example of the configuration of an emoticon/exaggeration information database according to the embodiment; 6 is a flowchart illustrating an example of dialogue support processing according to the embodiment; 4 is a front view showing an example of the configuration of a dialogue support image according to the embodiment; FIG. 9 is a flowchart showing an example of faction information display processing according to the embodiment; FIG. 5 is a front view showing an example of the configuration of a faction display image according to the embodiment; 7 is a time chart for explaining another method of determining correspondence information according to the embodiment;

Embodiments for carrying out the present invention will be described in detail below with reference to the drawings. In this embodiment, the present invention is implemented by a dialogue support device that comprehensively supports dialogue (speech in a conference) when a conference is held by a plurality of people, and a plurality of terminals that are individually used by each participant in the dialogue. A case of application to a dialogue support system including . Also, in the present embodiment, a case will be described in which the participants in the dialogue are dispersed in different remote locations.

First, the configuration of a dialogue support system 90 according to the present embodiment will be described with reference to FIGS. 1 and 2. FIG. As shown in FIG. 1 , a dialogue support system 90 according to this embodiment includes a dialogue support device 10 and a plurality of terminals 20 each of which can access a network 80 . Examples of the dialogue support device 10 include information processing devices such as personal computers and server computers. Examples of the terminal 20 include stationary and notebook personal computers, and portable terminals such as smartphones and tablet terminals.

The terminals 20 according to the present embodiment are terminals assigned to participants of dialogue in a conference using the dialogue support system 90 (hereinafter simply referred to as “participants”). The terminal 20 includes a CPU (Central Processing Unit) 21, a memory 22 as a temporary storage area, a nonvolatile storage unit 23, an input unit 24 such as a touch panel, a display unit 25 such as a liquid crystal display, and a medium reading/writing device (R/W). 26. The terminal 20 also includes a camera 28 , a microphone 29 and a wireless communication section 27 . The CPU 21, memory 22, storage section 23, input section 24, display section 25, medium read/write device 26, camera 28, microphone 29 and wireless communication section 27 are connected to each other via a bus B1. The medium read/write device 26 reads information written in the recording medium 96 and writes information to the recording medium 96 .

The storage unit 23 is implemented by a HDD (Hard Disk Drive), SSD (Solid State Drive), flash memory, or the like. In the dialogue support system 90 according to the present embodiment, the face of the participant using the terminal 20 fits within the angle of view of the camera 28 of each terminal 20 and the sound collection range of the microphone 29 of each terminal 20 Each terminal 20 is positioned to receive the speech of the participant using 20 .

On the other hand, the dialogue support device 10 is a device that comprehensively stores and manages various kinds of information handled by the dialogue support system 90 . The dialogue support device 10 includes a CPU 11, a memory 12 as a temporary storage area, a non-volatile storage section 13, an input section 14 such as a keyboard and a mouse, a display section 15 such as a liquid crystal display, a medium read/write device 16, and a communication interface (I/ F) a section 18; The CPU 11, memory 12, storage section 13, input section 14, display section 15, medium read/write device 16, and communication I/F section 18 are connected to each other via a bus B2. The medium read/write device 16 reads information written in the recording medium 17 and writes information to the recording medium 17 .

The storage unit 13 is implemented by an HDD, SSD, flash memory, or the like. A dialogue support program 13A is stored in the storage unit 13 as a storage medium. The recording medium 17 in which the dialogue support program 13A is written is set in the medium reading/writing device 16, and the medium reading/writing device 16 reads out the dialogue support program 13A from the recording medium 17. stored to The CPU 11 reads out the dialog support program 13A from the storage unit 13, develops it in the memory 12, and sequentially executes the processes of the dialog support program 13A.

The storage unit 13 also stores a situation correspondence information database 13B, a dialogue information database 13C, and an emoticon/exaggeration information database 13D. The situation correspondence information database 13B, the dialogue information database 13C, and the emoticon/exaggeration information database 13D will be described later in detail.

Next, functional configurations of the dialogue support device 10 and the terminal 20 according to the present embodiment will be described with reference to FIG. As shown in FIG. 2, the dialogue support device 10 includes an acquisition unit 11A, a derivation unit 11B, and a processing unit 11C. By executing the dialogue support program 13A, the CPU 11 of the dialogue support device 10 functions as an acquisition unit 11A, a derivation unit 11B, and a processing unit 11C.

The acquisition unit 11A according to the present embodiment acquires physical quantities from which situations in the dialogue of the participants can be derived. In the acquisition unit 11A according to the present embodiment, as the physical quantities, an image obtained by photographing the participant (hereinafter referred to as "captured image") and a voice indicating the participant's utterance (hereinafter referred to as "utterance voice") ) are applied, but the present invention is not limited to this. For example, only one of the captured image and the spoken voice may be applied as the physical quantity.

Further, the deriving unit 11B uses the physical quantity acquired by the acquiring unit 11A to derive the situation in the dialogue of the participants. In the derivation unit 11B according to the present embodiment, the physical quantity representing the degree of emotion of the participant (hereinafter referred to as "emotion level") and the physical quantity representing the movement of the participant (hereinafter referred to as "motion amount") ) are derived for each participant. More specifically, the derivation unit 11B according to the present embodiment uses the number of times of nodding of the receiver per predetermined period (10 seconds in the present embodiment) obtained from the captured image (hereinafter referred to as “nodding N, and a physical quantity (hereinafter referred to as "utterance level") H indicating the degree of utterance of the speaker obtained from the utterance voice are derived. The derivation unit 11B also derives a physical quantity (hereinafter referred to as "expression level") indicating the degree of expression of the participant obtained from the photographed image as the level of emotion.

More specifically, the deriving unit 11B according to the present embodiment uses, as the expression degrees, an anger level I indicating the degree of anger of the corresponding participant, a disgust level K indicating the degree of disgust of the corresponding participant, and a response A fear degree O, which indicates the degree of fear of the participants, is derived. Further, the derivation unit 11B according to the present embodiment uses, as the expression levels, a joy level Y indicating the degree of joy of the corresponding participant, a sadness level S indicating the degree of sadness of the corresponding participant, and a A degree of surprise B indicating the degree of surprise of is derived.

In addition, in this embodiment, these six types of emotion levels are based on images captured by the camera 28 of the terminal 20 used by the corresponding participant, for example, "Real-time face detection and emotion/gender classification ”, the Internet <URL: https://github.com/oarriaga/face_classification> or the like is derived by applying the known technology described.

In this technology, based on the neural network library Keras, facial features are extracted by a CNN (Convolutional Neural Network) to recognize each emotion. For example, in the case of a smile (with a degree of joy Y), a database of smile features is prepared. judge. In this embodiment, this degree of similarity is applied as the degree of emotion. Further, in this embodiment, the six types of emotion levels are derived as normalized values so that they all have a common range (a range from 0 to 100 in this embodiment).

The derivation of each emotion level is also possible by applying many known technologies such as the Emotion API (Application Programming Interface), which is a service provided by Microsoft's Azure (registered trademark). Therefore, further description is omitted here.

As described above, in the present embodiment, the above six types of emotion levels are applied. good too.

On the other hand, the derivation unit 11B according to the present embodiment calculates the speech degree H using the following equation (1). s(t) in Equation (1) represents the utterance speed (=the number of uttered characters/second) of the target speaker.

That is, the formula (1) calculates the speech rate H as the value obtained by integrating the speech speed s(t) for the most recent 10 minutes (600 seconds), with the weight value increased for speech closer to the time of calculation. . In this embodiment, as the number of uttered characters used when deriving the utterance speed s(t) of the speaker, the corresponding uttered voice is recognized by a known speech recognition technology, and the number of characters of the text information obtained by this is applied. However, it is not limited to this.

For example, as shown in FIG. 3 as an example, each participant's utterances in a conference are usually grouped together between other utterances. It is assumed that the speed s(t) is more important. As a result, the utterance level H can be calculated as a more accurate representation of the level of understanding of the topic during the conversation of the corresponding utterer.

Note that the formula for calculating the speech degree H is not limited to formula (1). For example, the most recent 10 minutes applied in formula (1) is an example, and it goes without saying that other periods may be used. In addition, in Equation (1), the closer the utterance to the calculation time is, the larger the weight value is. However, the utterance level H may be calculated without weighting. Also, in the present embodiment, a case has been described in which a mathematical formula is used to derive the speech degree H, but the present invention is not limited to this form, and a form in which the speech degree H is derived by table conversion, for example, is also possible. Furthermore, in the present embodiment, the utterance degree H is derived using the utterance voice, but the present invention is not limited to this. For example, using a photographed image, the utterance speed s(t) is derived from the movement of each participant's mouth, and the utterance rate H is calculated by substituting this utterance speed s(t) into Equation (1). etc.

Further, the derivation unit 11B according to the present embodiment calculates the nodding frequency N using the following equation (2). n(t) in equation (2) represents the number of nods of the target listener per predetermined time (=the number of nods/second).

That is, Equation (2) calculates the nodding frequency n(t) by multiplying the number of times n(t) of nodding per predetermined period of time over the most recent 10 minutes (600 seconds), with the weight value increased for the nodding closer to the calculation time. Calculate as N. In this embodiment, when the face image of the receiver included in the photographed image is in front of the receiver and tilts downward, and then returns to the original position, it is considered that the receiver has nodded once. Deciding. Note that in this embodiment, the detection of the tilt and return of the face image is performed when the image of a predetermined portion of the face image (eye image in this embodiment) moves downward by a predetermined distance (3 mm in this embodiment) or more. Although this is done by detecting the return to the original position later, it goes without saying that this is not the only option.

Note that the formula for calculating the nodding frequency N is not limited to formula (2). For example, the most recent 10 minutes applied in equation (2) is an example, and it goes without saying that other periods may be used. In addition, in equation (2), the weight value increases as the number of times n(t) of nodding is closer to the time of calculation, but the nodding frequency N may be calculated without this weighting. Further, in the present embodiment, the case where a formula is used to derive the nodding frequency N has been described, but the present invention is not limited to this form, and the nodding frequency N may be derived by table conversion, for example.

Then, the processing unit 11C performs both display processing of displaying the situation information corresponding to the situation derived by the deriving unit 11B on the display unit 25 of the terminal 20 and storage processing of storing the situation information in the storage unit 13. process. However, the present invention is not limited to this form, and may be a form in which either one of the display process and the storage process is performed.

In this embodiment, information representing the emotion of the corresponding participant is applied as the situation information. More specifically, in this embodiment, text information and image information representing the emotion, and information representing the emotional relationship between the participants are applied as the information representing the emotion.

In the present embodiment, as the text information, information representing the feelings of others with respect to statements made by one of the corresponding participants is applied. Further, in this embodiment, emoticons are applied as the image information. Further, in the present embodiment, as the image information, an image including face photographed image information obtained by photographing the face of the corresponding participant when the degree of emotion is maximized and information exaggerating the emotion. information is also applied.

On the other hand, the terminal 20 according to this embodiment includes a control section 21A. CPU 21 of terminal 20 functions as control section 21A by executing a dialogue support application program (not shown) pre-stored in storage section 23 .

The control unit 21A according to the present embodiment transmits the physical quantity from which the situation in the dialogue can be derived to the acquisition unit 11A of the dialogue support device 10 via the wireless communication unit 27 as a transmission unit. Further, when the processing unit 11C of the dialogue support device 10 performs the display processing, the control unit 21A controls the display unit 25 to be displayed in the display processing.

Next, the situation correspondence information database 13B according to this embodiment will be described with reference to FIG. As shown in FIG. 4, the situation correspondence information database 13B according to the present embodiment includes information indicating a situation and a text displayed in association with the participant as indicating the emotion of the corresponding participant in the corresponding situation. Correspondence information, which is information, is stored in association therewith. The corresponding information corresponds to the text information representing emotion of the present invention.

Here, as shown in FIG. 4, the information indicating the situation includes, in addition to speech level H by the speaker and six types of emotion levels by the listener, a predetermined time period of nodding frequency N by the listener (this embodiment). In this case, the nodding frequency decrease rate U, which indicates the decrease rate from 60 seconds ago, is included.

Further, the correspondence information corresponding to the corresponding situation is derived as follows as an example. That is, first, as shown in FIG. 5 as an example, a large number of "IF (situation) THEN (correspondence information)" assumed in the meeting place are prepared in advance as hypotheses. In the example shown in FIG. 5, the utterance level H of the speaker is 180 or more, the nodding frequency reduction rate U of the listener is 50% or more, and the anger level I of the listener is 50 or more. In the case of , "I can't keep up with what you're saying" is hypothesized as corresponding information indicating the listener's emotion.

Then, in the present embodiment, a combination of a large number of situations and response information prepared as hypotheses is applied in an actual meeting, and response information that satisfies the conditions of the situation is displayed on the terminal 20, and the display is effective. It learns by repeatedly evaluating whether or not there was. Then, the correspondence information obtained by this learning is reflected in the situation correspondence information database 13B. The evaluation performed here may be a subjective evaluation by the receiver, or may be an objective evaluation such as an actual improvement effect after the response information is displayed (for example, more smiles, more remarks, etc.).

As described above, in the present embodiment, the correspondence information corresponding to the situation is learned, but it is not always necessary to learn. It is good also as a form to apply.

Next, the dialogue information database 13C according to this embodiment will be described with reference to FIG. As shown in FIG. 6, the dialogue information database 13C according to the present embodiment stores information such as a terminal ID (IDentification), image data, voice data, text data, time, and maximum emotional level in association with each other.

The terminal ID is information assigned to identify the terminal 20 used by each participant. Note that in the present embodiment, the terminal ID and the information (name in the present embodiment) indicating the participant using the terminal 20 to which the terminal ID is assigned are associated and stored in the storage unit 13 . Therefore, the dialogue support device 10 can identify the participant using the terminal 20 when communicating with any terminal 20 .

Further, the image data is information indicating a photographed image acquired from the corresponding terminal 20, the audio data is information indicating the utterance voice acquired from the corresponding terminal 20, and the text data is information representing the corresponding It is information that converts the utterance voice to text. In this embodiment, the text data is obtained by converting the corresponding voice data into text data using a known voice recognition technique.

Further, the time is information indicating the date and time when the corresponding image data and audio data were acquired, and the maximum emotional level is the maximum value of the corresponding participant within the period during which the corresponding image data is obtained. This is information indicating the type of emotion level to be.

In this embodiment, as shown in FIG. 6, the terminal ID of the terminal 20 used by the participant is used as the information indicating each participant at the maximum emotional level. stomach. Also, in FIG. 6, the types of the maximum emotional level are represented only by symbols, but for example, 'I' represents the anger level I, and 'O' represents the fear level O. As shown in FIG. Furthermore, although FIG. 6 illustrates the case where the maximum emotional level is stored in the dialogue information database 13C only for the speaker, the maximum emotional level of the listener during the corresponding period is also stored in the dialogue information database 13C. 13C may be used.

Next, the emoticon/exaggeration information database 13D according to the present embodiment will be described with reference to FIG. As shown in FIG. 7, the emoticon/exaggerated information database 13D according to the present embodiment stores each piece of information of maximum emotional level, emoticon, and exaggerated information in association with each other.

The maximum emotion level is the same information as the maximum emotion level of the dialogue information database 13C, the emoticon is data indicating the emoticon corresponding to the corresponding maximum emotion level, and the exaggerated information is the corresponding This is information indicating the content of the exaggeration corresponding to the maximum degree of emotion.

For example, in the emoticon/exaggerated information database 13D shown in FIG. 7, when the type of emotion level that is the maximum emotion level is the fear level O, the corresponding emoticon is "(^_^;)". ing. Further, in the example shown in FIG. 7, the content of the exaggeration indicated by the exaggeration information corresponding to the case where the type of emotion level that is the maximum emotion level is the fear level O is the text information "Gang" and the image showing fear. It shows that The image showing the fear is, for example, an image 25G in which a plurality of vertical lines are superimposed near the forehead of the face image of the corresponding participant shown in FIG. 9, which will be described later.

Next, the operation of the dialogue support system 90 according to this embodiment will be described with reference to FIGS. 8 to 11. FIG. First, with reference to FIGS. 8 and 9, the operation of the dialogue support device 10 when executing dialogue support processing will be described. When the terminal 20 used by each participant in the conference starts executing the dialogue support application program, the CPU 11 of the dialogue support device 10 executes the dialogue support program 13A, whereby the dialogue shown in FIG. Support processing is performed. Here, in order to avoid complication, a case will be described in which dialogues by a plurality of participants proceed without overlapping in terms of time. Also, in order to avoid complication, a case where the situation correspondence information database 13B and the emoticon/exaggeration information database 13D have already been constructed will be described here.

When the execution of the dialogue support application program is started, the terminal 20 used by each participant starts photographing with its own camera 28 and operating the microphone 29, and the image data showing the photographed image obtained by this and the speech are displayed. Transmission of voice data representing voice to the dialogue support device 10 is started.

Therefore, in step 200 of FIG. 8, the acquisition unit 11A starts receiving image data and audio data transmitted from each terminal 20 and storing each received data in the storage unit 13 . When storing each piece of received data in the storage unit 13, the acquisition unit 11A associates and stores the terminal ID assigned to the terminal 20 that transmitted the corresponding data and the time at which the data was acquired.

At step 202, the acquiring unit 11A waits until the speech data of the speech data received from each terminal 20 is interrupted for a predetermined period (five seconds in this embodiment), thereby obtaining a series of speeches of each participant in the dialogue. (hereinafter referred to as "a series of statements") is completed.

At step 204 , the derivation unit 11</b>B reads from the storage unit 13 the image data and voice data for the most recent series of utterances of each participant. At step 206, derivation unit 11B converts the read speech data into text data for each participant using a known speech recognition technique.

In step 208, the derivation unit 11B uses the read image data to determine the six types of emotion levels for each participant (in this embodiment, the anger level I, the disgust level K, the fear level O, the joy level Y, and the sadness level). The degree S and the degree of surprise B) are derived as described above. In the present embodiment, the maximum value of the read image data for the most recent series of utterances is derived for each participant and for each emotion level. However, the present invention is not limited to this form, and for example, a form of derivation using the central image data in chronological order in the read image data of the most recent series of utterances, or a form of derivation using the image data of the most recent series of read image data A form of derivation using the last image data in the order may be applied.

At step 210, the derivation unit 11B extracts image data (still image data) corresponding to the derivation target time point of the maximum emotional level (maximum emotional level) among the six types of emotional levels for each participant. Identify for each participant. In step 212, the processing unit 11C uses the photographed image indicated by the still image data specified by the process of step 210 and the text data obtained by the process of step 206 to generate an image for supporting dialogue (hereinafter referred to as " (referred to as “dialogue support image”). At this time, as shown in FIG. 9 as an example, the processing unit 11C supports dialogue by displaying text 25D indicated by the text data in the form of a so-called speech balloon on the photographed image 25C of the corresponding participant. An image 30 is constructed.

At step 214, the derivation unit 11B uses the text data obtained by the processing at step 206 to calculate the remark level H for each participant using equation (1) as described above. At step 216, the derivation unit 11B uses the image data read out by the processing at step 204 to calculate the nodding frequency reduction rate U as described above. At this time, from the read image data, the nodding frequency N (hereinafter referred to as the "calculation frequency") before a predetermined time (in this embodiment, 60 seconds), which is applied when calculating the nodding frequency decrease rate U, is obtained. sometimes not. In this case, in this embodiment, the average value of past nodding frequencies N for each participant is applied as the starting frequency. However, it is not limited to this form, and for example, a form in which the most recent nodding frequency N for each participant is applied as the starting frequency may be employed.

At step 218, the processing unit 11C determines whether or not the maximum emotional level for each participant used in the processing at step 210 includes the maximum emotional level whose emoticons are registered in the emoticon/exaggerated information database 13D. is determined, and if the determination is affirmative, the process proceeds to step 220 .

At step 220, the processing unit 11C reads the emoticon corresponding to the maximum emotional level determined to be included in the processing at step 218 from the emoticon/exaggeration information database 13D. At step 222, the processing unit 11C converts the dialogue support image 30 so that the read emoticon 25E is included in the text 25D corresponding to the corresponding participant and displayed in a balloon, as shown in FIG. 9 as an example. update and then go to step 224 .

On the other hand, if the determination in step 218 is negative, the process proceeds to step 224 without executing the processes of steps 220 and 222 .

At step 224, the processing unit 11C determines whether the maximum emotional level for each participant used in the process at step 210 includes the maximum emotional level whose exaggerated information is registered in the emoticon/exaggerated information database 13D. is determined, and if the determination is affirmative, the process proceeds to step 226 .

At step 226, the processing section 11C reads out the exaggerated information corresponding to the maximum degree of emotion determined to be included in the processing at step 224 from the emoticon/exaggerated information database 13D. At step 228, as shown in FIG. 9 as an example, the processing unit 11C updates the dialogue support image 30 so that the information indicated by the read exaggerated information is displayed corresponding to the corresponding participant, and then Go to step 230 . In the example of the dialogue support image 30 shown in FIG. 9, as the information indicated by the exaggerated information, the text 25F of "Gang" is displayed above the corresponding participant's photographed image, and the corresponding participant's photographed image is displayed. An image 25G in which a plurality of vertical lines are superimposed near the forehead of the face is displayed.

On the other hand, if the determination in step 224 is negative, the process proceeds to step 230 without executing the processing of steps 226 and 228 .

At step 230, the processing unit 11C determines whether or not the situation correspondence information database 13B includes a condition that matches the combination of the remark level H, the nodding frequency decrease rate U, and the emotional level calculated by the above process for each participant. If the determination is affirmative, the process proceeds to step 232 .

At step 232, the processing unit 11C reads correspondence information corresponding to the conditions determined to be included in the processing at step 230 from the situation correspondence information database 13B. In step 234, the processing unit 11C causes the read correspondence information 25H to be displayed at a predetermined position (in the example shown in FIG. 9, the position near the upper end of the dialogue support image 30), as shown in FIG. 9 as an example. Then, the dialog support image 30 is updated as follows, and then the process proceeds to step 236 .

On the other hand, if the determination in step 230 is negative, the process proceeds to step 236 without executing the processes of steps 232 and 234 .

At step 236, the processing unit 11C updates the dialogue support image 30 so that other support information for supporting the dialogue is displayed. Note that, in the present embodiment, as the support information, as shown in FIG. 9 as an example, each participant's photographed image, speech frequency H (indicated as "speech" in FIG. 9), nodding frequency N (in FIG. 9) The dialog support image 30 is updated so that the support information 25I including the support information 25I including "affirmative") and emoticons (referred to as "mood" in FIG. 9) is displayed. In addition, in the present embodiment, the dialog support image 30 is updated so that the voice button 25J for receiving the instruction to reproduce the corresponding voice is displayed as the other support information. Furthermore, in the present embodiment, the dialog support image 30 is updated so that the scroll button 25K for receiving an instruction to scroll the displayed dialog support image 30 in the vertical direction is displayed as the other support information. . As other support information, as shown in FIG. 9, a photographed image 25P of the speaker while the speaker is speaking may be arranged and displayed near each balloon of the speaker.

At step 238, the processing unit 11C registers (stores) the various information obtained by the above processing in the dialogue information database 13C. Through the process of step 238, the dialogue information database 13C is constructed sequentially.

At step 240 , the processing unit 11</b>C controls the communication I/F unit 18 so as to transmit to each terminal 20 image information indicating the dialogue support image 30 obtained by the above processing. As a result of this process, the dialog support image 30 shown in FIG. 9 is displayed on the display unit 25 of each terminal 20 by the above-described dialog support application program. Each participant refers to the dialogue support image 30 displayed on the terminal 20 used by himself/herself, designates the corresponding sound button 25J when he wants to reproduce the sound, and scrolls the dialogue support image 30 in the vertical direction. moves the scroll button 25K in the desired direction. In response to this, the dialogue support application program running on each terminal 20 communicates state information indicating the state of operation when at least one of the voice button 25J and the scroll button 25K is operated by the participant. Send to the support device 10 .

Therefore, in step 242, the processing unit 11C determines whether or not state information indicating that the voice button 25J has been specified has been received from any terminal 20. If the determination is negative, the process proceeds to step 246. On the other hand, if the determination is affirmative, the process proceeds to step 244 .

At step 244, the processing unit 11C reads out the voice data corresponding to the designated voice button 25J from the storage unit 13 and transmits it to the terminal 20 that is the source of the corresponding status information, and then proceeds to step 246. By the processing of step 244, the terminal 20 that has transmitted the state information indicating that the voice button 25J has been designated reproduces the voice designated by the participant by the dialogue support application program.

At step 246, the processing unit 11C determines whether or not state information indicating that the scroll button 25K has been operated has been received from any terminal 20. If the determination is negative, the processing unit 11C proceeds to step 250. , and when the determination is affirmative, the process proceeds to step 248 .

At step 248, when the scroll button 25K is moved upward, the processing unit 11C stores information for scrolling the dialogue support image 30 upward by an amount corresponding to the amount of movement of the scroll button 25K. to the terminal 20 that sent the status information. Further, when the scroll button 25K is moved downward, the processing unit 11C stores information for scrolling the dialogue support image 30 downward by an amount corresponding to the amount of movement of the scroll button 25K. The information is transmitted to the terminal 20 that is the transmission source of the information. Then, the processing section 11C proceeds to the processing of step 250 after performing the above processing. In the terminal 20 that has transmitted the state information indicating that the scroll button 25K has been operated by the process of step 248, the dialog support application program changes the dialog support image 30 displayed on the display unit 25 to scrolled.

At step 250, the processing unit 11C determines whether or not the end timing of the dialogue support processing has arrived. do. In the present embodiment, the end timing of the dialogue support processing is determined when the dialogue support application programs running on the terminals 20 of all the participants participating in the conference targeted by the dialogue support processing are terminated. Although it is described as timing, it is not limited to this. For example, the end timing of the dialogue support process may be the timing at which the target meeting has stopped for a predetermined time (eg, 10 minutes) or longer, or the timing at which a preset time (eg, 1 hour) has passed for the target meeting. good.

In step 252, the processing unit 11C receives the image data and audio data transmitted from each terminal 20 and stores the received data in the storage unit 13. Terminate the dialogue support process.

On the other hand, in the dialogue support system 90 according to the present embodiment, when one of the participants designates the faction information display button 25A in the dialogue support image 30, the information graphically shows the emotional relationship between the participants. It has a faction display function for displaying a faction display image.

Next, with reference to FIGS. 10 and 11, the action of the dialogue support system 90 when executing the faction relationship display function will be described. FIG. 10 shows the operation of the dialogue support device 10 when information indicating that the faction information display button 25A has been specified is received from the terminal 20 of any of the participants participating in the target conversation. 4 is a flow chart showing the flow of faction information display processing executed by the CPU 11. FIG.

At step 300 in FIG. 10, the processing unit 11C stores the image data stored up to a predetermined time (10 minutes in the present embodiment) before that time, together with the corresponding terminal ID, in the dialogue information database 13C. read from At step 302, the processing unit 11C uses the read image data to compose a faction relation display image having a predetermined configuration. At step 304, the processing unit 11C transmits information indicating the configured faction relationship display image to the terminal 20 that sent the information indicating that the faction information display button 25A was designated. When the terminal 20 receives the information indicating the faction display image, the display unit 25 displays the faction display image 32 shown in FIG. 11 as an example. As shown in FIG. 11, the faction relationship display image 32 according to the present embodiment graphically displays the mutual feelings of the participants of the target conference.

In the present embodiment, mutual proximity SK _xy calculated by the following equation (3) is used as information indicating mutual feelings among participants. Note that x and y in equation (3) represent different participants, _nx represents the number of nods of participant x when participant y is speaking, and _ny is the number of times participant x is speaking. represents the number of nods of participant y when Here, the number of nods _nx and the number of nods _ny are derived in the same way as the number of nods n(t) used in the above equation (2) using the photographed image indicated by the read image data.

In the factional relationship display image 32 according to the present embodiment, the photographed images of the participants are arranged with the reciprocal of the calculated mutual proximity SK _xy as the separation distance. At this time, when the difference between the number of nods _nx and the number of nods _ny is greater than a predetermined value, as shown in FIG. An image 25L showing vision can be displayed. Also, in this case, an arrow can be displayed from the side with the higher number of nods to the side with the lower number of nods, and an image 25M showing a favorable impression can be displayed. Further, when the separation distance is less than the predetermined distance, the corresponding participants are connected by a straight line thicker than the others, and the image 25M showing favorable impression can be displayed on the straight line. Furthermore, when the separation distance is equal to or greater than the predetermined distance, the corresponding participants are connected by a straight line, and an image 25N showing a sense of collision can be displayed on the straight line.

In the example shown in FIG. 11, for example, Mr. A and other participants (Mr. B, Mr. C, Mr. D) indicate that they have a sense of conflict with each other. Mr. is hostile to Mr. D, but Mr. D shows that he has a favorable impression of Mr. C. Furthermore, the example shown in FIG. 11 indicates that Mr. B and Mr. D have a favorable impression of each other.

By referring to the factional relationship display image 32, the participants of the dialogue can grasp the estimation results of the feelings that others have toward them, so that they can make more effective remarks according to the situation. You can go and take an attitude. In addition, by referring to the factional relationship display image 32, the participants in the dialogue can determine whether or not the estimation result of their feelings toward others is appropriate. It becomes possible to

When the faction relationship display image 32 is displayed on the display unit 25, the participant refers to the faction relationship display image 32 and then designates the end button 25B. In response to this, the control unit 21A of the corresponding terminal 20 sends information (hereinafter referred to as “display end information”) to the dialogue support device 10 indicating that the display of the faction relationship display image 32 will end. Send via

Therefore, in the next step 306, the processing section 11C waits until the display end information is specified, and then ends the faction information display process.

As described above, according to the present embodiment, the obtaining unit 11A obtains the physical quantity from which the situation in the dialogue of the dialogue participant can be derived, and the physical quantity obtained by the obtaining unit 11A is used to obtain the dialogue of the participant. a derivation unit 11B for deriving the situation in the derivation unit 11B; and a processing unit 11C for performing both display processing for displaying the situation information corresponding to the situation derived by the derivation unit 11B and storage processing for storing the situation information. ing. Therefore, dialogue can be effectively activated.

Further, according to the present embodiment, the situation information is information representing the emotions of the participants. Therefore, activation of dialogue can be promoted more effectively.

Further, according to the present embodiment, the information representing emotion is text information representing emotion. Therefore, it is possible to grasp the emotions of the participants more specifically.

In addition, according to the present embodiment, the text information is information representing the feelings of others regarding the statement by one of the speakers among the participants. Therefore, it is possible to grasp the emotion of the participant listening to the speech.

Further, according to the present embodiment, the information representing emotion is image information representing emotion. Therefore, it is possible to grasp the emotions of the participants more intuitively.

Further, according to the present embodiment, the image information is emoticons. Therefore, it is possible to grasp the emotions of the participants more intuitively.

Further, according to the present embodiment, the image information is taken face image information obtained by photographing the face of the corresponding participant when the degree of emotion is maximized. Therefore, it is possible to grasp the emotions of the participants more effectively.

Further, according to the present embodiment, the image information is image information including information exaggerating emotions in addition to face photographed image information. Therefore, it is possible to grasp the emotions of the participants more effectively.

Further, according to the present embodiment, the information representing the emotion is information representing the relationship of emotions between the participants. Therefore, activation of dialogue can be promoted more effectively.

Further, according to the present embodiment, the physical quantity is an image obtained by photographing the participant and a sound indicating the speech of the participant. Therefore, activation of dialogue can be promoted at a lower cost.

Further, according to the present embodiment, the situation is a physical quantity representing the degree of emotion of the participant and a physical quantity representing the motion of the participant. Therefore, activation of dialogue can be promoted more easily.

Furthermore, according to the present embodiment, the above situations are represented by a physical quantity that indicates the frequency of nodding of the participant obtained from the image, a physical quantity that indicates the degree of expression of the participant obtained from the image, and a physical quantity that indicates the degree of expression of the participant obtained from the voice. It is a physical quantity that indicates the degree. Therefore, activation of dialogue can be promoted more easily.

In the above-described embodiment, a case has been described in which correspondence information to be displayed on the terminal 20 is determined using each participant's individual emotional level, but the present invention is not limited to this. For example, it is also possible to adopt a form in which correspondence information is determined using the degree of emotion of all participants. For example, as shown in FIG. 12, it can be assumed that all the participants are happy together when the degree of joy Y of each participant reaches a predetermined value (eg, 50) at the same time. In this case, by displaying on each terminal 20, for example, "there is a sense of unity and the situation is good" as correspondence information, the dialogue can be activated more effectively.

Further, in the above-described embodiment, a case has been described in which the present invention is applied to a form in which the participants of the dialogue are distributed to different locations for a meeting, but the present invention is not limited to this. For example, the present invention may be applied to a form in which each participant of the dialogue holds a conference in the same conference room or the like. In this case, instead of the camera 28 and microphone 29 provided in each terminal 20, one or more cameras and microphones configured separately from the terminal 20 are used to capture the images and voices of all conference participants. may be collected.

Also, in the above-described embodiment, the case where the dialogue support processing is executed in the dialogue support device 10 has been described, but the present invention is not limited to this. For example, at least one terminal 20 may execute the dialogue support process. In this form, the corresponding terminal 20 includes the dialogue support device of the present invention. Further, for example, at least one of the utterance level H, the nodding frequency decrease rate U, and each emotion level of each participant may be derived by the terminal 20 used by the corresponding participant.

Further, in the above embodiment, the case where the dialogue support image 30 is displayed on each terminal 20 has been described, but the present invention is not limited to this. For example, the dialogue support image 30 may be displayed on the dialogue support device 10 .

Also, in the above embodiment, the case where the present invention is applied to a conference has been described, but the present invention is not limited to this. For example, the present invention may be applied to a dialogue between a plurality of people other than a meeting, such as a personnel interview or a business negotiation.

Further, in the above-described embodiment, a case where emoticons are applied as image information representing emotions of the present invention has been described, but the present invention is not limited to this. For example, in addition to emoticons, at least one of pictograms and icons may be applied.

Further, in the above embodiment, a case has been described in which both a physical quantity representing the degree of emotion of the participant and a physical quantity representing the motion of the participant are applied as the situation, but the present invention is not limited to this. For example, only one of the physical quantity representing the degree of emotion of the participant and the physical quantity representing the motion of the participant may be applied.

In the above-described embodiment, the physical quantity indicating the frequency of nodding of the participant obtained from the image, the physical quantity indicating the degree of expression of the participant obtained from the image, and the degree of speech of the participant obtained from the voice are used as the above-described situations. Although the case where all of the indicated physical quantities are applied has been described, the present invention is not limited to this. For example, one of these physical quantities or a plurality of combinations excluding all of them may be applied.

Further, in the above-described embodiment, the case where the correspondence information is determined using the nodding frequency decrease rate U has been described, but the present invention is not limited to this. For example, the correspondence information may be determined using the nodding frequency N itself.

Further, in the above-described embodiment, a case has been described in which a database in which information on both the speaker and the receiver are mixed is applied as the situation correspondence information database 13B, but the present invention is not limited to this. For example, a form may be adopted in which different databases are constructed and applied for each of the speaker and the receiver.

In addition, Formulas (1) to (3) are all examples, and needless to say, they can be appropriately modified and applied without departing from the gist of the present invention.

Further, in the above-described embodiment, for example, the hardware structure of the processing unit that executes each process of the acquisition unit 11A, the derivation unit 11B, and the processing unit 11C includes the following various processors: can be used. As described above, the various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as a processing unit, as well as FPGAs (Field-Programmable Gate Arrays), etc., which have circuit configurations after manufacturing. Programmable Logic Device (PLD), ASIC (Application Specific Integrated Circuit), etc. etc. are included.

The processing unit may be configured with one of these various processors, or a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA). may consist of Also, the processing unit may be configured with a single processor.

As an example of configuring the processing unit with one processor, first, one processor is configured by combining one or more CPUs and software, as typified by computers such as clients and servers. There is a form that functions as a processing unit. Secondly, there is a form of using a processor that implements the functions of the entire system including the processing unit with a single IC (Integrated Circuit) chip, as typified by a System On Chip (SoC). In this way, the processing unit is configured using one or more of the above various processors as a hardware structure.

Furthermore, as the hardware structure of these various processors, more specifically, an electric circuit in which circuit elements such as semiconductor elements are combined can be used.

10 dialogue support device 11 CPU
11A acquisition unit 11B derivation unit 11C processing unit 12 memory 13 storage unit 13A dialogue support program 13B situation correspondence information database 13C dialogue information database 13D emoticon/exaggeration information database 14 input unit 15 display unit 16 medium read/write device 17 recording medium 18 communication I /F unit 20 terminal 21 CPU
22 Memory 23 Storage unit 24 Input unit 25 Display unit 25A Faction information display button 25B End button 25C Photographed image 25D Text 25E Emoticon 25F Text 25G Image 25H Support information 25I Support information 25J Voice button 25K Scroll buttons 25L, 25M, 25N Image 25P Photographed image 26 Medium read/write device 27 Wireless communication unit 28 Camera 29 Microphone 30 Dialogue support image 32 Faction relation display image 80 Network 90 Dialogue support system 96 Recording medium

Claims (11)

an acquisition unit that acquires a physical quantity from which a situation in the dialogue of a dialogue participant can be derived;
a derivation unit that derives a situation in the dialogue of the participant using the physical quantity acquired by the acquisition unit;
a processing unit that performs at least one of display processing for displaying the situation information corresponding to the situation derived by the deriving unit and storage processing for storing the situation information;
with
The situation information is information representing the emotions of the participants,
the information representing the emotion is image information representing the emotion;
The image information is photographed face image information obtained by photographing the face of the corresponding participant when the degree of emotion is maximum.
Dialogue support device. The image information is image information that includes information exaggerating the emotion in addition to the photographed face image information.
A dialogue support device according to claim 1 . The physical quantity is at least one of an image obtained by photographing the participant and a sound indicating the participant's remarks,
A dialogue support device according to claim 1 . The situation is at least one of a physical quantity representing the degree of emotion of the participant and a physical quantity representing the behavior of the participant,
4. A dialogue support device according to claim 3 . The situation is a physical quantity indicating the frequency of nodding of the participant obtained from the image, a physical quantity indicating the degree of expression of the participant obtained from the image, and the participant obtained from at least one of the image and the voice. is at least one of
5. A dialogue support device according to claim 4 . The degree of utterance is a degree according to the utterance speed of the most recent predetermined period,
6. A dialogue support device according to claim 5. The physical quantity indicating the frequency of nodding is information indicating the rate of decrease of the frequency,
6. A dialogue support device according to claim 5. learning the emotional information;
A dialogue support device according to claim 1. The situation information is information representing the emotions of all participants in the dialogue.
A dialogue support device according to claim 1. A dialogue support device according to any one of claims 1 to 9 ;
When the transmitting unit that transmits the physical quantity from which the situation in the dialogue can be derived to the acquisition unit of the dialogue support device and the processing unit of the dialogue support device perform the display processing, display targets of the display processing a terminal comprising a display;
Dialogue support system including. Acquiring a physical quantity from which a situation in the dialogue of a dialogue participant can be derived;
Deriving a situation in the dialogue of the participant using the obtained physical quantity,
performing at least one of a display process for displaying situation information corresponding to the derived situation and a storage process for storing the situation information;
A dialogue support program in which a computer executes processing ,
The situation information is information representing the emotions of the participants,
the information representing the emotion is image information representing the emotion;
The image information is photographed face image information obtained by photographing the face of the corresponding participant when the degree of emotion is maximum.
Dialogue support program.