JP7213857B2 - CONFERENCE SUPPORT DEVICE, CONFERENCE SUPPORT METHOD, AND CONFERENCE SUPPORT PROGRAM - Google Patents

Description

The present invention relates to a conference assistance device, a conference assistance method, and a conference assistance program for assisting progress of a conference.

As this type of device, there has been known a device that detects speech and facial expressions of participants in a conference and presents information to support the progress of the conference based on the detection results (for example, Patent Document 1, 2). In the device described in Patent Document 1, the psychological state of the participant is estimated based on images and voices, and information on the estimated mental state is displayed. The apparatus described in Patent Document 2 analyzes the contents of statements made by participants and displays the analysis results, thereby visualizing the discussion points and direction of the conference.

JP 2004-112518 A JP 2020-106687 A

By the way, in order to proceed smoothly with the conference, it is desirable to encourage the participants to speak as necessary. However, the devices described in Patent Documents 1 and 2 simply display the emotions of the participants and the content of the discussion, and the devices described in Patent Documents 1 and 2 have no idea how to prompt the participants to speak. Not proposed.

A conference support device that is one aspect of the present invention supports progress of a conference in which a plurality of participants including a first participant and a second participant participate. A conference support device includes an information acquisition unit that acquires biometric information of each of a plurality of participants, and based on the biometric information acquired by the information acquisition unit, determines whether or not each of the plurality of participants is speaking. an emotion estimating unit for estimating the emotion of each of the plurality of participants based on the biological information acquired by the information acquiring unit; and the utterance determining unit determining that the first participant is speaking. is determined by the utterance determination unit that the second participant has not made a statement continuously for a predetermined time or longer, and the second participant estimated by the emotion estimation unit during the utterance of the first participant an output unit for outputting a command for prompting the second participant to speak when the emotion changes.

Another aspect of the present invention is a conference support method for supporting progress of a conference in which a plurality of participants including a first participant and a second participant participate, wherein biometric information of each of the plurality of participants is an information acquisition step to be acquired; an utterance determination step for determining whether or not each of a plurality of participants is speaking based on the biometric information acquired in the information acquisition step; and the biometric information acquired in the information acquisition step an emotion estimation step of estimating the emotion of each of a plurality of participants based on ; When it is determined that the second participant is not speaking, and the emotion of the second participant estimated in the emotion estimation step changes during the speech of the first participant , the second participant is prompted to speak. and a command output step of outputting a command.

Still another aspect of the present invention is a conference support program for supporting progress of a conference in which a plurality of participants including a first participant and a second participant participate, wherein biometric information of each of the plurality of participants a speech determination step of determining whether or not each of a plurality of participants is speaking based on the biometric information acquired in the information acquisition step; and the biometric acquired in the information acquisition step Based on the information, an emotion estimation step of estimating the emotion of each of a plurality of participants; , to prompt the second participant to speak when it is determined that the second participant is not speaking and the emotion of the second participant estimated in the emotion estimation step changes during the speech of the first participant. and a command output step of outputting a command of .

According to the present invention, it is possible to support the smooth progress of a conference.

1 is a block diagram schematically showing an example of the overall configuration of a conference support system having a conference support device according to an embodiment of the present invention; FIG. FIG. 2 is a block diagram schematically showing an example of a main configuration of each terminal device in FIG. 1; 1 is a block diagram schematically showing an example of a configuration of essential parts of a conference support device according to an embodiment of the present invention; FIG. 4A and 4B are diagrams showing an example of the determination result of the utterance determination unit in FIG. 3 and the estimation result of the personal emotion estimation unit; FIG. FIG. 2 is a diagram showing an example of a display screen of the terminal device in FIG. 1 when an intervention command is received; The figure which shows the modification of FIG. 5A. 4 is a flowchart showing an example of intervention command output processing executed by the conference support device according to the embodiment of the present invention; FIG. 4 is a diagram showing an example of a calculation result of an activity calculation unit in FIG. 3; The figure which shows the modification of FIG. 7A. FIG. 3 is a diagram showing an example of a display screen of a terminal device used by the facilitator of FIG. 1 when activity level information is received; The figure which shows the modification of FIG. 8A. 4 is a flowchart showing an example of activity level information output processing executed by the conference support device according to the embodiment of the present invention; FIG. 4 is a diagram showing an example of an estimation result of an individual emotion estimating unit and an overall emotion estimating unit of FIG. 3; FIG. 10 is a view showing an example of a display screen of a terminal device used by the facilitator of FIG. 1 when information on overall emotion is received; The figure which shows the modification of FIG. 11A. 4 is a flowchart showing an example of overall emotion output processing executed by the conference support device according to the embodiment of the present invention;

An embodiment of the present invention will be described below with reference to FIGS. 1 to 12. FIG. A meeting support device according to an embodiment of the present invention supports progress of a meeting such as an internal meeting for generating new ideas. More specifically, the progress of the conference is indirectly supported by supporting the facilitator who manages the progress of the conference, or the progress of the conference is directly supported by taking on the role of the facilitator.

Meetings are held to generate ideas for new products and services, for example, and consist of multiple participants with diverse knowledge, such as experts in each field of the development department. Such a meeting progresses by exchanging opinions and debating among multiple participants. Inappropriate situations may arise where you do not respond or are unable to speak up. When such an inappropriate situation occurs, it becomes difficult to generate good ideas that reflect diverse knowledge.

A facilitator is a person in charge, such as a counselor who attends a meeting of the development department and who has expertise in psychology. More specifically, by observing the progress of the meeting and intervening as necessary to prevent inappropriate situations from occurring, participants are encouraged to speak, thereby promoting the progress of the meeting.

If the facilitator observes the amount of speech, facial expressions, actions, etc. of each participant, and intervenes in an appropriate manner by determining the appropriate intervention target and intervention timing, the progress of the conference will be promoted, but if the facilitator intervenes in an inappropriate manner , there is a possibility that the progress of the meeting will be hindered. Therefore, in this embodiment, the conference support apparatus is configured as follows so as to support the smooth progress of the conference by supporting intervention in an appropriate manner.

FIG. 1 is a block diagram schematically showing an example of the overall configuration of a conference support system 100 having a conference support device according to an embodiment of the invention. As shown in FIG. 1, a conference support system 100 includes terminal devices 10a to 10d used by a plurality of conference participants Pa to Pd, respectively, a terminal device 10f used by a facilitator Pf, and a terminal device 10f used by a facilitator Pf. and a server 20 constituting such a conference support device. Each terminal device 10 (10a to 10d, 10f) is composed of a general user terminal such as a personal computer or a smart phone.

Each terminal device 10 and server 20 are connected to a communication network 30 including a public wireless communication network such as the Internet network and a mobile phone network, and are configured to communicate with each other via the communication network 30 . The communication network 30 includes not only a public wireless communication network but also a closed communication network provided for each predetermined management area, such as a wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), and the like. .

Note that FIG. 1 shows the configuration of a Web conference in which each of the participants Pa to Pd and the facilitator Pf participate individually, but the conference is attended by a plurality of the participants Pa to Pd and some of the facilitators Pf as a group. It may be a teleconference or a conference other than an online conference. In this case, a single terminal device 10 may be shared by a plurality of participants Pa-Pd and facilitator Pf or some of them.

FIG. 2 is a block diagram schematically showing an example of a main configuration of each terminal device 10. As shown in FIG. As shown in FIG. 2, the terminal device 10 includes a computer having an arithmetic section 11 such as a CPU, a storage section 12 such as ROM and RAM, and peripheral circuits such as an I/O interface. The terminal device 10 further includes a microphone 13 connected to a computer, a camera 14, an input section 15, a speaker 16, and a display section 17. The storage unit 12 stores programs executed by the calculation unit 11, identification information of users (participants Pa to Pd, facilitator Pf) using the terminal device 10, and the like.

The microphone 13 acquires the voice uttered by the user of the terminal device 10 . The audio signal from the microphone 13 is processed by the calculation unit 11 and transmitted via the communication network 30 to the terminal devices 10 used by the other participants Pa to Pd and the facilitator Pf. In addition, the audio signal from the microphone 13 is temporarily stored in the storage unit 12, and processed by the calculation unit 11 as audio information for each user (participants Pa to Pd, facilitator Pf) together with the identification information. is transmitted to the server 20 at predetermined intervals via the . The microphone 13 can also receive voice operation inputs of various commands from the facilitator Pf. Information on instructions from the facilitator Pf input via the microphone 13 is processed by the calculation unit 11 and transmitted to the server 20 via the communication network 30 .

The camera 14 has an imaging element such as a CCD sensor or a CMOS sensor, and images the face of the user of the terminal device 10 . An image signal from the camera 14 is processed by the calculation unit 11 and transmitted via the communication network 30 to the terminal devices 10 used by the other participants Pa to Pd and the facilitator Pf. In addition, the image signal from the camera 14 is temporarily stored in the storage unit 12, and processed by the calculation unit 11 as image information for each user (participants Pa to Pd, facilitator Pf) together with identification information. is transmitted to the server 20 at predetermined intervals via the . The camera 14 can also receive operation inputs of various commands by gestures of the facilitator Pf. Information on instructions from the facilitator Pf input via the camera 14 is processed by the calculation unit 11 and transmitted to the server 20 via the communication network 30 .

The input unit 15 includes a keyboard, a mouse, a touch panel, and the like, and receives operation input from the user. For example, an operation input for turning on or off the microphone 13 or the camera 14 is accepted. It also accepts operation inputs of various commands from the facilitator Pf. Information on various commands from the facilitator Pf input via the input unit 15 is processed by the calculation unit 11 and transmitted to the server 20 via the communication network 30 .

The speaker 16 outputs sound according to the control signal from the calculation unit 11 . For example, based on the voice information acquired via the communication network 30, voices uttered by the other participants Pa to Pd or the facilitator Pf are output. The user of each terminal device 10 can interact with other participants Pa-Pd or facilitator Pf via microphone 13 and speaker 16 .

The display section 17 displays an image according to the control signal from the calculation section 11 . For example, images of other participants Pa to Pd or facilitator Pf are displayed based on image information obtained via communication network 30 . The user of each terminal device 10 can recognize the facial expressions and actions of the other participants Pa to Pd or the facilitator Pf via the display unit 17 .

FIG. 3 is a block diagram schematically showing an example of the main configuration of the server 20 that constitutes the conference support device according to the embodiment of the present invention. As shown in FIG. 3, the server 20 includes a computer having an arithmetic unit 21 such as a CPU, a storage unit 22 such as ROM and RAM, and peripheral circuits such as an I/O interface. Although a single server 20 is shown in FIG. 1, the server 20 may be configured by a plurality of servers in which the functions shown in FIG. 3 are distributed.

The storage unit 22 stores the conference support program executed by the calculation unit 21, the threshold used in the program, the participants Pa to Pd acquired from each terminal device 10, the audio information and image information for each facilitator Pf, the calculation unit 21 , and the like are stored. The calculation unit 21 includes an information acquisition unit 23, an utterance determination unit 24, an individual emotion estimation unit 25, a general emotion estimation unit 26, an activity level calculation unit 27, and an intervention recognition unit 28 as functional components of the CPU. , and an output unit 29 .

The information acquisition unit 23 acquires voice information and image information for each of the participants Pa to Pd and the facilitator Pf, that is, biometric information, from each terminal device 10 via the communication network 30, and various commands from the terminal device 10f of the facilitator Pf. Get information about The biometric information of each of the participants Pa to Pd acquired by the information acquisition unit 23 is stored in the storage unit 22 in association with the identification information of each of the participants Pa to Pd.

The utterance determination unit 24 determines whether or not each of the participants Pa to Pd is uttering based on the voice information acquired by the information acquisition unit 23 . More specifically, based on the voice information of each of the participants Pa to Pd, it is determined whether or not a voice signal has been input via the microphone 13 of each terminal device 10, so that the plurality of participants Pa to Pd The presence or absence of a statement is determined for each of The determination result by the utterance determination unit 24 is associated with the identification information of each of the participants Pa to Pd and stored in the storage unit 22 as time-series information.

Based on the image information acquired by the information acquisition unit 23, the personal emotion estimation unit 25 calculates the emotions (personal emotions) ea to ed of the participants Pa to Pd from the facial expressions (mouth corners, etc.) of the participants Pa to Pd. To which of a plurality of types of preset emotions belongs is estimated. For example, using Plutchik's circle of emotions, it is classified into eight basic emotions (joy, trust, fear, surprise, sadness, disgust, anger, and expectation) and applied emotions that are combinations of two adjacent emotions. fit to one of the patterns in the circle of

The personal emotion estimating unit 25 quantifies the personal emotions ea to ed in accordance with the application to the emotion circle. For example, a positive emotion such as joy (positive emotion) is represented by a positive value, and an unfavorable emotion (negative emotion) such as sadness is represented by a negative value. In this case, for example, the stronger the emotion (the closer it gets to the inner side of the circle of emotion), the larger the value (absolute value) representing the emotion. The estimation result by the personal emotion estimation unit 25 is associated with the identification information of each of the participants Pa to Pd and stored in the storage unit 22 as time-series information.

[Intervention targets and timing of intervention]
FIG. 4 is a diagram showing an example of the determination result of the utterance determination unit 24 and the estimation result of the personal emotion estimation unit 25, and an example of changes in the presence or absence of utterances of participants Pa to Pd and personal emotions ea to ed during the conference. indicate. In the example of FIG. 4, only participants Pa, Pb, and Pd speak, and participant Pc does not speak. If this situation continues, the discussion will proceed without reflecting the opinion of the participant Pc, so it is necessary to intervene to encourage the participant Pc to speak. On the other hand, if the participant Pc is thinking about the content of the opinion to be spoken or has no opinion to speak, and intervention is performed at the stage where the participant Pc cannot speak yet, the flow of the discussion will simply be interrupted. In addition, the intervened participant Pc becomes atrophied and becomes more difficult to speak.

As shown in FIG. 4, if the participant Pc does not speak even once from time t1 to t2 and does not speak continuously for a predetermined time T1 (for example, 5 minutes), the participant Pc should be intervened. Considered as a candidate (intervention candidate) of the target (intervention target). Furthermore, at time t3, when the personal emotion ec of participant Pc changes while other participant Pa is speaking (in the figure, positive emotion increases from "+1" to "+3"), participant Pc has some opinion. It is regarded as having occurred, and the participant Pc is made the subject of intervention. In other words, if the personal feeling changes in response to another participant's remarks, there is a high probability that some kind of opinion is generated. It is easy for the target audience to speak up and does not interfere with the flow of discussion. It should be noted that there is a high possibility that changes in personal feelings when no one is speaking are caused by personal matters unrelated to the content of the speech.

When the utterance determining unit 24 determines that the other participant Pa is speaking, the output unit 29 outputs a change in the personal emotion ec of the participant Pc (intervention candidate) estimated by the personal emotion estimating unit 25. , and outputs a command (intervention command) for prompting the participant Pc to speak. For example, through the communication network 30, the identification information for specifying the intervention candidate and the information on the change in personal emotion are transmitted to the terminal device 10f of the facilitator Pf. If the individual emotions of a plurality of intervention candidates who have not spoken continuously for a predetermined period of time T1 or longer have changed simultaneously, the intervention candidate who has not spoken for a longer period of time or the change in individual emotions has a greater change. An intervention command is output with the intervention candidate as the intervention target person.

FIG. 5A is a diagram showing an example of a display screen of the terminal device 10f when receiving an intervention command, which is displayed on the display unit 17 of the terminal device 10f used by the facilitator Pf in response to the intervention command from the server 20. An example of a display screen is shown. As shown in FIG. 5A, when an intervention command is transmitted from the server 20 to the terminal device 10f, among the plurality of (four in the figure) participants Pa to Pd displayed on the display unit 17, the participants identified by the intervention command Participant Pc is displayed with more emphasis than other participants Pa, Pb, and Pd.

Thus, as soon as the personal emotion ec of the participant Pc who has not spoken for a long time changes while the participant Pa is speaking, the facilitator Pf is notified that it is time to intervene in the participant Pc. Thus, for example, the facilitator Pf can prompt the participant Pc to speak before the participant Pa finishes speaking and the other participants Pb and Pd speak, so that the participant Pc smoothly participates in the discussion. can be done. Such a notification is sent not only when the facilitator Pf is unfamiliar and cannot judge the timing of intervention, but also when the facilitator Pf cannot observe the facial expressions and actions of the participants Pc because the materials are displayed on the full screen. Facilitator Pf can be assisted.

Further, the display unit 17 of the terminal device 10f displays information on changes in the personal emotion ec of the participant Pc specified by the intervention command. Information on changes in personal emotion ec is indicated using, for example, facial expression icons that indicate the type and intensity of emotion. In the example of FIG. 5A, when the personal emotion ec of the participant Pc changes from a weak positive emotion "+1" to a slightly strong positive emotion "+3", two facial expression icons corresponding to the emotions before and after the change are displayed. is shown using

By displaying the information on the change in the personal emotion ec in this way, the facilitator Pf can intervene after grasping the content of the change in the personal emotion ec of the participant Pc. You can encourage people to speak by using their voice. In addition, since the information displayed on the display unit 17 is minimized, such as the facial expression icons, the facilitator Pf can observe the facial expressions and movements of the participants Pa to Pd without hindrance.

FIG. 5B is a diagram showing a modified example of the display screen of the terminal device 10 when receiving an intervention command. Examples of display screens displayed on the display units 17 of 10d and 10f are shown. As shown in FIG. 5B, when an intervention command is transmitted from the server 20 to each terminal device 10, among the plurality of participants Pa to Pd displayed on the display unit 17 (four in the figure), the intervention command specifies The selected participant Pc is displayed with more emphasis than the other participants Pa, Pb, and Pd. For example, the image of participant Pc is displayed larger than the images of other participants Pa, Pb, and Pd.

In this way, the next speaker is clearly indicated not only to the facilitator Pf and the intervention target, but also to all of the participants Pa to Pd, so that the intervention target can be encouraged to speak in a more natural manner. When notifying all of the participants Pa to Pd in this manner, not only the intervention by the facilitator Pf is supported, but also the conference support device plays the role of the facilitator to directly support the progress of the conference. can be done.

FIG. 6 is a flow chart showing an example of intervention command output processing executed by the computing unit 21 of the server 20 according to the meeting support program pre-stored in the storage unit 22 of the server 20 . The processing shown in this flow chart is started, for example, when a start command is input from the facilitator Pf, and is repeatedly executed at a predetermined cycle until an end command is input.

First, in step S1, biometric information is acquired for each of the participants Pa to Pd. Next, in step S2, based on the voice information acquired in step S1, it is determined whether or not each of the participants Pa to Pd speaks. Next, in step S3, individual emotions ea to ed of each of the participants Pa to Pd are estimated based on the image information acquired in step S1. Next, in step S4, based on the determination result of step S2, it is determined whether or not there is an intervention candidate who has not made a statement continuously for a predetermined period of time T1 or longer. If the result in step S4 is affirmative, the process proceeds to step S5, and if the result is negative, the process ends. In step S5, based on the determination result of step S2 and the estimation result of step S3, it is determined whether or not there was a change in the emotions of the intervention candidate during the speech of the other participants. If the result in step S5 is affirmative, the process proceeds to step S6, and if the result is negative, the process ends. In step S6, an intervention command is transmitted to the terminal device 10 for prompting the intervention candidate to speak as an intervention candidate.

In this way, a participant who has not spoken for a long time is specified (steps S1, S2, S4), and an intervention command is output at the timing when the personal emotion changes in response to the speech of another participant (steps S1 to S4). S3, S5, S6), intervention can be performed at a timing when it is easy to speak. As a result, it is possible to effectively intervene without interfering with the flow of the discussion, thereby supporting the smooth progress of the conference.

[Activity of discussion and effect of intervention]
The facilitator Pf knows whether or not the intervention performed by him/herself was effective, so that the facilitator Pf can review the mode of subsequent intervention as necessary. By feeding back the effect of the intervention by the facilitator Pf to the facilitator Pf, it is possible to assist the effective intervention by the facilitator Pf and assist the smooth progress of the conference.

Based on the biometric information acquired by the information acquisition unit 23, the activity level calculation unit 27 calculates the activity level a, which represents the degree of activation of the discussion in the meeting. For example, based on the voice information acquired by the information acquiring unit 23, the frequency f of utterances by the participants Pa to Pd is calculated, and the activity level a is calculated based on the calculated frequency f of utterances.

FIG. 7A is a diagram showing an example of the calculation result of the activity level calculator 27, showing an example of the activity level a based on the utterance frequency f. As shown in FIG. 7A, when the intervention by the facilitator Pf is performed at time t6, the frequency f of utterances from time t5 to t6 before the intervention, the predetermined period T2 after the intervention (for example, about 5 minutes), the time t6 to The frequency f of utterances at t7 is calculated. In the example of FIG. 7A, there are two utterances in 3 minutes from time t5 to t6, the frequency f of utterances before intervention is calculated as 0.7 times/minute, and 7 times in 5 minutes from time t6 to t7. There is an utterance, and the utterance frequency f after the intervention is calculated as 1.4 times/minute. In this case, the activity level a before intervention is calculated as "0.7" and the activity level a after intervention is calculated as "1.4".

The activity level calculation unit 27 can also calculate the activity level a based on the individual emotions ea to ed of the participants Pa to Pd estimated by the individual emotion estimation unit 25 . For example, the average value of the individual emotions ea to ed of the participants Pa to Pd is calculated as the activity level a.

FIG. 7B is a diagram showing an example of the calculation result of the activity level calculator 27, showing an example of the activity level a based on the individual emotions ea to ed. In the example of FIG. 7B, when intervention is performed at time t6, the individual emotion ed of participant Pd changes from negative emotion “−3” to positive emotion “+3” at time t8, and participant Pc changes at time t9. changes from weak positive emotion "+1" to strong positive emotion "+5". In this case, the average value of the individual emotions ea to ed of the participants Pa to Pd, that is, the activity level a is "+1.5" at times t5 to t8, "+3" at times t8 to t9, and "+3" at times t9 to t7. +4".

The activity level calculator 27 may calculate the activity level a based on both the utterance frequency f and the individual emotions ea to ed of the participants Pa to Pd. In this case, for example, the activation level a based on the frequency of utterances f and the activation levels a based on individual emotions ea to ed are converted into numerical values af and ae that are the same minimum and maximum values, and the converted numerical values af and The average value of ae is calculated as the activity a.

The intervention recognition unit 28 recognizes the timing of intervention by the facilitator Pf (time t6 in FIGS. 7A and 7B). More specifically, based on the voice information of the facilitator Pf acquired by the information acquisition unit 23, the facilitator Pf intervenes by determining whether a voice signal is input via the microphone 13 of the terminal device 10f. Recognize when to do In this case, when the facilitator Pf intervenes, the timing of intervention is automatically recognized.

The intervention recognition unit 28 may recognize the timing of intervention based on a conscious operation input by the facilitator Pf, that is, an operation input predetermined for requesting feedback on the start of intervention and the effect of intervention. . More specifically, based on the information of various commands from the facilitator Pf acquired by the information acquisition unit 23, operation via the input unit 15 of the terminal device 10f, voice operation through the microphone 13, camera 14 It is determined whether or not an operation by a gesture is input. When recognizing the timing of intervention triggered by such a conscious operation input, for example, the effect of intervention is not calculated erroneously triggered by remarks other than intervention, and accurate intervention is performed in line with the intention of the facilitator Pf. effect can be fed back.

When the timing of the intervention is recognized by the intervention recognition unit 28, the output unit 29 generates an activity level representing a change in the activity level a based on the activity level a calculated by the activity level calculation unit 27 within a predetermined period T2 thereafter. The frequency information is generated and transmitted to the terminal device 10f via the communication network 30. FIG. For example, as shown in FIG. 7A, when the activity level a before the intervention is calculated by the activity level calculation unit 27 as "0.7" and the activity level a in the predetermined period T2 after the intervention is calculated as "1.4", Activation level information indicating that the activation level a has improved as an effect of the intervention is generated and output. Alternatively, as shown in FIG. 7B, when the activity level a in the predetermined period T2 after the intervention is calculated by the activity level calculation unit 27 as "+1.5", "+3", and "+4", the effect of the intervention is activity. Generate and output activity level information indicating the transition of the degree a.

8A and 8B are diagrams showing an example of the display screen of the terminal device 10f used by the facilitator Pf when the activity level information is received. In the example of FIG. 8A, as an effect of the intervention, an arrow indicating that the activity level a has improved before and after the intervention is displayed in the center of the screen. In the example of FIG. 8B, the effect of the intervention varies from "+1.5" corresponding to a weak positive emotion, to "+3" corresponding to a slightly strong positive emotion, to "+3" A facial expression icon is displayed in the center of the screen indicating that it has changed to "+4" corresponding to the emotion of . The effect of such an intervention is displayed after a predetermined period of time T2 has elapsed after the timing of intervention is recognized by the intervention recognition unit 28. may be continuously displayed for a predetermined period of time T2 after the timing of intervention is recognized.

FIG. 9 is a flow chart showing an example of activity level information output processing executed by the computing unit 21 of the server 20 according to the conference support program pre-stored in the storage unit 22 of the server 20 . The processing shown in this flow chart is started, for example, when a start command is input from the facilitator Pf, and is repeatedly executed at a predetermined cycle until an end command is input.

First, in step S1, biometric information is acquired for each of the participants Pa to Pd. Next, in step S7, the degree of activity a is calculated based on the biological information acquired in step S1. Next, in step S8, it is determined whether or not the timing of intervention by the facilitator Pf has been recognized. If the result in step S8 is affirmative, the process proceeds to step S9, and if the result is negative, the process ends. In step S9, activity level information representing changes in the activity level a calculated in step S7 is generated and output to the terminal device 10f of the facilitator Pf. Next, in step S10, it is determined whether or not a predetermined period of time T2 has elapsed from the intervention timing recognized in step S8. If the result in step S10 is affirmative, the process is terminated, and if the result is negative, the process returns to step S9.

In this way, in order to recognize the timing of intervention by the facilitator Pf (step S8) and feed back the change in the activity level a before and after the intervention and the transition in the activity level a after the intervention (steps S9 and S10), the facilitator Pf itself It is possible to grasp the effect of the intervention of As a result, effective intervention by the facilitator Pf can be supported, and smooth progress of the conference can be supported.

[Shaping the overall atmosphere of the meeting]
It is necessary to accurately grasp the overall atmosphere of the conference in order to smoothly proceed with the conference and arrive at a conclusion that reflects the opinions of all participants. The overall atmosphere of the meeting is formed by the general agreement of personal feelings of all the participants. When each participant speaks frankly and fully exchanges their opinions, the personal feelings of all participants are aligned, and reaching a conclusion in such a unanimous atmosphere reflects diverse knowledge. You can create well-designed and high-quality ideas.

Based on the individual emotions ea to ed of each of the participants Pa to Pd estimated by the individual emotion estimation unit 25, the overall emotion estimation unit 26 is based on the overall emotions (overall emotions) e of the participants Pa to Pd, that is, the conference Estimate the overall atmosphere. More specifically, it is determined whether or not the percentage of participants Pa to Pd that are estimated to belong to the same type of emotion by the personal emotion estimation unit 25 is equal to or greater than a predetermined percentage (eg, 70%), and If it is determined to be equal to or greater than the ratio, the emotion is estimated as the overall emotion e.

FIG. 10 is a diagram showing an example of estimation results of the individual emotion estimating section 25 and the overall emotion estimating section 26. As shown in FIG. As shown in FIG. 10, for example, immediately after the start of the conference, the individual emotions ea to ed of the participants Pa to Pd hardly match, and the general atmosphere of the conference is not formed. At this stage, even if the overall emotion e is estimated as an average value of the individual emotions ea to ed, the overall atmosphere of the meeting cannot be expressed accurately.

When the meeting proceeds smoothly, all participants Pa-Pd speak frankly, and mutual opinions are sufficiently exchanged, the personal emotions ea-ed of all participants Pa-Pd are unanimously expressed as positive emotions. They match, and the overall emotion estimation unit 26 estimates the overall emotion e as a positive emotion. Also, the average value of the individual emotions ea to ed is calculated as the absolute value of the overall emotion e.

On the other hand, when the conference progresses and most of the participants Pa to Pc speak frankly and exchange opinions, some of the participants Pd cannot speak sufficiently or frankly. , the personal emotion ed of some participants Pd is inconsistent with the overall emotion e. In this case, the positive emotion to which the individual emotions ea-ec of the participants Pa-Pc belong to a predetermined percentage or more is estimated as the overall emotion e, and the average value of the individual emotions ea-ec of the participants Pa-Pc is the total emotion e. Calculated as an absolute value.

When the overall emotion estimating unit 26 estimates the overall emotion e, the output unit 29 outputs the information of the participant Pd whose estimated overall emotion e and individual emotion ed do not match the overall emotion e via the communication network 30. It transmits to the terminal device 10f.

FIGS. 11A and 11B are diagrams showing an example of the display screen of the terminal device 10f used by the facilitator Pf when the information of the overall emotion e is received. In the example of FIG. 11A, the facial expression icon corresponding to the rather strong positive emotion “+4” estimated as the unanimously agreed overall emotion e in FIG. 10 is displayed in the center of the screen.

In the example of FIG. 11B, the facial expression icon corresponding to the somewhat strong positive emotion "+3" estimated as the overall emotion e when there is a participant Pd whose personal emotion ed does not match the overall emotion e in FIG. to be displayed. In addition, the participant Pd, whose personal emotion ed does not match the overall emotion e, is displayed with more emphasis than the other participants Pa to Pc, and the slightly strong negative value estimated as the individual emotion ed of the participant Pd is displayed. A facial expression icon corresponding to the emotion "-3" is displayed.

FIG. 12 is a flow chart showing an example of overall emotion output processing executed by the computing section 21 of the server 20 according to the meeting support program pre-stored in the storage section 22 of the server 20 . The processing shown in this flow chart is started, for example, when a start command is input from the facilitator Pf, and is repeatedly executed at a predetermined cycle until an end command is input.

First, in step S1, biometric information is acquired for each of the participants Pa to Pd. Next, in step S3, individual emotions ea to ed of each of the participants Pa to Pd are estimated based on the image information acquired in step S1. Next, in step S11, it is determined whether or not the individual emotions ea to ed of the participants Pa to Pd estimated in step S3 generally match each other. If the result in step S11 is affirmative, the process proceeds to step S12, and if the result is negative, the process ends. In step S12, the emotion determined to be generally matched in step S11 is estimated as the overall emotion e of all participants Pa to Pd, and the average value of the matched individual emotions ea to ec is calculated as the absolute value of the overall emotion e. .

Next, in step S13, it is determined whether or not there are participants Pa to Pd whose individual emotions ea to ed estimated in step S3 do not match the overall emotion e estimated in step S12. If the result in step S13 is affirmative, the process advances to step S14, where the total emotion e estimated in step S12, the identification information of the participant whose personal emotion is determined not to match the total emotion e in step S13, and the mismatched personal emotion. is output to the terminal device 10f of the facilitator Pf. If the result in step S13 is NO, the process proceeds to step S15, and only the unanimously agreed overall emotion e estimated in step S12 is output to the terminal device 10f of the facilitator Pf.

In this way, when the individual emotions ea to ed of all the participants Pa to Pd are generally matched, the overall emotion e is estimated and presented (steps S11, S12, S14, S15). can support the understanding of As a result, the facilitator Pf can accurately grasp the fact that the overall atmosphere of the conference is being formed and the nature of the atmosphere. can proceed smoothly. Also, by presenting the information of the participant Pd who has the individual emotion ed different from the overall emotion e (step S14), it is possible to assist the appropriate intervention by the facilitator Pf.

According to this embodiment, the following effects can be obtained.
(1) The server 20 supports progress of a conference in which a plurality of participants Pa to Pd including a first participant (participant Pa in FIG. 4) and a second participant (participant Pc in FIG. 4) participate. do. The server 20 includes an information acquisition unit 23 that acquires biometric information of each of the participants Pa to Pd, and based on the biometric information acquired by the information acquisition unit 23, each of the participants Pa to Pd makes a statement. and a personal emotion estimation unit for estimating the individual emotions ea to ed of each of the participants Pa to Pd based on the biometric information acquired by the information acquisition unit 23. 25, when the speech determination unit 24 determines that another participant Pa is speaking, if the personal emotion ec of the participant Pc estimated by the personal emotion estimation unit 25 changes, the participant Pc is made to speak. and an output unit 29 for outputting a prompting command (FIG. 3).

As a result, since the intervening target person is identified in a timely manner, it is possible to support the smooth progress of the conference. For example, as shown in FIG. 5A, it is possible to timely propose intervention candidates to a facilitator Pf who manages the progress of the conference. Alternatively, as shown in FIG. 5B, the next speaker can be made explicit to all conference participants Pa-Pd.

(2) The information acquisition unit 23 acquires voice information and image information as biological information of each of the participants Pa to Pd. The utterance determining unit 24 determines whether or not each of the participants Pa to Pd is speaking based on the voice information acquired by the information acquiring unit 23 (FIG. 4). Based on the image information acquired by the information acquiring section 23, the individual emotion estimating section 25 estimates individual emotions ea to ed of each of the participants Pa to Pd (FIG. 4). Since the biometric information of the participants Pa to Pd is acquired using the microphone 13 and camera 14 of general terminal devices 10 (10a to 10d) such as personal computers and smartphones, it can be suitably applied to online conferences. can.

(3) The output unit 29 urges the participant Pc to speak on the condition that the speech determination unit 24 continuously determines that the participant Pc (FIG. 4) has not spoken for a predetermined time T1 or more. output commands for As a result, the participant Pc, who has not spoken for a long time, can be urged to speak at the timing when the personal emotion ec changes in response to the speech of the other participant Pa, making it easier for the participant Pc to speak. You can intervene in time.

(4) The output unit 29 outputs information on changes in the personal emotion ec of the participant Pc estimated by the personal emotion estimation unit 25 when outputting a command to prompt the participant Pc to speak (FIG. 5A). . As a result, the facilitator Pf can intervene after grasping the content of the change in the personal emotion ec of the participant Pc, so that it is possible to intervene in an appropriate manner according to the psychological state of the participant Pc.

(5) The output unit 29 outputs a command for prompting the participant Pc to speak together with the identification information of the participant Pc to the terminal device 10f used by the facilitator Pf who manages the progress of the conference (FIG. 5A). As a result, the facilitator Pf can be assisted via the terminal device 10f.

(6) A plurality of participants Pa to Pd are displayed on the display units 17 of the terminal devices 10a to 10d used by the plurality of participants Pa to Pd (FIG. 5B). The output unit 29 instructs the terminal devices 10a to 10d used by each of the participants Pa to Pd to control the display unit 17 so that the participant Pc is displayed with more emphasis than the other participants. is output as a command to prompt the participant Pc to speak (FIG. 5B). As a result, the next speaker is clearly indicated not only to the facilitator Pf and the intervention target, but also to all of the participants Pa to Pd, so that the intervention target can be encouraged to speak in a more natural manner. Further, the conference support device can directly support the progress of the conference by playing the role of the facilitator.

In the above embodiment, an example was described in which the biometric information of each of the participants Pa to Pd was acquired via the microphone 13 and the camera 14 of each of the terminal devices 10a to 10d. The information acquisition unit is not limited to such. It is not limited to those attached to the terminal device, and may be an independent microphone or camera. It is not limited to being provided for each participant, but may be shared by a plurality of participants. In this case, voice information for each participant may be acquired using, for example, a microphone having directivity and different sensitivity for acquiring voice depending on the direction, and an individual may be recognized based on the voice information. Also, a plurality of participants may be photographed by a wide-angle camera, and individuals may be recognized based on the image information.

In the above embodiment, an example of determining whether or not participants Pa to Pd have spoken based on voice information has been described. Not limited to things. For example, the presence or absence of a statement in sign language may be determined based on image information.

In the above embodiment, an example has been described in which individual emotions ea to ed are estimated from facial expressions of participants Pa to Pd based on image information. It is not limited to something like For example, personal emotion may be estimated based on actions in addition to facial expressions. Instead of image information, or in addition to image information, biometric information such as heart rate, facial temperature, and voice information of each participant may be used to estimate personal emotion. In addition, although individual emotions are quantified by applying them to patterns classified by the circle of emotions, they may be quantified by applying them to other classified patterns of emotions.

In the above embodiment, an example was explained in which the activity level a is calculated based on the utterance frequency f or the individual emotions ea to ed of the participants Pa to Pd. The activity level calculation unit that performs the calculation is not limited to this type. For example, the degree of activity may be calculated by detecting a specific keyword based on the audio information of the meeting and estimating the progress of the discussion, the atmosphere, and the like.

In the above embodiment, an example of determining whether the personal emotions ea to ed of the participants Pa to Pd are positive emotions or negative emotions has been described with reference to FIG. The personal emotion estimating unit for estimating which of the types of emotions belongs to is not limited to this. For example, it may be classified into three or more emotions, such as eight basic emotions. In this case, the predetermined percentage for determining whether or not the personal feelings are generally in agreement is set to a percentage lower than 70% (for example, 60%).

Regarding a conference support method for supporting progress of a conference in which a plurality of participants including a first participant and a second participant participate, biometric information of each of the plurality of participants is acquired, and based on the acquired biometric information determines whether or not each of the plurality of participants is speaking, estimates the emotion of each of the plurality of participants, and when it is determined that the first participant is speaking, Any other configuration may be used as long as a command for prompting the second participant to speak is output when the emotions of the two participants change.

An information acquisition step S1 for acquiring biometric information of each of the plurality of participants, and an information acquisition for a conference assistance program for assisting progress of a conference attended by a plurality of participants including a first participant and a second participant. Based on the biometric information acquired in step S1, an utterance determination step S2 for determining whether or not each of the plurality of participants is speaking, and based on the biometric information acquired in the information acquisition step S1, a plurality of Emotion estimation step S3 for estimating the emotions of each of the participants, and when it is determined that the first participant is speaking in the utterance determination step S2, the emotions of the second participant estimated in the emotion estimation step S3 are: If command output steps S5 to S6 for outputting a command for prompting the second participant to speak when changed, the processing executed by the calculation unit 21 of the server 20 is the same as that shown in FIG. Not exclusively. Therefore, the configuration of the conference support program that causes the computer to execute these procedures is not limited to the one described above. Although the conference support program is stored in the storage unit 22 of the server 20 in the above embodiment, it may be stored in an external storage medium such as a flash memory. A conference assistance program can also be obtained via the communication network 30 .

The above description is merely an example, and the present invention is not limited by the above-described embodiments and modifications as long as the features of the present invention are not impaired. It is also possible to arbitrarily combine one or more of the above embodiments and modifications, and it is also possible to combine modifications with each other.

10 (10a to 10d, 10f) terminal device, 11 calculation unit, 12 storage unit, 13 microphone, 14 camera, 15 input unit, 16 speaker, 17 display unit, 20 server, 21 calculation unit, 22 storage unit, 23 information acquisition Section 24 Speech Judgment Section 25 Individual Emotion Estimation Section 26 Overall Emotion Estimation Section 27 Activity Calculation Section 28 Intervention Recognition Section 29 Output Section 30 Communication Network 100 Conference Support System

Claims (7)

A conference support device for supporting progress of a conference in which a plurality of participants including a first participant and a second participant participate,
an information acquisition unit that acquires biometric information of each of the plurality of participants;
an utterance determination unit that determines whether or not each of the plurality of participants is speaking based on the biometric information acquired by the information acquisition unit;
an emotion estimation unit that estimates the emotion of each of the plurality of participants based on the biometric information acquired by the information acquisition unit;
When the speech determination unit determines that the first participant is speaking, the speech determination unit continuously determines that the second participant has not spoken for a predetermined time or longer, and an output unit that outputs a command to prompt the second participant to speak when the emotion of the second participant estimated by the emotion estimation unit changes during the speech of the first participant. A meeting support device characterized by: The conference support device according to claim 1,
The information acquisition unit acquires audio information and image information as biological information of each of the plurality of participants,
The speech determination unit determines whether or not each of the plurality of participants is speaking based on the voice information acquired by the information acquisition unit;
The conference support apparatus, wherein the emotion estimation unit estimates the emotion of each of the plurality of participants based on the image information acquired by the information acquisition unit. The conference support device according to claim 1 or 2,
The conference, wherein the output unit outputs information on changes in the emotion of the second participant estimated by the emotion estimation unit when outputting a command for prompting the second participant to speak. support equipment. The conference support device according to any one of claims 1 to 3,
The output unit outputs a command for prompting the second participant to speak, together with the identification information of the second participant, to a terminal device used by a facilitator who manages progress of the conference. Conference support device. The conference support device according to any one of claims 1 to 4,
The plurality of participants are displayed on the display unit of the terminal device used by each of the plurality of participants,
The output unit instructs the terminal device used by each of the plurality of participants to control the display unit so that the second participant is displayed with more emphasis than the other participants. A conference support device that outputs as a command for prompting a second participant to speak. A conference support method for supporting progress of a conference in which a plurality of participants including a first participant and a second participant participate,
an information acquisition step of acquiring biometric information of each of the plurality of participants;
a statement determination step of determining whether or not each of the plurality of participants is speaking based on the biometric information acquired in the information acquisition step;
an emotion estimation step of estimating the emotion of each of the plurality of participants based on the biometric information acquired in the information acquisition step;
When it is determined in the statement determination step that the first participant has made a statement, it is determined in the statement determination step that the second participant has not made a statement continuously for a predetermined time or longer, and a command output step of outputting a command for prompting the second participant to speak when the emotion of the second participant estimated in the emotion estimation step changes during the speech of the first participant, by a computer; A conference support method, comprising executing. A conference support program for supporting progress of a conference in which a plurality of participants including a first participant and a second participant participate,
an information acquisition step of acquiring biometric information of each of the plurality of participants;
a statement determination step of determining whether or not each of the plurality of participants is speaking based on the biometric information acquired in the information acquisition step;
an emotion estimation step of estimating the emotion of each of the plurality of participants based on the biometric information acquired in the information acquisition step;
When it is determined in the statement determination step that the first participant has made a statement, it is determined in the statement determination step that the second participant has not made a statement continuously for a predetermined time or longer, and a command output step of outputting a command to prompt the second participant to speak when the emotion of the second participant estimated in the emotion estimation step changes during the speech of the first participant; A meeting support program to make it run.

Images