JP2022068905A - Conference support device, conference support method, and conference support program - Google Patents

Abstract

To support the smooth progress of a conference.SOLUTION: A server 20 supports the progress of a conference in which a plurality of participants participate. The server 20 comprises: an information acquisition unit 23 that acquires biological information on each of the plurality of participants; an individual feeling estimation unit 25 that, based on the biological information acquired by the information acquisition unit 23, estimates to any of a plurality of feelings set in advance an individual feeling of each of the plurality of participants belongs; an entire feeling estimation unit 26 that determines whether the ratio of the participants estimated to belong to the same type of feeling by the individual feeling estimation unit 25 is equal to or more than a predetermined ratio, when determining that the ratio is equal to or more than the predetermined ratio, estimates the feeling as the entire feeling of the entirety of the plurality of participants; and an output unit that outputs the entire feeling estimated by the entire feeling estimation unit 26.SELECTED DRAWING: Figure 3

Description

The present invention relates to a conference support device, a conference support method, and a conference support program that support the progress of a conference.

As a device of this type, conventionally, a device that detects the remarks and facial expressions of the participants of the conference and presents information that supports the progress of the conference based on the detection result is known (for example, Patent Document 1, 2). In the device described in Patent Document 1, the psychological state of the participant is estimated based on the image and the sound, and the information on the estimated psychological state is displayed. In the device described in Patent Document 2, the content of the remarks of the participants is analyzed and the analysis result is displayed to visualize the points of discussion and the direction of the meeting.

Japanese Unexamined Patent Publication No. 2004-112518 Japanese Unexamined Patent Publication No. 2020-106678

By the way, in order for the conference to proceed smoothly, it is desirable to encourage participants to speak as necessary. However, the devices described in Patent Documents 1 and 2 merely display the emotions and the contents of the discussion of the participants, and the devices described in Patent Documents 1 and 2 do not discuss how to encourage the participants to speak. Not proposed.

The conference support device, which is one aspect of the present invention, supports the progress of a conference in which a plurality of participants participate. The conference support device has an information acquisition unit that acquires the biometric information of each of the plurality of participants, and a plurality of preset personal emotions of the plurality of participants based on the biometric information acquired by the information acquisition unit. It is determined whether or not the ratio of the participants estimated to belong to the same type of emotion by the individual emotion estimation unit and the individual emotion estimation unit, which estimates which of the emotions of the same type belongs to, is equal to or higher than the predetermined ratio. When it is determined that the ratio is equal to or higher than a predetermined ratio, an overall emotion estimation unit that estimates the emotion as the overall emotion of a plurality of participants and an output unit that outputs the overall emotion estimated by the overall emotion estimation unit. Be prepared.

Another aspect of the present invention is a conference support method that supports the progress of a conference in which a plurality of participants participate, and is acquired in an information acquisition step of acquiring biometric information of each of the plurality of participants and an information acquisition step. The personal emotion estimation step that estimates which of the plurality of preset types of emotions each individual emotion of a plurality of participants belongs to based on the obtained biometric information, and the personal emotion estimation step of the same type. It is determined whether or not the proportion of participants estimated to belong to an emotion is equal to or greater than a predetermined proportion, and if it is determined to be equal to or greater than a predetermined proportion, the emotion is estimated as the overall emotion of a plurality of participants as a whole. It includes executing an emotion estimation step and an output step for outputting the total emotion estimated in the total emotion estimation step by a computer.

Yet another aspect of the present invention is a conference support program that supports the progress of a conference in which a plurality of participants participate, in an information acquisition step of acquiring biometric information of each of the plurality of participants, and an information acquisition step. Based on the acquired biometric information, the individual emotion estimation step that estimates which of the plurality of preset types of emotions each individual emotion of a plurality of participants belongs to, and the individual emotion estimation step are the same type. It is determined whether or not the ratio of the participants estimated to belong to the emotions of the above is equal to or more than the predetermined ratio, and if it is determined to be equal to or more than the predetermined ratio, the emotions are estimated as the total emotions of the entire plurality of participants. Have the computer execute the whole emotion estimation step and the output step that outputs the whole emotion estimated in the whole emotion estimation step.

According to the present invention, it is possible to support the smooth progress of the conference.

The block diagram schematically showing an example of the whole structure of the conference support system which has the conference support apparatus which concerns on embodiment of this invention. FIG. 3 is a block diagram schematically showing an example of a main configuration of each terminal device of FIG. 1. The block diagram schematically showing an example of the main part structure of the conference support apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the judgment result of the remark judgment unit of FIG. 3 and the estimation result of the individual emotion estimation unit. The figure which shows an example of the display screen of the terminal apparatus of FIG. 1 when an intervention command is received. The figure which shows the modification of FIG. 5A. The flowchart which shows an example of the intervention command output processing executed by the conference support apparatus which concerns on embodiment of this invention. The figure which shows an example of the calculation result of the activity calculation part of FIG. The figure which shows the modification of FIG. 7A. The figure which shows an example of the display screen of the terminal apparatus used by the facilitator of FIG. 1 when the activity information is received. The figure which shows the modification of FIG. 8A. The flowchart which shows an example of the activity information output processing executed by the conference support apparatus which concerns on embodiment of this invention. The figure which shows an example of the estimation result of the individual emotion estimation part and the whole emotion estimation part of FIG. The figure which shows an example of the display screen of the terminal apparatus used by the facilitator of FIG. 1 when the information of the whole emotion is received. The figure which shows the modification of FIG. 11A. The flowchart which shows an example of the whole emotion output processing executed by the conference support apparatus which concerns on embodiment of this invention.

Hereinafter, an embodiment of the present invention will be described with reference to FIGS. 1 to 12. The meeting support device according to the embodiment of the present invention supports the progress of a meeting such as an in-house meeting for generating a new idea. More specifically, it indirectly supports the progress of the meeting by supporting the facilitator who manages the progress of the meeting, or directly supports the progress of the meeting by taking the role of the facilitator.

The conference is held to generate ideas for new products and services, and is composed of multiple participants with diverse knowledge, such as experts in each field of the development department. Such meetings proceed by having multiple participants exchange and discuss each other's opinions, but when the progress of the meeting is left to the autonomy of the participants, certain participants continue to speak or speak. Inappropriate situations may occur, such as not doing so or being unable to speak up. When such an inappropriate situation occurs, it becomes difficult to generate good ideas that reflect various findings.

The facilitator is a person in charge such as a counselor who has a psychological specialty who witnesses a meeting in the development department, and is not involved in the discussion of the meeting itself, but manages only the progress of the meeting. More specifically, the progress of the meeting is promoted by observing the progress of the meeting and performing "intervention" to encourage the participants to speak as necessary so that an inappropriate situation does not occur.

If the facilitator observes each participant's speech volume, facial expressions, movements, etc., identifies the appropriate intervention target and intervention timing, and intervenes in an appropriate manner, the progress of the meeting will be promoted, but if the intervention is inappropriate. On the contrary, the progress of the meeting may be hindered. Therefore, in the present embodiment, the conference support device is configured as follows so that the smooth progress of the conference can be supported by supporting the intervention in an appropriate manner.

FIG. 1 is a block diagram schematically showing an example of an overall configuration of a conference support system 100 having a conference support device according to an embodiment of the present invention. As shown in FIG. 1, the conference support system 100 includes terminal devices 10a to 10d used by each of a plurality of participants Pa to Pd of the conference, terminal devices 10f used by the facilitator Pf, and embodiments of the present invention. It has a server 20 that constitutes the conference support device. Each terminal device 10 (10a to 10d, 10f) is composed of a general user terminal such as a personal computer or a smartphone.

Each terminal device 10 and the server 20 are connected to a communication network 30 including a public wireless communication network represented by an Internet network, a mobile phone network, or the like, and are configured to be able to communicate with each other via the communication network 30. The communication network 30 includes not only public wireless communication networks but also closed communication networks provided for each predetermined management area, such as wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark) and the like. ..

Note that FIG. 1 shows the configuration of a Web conference in which each participant Pa to Pd and the facilitator Pf participate individually. In the conference, a plurality of participants Pa to Pd and a part of the facilitator Pf participate in a group. It may be a conference other than a video conference or an online conference. In this case, a plurality of participants Pa to Pd and the facilitator Pf or a part thereof may share a single terminal device 10.

FIG. 2 is a block diagram schematically showing an example of a main part configuration of each terminal device 10. As shown in FIG. 2, the terminal device 10 includes a calculation unit 11 such as a CPU, a storage unit 12 such as a ROM and a RAM, and a computer having a peripheral circuit such as an I / O interface. The terminal device 10 further includes a microphone 13 connected to a computer, a camera 14, an input unit 15, a speaker 16, and a display unit 17. The storage unit 12 stores programs executed by the calculation unit 11, identification information of users (participants Pa to Pd, facilitator Pf) who use the terminal device 10, and the like.

The microphone 13 acquires the voice emitted by the user of the terminal device 10. The audio signal from the microphone 13 is transmitted to the terminal device 10 used by the other participants Pa to Pd and the facilitator Pf via the communication network 30 by processing by the arithmetic unit 11. Further, the audio signal from the microphone 13 is temporarily stored in the storage unit 12, and is processed by the calculation unit 11 as audio information for each user (participants Pa to Pd, facilitator Pf) together with the identification information to be processed by the communication network 30. It is transmitted to the server 20 at a predetermined cycle via the above. The microphone 13 can also receive voice operation inputs of various commands by the facilitator Pf. The command information from the facilitator Pf input via the microphone 13 is transmitted to the server 20 via the communication network 30 by processing by the arithmetic unit 11.

The camera 14 has an image pickup element such as a CCD sensor or a CMOS sensor, and captures an image of the user's face of the terminal device 10. The image signal from the camera 14 is transmitted to the terminal device 10 used by the other participants Pa to Pd and the facilitator Pf via the communication network 30 by processing by the arithmetic unit 11. Further, the image signal from the camera 14 is temporarily stored in the storage unit 12, and is processed by the calculation unit 11 as image information for each user (participants Pa to Pd, facilitator Pf) together with the identification information to be processed by the communication network 30. It is transmitted to the server 20 at a predetermined cycle via the above. The camera 14 can also receive operation inputs of various commands by the gesture of the facilitator Pf. The command information from the facilitator Pf input via the camera 14 is transmitted to the server 20 via the communication network 30 by processing by the arithmetic unit 11.

The input unit 15 is composed of a keyboard, a mouse, a touch panel, and the like, and receives operation input by the user. For example, it accepts an operation input for turning on or off the microphone 13 or the camera 14. It also accepts operation inputs for various commands from the facilitator Pf. Information of various commands from the facilitator Pf input via the input unit 15 is transmitted to the server 20 via the communication network 30 by processing by the arithmetic unit 11.

The speaker 16 outputs voice in response to a control signal from the calculation unit 11. For example, the voice emitted by another participant Pa to Pd or the facilitator Pf is output based on the voice information acquired via the communication network 30. The user of each terminal device 10 can interact with other participants Pa to Pd or the facilitator Pf via the microphone 13 and the speaker 16.

The display unit 17 displays an image in response to a control signal from the calculation unit 11. For example, an image of another participant Pa to Pd or a facilitator Pf is displayed based on the image information acquired via the communication network 30. The user of each terminal device 10 can recognize the facial expressions and actions of the other participants Pa to Pd or the facilitator Pf via the display unit 17.

FIG. 3 is a block diagram schematically showing an example of a main configuration of a server 20 constituting the conference support device according to the embodiment of the present invention. As shown in FIG. 3, the server 20 includes a calculation unit 21 such as a CPU, a storage unit 22 such as a ROM and a RAM, and a computer having a peripheral circuit such as an I / O interface. Although a single server 20 is shown in FIG. 1, the server 20 may be configured by a plurality of servers having the functions shown in FIG. 3 distributed.

The storage unit 22 contains a threshold used in a conference support program or program executed by the calculation unit 21, participants Pa to Pd acquired from each terminal device 10, audio information and image information for each facilitator Pf, and a calculation unit 21. The calculation result by is stored. The calculation unit 21 has an information acquisition unit 23, a remark determination unit 24, an individual emotion estimation unit 25, an overall emotion estimation unit 26, an activity calculation unit 27, and an intervention recognition unit 28 as functional configurations carried by the CPU. And an output unit 29.

The information acquisition unit 23 acquires voice information and image information for each participant Pa to Pd and facilitator Pf, that is, biometric information from each terminal device 10 via the communication network 30, and various commands from the terminal device 10f of the facilitator Pf. Get information about. The biological information of each participant Pa to Pd acquired by the information acquisition unit 23 is associated with the identification information of each participant Pa to Pd and stored in the storage unit 22.

The speech determination unit 24 determines whether or not each participant Pa to Pd is speaking based on the voice information acquired by the information acquisition unit 23. More specifically, by determining whether or not an audio signal is input via the microphone 13 of each terminal device 10 based on the audio information for each participant Pa to Pd, a plurality of participants Pa to Pd Judge whether or not there is a statement for each of. The determination result by the speech determination unit 24 is associated with the identification information of each participant Pa to Pd, and is stored in the storage unit 22 as time-series information.

The personal emotion estimation unit 25 is based on the image information acquired by the information acquisition unit 23, and from the facial expressions (mouth corners, etc.) of each participant Pa to Pd, the emotions (individual emotions) ea to ed of each participant Pa to Pd. Estimates which of the preset multiple types of emotions belongs to. For example, using the emotion circle of Plutchik, we classify eight basic emotions (joy, trust, fear, surprise, sadness, disgust, anger, expectation) and applied emotions by combining two adjacent emotions. Apply to any pattern of the ring.

The personal emotion estimation unit 25 quantifies the personal emotions ea to ed according to the fit to the emotion circle. For example, favorable emotions such as joy (positive emotions) are represented by positive values, and unfavorable emotions such as sadness (negative emotions) are represented by negative values. In this case, for example, the stronger the emotion (the inside of the circle of emotion), the larger the value representing the emotion (absolute value). The estimation result by the personal emotion estimation unit 25 is associated with the identification information of each participant Pa to Pd, and is stored in the storage unit 22 as time-series information.

[Target of intervention and timing of intervention]
FIG. 4 is a diagram showing an example of the determination result of the remark determination unit 24 and the estimation result of the personal emotion estimation unit 25, and is an example of the presence / absence of remarks by the participants Pa to Pd and the change of the personal emotions ea to ed during the meeting. Is shown. In the example of FIG. 4, only the participants Pa, Pb, and Pd speak, and the participant Pc does not speak. If such a situation continues, discussions will proceed without reflecting the opinions of the participants Pc, so it is necessary to intervene to encourage the participants Pc to speak. On the other hand, if the intervention is performed at a stage where the participant Pc cannot speak yet, such as when he / she is thinking about the content of the opinion to be expressed or when there is no opinion to speak, it simply hinders the flow of discussion. In addition, the intervened participant Pc contracts, making it difficult to speak.

As shown in FIG. 4, if the participant Pc has never spoken at times t1 to t2 and has not spoken continuously for a predetermined time T1 (for example, 5 minutes) or more, the participant Pc should intervene. It is regarded as a candidate (intervention candidate) of the target person (intervention target person). Furthermore, at time t3, when the personal emotion ec of the participant Pc changes during the remarks of the other participant Pa (the positive emotion increases from "+1" to "+3" in the figure), some opinion is given to the participant Pc. It is considered that it has occurred, and the participant Pc is the target of intervention. In other words, if the individual feelings change in response to the remarks of other participants, it is highly probable that some kind of opinion has arisen. It is easy for the subject to speak and does not interfere with the flow of discussion. It should be noted that the change in personal feelings when no one is speaking is likely to be caused by personal matters unrelated to the content of the statement.

When the output unit 29 determines that another participant Pa is speaking by the speech determination unit 24, the individual emotion ec of the participant Pc (intervention candidate) estimated by the individual emotion estimation unit 25 changes. , Outputs a command (intervention command) for encouraging participants to speak with the participant Pc as the intervention target. For example, identification information for identifying an intervention target person and information on changes in personal emotions are transmitted to the terminal device 10f of the facilitator Pf via the communication network 30. If the individual emotions of a plurality of intervention candidates who do not speak continuously for a predetermined time T1 or more change at the same time, the intervention candidate who does not speak for a longer time or the change in individual emotions is larger. An intervention command is output with the intervention candidate as the intervention target.

FIG. 5A is a diagram showing an example of a display screen of the terminal device 10f when an intervention command is received, and is displayed on the display unit 17 of the terminal device 10f used by the facilitator Pf in response to the intervention command from the server 20. An example of the display screen is shown. As shown in FIG. 5A, when an intervention command is transmitted from the server 20 to the terminal device 10f, it is specified by the intervention command among a plurality of participants Pa to Pd displayed on the display unit 17 (4 in the figure). Participant Pc is displayed with more emphasis than other participants Pa, Pb, Pd.

In this way, as soon as the personal emotion ec of the participant Pc who has not spoken for a long time changes during the speech of the participant Pa, the facilitator Pf is notified that it is time to intervene in the participant Pc. As a result, for example, the facilitator Pf can prompt the participant Pc to speak before the participant Pa finishes speaking and the other participants Pb and Pd speak, so that the participant Pc can smoothly join the discussion. Can be done. Such notifications are given not only when the facilitator Pf is unfamiliar and unable to determine the timing of intervention, but also when the facilitator Pf cannot observe the facial expressions and movements of the participants Pc due to the full screen display of materials and the like. The facilitator Pf can be supported.

Further, the display unit 17 of the terminal device 10f displays information on the change in personal emotion ec for the participant Pc specified by the intervention command. Information on changes in personal emotion ec is shown, for example, by using facial expression icons indicating the type and intensity of emotions. In the example of FIG. 5A, the change of the individual emotion ec of the participant Pc from a weak positive emotion “+1” to a slightly strong positive emotion “+3” corresponds to two facial expression icons before and after the change. Is shown using.

By displaying the information on the change in the personal emotion ec in this way, the facilitator Pf can intervene after grasping the content of the change in the personal emotion ec of the participant Pc, so that it is appropriate according to the psychological state of the participant Pc. It is possible to encourage remarks by speaking out. Further, since the information displayed on the display unit 17 is minimized such as facial expression icons, the facilitator Pf does not interfere with the observation of the facial expressions and movements of the participants Pa to Pd.

FIG. 5B is a diagram showing a modified example of the display screen of the terminal device 10 when the intervention command is received, and is a diagram showing the terminal devices 10a to used by the participants Pa to Pd and the facilitator Pf in response to the intervention command from the server 20. An example of the display screen displayed on the display unit 17 of 10d and 10f is shown. As shown in FIG. 5B, when an intervention command is transmitted from the server 20 to each terminal device 10, it is specified by the intervention command among a plurality of participants Pa to Pd displayed on the display unit 17 (4 in the figure). Participant Pc is displayed with more emphasis than other participants Pa, Pb, Pd. For example, the image of the participant Pc is displayed larger than the images of the other participants Pa, Pb, Pd.

In this way, since the next speaker is clearly shown not only to the facilitator Pf and the intervention target, but also to all the participants Pa to Pd, it is possible to encourage the intervention target to speak in a more natural manner. When notifying the entire participants Pa to Pd in this way, not only the intervention by the facilitator Pf is supported, but also the conference support device plays the role of the facilitator to directly support the progress of the conference. Can be done.

FIG. 6 is a flowchart showing an example of an intervention command output process executed by the calculation unit 21 of the server 20 according to the conference support program stored in advance in the storage unit 22 of the server 20. The process shown in this flowchart is started when, for example, a start command from the facilitator Pf is input, and is repeatedly executed at a predetermined cycle until the end command is input.

First, in step S1, biometric information for each participant Pa to Pd is acquired. Next, in step S2, it is determined whether or not there is a statement for each participant Pa to Pd based on the voice information acquired in step S1. Next, in step S3, the individual emotions ea to ed for each participant Pa to Pd are estimated based on the image information acquired in step S1. Next, in step S4, based on the determination result of step S2, it is continuously determined whether or not there is an intervention candidate who has not spoken for a predetermined time T1 or more. If it is affirmed in step S4, the process proceeds to step S5, and if it is denied, the process ends. In step S5, it is determined whether or not there is a change in the emotion of the intervention candidate during the remarks of other participants based on the determination result of step S2 and the estimation result of step S3. If affirmed in step S5, the process proceeds to step S6, and if denied, the process ends. In step S6, an intervention command for encouraging the intervention candidate to speak is transmitted to the terminal device 10.

In this way, in order to identify the participants who have not spoken for a long time (steps S1, S2, S4) and output the intervention command at the timing when the individual emotions change in response to the comments of other participants (steps S1 to S1). S3, S5, S6), intervention can be performed at a timing that makes it easy to speak. As a result, it is possible to effectively intervene without obstructing the flow of discussion, and it is possible to support the smooth progress of the meeting.

[Activity of discussion and effect of intervention]
By knowing whether or not the intervention performed by the facilitator Pf was effective, the facilitator Pf can review the mode of the intervention from the next time onward as necessary. By feeding back the effect of the intervention by the facilitator Pf to the facilitator Pf, it is possible to support the effective intervention by the facilitator Pf and support the smooth progress of the meeting.

The activity calculation unit 27 calculates the activity a indicating the degree of activation of the discussion in the conference based on the biological information acquired by the information acquisition unit 23. For example, the frequency f of remarks by the participants Pa to Pd is calculated based on the voice information acquired by the information acquisition unit 23, and the activity a is calculated based on the calculated frequency f of remarks.

FIG. 7A is a diagram showing an example of the calculation result of the activity calculation unit 27, and shows an example of the activity a based on the frequency f of remarks. As shown in FIG. 7A, when the intervention by the facilitator Pf is performed at time t6, the frequency f of remarks at times t5 to t6 before the intervention, the predetermined period T2 (for example, about 5 minutes) after the intervention, and the time t6 to The frequency f of remarks at t7 is calculated. In the example of FIG. 7A, there are two remarks in 3 minutes from time t5 to t6, the frequency f of remarks before the intervention is calculated as 0.7 times / minute, and 7 times in 5 minutes from time t6 to t7. There is a statement, and the frequency f of the statement after the intervention is calculated as 1.4 times / minute. In this case, the activity a before the intervention is calculated as "0.7" and the activity a after the intervention is calculated as "1.4".

The activity calculation unit 27 can also calculate the activity a based on the individual emotions ea to ed of the participants Pa to Pd estimated by the individual emotion estimation unit 25. For example, the average value of the individual emotions ea to ed of the participants Pa to Pd is calculated as the activity a.

FIG. 7B is a diagram showing an example of the calculation result of the activity calculation unit 27, and shows an example of the activity a based on the personal emotions ea to ed. In the example of FIG. 7B, when the intervention is performed at time t6, the individual emotion ed of the participant Pd changes from the negative emotion “-3” to the positive emotion “+3” at time t8, and the participant Pc at time t9. The personal emotion ec changes from a weak positive emotion "+1" to a strong positive emotion "+5". In this case, the average value of the individual emotions ea to ed of the participants Pa to Pd, that is, the activity a is "+1.5" at times t5 to t8, "+3" at times t8 to t9, and "+3" at times t9 to t7. It is calculated as +4 ".

The activity calculation unit 27 may calculate the activity a based on both the frequency f of the participants Pa to Pd and the personal emotions ea to ed. In this case, for example, the activity a based on the frequency f of speech and the activity a based on personal emotions ea to ed are converted into numerical values af and ae in which the minimum and maximum values are unified, and the converted numerical values af, The average value of ae is calculated as the activity a.

The intervention recognition unit 28 recognizes the timing of intervention by the facilitator Pf (time t6 in FIGS. 7A and 7B). More specifically, the facilitator Pf intervenes by determining whether or not an audio signal is input via the microphone 13 of the terminal device 10f based on the audio information of the facilitator Pf acquired by the information acquisition unit 23. Recognize when to do. In this case, when the facilitator Pf intervenes, the timing of the intervention is automatically recognized.

The intervention recognition unit 28 may recognize the timing of the intervention based on the conscious operation input by the facilitator Pf, that is, the operation input predetermined for requesting the start of the intervention and the feedback of the effect of the intervention. .. More specifically, based on the information of various commands from the facilitator Pf acquired by the information acquisition unit 23, the operation via the input unit 15 of the terminal device 10f, the voice operation via the microphone 13, and the camera 14 are performed. Judges whether or not an operation by a gesture through is input. When recognizing the timing of intervention triggered by such conscious operation input, for example, the effect of intervention is not erroneously calculated triggered by a statement other than intervention, and accurate intervention is performed according to the intention of the facilitator Pf. The effect of can be fed back.

When the intervention recognition unit 28 recognizes the timing of the intervention, the output unit 29 has an activity indicating a change in the activity a based on the activity a calculated by the activity calculation unit 27 within the subsequent predetermined period T2. The degree information is generated and transmitted to the terminal device 10f via the communication network 30. For example, as shown in FIG. 7A, when the activity a before the intervention is calculated as “0.7” and the activity a in the predetermined period T2 after the intervention is calculated as “1.4” by the activity calculation unit 27, Activity information indicating that the activity a is improved as an effect of the intervention is generated and output. Alternatively, as shown in FIG. 7B, when the activity a in the predetermined period T2 after the intervention is calculated by the activity calculation unit 27 as “+1.5”, “+3”, “+4”, the activity is active as the effect of the intervention. Generates and outputs activity information indicating the transition of degree a.

8A and 8B are diagrams showing an example of a display screen of the terminal device 10f used by the facilitator Pf when the activity information is received. In the example of FIG. 8A, as an effect of the intervention, an arrow indicating that the activity a is improved before and after the intervention is displayed in the center of the screen. In the example of FIG. 8B, the effect of the intervention is from “+1.5” corresponding to a positive emotion with a weak activity a to “+3” corresponding to a slightly strong positive emotion in a predetermined period T2 after the intervention, and a strong positive. A facial expression icon indicating that the emotion has changed to "+4" is displayed in the center of the screen. The effect of such an intervention is displayed after a predetermined period T2 has elapsed after the intervention recognition unit 28 recognizes the timing of the intervention, but when the transition of the activity a is displayed as shown in FIG. 8B. May be displayed continuously for a predetermined period T2 after the timing of the intervention is recognized.

FIG. 9 is a flowchart showing an example of activity information output processing executed by the calculation unit 21 of the server 20 according to the conference support program stored in advance in the storage unit 22 of the server 20. The process shown in this flowchart is started when, for example, a start command from the facilitator Pf is input, and is repeatedly executed at a predetermined cycle until the end command is input.

First, in step S1, biometric information for each participant Pa to Pd is acquired. Then, in step S7, the activity a is calculated based on the biological information acquired in step S1. Then, in step S8, it is determined whether or not the timing of the intervention by the facilitator Pf is recognized. If it is affirmed in step S8, the process proceeds to step S9, and if it is denied, the process ends. In step S9, activity information representing the change in activity a calculated in step S7 is generated and output to the terminal device 10f of the facilitator Pf. Then, in step S10, it is determined whether or not the predetermined period T2 has elapsed from the intervention timing recognized in step S8. If it is affirmed in step S10, the process ends, and if it is denied, the process returns to step S9.

In this way, the facilitator Pf itself recognizes the timing of the intervention by the facilitator Pf (step S8) and feeds back the change in the activity a before and after the intervention and the transition of the activity a after the intervention (steps S9 and S10). The effect of the intervention can be grasped. This can support effective intervention by the facilitator Pf and support the smooth progress of the meeting.

[Forming the overall atmosphere of the meeting]
In order for the conference to proceed smoothly and to reach a conclusion that reflects the opinions of all participants, it is necessary to have an accurate grasp of the overall atmosphere of the conference. The overall atmosphere of the conference is formed by the general agreement of the individual feelings of all the participants. When each participant speaks frankly and exchanges opinions sufficiently, the individual feelings of all the participants are in agreement, and various findings are reflected by reaching a conclusion in such a unanimous atmosphere. It is possible to generate good ideas that have been made.

The overall emotion estimation unit 26 is based on the individual emotions ea to ed of each participant Pa to Pd estimated by the individual emotion estimation unit 25, and the emotions (overall emotions) e of a plurality of participants Pa to Pd as a whole, that is, a conference. Estimate the overall atmosphere. More specifically, it is determined by the individual emotion estimation unit 25 whether or not the ratio of the participants Pa to Pd estimated to belong to the same type of emotion is a predetermined ratio (for example, 70%) or more, and a predetermined ratio is determined. If it is determined that the ratio is equal to or higher than the ratio, the emotion is estimated as the total emotion e.

FIG. 10 is a diagram showing an example of estimation results of the individual emotion estimation unit 25 and the overall emotion estimation unit 26. As shown in FIG. 10, for example, immediately after the start of the meeting, the individual emotions ea to ed of each participant Pa to Pd hardly match, and the overall atmosphere of the meeting is not formed. At such a stage, for example, even if the overall emotion e is estimated as the average value of the individual emotions ea to ed, the overall atmosphere of the conference cannot be accurately expressed.

When the meeting proceeds smoothly, all participants Pa to Pd speak frankly, and mutual opinions are sufficiently exchanged, the individual emotions ea to ed of all participants Pa to Pd are full as positive emotions. In agreement, the total emotion e is estimated as a positive emotion by the total emotion estimation unit 26. Further, the average value of the individual emotions ea to ed is calculated as the absolute value of the overall emotion e.

On the other hand, when the meeting progresses and most of the participants Pa to Pc speak frankly and exchange opinions, but some participants Pd are not able to speak sufficiently or can not speak frankly. Is that the individual emotion ed of some participants Pd does not match the overall emotion e. In this case, the positive emotion to which the individual emotions ea to ec of the participants Pa to Pc of a predetermined ratio or more belong is estimated as the total emotion e, and the average value of the individual emotions ea to ec of the participants Pa to Pc is the total emotion e. Calculated as an absolute value.

When the total emotion e is estimated by the total emotion estimation unit 26, the output unit 29 transfers the information of the participant Pd whose estimated total emotion e and individual emotion ed do not match the total emotion e via the communication network 30. It is transmitted to the terminal device 10f.

11A and 11B are diagrams showing an example of a display screen of the terminal device 10f used by the facilitator Pf when the information of the whole emotion e is received. In the example of FIG. 11A, the facial expression icon corresponding to the slightly strong positive emotion “+4” estimated as the unanimous overall emotion e in FIG. 10 is displayed in the center of the screen.

In the example of FIG. 11B, the facial expression icon corresponding to the slightly strong positive emotion “+3” estimated as the total emotion e when the individual emotion ed is inconsistent with the total emotion e in FIG. 10 is in the center of the screen. Is displayed in. In addition, the participant Pd whose individual emotion ed does not match the overall emotion e is displayed more emphasized than the other participants Pa to Pc, and is estimated as the individual emotion ed of the participant Pd, which is a slightly strong negative. A facial expression icon corresponding to the emotion "-3" is displayed.

FIG. 12 is a flowchart showing an example of the overall emotion output process executed by the calculation unit 21 of the server 20 according to the conference support program stored in advance in the storage unit 22 of the server 20. The process shown in this flowchart is started when, for example, a start command from the facilitator Pf is input, and is repeatedly executed at a predetermined cycle until the end command is input.

First, in step S1, biometric information for each participant Pa to Pd is acquired. Next, in step S3, the individual emotions ea to ed for each participant Pa to Pd are estimated based on the image information acquired in step S1. Next, in step S11, it is determined whether or not the individual emotions ea to ed for each participant Pa to Pd estimated in step S3 generally match. If affirmed in step S11, the process proceeds to step S12, and if denied, the process ends. In step S12, the emotions determined to be substantially matched in step S11 are estimated as the total emotion e of all the participants Pa to Pd, and the average value of the matched individual emotions ea to ec is calculated as the absolute value of the total emotion e. ..

Next, in step S13, it is determined whether or not there are participants Pa to Pd whose individual emotions ea to ed estimated in step S3 do not match the total emotions e estimated in step S12. If affirmed in step S13, the process proceeds to step S14, and the identification information and the mismatched individual emotions of the participants whose individual emotions are determined not to match the total emotions e in step S12 and the total emotion e estimated in step S12. Is output to the terminal device 10f of the facilitator Pf. If it is denied in step S13, the process proceeds to step S15, and only the unanimous whole emotion e estimated in step S12 is output to the terminal device 10f of the facilitator Pf.

In this way, in order to estimate and present the overall emotion e when the individual emotions ea to ed of all the participants Pa to Pd generally match (steps S11, S12, S14, S15), the overall atmosphere of the meeting by the facilitator Pf. Can help you understand. This allows the facilitator Pf to accurately grasp the overall atmosphere of the meeting and the nature of that atmosphere, for example, to appropriately determine the need for intervention, and the meeting. Can proceed smoothly. Further, by presenting the information of the participant Pd having the individual emotion ed different from the whole emotion e (step S14), it is possible to support the appropriate intervention by the facilitator Pf.

According to this embodiment, the following effects can be obtained.
(1) The server 20 supports the progress of a conference in which a plurality of participants Pa to Pd participate. The server 20 is an information acquisition unit 23 that acquires the biometric information of each of the plurality of participants Pa to Pd, and each individual of the plurality of participants Pa to Pd based on the biometric information acquired by the information acquisition unit 23. Participants Pa ~ estimated to belong to the same type of emotion by the individual emotion estimation unit 25 that estimates which of the plurality of preset types of emotions the emotions ea to ed belong to, and the individual emotion estimation unit 25. It is determined whether or not the ratio of Pd is a predetermined ratio (for example, 70%) or more, and if it is determined that the ratio is equal to or more than a predetermined ratio, the emotion is estimated as the total emotion e of the entire plurality of participants Pa to Pd. It includes an overall emotion estimation unit 26 and an output unit 29 that outputs the overall emotion e estimated by the overall emotion estimation unit 26 (FIG. 3).

In this way, the individual emotions ea to ed of all the participants Pa to Pd are almost the same, and the overall atmosphere of the conference is being formed, and by presenting the nature of the atmosphere, the smooth progress of the conference. Can be supported. For example, the facilitator Pf can adequately determine the need for intervention.

(2) The information acquisition unit 23 acquires image information as biometric information of each of the plurality of participants Pa to Pd. The personal emotion estimation unit 25 estimates the individual emotions ea to ed of each of the plurality of participants Pa to Pd based on the image information acquired by the information acquisition unit 23. Since the biometric information of the participants Pa to Pd is acquired by using the camera 14 of a general terminal device 10 such as a personal computer or a smartphone, it can be suitably applied to an online conference.

(3) The output unit 29 further outputs information on the participant Pd whose type of individual emotions ea to ed estimated by the individual emotion estimation unit 25 is different from the type of total emotion e estimated by the overall emotion estimation unit 26. (Fig. 11B). By grasping the participant Pd having the whole emotion e and the different kind of individual emotion ed, the facilitator Pf can intervene in the appropriate participant Pd.

In the above embodiment, an example of acquiring the biometric information of each participant Pa to Pd via the microphone 13 and the camera 14 of the terminal devices 10a to 10d has been described, but the biometric information of each of a plurality of participants is acquired. The information acquisition unit is not limited to such a unit. It is not limited to the one attached to the terminal device, and may be an independent microphone or camera. It is not limited to the one provided for each participant, but may be shared by a plurality of participants. In this case, for example, the voice information for each participant may be acquired by using a microphone having directivity and the sensitivity for acquiring the voice differs depending on the direction, and the individual may be recognized based on the voice information. Further, a plurality of participants may be photographed by a wide-angle camera and the individual may be recognized based on the image information.

In the above embodiment, an example of determining the presence or absence of a participant's speech based on voice information has been described, but the speech determination unit that determines the presence or absence of a participant's speech based on biological information is such. Not limited to things. For example, it may be determined whether or not there is a sign language statement based on the image information.

In the above embodiment, an example of estimating the individual emotions ea to ed from the facial expressions of the participants Pa to Pd based on the image information has been described, but the emotion estimation unit for estimating the emotions of the participants based on the biological information is this. Not limited to such things. For example, personal emotions may be estimated based on movements in addition to facial expressions. Individual emotions may be estimated by using biological information such as heart rate, facial temperature, and voice information for each participant instead of or in addition to the image information. Further, although the individual emotions are applied to the patterns classified by the circle of emotions and quantified, they may be applied to other classified emotion patterns and quantified.

In the above embodiment, an example of calculating the activity a based on the frequency f of the remarks of the participants Pa to Pd or the individual emotions ea to ed has been described, but the activity indicating the degree of activation of the discussion in the meeting is calculated. The activity calculation unit is not limited to such a unit. For example, the activity may be calculated by detecting a specific keyword based on the voice information of the meeting and estimating the degree of progress of the discussion, the atmosphere, and the like.

In the above embodiment, an example of determining whether the individual emotions ea to ed of the participants Pa to Pd are positive emotions or negative emotions has been described with reference to FIG. The personal emotion estimation unit that estimates which of the types of emotions belongs to is not limited to such. For example, it may be classified into three or more emotions such as eight basic emotions. In this case, the predetermined ratio for determining whether or not the individual emotions are substantially the same is a ratio lower than 70% (for example, 60%).

Regarding the conference support method that supports the progress of a conference in which multiple participants participate, the biometric information of each of the multiple participants is acquired, and based on the acquired biometric information, the individual emotions of each of the multiple participants are expressed. It estimates which of the plurality of preset types of emotions it belongs to, determines whether or not the proportion of participants estimated to belong to the same type of emotion is equal to or greater than the predetermined proportion, and the proportion is equal to or greater than the predetermined proportion. If it is determined to be present, any other configuration may be used as long as the emotion is estimated as the total emotion of a plurality of participants and the estimated total emotion is output.

Regarding a conference support program that supports the progress of a conference in which a plurality of participants participate, based on the information acquisition step S1 for acquiring the biometric information of each of the plurality of participants and the biometric information acquired in the information acquisition step S1. It is presumed that each of the individual emotions of the plurality of participants belongs to the same type of emotion in the individual emotion estimation step S3 for estimating which of the plurality of preset types of emotions belongs to and the individual emotion estimation step S3. It is determined whether or not the ratio of the participants is equal to or higher than the predetermined ratio, and if it is determined to be equal to or higher than the predetermined ratio, the emotion is estimated as the total emotion of the entire plurality of participants. And, if the whole emotion output steps S14 and S15 for outputting the whole emotion estimated in the whole emotion estimation steps S11 to S12 are included, the process executed by the calculation unit 21 of the server 20 is shown in FIG. Not limited to things. Therefore, the configuration of the conference support program that causes the computer to execute these procedures is not limited to the above. In the above embodiment, the conference support program is stored in the storage unit 22 of the server 20, but it may be stored in an external storage medium such as a flash memory. It is also possible to acquire a conference support program via the communication network 30.

The above description is merely an example, and the present invention is not limited to the above-described embodiments and modifications as long as the features of the present invention are not impaired. It is also possible to arbitrarily combine one or a plurality of the above-described embodiments and the modified examples, and it is also possible to combine the modified examples.

10 (10a-10d, 10f) Terminal device, 11 arithmetic unit, 12 storage unit, 13 microphone, 14 camera, 15 input unit, 16 speaker, 17 display unit, 20 server, 21 arithmetic unit, 22 storage unit, 23 information acquisition Department, 24 speech judgment unit, 25 individual emotion estimation unit, 26 overall emotion estimation unit, 27 activity calculation unit, 28 intervention recognition unit, 29 output unit, 30 communication network, 100 conference support system

Claims (5)

A conference support device that supports the progress of a conference in which multiple participants participate.
An information acquisition unit that acquires biometric information of each of the plurality of participants,
An individual emotion estimation unit that estimates which of a plurality of preset types of emotions each of the plurality of participants belongs to based on the biological information acquired by the information acquisition unit.
The individual emotion estimation unit determines whether or not the proportion of participants estimated to belong to the same type of emotion is equal to or greater than a predetermined proportion, and if it is determined to be equal to or greater than a predetermined proportion, the plurality of emotions are said to be present. The overall emotion estimation unit, which estimates the overall emotions of all the participants,
A conference support device including an output unit that outputs an overall emotion estimated by the overall emotion estimation unit. In the conference support device according to claim 1,
The information acquisition unit acquires image information as biometric information of each of the plurality of participants, and obtains image information.
The personal emotion estimation unit is a conference support device that estimates the individual emotions of each of the plurality of participants based on the image information acquired by the information acquisition unit. In the conference support device according to claim 1 or 2.
The output unit is further characterized in that the information of the participants whose type of the individual emotion estimated by the individual emotion estimation unit is different from the type of the total emotion estimated by the total emotion estimation unit is output. Conference support device. It is a conference support method that supports the progress of a conference in which multiple participants participate.
An information acquisition step for acquiring biometric information of each of the plurality of participants,
Based on the biological information acquired in the information acquisition step, an individual emotion estimation step for estimating which of a plurality of preset types of emotions each individual emotion of the plurality of participants belongs to, and a personal emotion estimation step.
In the individual emotion estimation step, it is determined whether or not the proportion of participants estimated to belong to the same type of emotion is equal to or greater than a predetermined proportion, and if it is determined to be equal to or greater than a predetermined proportion, the plurality of emotions are divided. Overall emotion estimation step to estimate as the overall emotion of all participants in
A conference support method comprising executing an output step for outputting the total emotion estimated in the total emotion estimation step by a computer. A conference support program that supports the progress of a conference in which multiple participants participate.
An information acquisition step for acquiring biometric information of each of the plurality of participants,
Based on the biological information acquired in the information acquisition step, an individual emotion estimation step for estimating which of a plurality of preset types of emotions each individual emotion of the plurality of participants belongs to, and a personal emotion estimation step.
In the individual emotion estimation step, it is determined whether or not the proportion of participants estimated to belong to the same type of emotion is equal to or greater than a predetermined proportion, and if it is determined to be equal to or greater than a predetermined proportion, the plurality of emotions are divided. Overall emotion estimation step to estimate as the overall emotion of all participants in
A conference support program for causing a computer to execute an output step that outputs the total emotion estimated in the total emotion estimation step.

Images