JP2022139436A - Conference support device, conference support system, conference support method, and program - Google Patents

Abstract

To provide a conference support device, a conference support system, a conference support method, and a program that support progress of an online conference.SOLUTION: In a conference support device 1, conference support devices 10 to 30 that support progress of a conference each includes: an emotion analysis unit that detects emotional expression of conference attendee; a prediction unit that predicts whether the attendee has intention of speaking on the basis of the emotional expression; and a questions unit that asks the attendee for an opinion thereof if the attendee is predicted to have an intention to speak.SELECTED DRAWING: Figure 1

Description

The present invention relates to a meeting support device, a meeting support system, a meeting support method, and a program, and more particularly to a meeting support device, a meeting support system, a meeting support method, and a program that support the progress of an online meeting.

In recent years, telework, remote work, telecommuting, and other forms in which employees perform their own work have rapidly spread. Along with this, there are increasing opportunities for employees to hold online meetings (also called web meetings or remote meetings) through networks such as the Internet instead of gathering employees in conference rooms. In an online conference, an online conference support system (hereinafter simply referred to as a conference support system) is used that enables conference attendees to make video calls using terminals equipped with microphones and cameras.

Patent Literature 1 discloses a meeting support system for making online meetings more efficient and comfortable. Specifically, in Patent Document 1, when attendees enter an online conference, instead of the attendees entering a user ID and password, pre-registered face image data and It describes face authentication of attendees by matching with face image data. Furthermore, Patent Literature 1 describes generating an avatar that represents an attendee's alter ego in a virtual space, and adding an emotion icon that indicates the attendee's emotion or state to the avatar.

JP 2019-061594 A

Discussions can often be stagnant, such as when participants in a meeting speak at the same time, or when no one speaks for a long period of time. Especially in an online conference, there is also the problem that it is difficult for attendees to grasp the timing of their remarks. There is a demand for a conference support system that can facilitate an online conference to proceed smoothly.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and an object of the present invention is to support the progress of a conference.

In order to solve the above problems, a conference support apparatus according to one aspect of the present invention includes emotion analysis means for detecting emotional expressions of attendees in a conference, and analysis of intentions of remarks by the attendees based on the emotional expressions. and a questioning means for asking an opinion of the attendee when it is predicted that the attendee has an intention to speak.

A conference support method according to an aspect of the present invention detects an emotional expression of a conference attendee, predicts whether the attendee has an intention to speak based on the emotional expression, and predicts whether the attendee has an intention to speak. , ask the attendees for their opinion.

A program according to an aspect of the present invention includes processing for detecting an emotional expression of a conference attendee, processing for predicting whether the attendee has an intention to speak based on the emotional expression, processing for predicting whether the attendee has an intention to speak, If the intention is predicted, the computer is caused to execute a process of asking the attendees for their opinions.

According to the aspect described above, it is possible to assist progress of the conference.

1 is a diagram schematically showing an example of configuration of a conference support system to which conference support devices according to Embodiments 1 to 3 can be applied; FIG. 1 is a block diagram showing the configuration of a conference support device according to Embodiment 1 or 2; FIG. 4 is a flow chart showing the operation of the conference support device according to the first embodiment; 9 is a flow chart showing the operation of the conference support device according to the second embodiment; 11 is a block diagram showing the configuration of a conference support device according to Embodiment 3; FIG. 11 is a flow chart showing the operation of the conference support device according to the third embodiment; 1 is an example of a database for predictor retraining; 2 is a block diagram showing a hardware configuration of a conference support device according to Embodiments 1-3; FIG.

(Meeting support system 1)
FIG. 1 shows a conference to which any one of conference support devices 10, 20, and 30 (hereinafter referred to as "conference support device 10 (20, 30)") according to Embodiments 1 to 3, which will be described later, can be applied. 1 is a diagram schematically showing an example of a configuration of a support system 1; FIG. As shown in FIG. 1, the conference support system 1 includes conference support devices 10 (20, 30) and terminals 100X, 100Y, 100Z. The conference support system 1 further includes a biometric authentication device 200 , a speech recognition device 300 and an avatar generation device 400 . Note that the biometric authentication device 200 , the voice recognition device 300 and the avatar generation device 400 are not essential components of the conference support system 1 . The meeting support system 1 may not include some or all of the biometric authentication device 200 , the speech recognition device 300 and the avatar generation device 400 .

In the conference support system 1, terminals 100X, 100Y, and 100Z of attendees x, y, and z are respectively connected to a conference support device 10 (20, 30) via a wide area network such as the Internet. Conference support device 10 ( 20 , 30 ) is connected to biometric authentication device 200 , speech recognition device 300 , and avatar generation device 400 . Biometric authentication device 200, speech recognition device 300, and avatar generation device 400 may be configured in the same cloud server as meeting support device 10 (20, 30), or may be configured in the same cloud server as meeting support device 10 (20, 30). may be configured in different cloud servers.

Each attendee registers face image data showing his/her own face in the conference support system 1. - 特許庁Also, each attendee registers the data of their avatar, which is their alter ego in the online conference. For example, attendee x selects his portrait as his avatar. On the other hand, attendee y selects a favorite character as an avatar. Attendee z selects a caricature as an avatar. The avatar registered by each attendee in the conference support system 1 is displayed on the screen (not shown) of each attendee's terminal 100 during the online conference.

Also, each attendee registers his/her name or identification name such as a nickname in the conference support system 1 . Attendee names are used for voice recognition purposes and display names. For example, attendee x registers his name as a nickname. Attendee y registers a nickname such as a nickname. Attendee z registers his/her position (such as "manager"). An avatar of each attendee and a name of each attendee are displayed on a screen (not shown) of the terminal 100 .

Terminals 100X, 100Y, and 100Z are user devices such as PCs (Personal Computers) owned and managed by attendees x, y, and z. Terminals 100X, 100Y, and 100Z have the same configuration. One of the terminals 100X, 100Y, and 100Z is abbreviated as the terminal 100 below. As shown in FIG. 1, the terminal 100 includes a sound collector 110, an imaging device 120, and a conference application . The sound collection device 110 is a device such as a microphone for collecting speech of attendees. The sound collecting device 110 converts vibration information generated in response to speech of attendees into audio data. The imaging device 120 is equipment such as a camera for photographing the faces of attendees. The imaging device 120 generates time-series face image data of the attendee's face by photographing the attendee's face.

The conference application 130 transmits audio data generated by the sound collector 110 of the terminal 100 and time-series face image data generated by the imaging device 120 to the conference support device 10 (20, 30) via the network. send and receive between The conference application 130 controls the sound collector 110 of the terminal 100 to switch on/off the input of the sound collector 110 at arbitrary timing. Normally, the conference application 130 turns off the input of the sound collector 110 unless the attendee performs an operation to unmute themselves. The conference application 130 also generates various UIs (User Interfaces) such as avatars and a browser for displaying conference materials, and displays the UIs on a screen (not shown) of the terminal 100 .

When the reserved time for the conference comes or a predetermined time comes before it, the conference application 130 automatically starts up and causes the imaging device 120 to photograph a person in front of the terminal 100 . The conference application 130 transmits face image data acquired by the imaging device 120 to the conference support device 10 (20, 30). The conference application 130 receives the authentication result by the biometric authentication device 200 from the conference support device 10 (20, 30). If the biometric authentication is successful, the conference application 130 automatically enters the online conference. Note that if the biometric authentication fails, the conference application 130 may request the attendee to manually log in using the user ID and password.

The biometric authentication device 200 performs biometric authentication using face, fingerprint, iris, vein, voiceprint, or other biometric information. In one example, the biometric authentication device 200 acquires face image data generated by the imaging device 120 of the terminal 100 from the conference support device 10 (20, 30). The biometrics authentication device 200 collates the face image data acquired from the meeting support device 10 (20, 30) with the face image data linked to the identification information of the attendees and registered in a DB (Data Base). The biometric authentication device 200 determines whether or not the person in front of the terminal 100 is the attendee registered in the DB by a matching technique that compares features extracted from face image data. Then, the biometric device 200 notifies the meeting support device 10 (20, 30) of the success or failure of the authentication. Based on the authentication result obtained from the biometric device 200, the conference support device 10 (20, 30) determines whether or not the terminal 100 can connect to the conference support device 10 (20, 30).

The speech recognition device 300 converts speech data into text data representing language based on the acoustic model and the language model. The acoustic model represents acoustic features of speech data. On the other hand, the language model expresses restrictions on the order of phonemes. In one example, the speech recognition device 300 acquires speech data generated by the sound collector 110 of the terminal 100 from the conference support device 10 (20, 30). The speech recognition device 300 converts speech data acquired from the conference support device 10 (20, 30) into text data. Then, the speech recognition device 300 transmits the text data converted from the speech data to the conference support device 10 (20, 30).

The avatar generation device 400 generates an avatar that represents the alter ego of the attendee in the virtual space (here, the online conference). The avatar generation device 400 transmits data of the generated avatar to the conference support device 10 (20, 30). The conference support device 10 ( 20 , 30 ) transmits avatar data to the terminal 100 . The conference application 130 installed in the terminal 100 uses the avatar data received from the conference support device 10 (20, 30) to display the avatar on the screen (not shown) of the terminal 100. FIG.

In addition, the avatar generation device 400 may add an icon representing the facial expression or emotion of the attendee to the avatar, like the related technology described in Patent Document 1. In this case, the avatar generation device 400 uses time-series face image data and voice data transmitted from the terminal 100, like the conference support device 10 (20, 30) according to Embodiments 1 to 3, which will be described later. , to detect the emotional expressions of attendees. Then, avatar generation device 400 determines whether the attendee has an intention to speak based on the analysis result of the emotional expression.

The conference support device 10 (20, 30) analyzes the facial expressions and actions of each attendee during the online conference, and reflects the emotional expression based on the analysis result on each attendee's avatar. In particular, if another attendee expresses an emotion after a certain attendee speaks, the conference support device 10 (20, 30) reflects the emotional expression on the avatar. This function makes it possible to reproduce the atmosphere of a real group meeting in an online meeting.

Also, the conference support device 10 ( 20 , 30 ) analyzes the text data obtained from the speech recognition device 300 . Then, the conference support device 10 (20, 30) facilitates the online conference based on the analysis result of the text data. For example, if the name of an attendee is included in the text data, the conference support device 10 (20, 30) asks the attendee for their opinion.

In Embodiments 1 to 3 below, an example in which the conference support system 1 includes three terminals 100X, 100Y, and 100Z will be described. However, the conference support system 1 may include any number of terminals 100 equal to or greater than two.

[Embodiment 1]
Embodiment 1 will be described with reference to FIGS. 2 and 3. FIG.

(Meeting support device 10)
FIG. 2 is a block diagram showing the configuration of the conference support device 10 according to the first embodiment. As shown in FIG. 2, the conference support device 10 includes an emotion analysis section 11, a prediction section 12, and a question section 13. As shown in FIG.

The emotion analysis unit 11 detects the emotional expressions of the attendees of the conference. The emotion analysis unit 11 is an example of emotion analysis means. In one example, the emotion analysis unit 11 receives time-series face image data from the terminal 100 . The emotion analysis unit 11 detects changes in faces of attendees appearing in time-series face image data. For example, the emotion analysis unit 11 associates pixels one-to-one between time-series face image data based on the degree of similarity of pixel values. Then, the emotion analysis unit 11 calculates how the positional coordinates of the pixels corresponding to each part of the face, such as the mouth, nose, and eyes, change in the time-series face image data.

The emotion analysis unit 11 detects specific facial expression changes of attendees in time-series face image data as emotional expressions. For example, a first evaluation table (not shown) is prepared in advance, which associates changes in each part of the face with emotions such as sadness, anger, and joy. For example, in the first rating table, raising the eyebrows on the attendee's face is identified as an emote involving some emotion (eg, anger). In this case, the emotion analysis unit 11 detects, in the emotion time-series facial image data, that the eyebrows of the attendee's face are raised as one emotional expression.

Furthermore, the emotion analysis unit 11 may detect specific actions of attendees in time-series face image data as emotional expressions. For example, a second evaluation table is prepared in advance, in which actions of attendees are associated with emotions such as sadness, anger, and joy. For example, in the second rating table, the attendee's head shaking is identified as an emotional expression involving some emotion. In this case, the emotion analysis unit 11 detects the head shaking of the attendee as one emotional expression in the emotion time-series face image data.

The emotion analysis unit 11 outputs the data of the emotion expression detected from the time-series face image data as described above to the prediction unit 12 .

The prediction unit 12 predicts whether the attendee has the intention of speaking based on the emotional expression. The prediction unit 12 is an example of prediction means. In one example, the predictor 12 uses a predictor to predict whether an attendee has an intention to speak. Emotional expression data of past statements made by attendees is stored in a database (not shown). The predictor is obtained by deep learning a CNN (Convolutional Neural Network) using the emotional expression data of the attendees' past remarks stored in this database. The process of machine learning the predictor may be executed by a learning unit (FIG. 6) not shown (third embodiment).

The prediction unit 12 inputs the emotional expression data received from the emotion analysis unit 11 to a machine learning predictor. A machine-learned predictor extracts features from input emotional expression data. A machine-learning predictor predicts whether the attendee has the intention of speaking based on the extracted features, and outputs the prediction result. The prediction unit 12 outputs to the question unit 13 a prediction result indicating whether or not the attendee has an intention to speak.

When it is predicted that the attendee has an intention to speak, the question section 13 asks the attendee for their opinion. The question part 13 is an example of question means. In one example, the question unit 13 receives from the prediction unit 12 a prediction result indicating whether or not the attendee has an intention to speak. When it is predicted that the attendee has the intention of speaking, for example, the question unit 13 combines the identification name of the attendee registered in advance with a standard sentence such as "I would like to speak." , to generate a voice message. Then, the questioning unit 13 instructs the conference application 130 of the terminal 100 to output a voice message asking the opinion of the attendee from a speaker (not shown) of the terminal 100 .

Alternatively, the question unit 13 may instruct the conference application 130 to display a message asking the opinions of attendees on a screen (not shown) of the terminal 100 . In addition, the question unit 13 instructs the conference application 130 to turn on the input of the sound collector 110 of the terminal 100 (that is, cancel the mute).

(Operation of conference support device 10)
The operation of the conference support device 10 according to the first embodiment will be described with reference to FIG. FIG. 3 is a flow chart showing the flow of processing executed by each unit of the conference support device 10. As shown in FIG. Any one of attendees x, y, and z of the conference is hereinafter referred to as "attendee x(y,z)."

As shown in FIG. 3, the emotion analysis section 11 detects the emotional expression of the attendee x(y,z) (S1). The emotion analysis unit 11 outputs the detected emotion expression data to the prediction unit 12 .

The prediction unit 12 predicts whether the attendee x(y, z) has an intention to speak (S2). The prediction unit 12 outputs to the question unit 13 a prediction result indicating whether or not the attendee x(y, z) intends to speak.

If it is predicted that the attendee x(y, z) has no intention of speaking (No in S3), the operation of the conference support device 10 ends.

On the other hand, when it is predicted that the attendee x(y,z) intends to speak (Yes in S3), the question unit 13 asks the attendee x(y,z) for his opinion (S4). For example, the question unit 13 instructs the conference application 130 installed in the terminal 100 to output a voice message prompting the attendee x(y,z) to speak from a speaker (not shown) of the terminal 100 . Alternatively, the questioning unit 13 causes the screen (not shown) of the terminal 100 to display a message asking the attendee x(y,z) for an opinion.

Thus, the operation of the conference support device 10 ends.

(Modification)
In a modified example, the conference support device 10 may cause the screen of the terminal 100 to display the emotional expression of the attendee and the corresponding emotion (for example, joy, joy, anger, sadness). Alternatively, the conference support device 10 may display text data obtained by speech recognition on the screen of the terminal 100 . As a result, even hearing-impaired people can more actively participate in the conference.

In other variations, a specific gesture for unmuting may be defined. For example, when the attendee blinks his eyes three times or raises his hand, the conference support device 10 unmutes the attendee's terminal 100 . This allows attendees to cancel muting with little effort.

(Effect of this embodiment)
According to the configuration of this embodiment, the emotion analysis unit 11 detects the emotional expressions of the attendees of the conference. The prediction unit 12 predicts whether the attendee has the intention of speaking based on the emotional expression. When it is predicted that the attendee has an intention to speak, the question section 13 asks the attendee for their opinion. Attendees are encouraged to speak up by being asked for their opinions. As a result, smooth progress of the conference can be supported.

[Embodiment 2]
Embodiment 2 will be described with reference to FIGS. 4 and 5. FIG. In Embodiment 2, the conference support device 20 performs facilitation based on text data obtained from the speech recognition device 300 (FIG. 1). Regarding the configuration common between the second embodiment and the first embodiment, the description of the first embodiment is quoted in the second embodiment, and the description thereof is omitted.

The configuration of the conference support device 20 according to the second embodiment is the same as that of the conference support device 10 (FIG. 2) according to the first embodiment.

(Operation of conference support device 20)
The operation of the conference support device 20 according to the second embodiment will be described with reference to FIG. FIG. 4 is a flow chart showing the flow of processing executed by each unit of the conference support device 20. As shown in FIG. In the following, either of the attendees y and z of the conference will be referred to as "attendee y(z)". First, attendee x is speaking. The attendee y(z) turns off the input of the sound collector 110 of the terminal 100 of the attendee y(z).

As shown in FIG. 4, the question unit 13 analyzes the statement of the attendee x (S201). For example, the question unit 13 receives voice data from the terminal 100X of the attendee x. The question unit 13 transmits voice data to the voice recognition device 300 . After that, the question unit 13 receives text data that has undergone speech recognition from the speech recognition device 300 . When the questioning unit 13 receives null data notifying that the speech recognition is completed from the speech recognition device 300, the questioning unit 13 determines that the attendee x has finished speaking.

Next, the questioning unit 13 searches the text data received from the speech recognition device 300 for the identification name of the attendee y(z). If attendee x called the identification name of attendee y(z) (Yes in S202), the flow proceeds to step S206, which will be described later.

The questioning unit 13 asks the opinion of the attendee y(z) whose identification name is called by the attendee x (S206). At this time, the question unit 13 instructs the conference application 130 installed in the terminal 100 of the attendee y(z) whose identification name is called to turn on the input of the sound collector 110 (unmute). may On the other hand, if attendee x has not called the identification name of attendee y(z) (No in S202), the flow proceeds to step S203.

Subsequently, the emotion analysis unit 11 detects the emotional expression of the attendee y(z) (S203). The emotion analysis unit 11 outputs the detected emotion expression data to the prediction unit 12 .

The prediction unit 12 predicts whether the attendee y(z) intends to speak (S204). The prediction unit 12 outputs to the question unit 13 a prediction result indicating whether or not the attendee y(z) intends to speak.

If it is predicted that the attendee y(z) has no intention of speaking (No in S205), the operation of the conference support device 10 ends.

On the other hand, if attendee x called the identification name of attendee y(z) (Yes in S202), or if it was predicted that attendee y(z) intended to speak (Yes in S205) ), and the questioning unit 13 asks the attendee y(z) for an opinion (S206).

For example, the question unit 13 instructs the conference application 130 installed in the terminal 100 to output a voice message prompting the attendee y(z) to speak from a speaker (not shown) of the terminal 100 . Alternatively, the questioning unit 13 causes the screen (not shown) of the terminal 100 to display a message asking the attendee y(z) for an opinion.

Thus, the operation of the conference support device 20 ends.

(Modification)
In a modified example, the question unit 13 of the conference support device 20 determines that the speech of the attendee is delayed when text data is not received from the speech recognition device 300 for a certain period of time. In this case, the questioning unit 13 asks the person with the highest position among the attendees to speak. For example, since the identification name of attendee z is "Manager", the questioning unit 13 presumes that z has the right to decide on the meeting. Therefore, the questioning unit 13 outputs voice data of fixed sentences such as "Director (identification name), please make a resolution."

(Effect of this embodiment)
According to the configuration of this embodiment, the emotion analysis unit 11 detects the emotional expressions of the attendees of the conference. The prediction unit 12 predicts whether the attendee has the intention of speaking based on the emotional expression. When it is predicted that the attendee has an intention to speak, the question section 13 asks the attendee for their opinion. Attendees are encouraged to speak up by being asked for their opinions. As a result, smooth progress of the conference can be supported.

Furthermore, according to the configuration of this embodiment, the conference support device 20 performs facilitation based on text data obtained from the speech recognition device 300 . Specifically, when attendee x calls attendee y(z) by name, question unit 13 asks attendee y(z), whose name is called by attendee x, about his/her opinion. As a result, smooth progress of the conference can be supported.

[Embodiment 3]
Embodiment 3 will be described with reference to FIGS. 5 to 7. FIG. In Embodiment 3, the conference support device 30 re-learns the predictor based on the prediction result (that is, hit or miss) regarding whether the attendee has the intention of speaking. Regarding the configuration common between the third embodiment and the first and second embodiments, in the third embodiment, the description of the first and second embodiments is cited, and the description thereof is omitted.

(Meeting support device 30)
FIG. 5 is a block diagram showing the configuration of the conference support device 30 according to the third embodiment. As shown in FIG. 5, the conference support device 30 includes an emotion analysis section 11, a prediction section 12, and a question section 13. The conference support device 30 according to the third embodiment further includes a learning section 34 .

The learning unit 34 re-learns the predictor using the prediction results regarding whether the attendee has the intention of speaking. The learning unit 34 is an example of learning means. In one example, the learner 34 feeds back to the predictor the result of whether the prediction about whether the attendee has the intention to speak was correct or not. Specifically, when the prediction is correct, the learning unit 34 maintains the weighting factor of the predictor. On the other hand, when the prediction is incorrect, the learning unit 34 reduces the weighting factor of the feature that greatly affects the prediction result. In addition, the learning unit 34 may correct the weighting factors of the predictors using any orientation method.

Furthermore, the learning unit 34 may re-learn the predictor based on the emotional expressions of the attendees after asking their opinions. Specifically, after asking the attendees for their opinions, the learning unit 34 causes the emotion analysis unit 11 to detect the emotional expressions of the attendees again. The learning unit 34 receives the second emotional expression data of the attendee from the emotion analysis unit 11 . Then, the learning unit 34 re-learns the predictor using the second emotional expression data of the attendee. For example, the learning unit 34 deep-learns the predictor using the second emotional expression data of the attendee received from the emotion analysis unit 11 .

(Operation of conference support device 30)
The operation of the conference support device 30 according to the third embodiment will be described with reference to FIG. FIG. 6 is a flow chart showing the flow of processing executed by each unit of the conference support device 30. As shown in FIG. Any one of attendees x, y, and z of the conference is hereinafter referred to as "attendee x(y,z)."

As shown in FIG. 6, the emotion analysis unit 11 detects the first emotion expression of the attendee x(y, z) (S301). The emotion analysis unit 11 outputs the data of the detected emotion expression for the first time to the prediction unit 12 .

The prediction unit 12 predicts whether the attendee x(y,z) intends to speak based on the first emotional expression of the attendee x(y,z) (S302). The prediction unit 12 outputs to the question unit 13 a prediction result indicating whether or not the attendee x(y, z) intends to speak.

If it is predicted that the attendee x(y, z) has no intention of speaking (No in S303), the operation of the conference support device 10 ends.

On the other hand, when it is predicted that attendee x(y,z) intends to speak (Yes in S303), question unit 13 asks attendee x(y,z) for his opinion (S304). For example, the question unit 13 instructs the conference application 130 installed in the terminal 100 to output a voice message prompting the attendee x(y,z) to speak from a speaker (not shown) of the terminal 100 . Alternatively, the questioning unit 13 causes the screen (not shown) of the terminal 100 to display a message asking the attendee x(y,z) for an opinion.

Furthermore, the emotion analysis unit 11 again detects the emotional expression of the attendee x(y, z) who asked the question in step S304 (S305).

The learning unit 34 updates the database for re-learning (FIG. 7) based on the prediction result in step S303 and the second emotional expression of the attendee x(y, z) detected in step S305. At the same time, the predictor is re-learned (S306). The result of the prediction in step S303 means whether the prediction that attendee x(y, z) has an intention to speak is true or false.

After step S304, if voice data is input from the terminal 100 of the attendee x(y, z) within a certain period of time, the learning unit 34 determines that the prediction is correct. On the other hand, after step S304, if voice data is not input from the terminal 100 of the attendee x(y,z) within a certain period of time, the learning unit 34 determines that the prediction has failed.

Alternatively, the learning unit 34 analyzes the text data received from the speech recognition device 300, and if negative words such as "nothing in particular", "nothing", "nothing", etc. are detected, it is determined that the prediction is wrong. good too.

If the emotional expression of the attendee x(y, z) detected in step S305 partially overlaps the emotional expression detected in step S301, the learning unit 34 uses the difference between them to Relearn. For example, the first emotional expression detected in step S301 is "eyebrows rise" and "nose puffs up", and the second emotional expression of attendee x(y, z) detected in step S305 is " Suppose that it is "eyebrows rise" and "cheeks rise". In this case, the learning unit 34 uses the difference between the second emotional expression and the first emotional expression, ie, “cheek rises” to re-learn the predictor.

Thus, the operation of the conference support device 30 ends.

(Example of database for re-learning)
FIG. 7 shows an example of a database used by the learning unit 34 to re-learn the predictor. In the example shown in FIG. 7, the leftmost column indicates the number of times predictions were made by the predictor. The second to fourth columns counted from the left end represent the emotional expressions of the first attendee detected by the emotion analysis unit 11 . The 5th and 6th columns counted from the left end indicate whether the prediction that the attendee has the intention to speak was true or false. The rightmost column represents the attendee's second emotional expression (denoted as “emotional expression (difference)” in FIG. 7) detected by the emotion analysis unit 11 .

In one example, the learning unit 34 re-learns the predictor when the same emotional expression is detected a predetermined number of times in step S305. In one example, as shown in FIG. 7, the learning unit 34 re-learns the predictor when the emotional expression "cheeks go up" is detected four times.

(Effect of this embodiment)
According to the configuration of this embodiment, the emotion analysis unit 11 detects the emotional expressions of the attendees of the conference. The prediction unit 12 predicts whether the attendee has the intention of speaking based on the emotional expression. When it is predicted that the attendee has an intention to speak, the question section 13 asks the attendee for their opinion. Attendees are encouraged to speak up by being asked for their opinions. As a result, smooth progress of the conference can be supported.

Furthermore, according to the configuration of this embodiment, the learning unit 34 re-learns the predictor using the result of prediction regarding whether the attendee has an intention to speak. By re-learning the predictor, it is possible to improve the accuracy of the prediction as to whether the attendee has the intention to speak.

(About hardware configuration)
Each component of the conference support devices 10, 20, and 30 described in the first to third embodiments represents a functional unit block. Some or all of these components are realized by an information processing device 900 as shown in FIG. 8, for example. FIG. 8 is a block diagram showing an example of the hardware configuration of the information processing device 900. As shown in FIG.

As shown in FIG. 8, the information processing apparatus 900 includes, as an example, the following configuration.

- CPU (Central Processing Unit) 901
・ROM (Read Only Memory) 902
・RAM (Random Access Memory) 903
・Program 904 loaded into RAM 903
- Storage device 905 for storing program 904
A drive device 907 that reads and writes the recording medium 906
- A communication interface 908 that connects to the communication network 909
- An input/output interface 910 for inputting/outputting data
A bus 911 connecting each component
Each component of the conference support apparatuses 10, 20, and 30 described in the first to third embodiments is implemented by the CPU 901 reading and executing the program 904 that implements these functions. A program 904 that implements the function of each component is stored in advance in the storage device 905 or the ROM 902, for example, and is loaded into the RAM 903 and executed by the CPU 901 as necessary. The program 904 may be supplied to the CPU 901 via the communication network 909 or may be stored in the recording medium 906 in advance, and the drive device 907 may read the program and supply it to the CPU 901 .

According to the above configuration, the conference support devices 10, 20, and 30 described in the first to third embodiments are implemented as hardware. Therefore, the same effects as those described in the above embodiment can be obtained.

(Appendix)
One aspect of the present invention can also be described in the following supplementary remarks, but is not limited to the following.

(Appendix 1)
sentiment analysis means for detecting emotional expressions of conference attendees;
Prediction means for predicting whether the attendee has an intention to speak based on the emotional expression;
and questioning means for asking an opinion of the attendee when it is predicted that the attendee has an intention to speak.

(Appendix 2)
The conference support apparatus according to appendix 1, wherein the emotion analysis means detects the emotional expression of the attendee from face image data obtained by photographing the face of the attendee.

(Appendix 3)
3. The conference support device according to appendix 1 or 2, wherein the emotional expression includes an action of the attendee and a change in facial expression of the attendee.

(Appendix 4)
4. The conference support device according to any one of appendices 1 to 3, wherein the question means sends a voice message for prompting the attendee to speak.

(Appendix 5)
Supplements 1 to 4, wherein the prediction means predicts whether or not the attendee has an intention to speak by using a predictor that performs machine learning of emotional expressions when the attendee has spoken in the past. The conference support device according to any one of 1.

(Appendix 6)
The question means uses a speech recognition device to analyze the speech of the attendee,
6. The conference support device according to any one of appendices 1 to 5, wherein the emotion analysis means detects the emotional expression of the attendee after the identification name of the attendee is called.

(Appendix 7)
The conference support device according to appendix 5, further comprising learning means for re-learning the predictor using a prediction result regarding whether the attendee has an intention to speak.

(Appendix 8)
Detects the emotional expressions of meeting attendees,
predicting whether the attendee has an intention to speak based on the emotional expression;
A method of supporting a meeting, which asks an opinion of the attendee when it is predicted that the attendee has an intention to speak.

(Appendix 9)
a process of detecting an emotional expression of a meeting attendee;
a process of predicting whether the attendee has an intention to speak based on the emotional expression;
A program for causing a computer to execute a process of asking an opinion of said attendee when said attendee is predicted to have an intention to speak.

(Appendix 10)
The conference support device according to any one of Appendices 1 to 7;
an imaging device that captures the face of the attendee;
a sound collecting device that collects the speech of the attendee;
A conference support system with

(Appendix 11)
11. The meeting support system according to appendix 10, further comprising a biometric authentication device that biometrically authenticates the attendee.

(Appendix 12)
12. The conference support system according to appendix 11, wherein the biometric authentication device authenticates the face of the attendee by matching face image data of the attendee with face image data registered in advance. .

(Appendix 13)
13. The conference support system according to any one of appendices 10 to 12, further comprising an avatar generation device that generates an avatar representing the alter ego of the attendee in the virtual space.

(Appendix 14)
14. The conference support system according to Supplementary Note 13, wherein the avatar generation device changes the mode of the avatar according to the emotional expression of the attendee.

(Appendix 15)
15. The conference support system according to any one of appendices 10 to 14, further comprising a voice recognition device that recognizes voice data input from the sound collector.

INDUSTRIAL APPLICABILITY The present invention can be used, for example, in a conference support system for supporting an online conference through a network such as the Internet.

1 meeting support system 10 meeting support device 11 emotion analysis unit 12 prediction unit 13 question unit 20 meeting support device 30 meeting support device 34 learning unit 110 sound collector 120 imaging device 200 biometric authentication device 300 voice recognition device 400 avatar generation device

Claims (10)

sentiment analysis means for detecting emotional expressions of conference attendees;
Prediction means for predicting whether the attendee has an intention to speak based on the emotional expression;
and questioning means for asking an opinion of the attendee when it is predicted that the attendee has an intention to speak. 2. The conference support apparatus according to claim 1, wherein said emotion analysis means detects an emotional expression of said attendee from face image data corresponding to an image of said attendee's face. 3. The conference support apparatus according to claim 1, wherein said emotional expression includes behavior of said attendee and changes in facial expression of said attendee. 4. The conference support apparatus according to any one of claims 1 to 3, wherein said question means sends out a voice message for prompting said attendee to speak. The prediction means predicts whether or not the attendee has an intention to speak by using a predictor that has undergone machine learning of emotional expressions when the attendee has spoken in the past. 5. The conference support device according to any one of 4. The question means uses a speech recognition device to analyze the speech of the attendee,
6. The conference support device according to any one of claims 1 to 5, wherein the emotion analysis means detects the emotional expression of the attendee after the identification name of the attendee is called. 6. The conference support apparatus according to claim 5, further comprising learning means for re-learning said predictor using a prediction result regarding whether said attendee has an intention to speak. Detects the emotional expressions of meeting attendees,
predicting whether the attendee has an intention to speak based on the emotional expression;
A method of supporting a meeting, which asks an opinion of the attendee when it is predicted that the attendee has an intention to speak. a process of detecting an emotional expression of a meeting attendee;
a process of predicting whether the attendee has an intention to speak based on the emotional expression;
A program for causing a computer to execute a process of asking an opinion of said attendee when said attendee is predicted to have an intention to speak. a meeting support device according to any one of claims 1 to 7;
an imaging device that captures the face of the attendee;
a sound collecting device that collects the speech of the attendee;
A conference support system with

Images