JP7036046B2 - Information processing equipment, information processing methods, and information processing programs - Google Patents

Description

The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

A service that supports smooth communication between users is known as a conventional technique. Patent Document 1 describes a telephone voice monitoring evaluation system that evaluates operator performance by recognizing emotions based on voices in conversations between call center operators and customers, and performing voice analysis by combining voices and emotions. Has been done. Patent Document 2 describes an emotion matching device that recognizes emotions indicated by a chat sentence input by a user and obtains the degree of similarity of emotions between users.

Japanese Unexamined Patent Publication No. 2017-135642 (published on August 3, 2017) Japanese Unexamined Patent Publication No. 2005-284822 (published on October 13, 2005)

However, the conventional technique as described above has a problem that multifaceted emotion recognition cannot be performed because the speaker's emotion is recognized only based on the voice in the conversation or based only on the input chat sentence. ..

One aspect of the present invention is to provide a communication support technique for recognizing the emotions of each user during a conversation from various aspects and notifying the evaluation of the conversation based on the recognized emotions.

In order to solve the above-mentioned problems, the information processing apparatus according to one aspect of the present invention includes the first facial expression information regarding the facial expressions of the first participant among the plurality of participants and the first among the plurality of participants. The facial expression information acquisition unit that acquires the second facial expression information regarding the facial expressions of the second participant, the first speech information regarding the speech of the first participant, and the second participant among the plurality of participants. The first participant and the second participant with reference to the voice information acquisition unit that acquires the second utterance information related to the utterance, the first facial information, and the second facial information. With reference to the facial relationship information generation unit that generates facial relationship information indicating the relationship between the first and the first spoken information and the second spoken information, the first participant and the said first participant. The first participant refers to the utterance relationship information generation unit that generates utterance relationship information indicating the relationship with the second participant, and the facial expression relationship information and the utterance relationship information. It is characterized by including a relationship information generation unit that generates relationship information, which is information indicating the relationship between the second participant and the second participant.

According to this, it is possible to evaluate the relationship between the participants during the meeting based on both the voice information and the facial expression information of each participant.

In the information processing apparatus according to the one aspect, the relationship information is real-time or temporal information indicating the relationship between the first participant and the second participant.

According to this, it is possible to evaluate the relationship between the participants during the meeting in real time based on both the voice information and the facial expression information of each participant.

In the information processing apparatus according to the one aspect, the first facial expression information includes a plurality of first indexes expressing the facial expressions of the first participant, and the second facial expression information includes a plurality of first indexes. , A plurality of second indexes expressing the facial expressions of the second participant are included, and the facial expression relationship information generation unit uses the facial expression difference regarding the difference between the first index and the second index. Information is generated, and the generated facial expression difference information is included in the facial expression relationship information.

According to this, since a plurality of indexes are used to express the facial expressions of the participants referred to for generating the facial expression relationship information, the facial expressions of the participants can be expressed more accurately.

In the information processing apparatus according to the one aspect, the first facial expression information includes the first line-of-sight information regarding the line-of-sight direction of the first participant, and the second facial expression information includes the above-mentioned second facial expression information. The second line-of-sight information regarding the line-of-sight direction of the second participant is included, and the facial expression relationship information generation unit refers to the first line-of-sight information and the second line-of-sight information to form a line-of-sight relationship. Information is generated, and the generated line-of-sight relationship information is included in the facial expression relationship information.

According to this, since the facial expression information of the participant referred to for generating the facial expression relationship information includes the line-of-sight information of the participant, the facial expression of the participant can be expressed more accurately.

In the information processing apparatus according to the one aspect, the utterance relationship information generation unit has the utterance time of the first participant indicated by the first utterance information and the second utterance information indicated by the second utterance information. The utterance time relationship information indicating the relationship with the utterance time of the participant is generated, and the generated utterance time relationship information is included in the utterance relationship information.

According to this, since the utterance time relationship information of the participant referred to for generating the utterance relationship information is also included, the utterance relationship information of the participant can be generated more accurately.

In the information processing apparatus according to the one aspect, the utterance relationship information generation unit includes, at least one of the first utterance information and the second utterance information, the utterance content included in a specific category. It is determined whether or not the information is present, and the information corresponding to the result of the determination is included in the utterance relationship information.

According to this, since the utterance relationship information includes information according to the judgment result of whether or not the utterance content included in a specific category is included, the relationship information between the participants can be generated more accurately. Can be done.

In the information processing apparatus according to the one aspect, the utterance relationship information generation unit is a word having a relatively high frequency of appearance within a predetermined time from at least one of the first utterance information and the second utterance information. Is extracted, and the extracted words are included in the utterance relationship information.

According to this, since the utterance relationship information includes the upper part of the frequently used word, the relationship information between the participants can be generated more accurately.

In the information processing apparatus according to the one aspect, the relationship information generation unit refers to the relationship information and presents presentation information to at least one of the first participant and the second participant. Generate.

According to this, by presenting the presented information to the participants, it is possible to make the participants recognize the relationship information.

The presented information includes information indicating the ratio between the utterance time of the first participant and the utterance time of the second participant, the line-of-sight direction of the first participant, and the second. It contains information about the time course of the match rate with the participant's line-of-sight direction.

According to this, by presenting the presented information to the participants, it is possible to make the participants recognize the ratio of the utterance time of each participant and the change with time of the matching rate in the line-of-sight direction of each participant.

In the information processing apparatus according to the one aspect, the facial expression relationship information generation unit and the utterance relationship information generation unit further refer to the participant information indicating the attributes of the first and second participants. The facial expression relationship information and the utterance relationship information are generated.

According to this, since the attributes of the participants are also referred to in the generation of the facial expression relationship information and the utterance relationship information, more accurate facial expression relationship information and the utterance relationship information can be generated.

Further, in order to solve the above-mentioned problems, the information processing method according to one aspect of the present invention includes the first facial expression information regarding the facial expressions of the first participant among the plurality of participants and the first facial expression information regarding the facial expressions of the first participant among the plurality of participants. The facial information acquisition step for acquiring the second facial information regarding the facial expressions of the second participant, the first utterance information regarding the utterance of the first participant, and the second participation among the plurality of participants. The first participant and the second participation with reference to the voice information acquisition step of acquiring the second utterance information regarding the utterance of the person and the first facial information and the second facial information. With reference to the first utterance information and the second utterance information, and the first participant The first participation with reference to the utterance relationship information generation step for generating the utterance relationship information indicating the utterance relationship information with the second participant, and the facial expression relationship information and the utterance relationship information. It is characterized by including a relationship information generation step of generating relationship information, which is real-time or temporal information indicating the relationship between the person and the second participant.

According to this, it is possible to evaluate the relationship between the participants during the meeting based on the voice information and the facial expression information of each participant.

Further, in order to solve the above-mentioned problems, the information processing program according to one aspect of the present invention is an information processing program for operating a computer as the information processing device according to any one of the above, and is the facial expression information acquisition unit. The computer functions as the voice information acquisition unit, the facial expression relationship information generation unit, the speech relationship information generation unit, and the relationship information generation unit.

According to this, it is possible to evaluate the relationship between the participants during the meeting based on the voice information and the facial expression information of each participant.

According to one aspect of the present invention, the relationship between the participants during the meeting can be evaluated based on the voice information and the facial expression information of each participant.

It is a block diagram which shows an example of the component of the information processing system including the information processing apparatus which concerns on one Embodiment of this invention. It is a figure which shows the outline of the information processing system which includes the information processing apparatus which concerns on one Embodiment of this invention. It is a figure which shows the outline of the data flow in the information processing system including the information processing apparatus which concerns on one Embodiment of this invention. It is a figure which shows an example of the information presented by the information processing system including the information processing apparatus which concerns on one Embodiment of this invention. It is a figure which shows the other example of the information which the information processing system including the information processing apparatus which concerns on one Embodiment of this invention presents.

[Embodiment 1]
Hereinafter, one embodiment of the present invention will be described in detail. FIG. 1 is a diagram showing an outline of an information processing system 100 including the information processing device 10 of the present embodiment. As shown in FIG. 1, the information processing system 100 includes an information processing device 10, a first terminal device 20, and a second terminal device 30. The number of terminal devices is not limited to this embodiment, and may be 3 or more.

FIG. 2 is a diagram showing an outline of an information processing system 100 including an information processing device 10 according to an embodiment of the present invention. In the information processing system 100, as shown in FIG. 2, between the first participant 200 who uses the first terminal device 20 and the second participant 201 who uses the second terminal device 30. Evaluate the relationship.

In the information processing system 100, information is obtained based on the facial expression information and the spoken information of the first participant 200 and the second participant 201 obtained from the first terminal device 20 and the second terminal device 30 during the meeting. The processing device 10 evaluates the relationship between the first participant 200 and the second participant 201 during the meeting. The information processing system 100 displays the result of the evaluation of the relationship on at least one of the first terminal device 20 and the second terminal device 30, so that the first participant 200 and the second participant 201 can display the result. The evaluation results are fed back to at least one person in real time to promote improvement of the communication status during the meeting.

In the present embodiment, the “meeting” is not limited to a meeting in a narrow sense, but includes interviews, interviews, counseling, interviews, customer service, interviews, consultations, and the like. As an example,
・ Interviews between superiors and subordinates ・ Interviews with patients by doctors ・ Counseling with target persons by counselors ・ Customer service and counter consultation for customers ・ Remote communication such as Web Meeting ・ For video images such as e-Learning Communication etc. are included.

Further, FIG. 3 is a diagram showing an outline of the data flow in the information processing system 100.

[First terminal device 20]
As shown in FIG. 1, the first terminal device 20 includes a camera 21, a microphone 22, a display unit 23, a control unit 24, a speaker 25, and a communication unit 26.

<Video acquisition process>
The camera 21 captures the first participant and supplies the captured image to the control unit 24. Here, the image captured by the camera 21 is preferably a moving image, and in the case of the configuration, as shown in FIG. 3, the camera 21 is included in the moving image file and the moving image file with respect to the control unit 24. Supply at least one of a moving image file list, which is a list of each image. The camera 21 supplies the control unit 24 with a time stamp indicating the imaging time of each image included in the moving image file.

The first terminal device 20 may be configured to include a plurality of cameras, and in the case of the configuration, the control unit 24 can specify the camera 21 by referring to the camera device identification information.

<Image recognition processing>
The control unit 24 performs image recognition processing with reference to the moving image file, the moving image file list, and the time stamp supplied from the camera 21.

As an example, as shown in FIG. 3, the control unit 24 performs image recognition processing with reference to a moving image file, a moving image file list, and a time stamp, thereby performing time-series facial expression values, time-series face part coordinates, and time-series line-of-sight. Calculate the coordinates. The time-series facial expression value, the time-series face part coordinates, and the time-series line-of-sight coordinates are examples of the first facial expression information regarding the facial expressions of the first participant.

<Voice acquisition processing>
The microphone 22 mainly collects the voice uttered by the first participant, and provides a voice file indicating the collected voice and a time stamp for specifying the time point of the utterance in the voice file to the control unit 24. Supply.

The first terminal device 20 may be configured to include a plurality of microphones, and in the case of the configuration, the control unit 24 can specify the microphone 22 by referring to the voice device identification information.

<Utterance recognition processing>
The control unit 24 performs the utterance recognition process with reference to the audio file and the time stamp supplied from the microphone 22.

As an example, as shown in FIG. 3, the control unit 24 generates section time-series text data by performing an utterance recognition process with reference to an audio file and a time stamp. Here, the section time-series text data is information that mainly indicates the contents spoken by the first participant as text data in chronological order. The section time-series text data is an example of the first utterance information indicating the utterance of the first participant.

[Second terminal device 30]
The second terminal device 30 includes a camera 31, a microphone 32, a display unit 33, a control unit 34, a speaker 35, and a communication unit 36.

<Video acquisition process>
The camera 31 takes an image of the second participant and supplies the captured image to the control unit 34. Here, the image captured by the camera 31 is preferably a moving image, and in the case of the configuration, as shown in FIG. 3, the camera 31 is included in the moving image file and the moving image file with respect to the control unit 34. Supply at least one of a moving image file list, which is a list of each image. The camera 31 supplies the control unit 34 with a time stamp indicating the imaging time of each image included in the moving image file.

The second terminal device 30 may be configured to include a plurality of cameras, and in the case of the configuration, the control unit 34 can specify the camera 31 by referring to the camera device identification information.

<Image recognition processing>
The control unit 34 performs image recognition processing with reference to the moving image file, the moving image file list, and the time stamp supplied from the camera 31.

As an example, as shown in FIG. 3, the control unit 34 performs image recognition processing with reference to a moving image file, a moving image file list, and a time stamp, thereby performing time-series facial expression values, time-series face part coordinates, and time-series line-of-sight. Calculate the coordinates. The time-series facial expression value, the time-series face part coordinates, and the time-series line-of-sight coordinates are examples of the second facial expression information regarding the facial expressions of the second participant.

<Voice acquisition processing>
The microphone 32 mainly collects the voice spoken by the second participant, and provides a voice file indicating the collected voice and a time stamp for specifying the time point of the utterance in the voice file to the control unit 34. Supply.

The second terminal device 30 may be configured to include a plurality of microphones, and in the case of the configuration, the control unit 34 can specify the microphone 32 by referring to the voice device identification information.

<Utterance recognition processing>
The control unit 34 performs an utterance recognition process with reference to the audio file and the time stamp supplied from the microphone 32.

As an example, as shown in FIG. 3, the control unit 34 generates section time-series text data by performing an utterance recognition process with reference to an audio file and a time stamp. Here, the section time-series text data is information that mainly indicates the contents spoken by the second participant as text data in chronological order. The section time-series text data is an example of the second utterance information indicating the utterance of the second participant.

[Information processing device 10]
The information processing apparatus 10 includes a facial expression information acquisition unit 13, a voice information acquisition unit 14, a facial expression relationship information generation unit 15, an utterance relationship information generation unit 16, and a relationship information generation unit 17. The information processing device 10 further includes a communication unit 11. The calculation unit 12 includes a facial expression information acquisition unit 13, a voice information acquisition unit 14, a facial expression relationship information generation unit 15, an utterance relationship information generation unit 16, and a relationship information generation unit 17.

(Facial expression information acquisition unit 13)
The facial expression information acquisition unit 13 relates to the first facial expression information regarding the facial expression of the first participant among the plurality of conference participants and the facial expression of the second participant among the plurality of conference participants via the communication unit 11. Acquire the second facial expression information.

<Numerical data cleaning process>
As an example, the facial expression information acquisition unit 13 via the communication unit 11 has time-series facial expression values, time-series face part coordinates, and time-series line-of-sight coordinates, which are time-series numerical data included in the facial expression information regarding the first participant. As an example, the numerical data cleaning process is performed by performing the following process on the time-series numerical data with reference to.
-Delete the invalid data section-Average the data in the valid data section-Convert to the distribution and the number of terms The facial expression information acquisition unit 13 performs the above-mentioned numerical data cleaning process to perform the section time related to the first participant. Generate series numerical data. The section time-series numerical data includes time-series facial expression values, time-series face part coordinates, and time-series line-of-sight coordinates in the effective section.

The facial expression information acquisition unit 13 performs the same processing on the facial expression information regarding the second participant, and generates interval time-series numerical data regarding the second participant.

<Facial expression detection>
The facial expression information acquisition unit 13 calculates a plurality of first indexes expressing the facial expressions of the first participant with reference to the interval time-series numerical data relating to the first participant. Further, the facial expression information acquisition unit 13 calculates a plurality of second indexes expressing the facial expressions of the second participant by referring to the section time-series numerical data relating to the second participant.

Here, as an example of the index expressing the facial expression, the following index can be mentioned.
・ Anger
・ Insult (contempt)
・ Disgust
・ Fear
・ Happiness
・ Neutral
・ Sadness
・ Surprise
Therefore, the index expressing the facial expression can also be said to be an index expressing the emotion indicated by the facial expression.

The facial expression information acquisition unit 13 may use the time-series facial expression value included in the section time-series numerical data relating to the first participant as a plurality of first indexes for expressing the facial expression of the first participant as it is. good. Similarly, the facial expression information acquisition unit 13 uses the time-series facial expression values included in the interval time-series numerical data relating to the second participant as a plurality of second indexes for expressing the facial expressions of the second participant as they are. May be good.

Further, the facial expressions of the first participant and the facial expressions of the second participant can also be expressed as a vector having the above index as each component. These vectors are sometimes called facial expression vectors.

The technique for detecting the facial expressions of each participant and the technique for indexing and expressing the emotions indicated by the detected facial expressions are not limited to the present embodiment, and for example, known techniques can be used.

<Detection of line of sight>
Further, the facial expression information acquisition unit 13 acquires information regarding the line-of-sight directions of the first participant and the second participant from the first terminal device 20 and the second terminal device 30 via the communication unit 11. .. Specifically, as an example, the facial expression information acquisition unit 13 acquires the time-series line-of-sight coordinates included in the above-mentioned section time-series numerical data regarding the first participant as information regarding the line-of-sight direction of the first participant. .. Similarly, the facial expression information acquisition unit 13 acquires the time-series line-of-sight coordinates included in the above-mentioned section time-series numerical data regarding the second participant as information regarding the line-of-sight direction of the second participant.

The method of acquiring the line-of-sight coordinates is not particularly limited, but a point light source (not shown) is provided in the first terminal device 20 and the second terminal device 30, and the corneal reflex image of the light from the point light source is captured by the camera. A method of acquiring the user's line-of-sight coordinates by taking a picture with the 21 and the camera 31 for a predetermined time can be mentioned. The type of the point light source is not particularly limited, and examples thereof include visible light and infrared light. For example, by using an infrared LED, the line-of-sight coordinates can be acquired without causing discomfort to the user.

<Distance detection>
Further, the facial expression information acquisition unit 13 acquires the time-series face part coordinates of the first participant included in the section time-series numerical data, and the distance between the first participant and the image pickup means (camera 21). May be calculated. Further, the facial expression information acquisition unit 13 acquires the time-series face part coordinates of the second participant included in the section time-series numerical data, and the distance between the second participant and the image pickup means (camera 31). May be calculated. The distance between the participant and the imaging means can be calculated as the reciprocal of the outer corner distance of the face, for example, the outer corner distance of the face corrected by the face angle in the captured image obtained from the coordinates of the face parts. ..

(Voice information acquisition unit 14)
The voice information acquisition unit 14 acquires the first utterance information regarding the utterance of the first participant and the second utterance information regarding the utterance of the second participant among the plurality of conference participants. That is, the voice information acquisition unit 14 acquires information regarding the utterances of the first participant and the second participant from the first terminal device 20 and the second terminal device 30 via the communication unit 11.

As an example, the voice information acquisition unit 14 acquires the time-series utterance text included in the section time-series text data relating to the first participant described above. Similarly, the voice information acquisition unit 14 acquires, as an example, the time-series utterance text included in the section time-series text data relating to the second participant described above.

Further, as an example, the voice information acquisition unit 14 acquires the time-series utterance text regarding the first participant and the time-series face part coordinates at the time of utterance of the time-series utterance text. The voice information acquisition unit 14 refers to the time-series face parts coordinates, and if the mouth of the first participant is open at the time of utterance of the time-series utterance text, the first participant uses the time-series utterance text. Included in the first utterance information regarding the utterance of. Similarly, the voice information acquisition unit 14 acquires the time-series utterance text regarding the second participant and the time-series face part coordinates at the time of utterance of the time-series utterance text. The voice information acquisition unit 14 refers to the time-series face parts coordinates, and if the second participant's mouth is open at the time of utterance of the time-series utterance text, the second participant uses the time-series utterance text. Included in the second utterance information regarding the utterance of. Thereby, even when a simple microphone having no directivity is used as the microphone 22 or the microphone 32, the person who has spoken can be specified.

(Facial expression relationship information generation unit 15)
The facial expression relationship information generation unit 15 refers to the first facial expression information and the second facial expression information, and obtains facial expression relationship information indicating the relationship between the first participant and the second participant regarding the facial expression. Generate.

Meeting participants' satisfaction with the meeting depends not only on the content and conclusions of the meeting, but also on the presence or absence of good communication between the participants. The state of communication between participants is represented by the relationships between participants during the meeting, and the relationships between participants can be evaluated by the degree of emotional concordance. The facial expression relationship information generation unit 15 acquires the first facial expression information and the second facial expression information from the facial expression information acquisition unit 13, and based on these facial expression information, determines the degree of emotional matching between the participants during the meeting. By evaluating from the facial expressions of both participants, the communication status between the participants is evaluated in real time.

Each facial expression information acquired by the facial expression relationship information generation unit 15 from the facial expression information acquisition unit 13 is calculated based on the interval time series numerical data, that is, regarding the real-time or temporal facial expression of each participant. It was calculated based on the information. Since the facial expression relationship information generation unit 15 generates facial expression relationship information based on the facial expression information of each participant in real time or over time, the generated facial expression relationship information can be obtained in real time or over time between the participants. It shows the relationship related to various facial expressions.

<Facial expression match rate judgment>
As described above, the first facial expression information includes a plurality of first indexes expressing the facial expressions of the first participant, and the second facial expression information includes the facial expressions of the second participant. A plurality of second indicators expressing the above are included.

The facial expression relationship information generation unit 15 may generate facial expression difference information regarding the difference between the first index and the second index, and may include the generated facial expression difference information in the facial expression relationship information.

As an example, the facial expression relationship information generation unit 15 uses a first facial expression vector having a plurality of indexes expressing the facial expressions of the first participant as elements, and a plurality of indexes expressing the facial expressions of the second participant. The amount of facial expression mismatch is calculated using the absolute value of the difference from the second facial expression vector as an element. The calculated facial expression mismatch amount can be said to be an index showing the state of emotional harmony between the participants during the meeting. Further, the facial expression relationship information generation unit 15 may calculate the facial expression matching rate as an index indicating the ratio of matching facial expressions from the start of the meeting to the present time. The facial expression match rate is obtained, for example, by subtracting the time when the facial expressions did not match from the time from the start of the conference to the present time and dividing by the time from the start of the conference to the present time.

<Judgment of line-of-sight matching rate>
Further, the first facial expression information includes the first line-of-sight information regarding the line-of-sight direction of the first participant, and the second facial expression information includes the second line-of-sight information regarding the line-of-sight direction of the second participant. It may be a configuration including. The facial expression relationship information generation unit 15 may generate line-of-sight relationship information with reference to the first line-of-sight information and the second line-of-sight information, and may include the generated line-of-sight relationship information in the facial expression relationship information. ..

As an example, the facial expression relationship information generation unit 15 calculates the line-of-sight matching rate between the first participant and the second participant as the line-of-sight relationship information. The calculated line-of-sight matching rate can be said to be an index showing the state of being concerned about the state of other participants during the meeting. More specifically, first, the control unit 24 or the facial expression relationship information generation unit 15 identifies the position of the eyes of the first participant in the conference room by analyzing the captured image of the camera 21, and the control unit. 34 or the facial expression relationship information generation unit 15 identifies the position of the eyes of the second participant in the conference room by analyzing the captured image of the camera 31.

Then, the facial expression relationship information generation unit 15 determines at each time point whether or not the line-of-sight direction of the first participant indicated by the first line-of-sight information is toward the eyes of the second participant, and the second By determining whether or not the line-of-sight direction of the second participant indicated by the line-of-sight information of 2 is toward the eyes of the first participant, the line-of-sight of the first participant and the second line of sight are determined at each time point. Determine if the line of sight of the participant matches.

As an example, the facial expression relationship information generation unit 15 sets the line-of-sight flag of the first participant to 1 when it is determined that the line of sight of the first participant is toward the eyes of the second participant. .. Further, the facial expression relationship information generation unit 15 sets the line-of-sight flag of the second participant to 1 when it is determined that the line of sight of the second participant is toward the eyes of the first participant. Then, the facial expression relationship information generation unit 15 determines that the lines of sight match when both line-of-sight flags are 1.

Then, the facial expression relationship information generation unit 15 calculates the line-of-sight matching rate as an index indicating the ratio of the line-of-sight matching from the start of the meeting to the present time. The line-of-sight matching rate is obtained, for example, by dividing the time when the line-of-sight is matched by the time from the start of the conference to the present time.

In addition, in order to determine whether or not the lines of sight are directed toward each other's eyes, a configuration may be configured in which the position information indicating the relative positional relationship between the first terminal device 20 and the second terminal device 30 is further referred to. ..

Further, when the eyes of each other are not necessarily the eyes of the other party but are facing the face of the other party or the body of the other party, it may be determined that the lines of sight match.

Further, when the participants have a meeting via the Internet or the like, the line-of-sight matching rate between the participants is calculated through the screen of the terminal device. More specifically, as an example, the position of the face of the second participant displayed on the display screen of the first terminal device 20 is specified as the coordinates on the display screen, and the second participant is designated with respect to the specified coordinates. When the line of sight of one participant is directed, the line of sight flag of the first participant is set to 1. Similarly, the position of the face of the first participant displayed on the display screen of the second terminal device 30 is specified as the coordinates on the display screen, and the line of sight of the second participant with respect to the specified coordinates. Is pointed to, the line-of-sight flag of the second participant is set to 1.

<Judgment of front leaning rate>
Further, the facial expression relationship information generation unit 15 may calculate the front turning rate between the first participant and the second participant, and include the calculated front turning rate in the facial expression relationship information. The calculated pre-flipping rate can be said to be an index showing the state of being interested in the utterances of other participants during the meeting. As an example, in the facial expression relationship information generation unit 15, when the distances of the first participant and the second participant from the respective imaging means fall below the threshold value within a preset fixed time. In addition, it is determined that the first participant and the second participant are in a forward leaning state.

Then, the facial expression relationship information generation unit 15 specifies the ratio of the time when the first participant is leaning forward in the time from the start of the meeting to the present time as the front leaning rate regarding the first participant, and the second. The percentage of time that a participant is leaning forward is specified as the percentage of time that a second participant is leaning forward.

Further, the facial expression relationship information generation unit 15 has a meeting from a preset change within a certain period of time regarding the face image size obtained based on the distances from the imaging means of each of the first participant and the second participant. The postures of the participants inside may be calculated and included in the facial expression relationship information. The calculated attitude of the participants can be said to be an index showing the attitude suitable for listening to the utterances of other participants during the meeting.

Further, the facial expression relationship information generation unit 15 calculates the correlation between the change in the posture of the first participant and the change in the second facial expression vector of the second participant, and includes the correlation in the facial expression relationship information. You may. The correlation between the change in posture and the change in facial expression vector can be said to be an index showing the influence of the posture of one participant on the facial expressions of other participants. Similarly, the facial expression relationship information generation unit 15 calculates the correlation between the change in the posture of the second participant and the change in the first facial expression vector of the first participant, and uses the correlation as the facial expression relationship information. May be included.

Further, the facial expression relationship information generation unit 15 refers to the posture of the first participant and the posture of the second participant, and refers to the degree of similarity between the posture states of the first participant and the second participant. May be calculated and the calculated similarity may be included in the facial expression relationship information. The similarity of the posture states represents the mirroring state, and can be said to be an index showing the state of being interested in the utterances of other participants during the meeting.

The facial expression relationship information generation unit 15 may further refer to the participant information indicating the attributes of the first and second participants to generate the facial expression relationship information. Participant information indicating the attributes of a participant includes at least one of the participant's age, gender, blood type, personality, place of origin, family relationship, job title, years of service, number of job changes, job history, and the like. Participant information also includes the usage history of the system.

As an example, when the facial expression relationship information generation unit 15 refers to the participant information and determines that the participant is likely to produce a specific facial expression, the index corresponding to the specific facial expression has a weighting coefficient smaller than 1. The facial expression vector of the participant may be corrected by performing a correction by multiplying by, and the facial expression relationship information may be generated using the corrected facial expression vector.
For example, when the participant information indicating the attribute of the first participant indicates that the first participant is shy, the facial expression relationship information generation unit 15 is an index of "neutral". Is multiplied by a weight of 0.8, and the remaining 0.2 weight is proportionally distributed to other indexes to correct the facial expression vector of the first participant, and the corrected facial expression. It may be configured to generate facial expression relationship information using a vector.

The information processing device 10 is configured to further acquire biological information such as pulse waves and brain waves of the participant and environmental information such as temperature, humidity, carbon dioxide concentration, and illuminance around the participant, and is a facial expression relationship information generation unit. Reference numeral 15 may generate facial expression relationship information by further referring to biological information and environmental information.

As an example, the facial expression relationship information generation unit 15 expresses the stress state of the participant determined from the pulse wave or respiration of the first participant and the facial expression of the second participant immediately before or at that time. With reference to the index of, the facial expression of the second participant who stresses the first participant is estimated. Then, the facial expression relationship information generation unit 15 may recognize the estimated facial expression of the second participant as an NG facial expression for the first participant, and may include the information in the facial expression relationship information. The NG facial expression of one participant with respect to the other participants can be said to be an index showing the influence of the facial expression of one participant on the stress state of the other participants. Similarly, the facial expression relationship information generation unit 15 may estimate the facial expression of the first participant who stresses the second participant, and may recognize the NG facial expression for the second participant.

Further, the facial expression relationship information generation unit 15 changes the environmental information around the participant within a predetermined fixed period, the first facial expression vector of the first participant, and the second participant's second. A correlation with a change in the average value of the facial expression vectors of 2 may be calculated, and the correlation may be included in the facial expression relationship information. The correlation between the change in environmental information and the change in the average value of the facial expression vector can be said to be an index showing the influence of the environment around the participants on the communication state between the participants.

<Dialogue management process>
As an example, the facial expression relationship information generation unit 15 has the user IDs of the first participant and the second participant, and the time and end of the meeting represented by the user ID, via the communication unit 11. Dialog management processing is performed with reference to the time stamp indicating the time stamp. The facial expression relationship information generation unit 15 extracts the user IDs of other participants who are interacting with each other from the data at a certain time point in the section time series numerical data relating to one participant, and is interacting with which participant. It may be determined whether the data is obtained in the above, and the result may be included in the facial expression relationship information.

(Utterance relationship information generation unit 16)
The utterance relationship information generation unit 16 refers to the first utterance information and the second utterance information, and provides the utterance relationship information indicating the relationship between the first participant and the second participant regarding the utterance. Generate. The utterance relationship information generation unit 16 acquires the first utterance information and the second utterance information from the voice information acquisition unit 14, and based on these utterance information, the degree of matching of emotions among the participants during the meeting is determined. By evaluating from the utterances of both participants, the communication status between the participants is evaluated.

Each utterance information acquired by the utterance relationship information generation unit 16 from the voice information acquisition unit 14 is calculated based on the section time-series text data, that is, regarding real-time or temporal utterances of each participant. It was calculated based on the information. Since the utterance relationship information generation unit 16 generates utterance relationship information based on the real-time or temporal utterance information of each participant, the generated utterance relationship information is real-time or temporal between the participants. It shows the relationship between various utterances.

<Utterance ratio judgment>
The utterance relationship information generation unit 16 shows the utterance time relationship showing the relationship between the utterance time of the first participant indicated by the first utterance information and the utterance time of the second participant indicated by the second utterance information. Information may be generated, and the generated utterance time relationship information may be included in the utterance relationship information.

As an example, the utterance relationship information generation unit 16 calculates the utterance ratio between the utterance time of the first participant and the utterance time of the second participant within a predetermined fixed time, and uses it as the utterance relationship information. include. The calculated utterance ratio can be said to be an index showing the equality of the relationships between the participants.

<Utterance frequency judgment>
Further, the utterance relationship information generation unit 16 determines whether or not at least one of the first utterance information and the second utterance information includes the utterance content included in a specific category, and the result of the determination. The information corresponding to the above may be included in the utterance relationship information.

Examples of specific categories in the utterance include open questions, action-prompting words (so? Well, sure), parrots, utterance interruptions, utterance sufferings, and negative words (but, though). .. As an example, the utterance relationship information generation unit 16 calculates the frequency at which the utterance content included in such a specific category is uttered within a predetermined fixed time, and the information regarding the calculated frequency is used for the utterance relationship. Include in information.

Specifically, as an example, a specific category included in the utterance content is set as an open question, and the utterance relationship information generation unit 16 is included within a certain period of time from the section time-series text data of the first participant. Extract text data that represents an open question. Then, the utterance relationship information generation unit 16 calculates the frequency at which the open question is spoken as the open question rate by dividing the number of words in the extracted text data by the number of words in all the text data within the above-mentioned fixed time. do. Similarly, the utterance relationship information generation unit 16 calculates the open question rate from the section time-series text data of the second participant. Then, the utterance relationship information generation unit 16 compares the open question rate of the first participant with the open question rate of the second participant, calculates the open question ratio, and includes it in the utterance relationship information. The calculated open question ratio can be said to be an index showing the equality of relationships between participants. Further, the utterance relationship information generation unit 16 may include the open question rate of the first participant and the open question rate of the second participant in the utterance relationship information.

Similarly, a specific category included in the utterance content is set as an action urging word, and the utterance relationship information generation unit 16 sets the action urging word included within a certain period of time from the section time-series text data of the first participant. Extract the text data to represent. Then, the utterance relationship information generation unit 16 divides the number of words in the extracted text data by the number of words in all the text data within the above-mentioned fixed time to promote the action and the frequency at which the words are spoken as a question rate. calculate.
Similarly, the utterance relationship information generation unit 16 calculates the prompting question rate from the section time-series text data of the second participant. Then, the utterance relationship information generation unit 16 compares the urged question rate of the first participant with the urged question rate of the second participant, calculates the urged question ratio, and includes it in the utterance relationship information. The calculated prompting question ratio can be said to be an index showing the equality of the relationships between the participants. Further, the utterance relationship information generation unit 16 may include the urged question rate of the first participant and the urged question rate of the second participant in the utterance relationship information.

<Word-based evaluation>
Further, the utterance relationship information generation unit 16 extracts words having a relatively high frequency of appearance within a predetermined time from at least one of the first utterance information and the second utterance information, and the extracted words are utterance-related. It may be included in sexual information.

As an example, the utterance relationship information generation unit 16 participates in the number of appearances of each word included in a predetermined fixed time from the section time-series text data of each of the first participant and the second participant. It is calculated and ranked for each person, and multiple words with relatively high frequency of appearance are extracted from the top. Then, the utterance relationship information generation unit 16 includes the extracted words having a higher frequency of appearance as frequently occurring words in the utterance relationship information. Further, the utterance relationship information generation unit 16 determines whether the frequently-used words of the first participant and the second participant and their ranks match, and includes the determination result in the utterance relationship information. May be good.

Further, the utterance relationship information generation unit 16 extracts words included within a predetermined fixed time for each participant from the section time-series text data of each of the first participant and the second participant. , The matching rate of the extracted words may be calculated and included in the utterance relationship information. The word match rate can be calculated as the ratio of words that match between the first participant and the second participant among all the words contained in the time-series text data in a predetermined fixed time period. can. The calculated word match rate can be said to be an index of whether or not the parrot is returned.

Further, the utterance relationship information generation unit 16 may extract the utterance timing from the section time-series text data of each of the first participant and the second participant, and calculate the temporal overlap of the utterance timing. good. Then, the utterance relationship information generation unit 16 may calculate the number of overlaps within a predetermined fixed time as the overlap frequency and include it in the utterance relationship information. The calculated overlap frequency can be said to be an index showing the frequency of interrupting the utterances of other participants.

Further, the utterance relationship information generation unit 16 may further refer to the participant information indicating the attributes of the first and second participants to generate the utterance relationship information. Participant information indicating the attributes of a participant includes at least one of the participant's age, gender, blood type, personality, place of origin, family relationship, job title, years of service, number of job changes, job history, and the like. Participant information also includes the usage history of the system.

The information processing device 10 is configured to further acquire biological information such as pulse waves and brain waves of the participant and environmental information such as temperature, humidity, carbon dioxide concentration, and illuminance around the participant, and is a speech-relationship information generation unit. Reference numeral 16 may generate speech-related information by further referring to biological information and environmental information.

Further, as an example, the utterance relationship information generation unit 16 determines the stress state of the participant determined from the pulse wave or respiration of the first participant, and the section time series text of the second participant immediately before or at that time. With reference to the data, the text data of the second participant that stresses the first participant is estimated. Then, the utterance relationship information generation unit 16 may recognize the estimated text data of the second participant as an NG word for the first participant, and include the information in the utterance relationship information. The NG word for one participant to another can be said to be an index showing the influence of one participant's remark on the stress state of another participant. Similarly, the speech relationship information generation unit 16 may estimate the text data of the first participant that stresses the second participant and certify the NG word for the second participant.

Further, as an example, the utterance relationship information generation unit 16 determines the activity of the participant's thinking determined from the brain waves of the first participant, and the section time-series text data of the second participant immediately before or at that time. And, the text data of the second participant that activates the thinking of the first participant is estimated. Then, the utterance relationship information generation unit 16 may recognize the estimated text data of the second participant as an important word for the first participant, and may include the information in the utterance relationship information. An important word for one participant to another can be said to be an index showing the influence of one participant's remarks on the activation of other participants' thoughts. Similarly, the utterance relationship information generation unit 16 may estimate the text data of the first participant that activates the thinking of the second participant, and may recognize the important word for the second participant.

Further, the utterance relationship information generation unit 16 may calculate the degree of matching of voice tones between participants, the degree of matching of utterance speed, the degree of matching of volume, and the like.

Further, the utterance relationship information generation unit 16 extracts logs representing the utterances of the first participant and the second participant from the accumulated section time-series text data, and acquires the morphological analysis data thereof. , Frequently-used words in past utterances may be listed and included in the utterance relationship information. Frequently spoken words in the past can be used to assist in meeting themes by presenting them to both participants during the meeting.

<Dialogue management process>
As an example, the utterance relationship information generation unit 16 has the user IDs of the first participant and the second participant, and the time and end of the conference represented by the user ID, via the communication unit 11. Dialog management processing is performed with reference to the time stamp indicating the time stamp. The utterance relationship information generation unit 16 extracts the user IDs of other participants who are interacting with each other from the data at a certain point in the section time series text data relating to one participant, and is interacting with which participant. It may be determined whether the data is obtained in the above, and the result may be included in the utterance relationship information.

(Relationship information generation unit 17)
The relationship information generation unit 17 refers to the facial expression relationship information and the utterance relationship information, and is a relationship that is real-time or temporal information indicating the relationship between the first participant and the second participant. Generate sexual information. By evaluating both the facial expressions and utterances of both participants during the meeting, the communication status between the participants can be evaluated in more detail. Further, since the relationship information generation unit 17 generates relationship information based on real-time or temporal facial expression information and utterance information between participants, it is necessary to evaluate the real-time or temporal communication state between participants. Can be done.

The relationship information generation unit 17 may generate presentation information to be presented to at least one of the first participant and the second participant. The presented information may include the degree of emotional matching between the participants, which is comprehensively evaluated based on the facial expression relationship information and the utterance relationship information (for example, the line-of-sight matching rate is high and the utterance ratio is high). If they are equal, increase the degree of emotional concordance, etc.).

By presenting the presentation information generated by the relationship information generation unit 17 to the participants, the relationship between the participants can be fed back to the participants. By presenting the presented information to the participants in real time, the relationship can be confirmed in real time during the conversation, so it is possible to promote the improvement of communication in real time.

The presented information may be presented to both the first participant and the second participant, or may be presented to either one. Further, the relationship information may present the same content to the first participant and the second participant, or may present different content. By generating relationship information that presents the same content to the first participant and the second participant, it can be expected to build a flat relationship between the participants. In addition, the presented information may be selectable by the participants themselves, or the presented information may be changed by a rule or an agreement between the participants.

The relationship information includes information indicating the ratio between the utterance time of the first participant and the utterance time of the second participant, the line-of-sight direction of the first participant, and the line-of-sight of the second participant. It may contain information about the time course of the match rate with the direction. In addition, the relationship information may include information on changes in the facial expression match rate or the facial expression mismatch rate over time, the forward lean rate, the text of the utterance content, frequently-used words, and the like. Furthermore, the relationship information includes the participant's ID, the participant's own facial image, the avatar image showing the facial expressions of other participants, the display of the recommended agenda or recommended words extracted from the accumulated data based on the utterance content, and the like. It may be.

In addition, the presented information may include evaluation results for improving the communication skills of the conference participants. As an example, along with the facial expression matching rate, information that encourages improvement of reflexive listening skills by increasing the facial expression matching rate can be presented, and by increasing the line-of-sight matching rate, the posture and attitude suitable for communication can be achieved. You may also present information that encourages you to take. In addition, recommended words and question contents may be presented to encourage dialogue-level and question-level deductions.

Specifically, as an example, the method of presenting the presented information is a method of displaying on the display unit of the conference participants (each of the display unit 23 and the display unit 33), and a common display unit that can be visually recognized by all the conference participants. Display method, presentation method to other than conference participants by network distribution, presentation method by physical action (vibration, electrical stimulation, etc.) from wearable devices such as watch-type devices, environmental equipment (certification, air conditioning, etc.) , Music, etc.) Physical action (such as illuminating the room red when the discussion heats up), Image image corresponding to an emotional index (such as a volcanic eruption that expresses anger) , A method of presenting by the facial expression of the avatar corresponding to the index expressing emotions and the like.

A screen example in which presentation information is presented to at least one of the display unit 23 and the display unit 33 will be described with reference to FIGS. 4 and 5. FIG. 4 is a diagram showing an example of information presented by an information processing system including an information processing device according to an embodiment of the present invention, and FIG. 5 is a diagram showing information including an information processing device according to an embodiment of the present invention. It is a figure which shows other example of the information which a processing system presents.

As shown in FIG. 4, on the screen 400, the user ID of the conference participant is displayed in the area 401, the face image of the conference participant is displayed in the area 402, and the target to be presented with the presentation information is specified. The utterance ratio is displayed as a talk ratio in the area 403, for example, in a pie chart, the facial expression of the avatar corresponding to the index representing the emotion of the participant during the dialogue is displayed in the area 404, and the facial expression matching rate with time is displayed in the area 405. By displaying the changes in a graph, it is possible to instantly check the current state of communication during the meeting. Further, in the area 406, a text representing the utterance content is displayed as a Talk Stream, and a conversation theme or a word recommended during the meeting is displayed. The facial expression matching rate displayed in the area 405 is an example of the presentation information generated based on the time-dependent information from the past to the present with respect to the facial expression relationship information. The facial expression of the avatar displayed in the area 404 is an example of the presentation information generated based on the real-time information regarding the facial expression relationship information.

Further, as shown in FIG. 5, on the screen 500, as in the screen 400, the user ID is displayed in the area 501, the facial expression is displayed in the area 502, the talk ratio is displayed in the area 503, and the avatar is displayed in the area 504. The facial expression may be displayed, the utterance content, the recommended theme, and the like may be displayed in the area 505, and the line-of-sight matching rate may be displayed in the area 505 instead of the facial expression matching rate.

<Appendix 1>
A part or all of the processing in the control unit of the terminal device may be performed in the arithmetic unit 12 provided in the information processing device 10. For example, the calculation unit 12 acquires the captured image of the camera 21 via the communication unit 11, and the facial expression information acquisition unit 13 obtains the first facial expression information regarding the facial expressions of the first participant and the second participation. It may be configured to generate a second facial expression information regarding a person's facial expression.

<Appendix 2>
Further, in the above example, a meeting of two participants, a first participant and a second participant, is taken as an example, but the present embodiment is not limited to this. As a matter of course, the invention described in the present specification can be applied to a meeting with N names (N is 3 or more). In that case, the configurations described herein can be applied individually to any two pairs of N names. For example, for a meeting of three people (A, B, C), the inventions described herein are individually applied to the three sets (A, B) (A, C) (B, C). can do.
As described above, the invention described in the present embodiment uses data representing the state of N persons, history data of the state of N persons, and environmental information of N persons, and participates in a part or all of the N persons. It can also be expressed as generating "relationship information" indicating the relationship between persons.
<Appendix 3>
Further, in the above example, the case where the first participant and the second participant are both human beings is taken as an example, but this does not limit the present embodiment.
For example, the second participant may not be a human but a pseudo-human represented by a computer such as a preset avatar or BOT. In the case of such a configuration, the second terminal device is not indispensable, and the facial expression information acquisition unit 13 and the voice information acquisition unit 14 input the facial expression and the utterance content represented by the BOT created in advance to the second participant. It may be configured to be acquired as facial expression information and voice information.
The facial expressions and utterances represented by the BOT may use data created in advance before the meeting, or may be adaptively changed according to the facial expressions and utterances of the first participant during the meeting. May be good.

[Example of implementation by software]
The control block (particularly, the arithmetic unit 12) of the information processing apparatus 1 may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software.

In the latter case, the information processing apparatus 1 includes a computer that executes instructions of a program that is software that realizes each function. The computer includes, for example, one or more processors and a computer-readable recording medium that stores the program. Then, in the computer, the processor reads the program from the recording medium and executes the program, thereby achieving the object of the present invention. As the processor, for example, a CPU (Central Processing Unit) can be used. As the recording medium, a "non-temporary tangible medium", for example, a ROM (Read Only Memory) or the like, a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. Further, a RAM (Random Access Memory) for expanding the above program may be further provided. Further, the program may be supplied to the computer via any transmission medium (communication network, broadcast wave, etc.) capable of transmitting the program. It should be noted that one aspect of the present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the above program is embodied by electronic transmission.

The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims, and the embodiments obtained by appropriately combining the technical means disclosed in the different embodiments. Is also included in the technical scope of the present invention.

100 Information processing system 10 Information processing device 20 First terminal device 30 Second terminal device 13 Expression information acquisition unit 14 Voice information acquisition unit 15 Expression relationship information generation unit 16 Speech relationship information generation unit 17 Relationship information generation unit

Claims (12)

A facial expression information acquisition unit that acquires the first facial expression information regarding the facial expression of the first participant among the plurality of participants and the second facial expression information regarding the facial expression of the second participant among the plurality of participants. ,
A voice information acquisition unit that acquires the first utterance information regarding the utterance of the first participant and the second utterance information regarding the utterance of the second participant among the plurality of participants.
A facial expression relationship that generates facial expression relationship information indicating a relationship between the first participant and the second participant with reference to the first facial expression information and the second facial expression information. Information generator and
The utterance relationship that generates the utterance relationship information showing the relationship between the first participant and the second participant by referring to the first utterance information and the second utterance information. Information generator and
A relationship information generation unit that generates relationship information, which is information indicating the relationship between the first participant and the second participant, with reference to the facial expression relationship information and the utterance relationship information.
An information processing device characterized by being equipped with. The information processing apparatus according to claim 1, wherein the relationship information is real-time or temporal information indicating the relationship between the first participant and the second participant. The first facial expression information includes a plurality of first indexes expressing the facial expressions of the first participant.
The second facial expression information includes a plurality of second indexes expressing the facial expressions of the second participant.
The facial expression relationship information generation unit is characterized in that it generates facial expression difference information regarding the difference between the first index and the second index, and includes the generated facial expression difference information in the facial expression relationship information. The information processing apparatus according to claim 1 or 2. The first facial expression information includes the first line-of-sight information regarding the line-of-sight direction of the first participant.
The second facial expression information includes the second line-of-sight information regarding the line-of-sight direction of the second participant.
The facial expression relationship information generation unit generates line-of-sight relationship information with reference to the first line-of-sight information and the second line-of-sight information, and includes the generated line-of-sight relationship information in the facial expression relationship information. The information processing apparatus according to any one of claims 1 to 3, wherein the information processing apparatus is characterized by the above. The utterance relationship information generation unit
The utterance time relationship information indicating the relationship between the utterance time of the first participant indicated by the first utterance information and the utterance time of the second participant indicated by the second utterance information is generated. The information processing apparatus according to any one of claims 1 to 4, wherein the generated utterance time relationship information is included in the utterance relationship information. The utterance relationship information generation unit
It is determined whether or not at least one of the first utterance information and the second utterance information includes the utterance content included in a specific category, and the information according to the determination result is the utterance relationship. The information processing apparatus according to any one of claims 1 to 5, wherein the information processing apparatus is included in the information. The utterance relationship information generation unit
A feature is that words having a relatively high frequency of appearance are extracted from at least one of the first utterance information and the second utterance information within a predetermined time, and the extracted words are included in the utterance relationship information. The information processing apparatus according to any one of claims 1 to 6. Claim 1 is characterized in that the relationship information generation unit refers to the relationship information and generates presentation information to be presented to at least one of the first participant and the second participant. The information processing apparatus according to any one of 7 to 7. The presented information includes
Information indicating the ratio between the utterance time of the first participant and the utterance time of the second participant,
as well as,
The information processing apparatus according to claim 8, wherein the information processing apparatus includes information on a change over time in a matching rate between the line-of-sight direction of the first participant and the line-of-sight direction of the second participant. The facial expression relationship information generation unit and the utterance relationship information generation unit further refer to the participant information indicating the attributes of the first and second participants, and further refer to the facial expression relationship information and the utterance relationship information. The information processing apparatus according to any one of claims 1 to 9, wherein information is generated. It is an information processing method performed by a computer under the control of software.
A facial expression information acquisition step for acquiring the first facial expression information regarding the facial expression of the first participant among the plurality of participants and the second facial expression information regarding the facial expression of the second participant among the plurality of participants. ,
A voice information acquisition step for acquiring the first utterance information regarding the utterance of the first participant and the second utterance information regarding the utterance of the second participant among the plurality of participants.
A facial expression relationship that generates facial expression relationship information indicating a relationship between the first participant and the second participant with reference to the first facial expression information and the second facial expression information. Information generation step and
The utterance relationship that generates the utterance relationship information showing the relationship between the first participant and the second participant by referring to the first utterance information and the second utterance information. Information generation steps and
A relationship that generates relationship information that is real-time or temporal information indicating the relationship between the first participant and the second participant by referring to the facial expression relationship information and the utterance relationship information. Information generation step and
An information processing method, characterized in that the computer performs the above. An information processing program for operating a computer as the information processing device according to any one of claims 1 to 10, wherein the facial information information acquisition unit, the voice information acquisition unit, and the facial expression relationship information generation unit. An information processing program for operating a computer as the speech-relationship information generation unit and the relationship information generation unit.

Images