JP7465040B1 - Communication visualization system - Google Patents

Abstract

[Problem] To visualize communications in order to communicate more efficiently in situations where online communication is the norm, such as in meetings and lectures. [Solution] The communication visualization system disclosed herein is a communication visualization server having a video acquisition unit that acquires video images of each of a plurality of users by filming the user during an online session, an analysis unit that analyzes communication between the plurality of users based on the video images acquired by the video acquisition unit, and a display unit that displays the results of the analysis, in which the display unit displays user icons representing the users and connectors that connect the user icons according to the communication between the users. [Selected Figure] Figure 1

Description

The present invention relates to a communication visualization system that analyzes the reactions of participants in an environment where an online session is held with multiple participants, based on video images obtained by photographing the participants, regardless of whether the participants are displayed on the screen during the online session.

There is known a technique for analyzing the emotions felt by others in response to a speaker's statement (see, for example, Patent Document 1). There is also known a technique for chronologically analyzing changes in a subject's facial expression over a long period of time and estimating the emotions felt during that time (see, for example, Patent Document 2). There is also known a technique for identifying the factor that most influenced changes in emotions (see, for example, Patent Documents 3 to 5). There is also known a technique for comparing a subject's usual facial expression with their current facial expression and issuing an alert if the facial expression is gloomy (see, for example, Patent Document 6). There is also known a technique for comparing a subject's normal (expressionless) facial expression with their current facial expression to determine the degree of the subject's emotions (see, for example, Patent Documents 7 to 9). There is also known a technique for analyzing organizational emotions and the atmosphere within a group that an individual feels (see, for example, Patent Documents 10 and 11).

JP 2019-58625 A JP 2016-149063 A JP 2020-86559 A JP 2000-76421 A JP 2017-201499 A JP 2018-112831 A JP 2011-154665 A JP 2012-8949 A JP 2013-300 A JP 2011-186521 A Publication No. WO15/174426

All of the above technologies are merely secondary functions in situations where communication in the real world is the main focus. In other words, they were not created in a situation where communication for work, classes, etc. is primarily conducted online, in response to the recent digital transformation (DX) of work and the global pandemic.

The present invention aims to visualize communications in order to make them more efficient in situations where online communication is the norm, such as meetings and lectures.

The communication visualization system of the present invention is a communication visualization server having a video acquisition unit that acquires video for each of a plurality of users by photographing the user during an online session, an analysis unit that analyzes communication between the plurality of users based on the video acquired by the video acquisition unit, and a display unit that displays the results of the analysis, in which the display unit displays user icons representing the users and connectors that connect the user icons according to the communication between the users.

According to the present disclosure, in situations where online communication is the norm, such as meetings and lectures, it is possible to visualize such communications in order to make them more efficient.

FIG. 1 is a diagram showing an overall system according to an embodiment of the present invention. FIG. 2 is a block diagram showing a configuration example according to the present embodiment. FIG. 2 is a block diagram showing a configuration example according to the present embodiment. FIG. 1 illustrates an example of a communication diagram. FIG. 1 illustrates an example of a communication diagram. FIG. 1 illustrates an example of a communication diagram. FIG. 1 illustrates an example of a communication diagram. FIG. 1 illustrates an example of a communication diagram.

The contents of the embodiments of the present disclosure will be described below. The present disclosure has the following configuration.
[Item 1] <<User icon and connector (related to Figure 4)>>
a video acquisition unit that acquires, for each of a plurality of users, a video obtained by photographing the user during an online session;
an analysis unit that analyzes communications between the plurality of users based on the video acquired by the video acquisition unit;
a communication visualization server having a display unit that displays a result of the analysis,
the display unit displays user icons representing the users and connectors connecting the user icons in response to communication between the users.
Communication visualization system.
[Item 2] <<When an icon is selected, the connector with the person you communicated with turns orange. Related to Figure 5/Figure 6>>
2. The communication visualization system according to claim 1,
the display unit displays the connector connected to the user icon in a distinguishable manner when the user icon is selected.
Communication visualization system.
[Item 3] <<Display supplementary information when an icon is selected (see Figure 7)>>
2. The communication visualization system according to claim 1,
the display unit displays, when the user icon is selected, statistical information regarding communication of the user corresponding to the user icon.
Communication visualization system.
[Item 4] <<Display connectors by department>>
2. The communication visualization system according to claim 1,
Each of the plurality of users is associated with a predetermined group;
the display unit displays a group icon indicating the group and a connector connecting the group icons in accordance with communication between the users belonging to the group.
Communication visualization system.
[Item 5] <<Calculation of loneliness rate>>
2. The communication visualization system according to claim 1,
the display unit calculates and displays a ratio of the users who have a predetermined number of communication partners or less among the plurality of users.
Communication visualization system.
[Item 6] <<Estimation of communication load by order>>
2. The communication visualization system according to claim 1,
The method further includes a communication load calculation unit that calculates a communication load according to a communication order between the users,
The display unit is a communication visualization system that displays the calculated communication load.
[Item 7] <<Communication load estimated by conversation volume>>
2. The communication visualization system according to claim 1,
The communication load calculation unit further includes a communication load calculation unit that calculates a communication load according to a bias in the amount of communication between the users,
The display unit is a communication visualization system that displays the calculated communication load.

A preferred embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configuration are designated by the same reference numerals to avoid redundant description.

<Basic functions>
The communication visualization system of the present embodiment is a system that analyzes and evaluates a unique emotion (feelings caused by one's own or other's words and actions, such as pleasantness or unpleasantness, or the degree of pleasantness) of an analysis target among a plurality of people in an environment where the plurality of people are engaged in a video session (hereinafter, one-way and two-way sessions are both referred to as online sessions) that is different from that of other people. An online session is, for example, an online conference, an online class, an online chat, etc., and terminals installed in a plurality of locations are connected to a server via a communication network such as the Internet, and moving images can be exchanged between the plurality of terminals through the server. The moving images handled in the online session include facial images and voices of users using the terminals. In addition, the moving images also include images such as materials shared and viewed by a plurality of users. It is possible to switch between a facial image and a material image on the screen of each terminal to display only one of them, or to divide the display area to display a facial image and a material image simultaneously. It is also possible to display an image of one of the plurality of people in full screen, or to display images of some or all of the users divided into small screens. It is possible to specify one or more of the plurality of users who participate in the online session using a terminal as the analysis target. For example, the leader, facilitator, or manager of the online session (hereinafter collectively referred to as the organizer) designates one of the users as the analysis target. The organizer of the online session is, for example, a lecturer of an online class, a chairperson or facilitator of an online conference, or a coach of a session for coaching purposes. The organizer of the online session is usually one of the multiple users who participate in the online session, but may be a different person who does not participate in the online session. It is also possible to not designate an analysis target and to use all participants as the analysis target. It is also possible for the leader, facilitator, or manager of the online session (hereinafter collectively referred to as the organizer) to designate one of the users as the analysis target. The organizer of the online session is, for example, a lecturer of an online class, a chairperson or facilitator of an online conference, or a coach of a session for coaching purposes. The organizer of the online session is usually one of the multiple users who participate in the online session, but may be a different person who does not participate in the online session.

In the communication visualization system according to the present embodiment, when a video session is established between multiple terminals, at least a moving image acquired from the video session is displayed. The displayed moving image is acquired by the terminal, and at least a facial image contained in the moving image is identified for each predetermined frame unit. An evaluation value for the identified facial image is then calculated. The evaluation value is shared as necessary. In particular, in the present embodiment, the acquired moving image is stored in the terminal, analyzed and evaluated on the terminal, and the results are provided to the user of the terminal. Therefore, even if a video session includes personal information or confidential information, for example, the video itself can be analyzed and evaluated without providing the video itself to an external evaluation agency, etc. Also, if necessary, the evaluation result (evaluation value) alone can be provided to an external terminal, allowing the results to be visualized and cross-analysis, etc. to be performed.

As shown in FIG. 1, the communication visualization system according to this embodiment includes user terminals 10 and 20 having at least an input unit such as a camera unit and a microphone unit, a display unit such as a display, and an output unit such as a speaker, a video session service terminal 30 that provides a two-way video session to the user terminals 10 and 20, and an evaluation terminal 40 that performs part of the evaluation of the video session.

<Hardware configuration example>
Each functional block, functional unit, and functional module described below can be configured, for example, by hardware, DSP (Digital Signal Processor), or software provided in a computer. For example, when configured by software, it is actually configured with a computer's CPU, RAM, ROM, etc., and is realized by the operation of a program stored in a recording medium such as a RAM, ROM, hard disk, or semiconductor memory. The series of processes by the system and terminal described in this specification may be realized using any of software, hardware, and a combination of software and hardware. It is possible to create a computer program for realizing each function of the user terminal 10 according to this embodiment and implement it in a PC or the like. It is also possible to provide a computer-readable recording medium in which such a computer program is stored. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, etc. In addition, the above computer program may be distributed, for example, via a network, without using a recording medium.

The evaluation terminal in this embodiment acquires a moving image from a video session service terminal, identifies at least a facial image contained in the moving image for each predetermined frame unit, and calculates an evaluation value for the facial image (details will be described later).

<How to get the video>
As shown in FIG. 2, the video session service provided by the video session service terminal (hereinafter, sometimes simply referred to as "this service") enables two-way communication with user terminals 10 and 20 using images and audio. This service is capable of displaying video images captured by the camera unit of the other user terminal on the display of the user terminal, and outputting audio captured by the microphone unit of the other user terminal from the speaker. This service is also configured to enable both or any of the user terminals to record (record) video images and audio (collectively referred to as "video images, etc.") in the memory unit of at least one of the user terminals. The recorded video information Vs (hereinafter referred to as "recorded information") is cached in the user terminal that started the recording, and is recorded only locally in one of the user terminals. If necessary, the user can view the recorded information himself or share it with others within the scope of use of this service.

<Functional configuration example 1>
Fig. 3 is a block diagram showing an example of the configuration according to this embodiment. As shown in Fig. 3, the communication visualization system of this embodiment is realized as a functional configuration of a user terminal 10. That is, the user terminal 10 includes, as its functions, a moving image acquisition unit 11, a biological reaction analysis unit 12, a display unit 13, and a communication load calculation unit 14.

The video acquisition unit 11 acquires video from each terminal, obtained by photographing multiple people (multiple users) with a camera provided on each terminal during an online session. The video acquired from each terminal may or may not be set to be displayed on the screen of each terminal. In other words, the video acquisition unit 11 acquires video from each terminal, including video being displayed and video not being displayed on each terminal.

The bioreaction analysis unit 12 analyzes changes in bioreactions for each of the multiple people based on the video images (whether or not they are being displayed on the screen) acquired by the video image acquisition unit 11. In this embodiment, the bioreaction analysis unit 12 is an analysis unit that analyzes communication between the multiple users (who was talking to whom, when, and for how long) based on the video images acquired by the video image acquisition unit 11, separates the video images acquired by the video image acquisition unit 11 into a set of images (a collection of frame images) and audio, and analyzes changes in bioreactions from each of them.

For example, the biological reaction analysis unit 12 analyzes a change in a biological reaction related to at least one of facial expression, eye line, pulse rate, and facial movement by analyzing the user's facial image using frame images separated from the video acquired by the video acquisition unit 11. The biological reaction analysis unit 12 also analyzes a change in a biological reaction related to at least one of the user's speech content and voice quality by analyzing the sound separated from the video acquired by the video acquisition unit 11.

When a person's emotions change, this is reflected in changes in biological reactions such as facial expressions, gaze, pulse rate, facial movements, speech, and voice quality. In this embodiment, the change in the user's emotions is analyzed by analyzing the change in the user's biological reactions. In this embodiment, the emotion analyzed is, for example, the degree of comfort/discomfort. In this embodiment, the biological reaction analysis unit 12 quantifies the change in the biological reaction according to a predetermined standard, thereby calculating a biological reaction index value that reflects the content of the change in the biological reaction.

Analysis of changes in facial expressions is performed, for example, as follows. That is, for each frame image, a facial area is identified within the frame image, and the identified facial expressions are classified into multiple categories according to an image analysis model that has been trained by machine learning in advance. Then, based on the classification results, it is analyzed whether a positive or negative facial expression change has occurred between consecutive frame images, and the extent of the facial expression change, and a facial expression change index value according to the analysis results is output.

Analysis of changes in eye line is performed, for example, as follows. That is, for each frame image, the eye area is identified from within the frame image, and the direction of both eyes is analyzed to analyze where the user is looking. For example, it is analyzed whether the user is looking at the face of the speaker being displayed, at the shared material being displayed, or looking outside the screen. It may also be possible to analyze whether the eye movement is large or small, and whether the movement is frequent or infrequent. Changes in eye line are also related to the user's level of concentration. The biological response analysis unit 12 outputs an eye line change index value according to the analysis result of changes in eye line.

The analysis of changes in pulse rate is performed, for example, as follows. That is, for each frame image, the facial area is identified from within the frame image. Then, using a trained image analysis model that captures the numerical values of facial color information (G in RGB), changes in the G color of the face surface are analyzed. The results are arranged along the time axis to form a waveform representing changes in color information, and the pulse rate is identified from this waveform. When a person is nervous, their pulse rate increases, and when they feel calm, their pulse rate decreases. The biological response analysis unit 12 outputs a pulse rate change index value according to the analysis results of changes in pulse rate.

The analysis of changes in facial movement is performed, for example, as follows. That is, for each frame image, the facial area is identified from within the frame image, and the facial direction is analyzed to analyze where the user is looking. For example, it is analyzed whether the user is looking at the face of the speaker being displayed, the shared material being displayed, or looking outside the screen. It is also possible to analyze whether the facial movement is large or small, and whether the movement occurs frequently or infrequently. It is also possible to analyze the facial movement together with the eye movement. For example, it is possible to analyze whether the user is looking straight at the speaker being displayed, looking up or down, or looking at an angle. The biological response analysis unit 12 outputs a facial direction change index value according to the analysis result of the change in facial direction.

The analysis of the speech content is performed, for example, as follows. That is, the biological response analysis unit 12 converts the speech into a string of characters by performing known speech recognition processing on the speech for a specified period of time (for example, about 30 to 150 seconds), and performs morphological analysis on the string of characters to remove words that are unnecessary for expressing the conversation, such as particles and articles. The remaining words are then vectorized, and an analysis is performed to determine whether a positive or negative emotional change has occurred, and to what extent the emotional change has occurred, and a speech content index value corresponding to the analysis result is output.

Voice quality analysis is performed, for example, as follows. That is, the biological response analysis unit 12 identifies the acoustic characteristics of the voice by performing a known voice analysis process on the voice for a specified period of time (for example, about 30 to 150 seconds). Then, based on the acoustic characteristics, it analyzes whether a positive or negative voice quality change has occurred and to what extent the voice quality change has occurred, and outputs a voice quality change index value according to the analysis result.

The biological reaction analysis unit 12 calculates a biological reaction index value using at least one of the facial expression change index value, eye gaze change index value, pulse rate change index value, face direction change index value, speech content index value, and voice quality change index value calculated as described above. For example, the biological reaction index value is calculated by weighting the facial expression change index value, eye gaze change index value, pulse rate change index value, face direction change index value, speech content index value, and voice quality change index value.

The display unit 13 displays the results of the communication analysis performed by the biological reaction analysis unit 12. The display unit 13 displays a screen that visualizes communication. Hereinafter, a diagram that visualizes communication is referred to as a communication diagram. FIG. 4 is a diagram showing an example of a communication diagram. In the illustrated communication diagram, the amount of communication between each user is represented by the thickness of the line connecting each user. As shown in FIG. 4, the display unit 13 displays user icons 50 representing users, and connectors 51 that connect the user icons 50 according to the communication between the users.

When a user icon is selected, the display unit 13 displays the connectors connected to the user icon so that they can be distinguished. For example, as shown in Fig. 5, when the user icon of user A is selected, the connectors connecting the user icon of user A to each of the user icons of users B to N with whom user A communicated turn colored, for example. Also, for example, as shown in Fig. 6, when the user icon of user G is selected, the connectors connecting the user icon of user G to each of the user icons of users A, E, F, I, and H with whom user G communicated turn colored, for example.

When a user icon is selected, the display unit 13 displays statistical information regarding the communication of the user corresponding to the user icon. As shown in FIG. 7, when the user icon of user A is selected, "Collaboration (number of people who communicated)/22 people" and "Total meeting time/6 hours 36 minutes" are displayed as statistical information regarding user A's communication.

In addition, in the communication diagram, as shown in FIG. 8, each department such as "Sales" and "Design" may be treated as a "group" to which multiple users belong. Each of the multiple users is associated with a specific group. As shown in FIG. 8, each department may be displayed connected by a connector just like each user. That is, as shown in FIG. 8, the display unit 13 displays a group icon indicating the group, and connectors that connect the group icons according to the communication between users belonging to the group.

The display unit 13 can calculate and display the ratio (loneliness rate) of users who have a certain number of communication partners or less among multiple users. This makes it possible to visualize users who are at the center of a conversation and those who are not included in the conversation and are isolated.

The communication visualization system according to this embodiment further includes a communication load calculation unit 14. The communication load calculation unit 14 calculates the communication load according to the order of communication between users. The display unit 13 displays the calculated communication load. The method of calculating the load is such that, for example, if user B replies after user A makes a statement, it is determined that user A has imposed a load on user B, and 1 point is added up for user B.

The communication load calculation unit 14 may calculate the communication load according to the imbalance in the amount of communication between users. The method of calculating the load is such that, if the amount of conversation of user A is greater than the amount of conversation of user B, it is estimated that user A has imposed a load on user B and a score according to the conversation time is added.

The processes described in this specification using flow chart diagrams do not necessarily have to be performed in the order shown. Some process steps may be performed in parallel. Additional process steps may also be employed, and some process steps may be omitted.

The above-described embodiments may be implemented in appropriate combinations. Furthermore, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology disclosed herein may achieve other effects that are apparent to a person skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

10, 20 User terminal 11 Video image acquisition unit 12 Biological reaction analysis unit 13 Display unit 50 User icon 51 Connector

Claims (6)

a video acquisition unit that acquires, for each of a plurality of users, a video obtained by photographing the user during an online session;
an analysis unit that analyzes communications between the plurality of users based on the video acquired by the video acquisition unit;
a communication visualization server having a display unit that displays a result of the analysis,
the display unit displays user icons representing the users and connectors connecting the user icons according to communications between the users, and calculates and displays a ratio of the users who have a predetermined number of communication partners or less among the plurality of users.
Communication visualization system. a video acquisition unit that acquires, for each of a plurality of users, a video obtained by photographing the user during an online session;
an analysis unit that analyzes communications between the plurality of users based on the video acquired by the video acquisition unit;
A display unit that displays the results of the analysis;
a communication load calculation unit that calculates a communication load according to an order of communications between the users,
the display unit displays user icons representing the users and connectors connecting the user icons in accordance with communications between the users, and displays the calculated communication load.
Communication visualization system. a video acquisition unit that acquires, for each of a plurality of users, a video obtained by photographing the user during an online session;
an analysis unit that analyzes communications between the plurality of users based on the video acquired by the video acquisition unit;
A display unit that displays the results of the analysis;
a communication load calculation unit that calculates a communication load according to a bias in the amount of communication between the users,
the display unit displays user icons representing the users and connectors connecting the user icons in accordance with communications between the users, and displays the calculated communication load.
Communication visualization system. The communication visualization system according to any one of claims 1 to 3,
the display unit displays the connector connected to the user icon in a distinguishable manner when the user icon is selected.
Communication visualization system. The communication visualization system according to any one of claims 1 to 3,
the display unit displays, when the user icon is selected, statistical information regarding communication of the user corresponding to the user icon.
Communication visualization system. The communication visualization system according to any one of claims 1 to 3,
Each of the plurality of users is associated with a predetermined group;
the display unit displays a group icon indicating the group and a connector connecting the group icons in accordance with communication between the users belonging to the group.
Communication visualization system.