JP7465013B2 - Video meeting evaluation terminal, video meeting evaluation system and video meeting evaluation program - Google Patents

Description

The present disclosure relates to a video meeting evaluation terminal, a video meeting evaluation system, and a video meeting evaluation program.

Conventionally, systems have been known for teaching knowledge or providing explanations online (see, for example, Patent Document 1).

In addition, as a method for measuring the effectiveness of such online video meetings, for example, a method of conducting a survey after the video meeting has been proposed (see, for example, Patent Document 2).

JP 2019-58625 A

The above-mentioned method of measuring effectiveness through surveys tends to be subjective and is insufficient as a way to objectively measure the effectiveness of the content of video meetings.

Another option would be to have a third party monitor the video meeting and obtain an objective evaluation, but this would be too time-consuming and laborious to be practical.

Therefore, the present invention aims to objectively evaluate video meetings, particularly their content.

According to the present invention,
An acquisition means for acquiring video images of a video conference held between the terminal and another terminal;
a face recognition means for recognizing at least a face image of a target person included in the moving image for each predetermined frame;
a voice recognition means for recognizing at least the voice of the subject contained in the moving image; and an evaluation means for calculating an evaluation value from a plurality of viewpoints based on both the recognized face image and the recognized voice.
and a notification means for issuing a notification when the relative deviation of the evaluation values from the plurality of viewpoints reaches or exceeds a certain level.

According to the present disclosure, by evaluating the video footage of a video meeting, it is possible to objectively evaluate the content in particular.

In particular, according to the present invention, since the acquired video is assumed to be stored on the terminal, it is analyzed and evaluated on the terminal, and the results are provided to the user of the terminal. Therefore, even if a video meeting contains personal information or confidential information, for example, the video itself can be analyzed and evaluated without providing it to an external evaluation company, etc.

In addition, convenience can be further improved by displaying the analysis and evaluation results in real time or after the fact by superimposing them on video images.

FIG. 1 is a diagram showing an overall system according to an embodiment of the present invention. FIG. 2 is a diagram illustrating a configuration example of a terminal according to an embodiment of the present invention. FIG. 2 is an example of a functional block diagram of an evaluation terminal according to the first embodiment of the present invention; FIG. 1 is a functional block diagram according to a first embodiment of the present invention. FIG. 1 is a functional block diagram according to a first embodiment of the present invention. FIG. 1 is a functional block diagram according to a first embodiment of the present invention. FIG. 1 is a functional block diagram according to a first embodiment of the present invention. FIG. 1 is a functional block diagram according to a first embodiment of the present invention. FIG. 11 is a functional block diagram according to a second embodiment of the present invention. FIG. 11 is a functional block diagram according to a second embodiment of the present invention. 13 is an implementation example according to the third embodiment of the present invention. 13 is an implementation example according to the fourth embodiment of the present invention. 13 is an implementation example according to the fifth embodiment of the present invention. 13 is an implementation example according to the sixth embodiment of the present invention.

The contents of the embodiments of the present disclosure will be described below. The present disclosure has the following configuration.
[Item 1]
An acquisition means for acquiring video images of a video conference held between the terminal and another terminal;
a face recognition means for recognizing at least a face image of a target person included in the moving image for each predetermined frame;
a voice recognition means for recognizing at least the voice of the subject contained in the moving image; and an evaluation means for calculating an evaluation value from a plurality of viewpoints based on both the recognized face image and the recognized voice.
and a notification means for issuing a notification when a relative deviation of the evaluation values from the plurality of viewpoints reaches or exceeds a certain level.
[Item 2]
Item 1 is a video meeting evaluation terminal according to the present invention,
the notification means issues a notification when the evaluation values obtained from the face image and the voice respectively deviate from each other by a certain amount or more.
Video meeting evaluation terminal.
[Item 3]
Item 1 is a video meeting evaluation terminal according to the present invention,
the notification means issues a notification when the evaluation values obtained from the facial movement obtainable from the facial image and the voice respectively deviate from each other by a certain amount or more.
Video meeting evaluation terminal.
[Item 4]
Item 1 is a video meeting evaluation terminal according to the present invention,
the notification means notifies when the degree of emotion evaluated from the face image and the evaluation value obtained from the voice deviate by a certain amount or more.
Video meeting evaluation terminal.
[Item 5]
Computer,
An acquisition means for acquiring video images of a video conference held between the terminal and another terminal;
a face recognition means for recognizing at least a face image of a target person included in the moving image for each predetermined frame;
a voice recognition means for recognizing at least the voice of the subject contained in the moving image;
an evaluation means for calculating an evaluation value from a plurality of viewpoints based on both the recognized face image and the recognized voice;
and a video meeting evaluation program that functions as a notification means for issuing a notification when the relative deviation of the evaluation values from the multiple viewpoints reaches a certain level or more.

A preferred embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings. Note that in this specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals to avoid redundant description.

<Basic functions>
The video meeting evaluation system of this embodiment (hereinafter sometimes simply referred to as "this system") is a system that analyzes and evaluates the unique emotions (feelings that arise in response to one's own or others' words and actions, such as pleasant or unpleasant, or the degree of such feelings) of a person being analyzed among multiple people in an environment where the person is holding a video meeting (hereinafter referred to as an online session, including both one-way and two-way sessions) that are different from those of the other people.

Online sessions include, for example, online conferences, online classes, and online chats, in which terminals installed in multiple locations are connected to a server via a communications network such as the Internet, allowing video and images to be exchanged between multiple terminals through the server.

The video images handled in an online session include facial images and audio of the users using the terminals. The video images also include images of documents shared and viewed by multiple users. It is possible to switch between facial images and document images on the screen of each terminal and display only one of them, or to divide the display area and display facial images and document images simultaneously. It is also possible to display an image of one of multiple people on the full screen, or to split the images of some or all users onto smaller screens and display them.

Of multiple users who use terminals to participate in an online session, it is possible to specify one or more users as the subject of analysis. For example, the leader, facilitator, or manager of the online session (hereinafter collectively referred to as the organizer) specifies one of the users as the subject of analysis. The organizer of an online session may be, for example, a lecturer of an online class, a chairperson or facilitator of an online conference, or a coach of a session for coaching purposes. The organizer of an online session is usually one of the multiple users who participate in the online session, but may also be a different person who does not participate in the online session. Note that it is also possible to analyze all participants without specifying an analysis subject.

It is also possible for the leader, facilitator or manager (hereinafter collectively referred to as the organizer) of an online session to specify a user as the subject of analysis. The organizer of an online session may be, for example, a lecturer of an online class, a chairperson or facilitator of an online conference, or a coach in a session for coaching purposes. The organizer of an online session is usually one of multiple users participating in the online session, but may also be a different person who does not participate in the online session.

In the video meeting evaluation system according to this embodiment, when a video meeting session is established between multiple terminals, at least a moving image acquired from the video meeting is displayed. The displayed moving image is acquired by the terminal, and at least a facial image included in the moving image is identified for each predetermined frame unit. An evaluation value for the identified facial image is then calculated. The evaluation value is shared as necessary.

In particular, in this embodiment, the acquired video is stored on the terminal, analyzed and evaluated on the terminal, and the results are provided to the user of the terminal. Therefore, even if the video is a video meeting that contains personal information or confidential information, it can be analyzed and evaluated without providing the video itself to an external evaluation agency, etc. Furthermore, if necessary, the results can be visualized or cross-analysis can be performed by providing only the evaluation results (evaluation values) to an external terminal.

As shown in FIG. 1, the video meeting evaluation system of this embodiment includes user terminals 10, 20 having at least an input unit such as a camera unit and a microphone unit, a display unit such as a display, and an output unit such as a speaker, a video meeting service terminal 30 that provides two-way video meetings to the user terminals 10, 20, and an evaluation terminal 40 that performs part of the evaluation related to the video meeting.

<Hardware configuration example>
2 is a diagram showing an example of the hardware configuration of a computer that realizes each of the terminals 10 to 40 according to this embodiment. The computer includes at least a control unit 110, a memory 120, a storage 130, a communication unit 140, and an input/output unit 150. These are electrically connected to each other via a bus 160.

The control unit 110 is a computing device that controls the overall operation of each terminal, controls the transmission and reception of data between each element, and performs information processing necessary for application execution and authentication processing. For example, the control unit 110 is a processor such as a CPU, and executes programs stored in the storage 130 and deployed in the memory 120 to perform each information processing.

The memory 120 includes a main memory configured with a volatile storage device such as a DRAM, and an auxiliary memory configured with a non-volatile storage device such as a flash memory or a HDD. The memory 120 is used as a work area for the control unit 110, and also stores the BIOS executed when each terminal is started up, various setting information, etc.

The storage 130 stores various programs such as application programs. A database storing data used for each process may be constructed in the storage 130. In particular, in this embodiment, the video images in the online session are not recorded in the storage 130 of the video meeting service terminal 30, but are stored in the storage 130 of the user terminal 10. The evaluation terminal 40 also stores applications and other programs necessary for evaluating the video images acquired on the user terminal 10, and provides them appropriately so that the user terminal 10 can use them. Note that, for example, only the results of analysis and evaluation by the user terminal 10 may be shared in the storage 13 managed by the evaluation terminal 40.

The communication unit 140 connects the terminal to a network. The communication unit 140 communicates with external devices directly or via a network access point, for example, by a wired LAN, a wireless LAN, Wi-Fi (registered trademark), infrared communication, Bluetooth (registered trademark), short-distance or contactless communication method, etc.

The input/output unit 150 is, for example, an information input device such as a keyboard, a mouse, a touch panel, etc., and an output device such as a display.

Bus 160 is commonly connected to each of the above elements and transmits, for example, address signals, data signals, and various control signals.

In particular, the evaluation terminal according to this embodiment acquires moving images from a video meeting service terminal, identifies at least facial images contained in the moving images for each predetermined frame unit, and calculates an evaluation value for the facial images (details will be described later).
<How to get the video>
As shown in Fig. 3, the video meeting service provided by the video meeting service terminal (hereinafter sometimes simply referred to as "the service") enables two-way communication with user terminals 10 and 20 using images and audio. The service enables a user terminal to display video images captured by the camera unit of the other user terminal on its display, and to output audio captured by the microphone unit of the other user terminal from its speaker.

The service is also configured to enable both or either of the user terminals to record (record) video and audio (collectively referred to as "video, etc.") in the memory of at least one of the user terminals. The recorded video information Vs (hereinafter referred to as "recorded information") is cached in the user terminal that initiated the recording and is recorded only locally on one of the user terminals. If necessary, the user can view the recorded information themselves or share it with others within the scope of their use of the service.

The user terminal 10 evaluates the video acquired in the above manner through the following analysis.

An embodiment of the present invention will now be described with reference to the drawings. FIG. 4 is a block diagram showing an example configuration according to this embodiment. As shown in FIG. 4, the video meeting evaluation system of this embodiment is realized as a functional configuration possessed by a user terminal 10. That is, the user terminal 10 has, as its functions, a video image acquisition unit 11, a biological reaction analysis unit 12, a peculiar determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification unit 16.

Each of the above-mentioned functional blocks 11 to 16 can be configured, for example, by hardware, a DSP (Digital Signal Processor), or software provided in the user terminal 10. For example, when configured by software, each of the above-mentioned functional blocks 11 to 16 is actually configured with a computer's CPU, RAM, ROM, etc., and is realized by the operation of a program stored in a recording medium such as the RAM, ROM, hard disk, or semiconductor memory.

The video acquisition unit 11 acquires video from each terminal, obtained by photographing multiple people (multiple users) with a camera provided on each terminal during an online session. The video acquired from each terminal may or may not be set to be displayed on the screen of each terminal. In other words, the video acquisition unit 11 acquires video from each terminal, including video being displayed and video not being displayed on each terminal.

The biological reaction analysis unit 12 analyzes changes in biological reactions for each of multiple people based on the moving images (whether or not they are being displayed on the screen) acquired by the moving image acquisition unit 11. In this embodiment, the biological reaction analysis unit 12 separates the moving images acquired by the moving image acquisition unit 11 into a set of images (a collection of frame images) and audio, and analyzes changes in biological reactions from each.

For example, the biological reaction analysis unit 12 analyzes a change in a biological reaction related to at least one of facial expression, eye line, pulse rate, and facial movement by analyzing the user's facial image using frame images separated from the video acquired by the video acquisition unit 11. The biological reaction analysis unit 12 also analyzes a change in a biological reaction related to at least one of the user's speech content and voice quality by analyzing the sound separated from the video acquired by the video acquisition unit 11.

When a person's emotions change, this is reflected in changes in biological reactions such as facial expressions, gaze, pulse rate, facial movements, speech, and voice quality. In this embodiment, the change in the user's emotions is analyzed by analyzing the change in the user's biological reactions. In this embodiment, the emotion analyzed is, for example, the degree of comfort/discomfort. In this embodiment, the biological reaction analysis unit 12 quantifies the change in the biological reaction according to a predetermined standard, thereby calculating a biological reaction index value that reflects the content of the change in the biological reaction.

Analysis of changes in facial expressions is performed, for example, as follows. That is, for each frame image, a facial area is identified within the frame image, and the identified facial expressions are classified into multiple categories according to an image analysis model that has been trained by machine learning in advance. Then, based on the classification results, it is analyzed whether a positive or negative facial expression change has occurred between consecutive frame images, and the extent of the facial expression change, and a facial expression change index value according to the analysis results is output.

Analysis of changes in eye line is performed, for example, as follows. That is, for each frame image, the eye area is identified from within the frame image, and the direction of both eyes is analyzed to analyze where the user is looking. For example, it is analyzed whether the user is looking at the face of the speaker being displayed, at the shared material being displayed, or looking outside the screen. It may also be possible to analyze whether the eye movement is large or small, and whether the movement is frequent or infrequent. Changes in eye line are also related to the user's level of concentration. The biological response analysis unit 12 outputs an eye line change index value according to the analysis result of changes in eye line.

The analysis of changes in pulse rate is performed, for example, as follows. That is, for each frame image, the facial area is identified from within the frame image. Then, using a trained image analysis model that captures the numerical values of facial color information (G in RGB), changes in the G color of the face surface are analyzed. The results are arranged along the time axis to form a waveform representing changes in color information, and the pulse rate is identified from this waveform. When a person is nervous, their pulse rate increases, and when they feel calm, their pulse rate decreases. The biological response analysis unit 12 outputs a pulse rate change index value according to the analysis results of changes in pulse rate.

The analysis of changes in facial movement is performed, for example, as follows. That is, for each frame image, the facial area is identified from within the frame image, and the facial direction is analyzed to analyze where the user is looking. For example, it is analyzed whether the user is looking at the face of the speaker being displayed, the shared material being displayed, or looking outside the screen. It is also possible to analyze whether the facial movement is large or small, and whether the movement occurs frequently or infrequently. It is also possible to analyze the facial movement together with the eye movement. For example, it is possible to analyze whether the user is looking straight at the speaker being displayed, looking up or down, or looking at an angle. The biological response analysis unit 12 outputs a facial direction change index value according to the analysis result of the change in facial direction.

The analysis of the speech content is performed, for example, as follows. That is, the biological response analysis unit 12 converts the speech into a string of characters by performing a known speech recognition process on the speech for a specified period of time (for example, about 30 to 150 seconds), and performs morphological analysis on the string of characters to remove words that are unnecessary for expressing the conversation, such as particles and articles. The remaining words are then vectorized, and an analysis is performed to determine whether a positive or negative emotional change has occurred, and to what extent the emotional change has occurred, and a speech content index value corresponding to the analysis result is output.

Voice quality analysis is performed, for example, as follows. That is, the biological response analysis unit 12 identifies the acoustic characteristics of the voice by performing a known voice analysis process on the voice for a specified period of time (for example, about 30 to 150 seconds). Then, based on the acoustic characteristics, it analyzes whether a positive or negative voice quality change has occurred and to what extent the voice quality change has occurred, and outputs a voice quality change index value according to the analysis result.

The biological reaction analysis unit 12 calculates a biological reaction index value using at least one of the facial expression change index value, eye gaze change index value, pulse rate change index value, face direction change index value, speech content index value, and voice quality change index value calculated as described above. For example, the biological reaction index value is calculated by weighting the facial expression change index value, eye gaze change index value, pulse rate change index value, face direction change index value, speech content index value, and voice quality change index value.

The uniqueness determination unit 13 determines whether the change in the biological reaction analyzed for the subject of analysis is unique compared to the change in the biological reaction analyzed for other people other than the subject of analysis. In this embodiment, the uniqueness determination unit 13 determines whether the change in the biological reaction analyzed for the subject of analysis is unique compared to other people based on the biological reaction index values calculated for each of the multiple users by the biological reaction analysis unit 12.

For example, the uniqueness determination unit 13 calculates the variance of the biological reaction index values calculated for each of multiple individuals by the biological reaction analysis unit 12, and by comparing the biological reaction index value calculated for the subject of analysis with the variance, determines whether or not the changes in the biological reactions analyzed for the subject of analysis are unique compared to others.

There are three possible cases where the changes in the analyzed biological reactions of the subject are unique compared to others. The first is when no particularly significant changes in the biological reactions of others occur, but a relatively large change in the biological reaction of the subject occurs. The second is when no particularly significant changes in the biological reactions of the subject occur, but a relatively large change in the biological reaction of others occurs. The third is when a relatively large change in the biological reaction occurs in both the subject and others, but the content of the change differs between the subject and others.

The related event identification unit 14 identifies an event occurring with respect to at least one of the subject of analysis, other people, and the environment when a change in a biological reaction determined to be unique by the uniqueness determination unit 13 occurs. For example, the related event identification unit 14 identifies the behavior of the subject of analysis himself/herself from a video when a unique change in a biological reaction occurs in the subject of analysis. The related event identification unit 14 also identifies the behavior of other people from the video when a unique change in a biological reaction occurs in the subject of analysis. The related event identification unit 14 also identifies the environment from the video when a unique change in a biological reaction occurs in the subject of analysis. The environment is, for example, a shared document being displayed on the screen, something that appears in the background of the subject of analysis, etc.

The clustering unit 15 analyzes the degree of correlation between a change in a biological reaction determined to be unique by the unique determination unit 13 (e.g., one or more combinations of eye movement, pulse rate, facial movement, speech content, and voice quality) and an event occurring when the unique change in the biological reaction occurs (an event identified by the related event identification unit 14), and if it is determined that the correlation is at or above a certain level, it clusters the person or event being analyzed based on the results of the correlation analysis.

For example, if a specific change in a biological reaction corresponds to a negative emotional change, and the event occurring when the specific change in the biological reaction occurs is also a negative event, a correlation above a certain level is detected. The clustering unit 15 clusters the subject of analysis or the event into one of multiple pre-segmented classifications according to the content of the event, the degree of negativity, the magnitude of correlation, etc.

Similarly, if a specific change in biological reaction corresponds to a positive emotional change, and the event occurring when the specific change in biological reaction occurs is also a positive event, a correlation above a certain level is detected. The clustering unit 15 clusters the subject of analysis or the event into one of multiple pre-segmented classifications according to the content of the event, the degree of positivity, the magnitude of correlation, etc.

The analysis result notification unit 16 notifies the person designating the analysis subject (the analysis subject or the organizer of the online session) of at least one of the changes in biological reactions determined to be specific by the uniqueness determination unit 13, the events identified by the related event identification unit 14, and the classifications clustered by the clustering unit 15.

For example, the analysis result notification unit 16 notifies the analysis subject of the analysis subject's own words and actions as an event occurring when a unique change in the analysis subject's biological reaction that differs from that of others occurs (any of the three patterns described above; the same applies below). This allows the analysis subject to understand that when he or she behaves in a certain way, he or she has different emotions than others. At this time, the analysis subject may also be notified of the unique changes in the analysis subject's biological reaction identified. Furthermore, the analysis subject may be further notified of the changes in the biological reaction of others for comparison.

For example, if the emotions felt by others in response to words or actions made by the subject without any particular awareness and with normal emotions, or words or actions made by the subject with particular emotions and awareness differ from the emotions felt by the subject himself at the time of the words or actions, the subject will be notified of the words or actions of the subject at that time. This makes it possible to discover words or actions that are well-received by others or not well-received by others despite one's own awareness.

Furthermore, the analysis result notification unit 16 notifies the organizer of the online session of events that occur when a unique change in the biological reaction of the analysis subject occurs that is different from that of others, along with the unique change in the biological reaction. This allows the organizer of the online session to know what events are influencing what emotional changes as a phenomenon unique to the specified analysis subject. Then, it becomes possible to take appropriate measures for the analysis subject depending on the content thus understood.

Furthermore, the analysis result notification unit 16 notifies the organizer of the online session of events occurring when a unique change in the biological reaction of the analysis subject occurs that is different from that of others, or the clustering results of the analysis subject. This allows the organizer of the online session to understand the behavioral tendencies unique to the analysis subject and predict possible future behaviors and conditions, etc., depending on which classification the specified analysis subject is clustered into. It then becomes possible to take appropriate measures for the analysis subject.

In the above embodiment, a bioreaction index value is calculated by quantifying the change in bioreaction according to a predetermined standard, and based on the calculated bioreaction index value for each of a plurality of people, it is determined whether or not the change in the bioreaction analyzed for the subject of analysis is unique compared to others. However, the present invention is not limited to this example. For example, the following may be used.

That is, the biological reaction analysis unit 12 analyzes the eye movement of each of the multiple people to generate a heat map showing the direction of the eye movement. The peculiar determination unit 13 compares the heat map generated by the biological reaction analysis unit 12 for the analysis subject with the heat map generated for other people to determine whether the change in the biological reaction analyzed for the analysis subject is unique compared to the change in the biological reaction analyzed for other people.

In this manner, in this embodiment, the video of the video meeting is stored in the local storage of the user terminal 10, and the above-mentioned analysis is performed on the user terminal 10. Although this may depend on the machine specifications of the user terminal 10, it is possible to analyze the video information without providing it to an external party.

The biological response analysis unit 12 may have an emotion evaluation unit that evaluates the degree of emotion of the subject according to an evaluation standard that is standardized among multiple subjects based on the changes in biological responses analyzed for the subject. For example, the emotion evaluation unit calculates an emotional response absolute value based on an evaluation standard that is standardized among multiple subjects based on the changes in biological responses (bioreaction index values) analyzed for the subject by the biological response analysis unit 12.

The emotional response absolute value calculated by the emotion evaluation unit is, for example, a value obtained by adjusting the biological response index value calculated by the biological response analysis unit 12 according to the likelihood of the subject experiencing the same emotion. For example, the emotion evaluation unit calculates the emotional response absolute value by multiplying the biological response index value calculated by the biological response analysis unit 12 by a weighting value according to the frequency with which the same emotion is experienced.

For example, even if the biological response index value calculated for subject A and the biological response index value calculated for subject B are the same value, if the ease with which the same emotion occurs (the frequency with which the same emotion occurs) differs between subject A and subject B, the emotional response absolute value calculated by the emotion evaluation unit will be different between subject A and subject B. As an example, the emotion evaluation unit calculates the emotional response absolute value according to a function such that the weighting value is smaller the more likely it is that the same emotion will occur, and the weighting value is larger the more unlikely it is that the same emotion will occur.

By using the absolute emotional response value calculated in this way, it is possible to evaluate the true degree of emotion of the subject, and to objectively compare the degrees of emotion between different subjects.

The emotion evaluation unit may evaluate the degree of emotion based on the magnitude of difference between the current biological reaction and the normal biological reaction, and adjusted according to the ease with which the subject will experience the same emotion. For example, the emotion evaluation unit calculates the absolute emotional reaction value by adjusting the biological reaction index value calculated by the biological reaction analysis unit 12 according to the magnitude of difference between the current biological reaction and the normal biological reaction and the ease with which the subject will experience the same emotion. The absolute emotional reaction value calculated in this manner is a value that represents the degree of emotion based on the magnitude of difference between the current biological reaction and the normal biological reaction, and is adjusted according to the ease with which the subject will experience the same emotion.

In the above embodiment, the frequency of occurrence of the same emotion is used as a measure of the likelihood of occurrence of the same emotion, but the present invention is not limited to this. For example, the subject's nature or character may be used instead of or in addition to the frequency of occurrence of the same emotion.

As shown in FIG. 5, the system may be provided with a reaction information presentation unit 13a that presents information indicating changes in biological reactions analyzed by the biological reaction analysis unit 12a, including participants not displayed on the screen. For example, the reaction information presentation unit 13a presents information indicating changes in biological reactions to a leader, facilitator, or manager of the online session (hereinafter collectively referred to as the organizer). The organizer of the online session may be, for example, a lecturer of an online class, a chairperson or facilitator of an online conference, or a coach of a session for coaching purposes. The organizer of the online session is usually one of multiple users participating in the online session, but may also be a different person who does not participate in the online session.

In this way, the organizer of an online session can keep track of participants who are not displayed on the screen in an environment where an online session is being held with multiple people.

First Embodiment
A first embodiment of the present system based on the above-mentioned configuration will be described with reference to Fig. 6 to Fig. 8. In this embodiment, a face image of a subject included in a video is recognized for each predetermined frame, and the voice of the subject is also recognized. Recognition may be performed for multiple subjects.

Next, the subject's emotions from multiple perspectives are quantified and evaluated based on both the recognized facial image and voice. The evaluated emotions are plotted on a graph along with their degree.

The graph is plotted along the timeline of the video. As shown in FIG. 6, a numerical value evaluated from one perspective, the happiness level (Happy Score), may be plotted for one subject in the video at a position. Also, as shown in FIG. 7, the degree of emotion from multiple perspectives may be plotted for multiple subjects in the same video.

Furthermore, as shown in Figure 8, the average level of emotion (which may be the highest, lowest, most frequent, etc.) in multiple videos that include a certain subject (e.g., videos of multiple classes taken online by a user, or videos of multiple online conferences that the user participated in) may be plotted with the title of the video on the horizontal axis and the level of emotion on the vertical axis. This makes it possible to visualize how the subject's emotions have changed as he or she has participated in multiple video meetings.

The graph shown is a line graph, but it could also be a bar graph, heat map, or any other type. It could also be color-coded to show different types of emotions.

The vertical axis of each graph may plot the degree of emotion of the subject for each type of emotion, for example, according to an evaluation standard that is standardized across multiple subjects. This allows for objective evaluation even on a uniform axis scale (e.g., 0 to 100). The same effect can also be achieved by evaluating the degree of emotion based on the magnitude of difference between the subject's current biological response and their normal biological response, adjusted according to the likelihood of the same emotion occurring, and plotting the adjusted degree of emotion for each type of emotion.

Second Embodiment
A second embodiment of the present invention will be described with reference to Figures 9 and 10. As shown in the figures, a search word box for accepting a search word is displayed on the screen according to this embodiment. When a word is input into the search word box (for example, when "base" is input), a predetermined range in a video image that includes a sound corresponding to the input search word is extracted and displayed.

The system according to the present embodiment includes a face recognition means for recognizing at least the face image of a subject included in a video for each predetermined frame, a voice recognition means for recognizing at least the voice of the subject included in the video, and a search reception means for receiving input of a search word. With this configuration, as shown in the figure, it is possible to display a part of the video corresponding to the part where "base" is spoken in the video file "20201230_Biology_Tanaka" and text information.

By displaying the extracted video and corresponding text information on the same screen, it is easy to check the situation and facial expression when the word was spoken.

In addition, the example screen shown in the figure has the word "base" extracted in three places. When a displayed search word (for example, the very first "base") is selected, a video containing frames when that word was spoken is (partially) played.

If multiple search words are extracted from a video, a digest video may be generated by stitching together multiple partial videos that contain the search words. This allows users to quickly and efficiently check the video surrounding the searched word.

It is also possible to accept search words to be registered in advance, and to issue an alert if the registered search word is extracted within a video, or to automatically generate a digest video by stitching together multiple partial videos that contain the search word.

For example, as shown in Figure 10, words such as those shown in the alert value column can be registered and associated with information indicating the location of video information in which the word appears.

Furthermore, when the word is spoken during a video meeting, a playback link to that point in the video may be generated.

Registered words can be easily managed by tagging them in advance (alert pattern).

It is also possible to count how many times a word occurs in a video, for example by counting the number of times the word appears in the recognized speech (or in the converted text information).

Third Embodiment
A system according to a third embodiment of the present invention will be described with reference to Fig. 11. In this embodiment, even when the other party has turned off the camera (a state in which visual information obtained from the camera is not provided), it is possible to provide the other party with information that the user is within the range of the camera (i.e., in front of the computer) and his/her facial expression at that time.

When multiple participants participate in a video conference, such as a class or meeting, it is possible that information obtained from the camera could be used for malicious purposes. For example, someone could take screenshots of other people's faces or view private spaces, such as rooms, reflected in the background. For this reason, many participants turn off their camera functions when participating.

However, if the camera is turned off, the host cannot see the guest's face and cannot be sure whether they are properly participating in the lecture or meeting. In this way, a situation may arise where the host wants to know whether the guest is properly participating in front of the camera, while the guest wants to convey that they are properly participating but does not want to turn the camera on.

Therefore, in this embodiment, the guest user's terminal acquires video images from the guest user's camera, identifies at least the facial image contained in the video images for each specified frame, and converts the identified facial image into facial information and provides it to the host user terminal.

Examples of facial information include, but are not limited to, whether the person is in front of the camera or not, the direction of the face, emotions derived from facial expressions captured by the camera, and object information generated based on those emotions.

With this configuration, the guest user's private information (visual information obtained by the camera) is not shared with the host side, but information such as that the guest has been detected by the camera, that they are looking directly at the screen, and their facial expressions can be provided.

<Fourth embodiment>
A system according to a fourth embodiment of the present invention will be described with reference to Fig. 12. In this embodiment, information relating to emotions is added to text information obtained from a video image.

The system according to this embodiment outputs text by associating it with an evaluation value related to emotions. For example, as shown in FIG. 12, content spoken in a voice louder than a certain volume is displayed in a larger font size, and words spoken while looking directly at the camera are underlined.

In other words, according to this embodiment, it becomes possible to apply predetermined processing to the text based on the user's direct words and actions obtained from video images, the results of analyzing such words and actions, etc.

User behavior can be registered in advance, including gestures, hand movements, facial direction, facial movements, as well as voice intonation, volume, pitch, speed, etc. Examples of processing applied to text include changing the font size, thickness, italics, changing the text color, adding a shadow, changing the font type, etc.

In addition, for example, the system may accept registration of conditions related to audio (e.g., a certain volume or greater), and change the text corresponding to audio that meets the registered conditions to a specified display format (displayed larger).

Fifth embodiment
A system according to a fifth embodiment of the present invention will be described with reference to Fig. 13. In this embodiment, latent emotions are analyzed based on discrepancies (divergences in evaluation values) between pieces of analytical evaluation information from a plurality of perspectives obtained from video images.

Specifically, the system calculates evaluation values from multiple perspectives based on both the facial image and the voice. If the relative deviation between the evaluation values from the multiple perspectives exceeds a certain level, a notification is issued by the notification means.

Examples of patterns of deviation include when the evaluation values obtained from the facial image and voice respectively deviate by more than a certain amount, when the evaluation values obtained from the facial movement obtained from the facial image and voice respectively deviate by more than a certain amount, and when the degree of emotion evaluated from the facial image and the evaluation value obtained from the voice deviate by more than a certain amount.

For example, as shown in FIG. 13, when comparing a point (H01) on a graph representing happiness with a point (A01) on a graph representing anger at a certain point, happiness shows a significantly higher value, and it is presumed that the user is feeling happy (good communication). On the other hand, when comparing a point (H02) on a graph representing happiness with a point (A02) on a graph representing anger at another point, both show similar values, resulting in emotional contradiction. In such a case, for example, it is possible that the person is smiling on the face, but is secretly suspicious or feeling suspicious.

According to this embodiment, it is possible to estimate true emotions based on the deviation of evaluation values. The deviation can be determined based on a predetermined correlation, or a machine learning-based determination can be used.

Sixth embodiment
A system according to a sixth embodiment of the present invention will be described with reference to Fig. 14. In this embodiment, evaluation values obtained by analyzing and evaluating videos of multiple users participating at the same time are displayed in association with each of the videos. This allows the status of all participants to be understood at once, like a heat map.

The system includes a display means for displaying a list of video images obtained from the cameras of multiple users simultaneously participating in a video meeting, and an association means for displaying an object associated with an evaluation value for each of the multiple video images.

For example, the association means may generate a heat map according to the evaluation values and overlay the corresponding colors on each of the videos. As shown in FIG. 14, users with high anger levels may be grayed out.

The association means may also generate joy, anger, sadness, and happiness icons according to the evaluation value and display the corresponding icons alongside each of the moving images.

Seventh embodiment
A seventh embodiment of the present invention will be described with reference to Fig. 4 again. In this embodiment, a notification means is provided for issuing an alert on a predetermined terminal (screen or the like) when a unique reaction different from the previous reaction is analyzed for the same user determined by the above-mentioned unique determination unit 13.

The notification means issues a notification when the reaction exceeds a threshold range. For example, a notification may be issued when the anger level of a user who normally does not get angry becomes significantly higher than before, or when a user who normally does not laugh laughs. It is possible to register in advance what kind of notification will be issued under what conditions.

The notification means may also be configured to issue the notification when a particular user's reaction exceeds a predetermined number of times during an online meeting held during the same time period (a single conference, a single class, etc.).

<Hardware configuration supplement>
The series of processes performed by the device described in this specification may be realized using any of software, hardware, and a combination of software and hardware. A computer program for realizing each function of the information sharing support device 10 according to this embodiment may be created and installed in a PC or the like. It is also possible to provide a computer-readable recording medium in which such a computer program is stored. The recording medium may be, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like. The above computer program may also be distributed, for example, via a network, without using a recording medium.

In addition, the processes described in this specification using flow chart diagrams do not necessarily have to be performed in the order shown. Some processing steps may be performed in parallel. Also, additional processing steps may be employed, and some processing steps may be omitted.

In addition, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology disclosed herein may provide other effects that are apparent to a person skilled in the art from the description herein, in addition to or in place of the above effects.

10, 20 User terminal 30 Video meeting service terminal 40 Evaluation terminal

Claims (1)

An acquisition means for acquiring video images of a video conference held between the terminal and another terminal;
a face recognition means for recognizing at least a face image of a target person included in the moving image for each predetermined frame;
a voice recognition means for recognizing at least the voice of the subject contained in the moving image; and an evaluation means for calculating an evaluation value from a plurality of viewpoints based on both the recognized face image and the recognized voice.
a notification means for notifying the user when a relative deviation of the evaluation values from the plurality of viewpoints becomes equal to or exceeds a certain level ;
the evaluation means estimates a third emotion including the first and second emotions when a first emotion evaluated from the face image and a second emotion obtained from the voice are contradictory to each other;
Video meeting evaluation terminal.

Images