JP7100938B1 - Video analysis program - Google Patents

Abstract

[Problem] To objectively evaluate these communications in order to conduct more efficient communication in situations such as meetings and lectures where online communication is the main. A system of the present disclosure includes a moving image acquisition unit that acquires a moving image obtained by shooting a participant during an online session; a word analysis unit that analyzes the words contained in the moving image and converts them into predetermined document vectors; and a plot that plots the converted document vectors along with the words in a predetermined dimension. and a generator. [Selection drawing] Fig. 1

Description

The present invention also provides a moving image obtained by photographing a participant in an environment where an online session is held by a plurality of participants, regardless of whether or not the participant is displayed on the screen during the online session. It relates to a moving image analysis system that analyzes the reactions of participants.

A technique for analyzing emotions received by others in response to a speaker's remark is known (see, for example, Patent Document 1). Further, there is also known a technique of analyzing changes in a subject's facial expression over a long period of time in a time series and estimating emotions held during that period (see, for example, Patent Document 2). Further, a technique for identifying an element that has the greatest influence on emotional changes is also known (see, for example, Patent Documents 3 to 5). Furthermore, there is also known a technique of comparing a subject's normal facial expression with the current facial expression and issuing an alert when the facial expression is dark (see, for example, Patent Document 6). Further, there is also known a technique of comparing a normal (expressionless) facial expression of a subject with a current facial expression to determine the degree of emotion of the subject (for example, Patent Documents 7 to 9). reference). Furthermore, techniques for analyzing the emotions of an organization and the atmosphere within a group felt by an individual are also known (see, for example, Patent Documents 10 and 11).

Japanese Unexamined Patent Publication No. 2019-58625 Japanese Unexamined Patent Publication No. 2016-149063 Japanese Unexamined Patent Publication No. 2020-86559 Japanese Unexamined Patent Publication No. 2000-76421 Japanese Unexamined Patent Publication No. 2017-201499 Japanese Unexamined Patent Publication No. 2018-112831 Japanese Unexamined Patent Publication No. 2011-154665 Japanese Unexamined Patent Publication No. 2012-8949 Japanese Unexamined Patent Publication No. 2013-300 Japanese Unexamined Patent Publication No. 2011-186521 WO15 / 174426

All of the techniques mentioned above are only sub-functions in situations where communication in real space is predominant. In other words, due to the recent shift to DX (Digital Transformation) in business and the worldwide epidemic of infectious diseases, it was not born in a situation where communication such as business and classes is mainly conducted online.

An object of the present invention is to objectively evaluate these communications in order to perform more efficient communication in a situation where online communication is the main such as a conference or a lecture.

According to the present invention
In an environment where an online session is held by a plurality of participants, regardless of whether or not the participants are displayed on the screen during the online session, the above is based on the moving image obtained by photographing the participants. It is a moving image analysis system that analyzes the reaction of participants.
A moving image acquisition unit that acquires a moving image obtained by photographing the participant during the online session, and a moving image acquisition unit.
An analysis unit that analyzes changes in biological reactions of the participants based on the moving images acquired by the moving image acquisition unit.
A word analysis unit that analyzes words contained in the moving image and converts them into a predetermined document vector.
A plot generator that plots the converted document vector in a given dimension with the word,
A moving image analysis system can be obtained.

According to the present disclosure, by analyzing and evaluating the moving image of the video session, it is possible to objectively evaluate the content in particular.

In particular, according to the present invention, in a situation where online communication is the main aspect, it is possible to objectively evaluate the exchanged communication in order to perform more efficient communication.

It is a figure which shows the whole system by embodiment of this invention. This is an example of a functional block diagram of an evaluation terminal according to an embodiment of the present invention. It is a figure which shows the functional structure example 1 of the evaluation terminal by embodiment of this invention. It is a figure which shows the functional structure example 2 of the evaluation terminal by embodiment of this invention. It is a figure which shows the functional structure example 3 of the evaluation terminal by embodiment of this invention. It is a screen display example by the function configuration example 3 of FIG. 6 is another screen display example according to the functional configuration example 3 of FIG. It is a figure which shows the other configuration of the functional configuration example 3 of the evaluation terminal according to the embodiment of the present invention. It is a figure which shows the other configuration of the functional configuration example 3 of the evaluation terminal according to the embodiment of the present invention. It is a functional block diagram of the system by embodiment of this invention. It is an image diagram of the plot generated by the system by embodiment of this invention. It is a figure which shows the image of the processing of the system by the embodiment of this invention. It is an image diagram of the plot generated by the system by embodiment of this invention.

The contents of the embodiments of the present disclosure will be listed and described. The present disclosure comprises the following configurations.
[Item 1]
In an environment where an online session is held by a plurality of participants, regardless of whether or not the participants are displayed on the screen during the online session, the above is based on the moving image obtained by photographing the participants. It is a moving image analysis system that analyzes the reaction of participants.
A moving image acquisition unit that acquires a moving image obtained by photographing the participant during the online session, and a moving image acquisition unit.
An analysis unit that analyzes changes in biological reactions of the participants based on the moving images acquired by the moving image acquisition unit.
A word analysis unit that analyzes words contained in the moving image and converts them into a predetermined document vector.
A plot generator that plots the converted document vector in a given dimension with the word,
Video analysis system.
[Item 2]
The moving image analysis system according to item 1.
A reproduction unit for reproducing a moving image including a corresponding word when the plot is manipulated from a frame containing the word is further provided.
Video analysis system.
[Item 3]
The moving image analysis system according to item 1 or item 2.
Further provided with an evaluation unit for evaluating the degree of overlap by comparing the document vector included in the prior information prepared before the start of the online session with the document vector of the moving image of the online session.
Video analysis system.
[Item 4]
A moving image analysis device comprising the configuration of the moving image analysis system according to any one of items 1 to 3.
[Item 5]
A moving image analysis program that causes the moving image analysis device to function the configuration of the moving image analysis system according to any one of items 1 to 3.
[Item 6]
A moving image analysis method for executing the configuration of the moving image analysis system according to any one of items 1 to 3 as a step.

Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, and duplicate description will be omitted.

<Basic function>
The video session evaluation system of the present embodiment is different from others in the analysis target person among the plurality of people in an environment where a video session (hereinafter referred to as an online session including one-way and two-way) is performed by a plurality of people. It is a system that analyzes and evaluates specific emotions (feelings that occur in the words and actions of oneself or others, such as comfort / discomfort or the degree thereof). An online session is, for example, an online conference, an online class, an online chat, etc., in which terminals installed in multiple locations are connected to a server via a communication network such as the Internet, and moving images are transmitted between the terminals through the server. It is designed to be able to communicate. The moving images handled in the online session include facial images and sounds of the user who uses the terminal. In addition, the moving image also includes an image such as a material shared and viewed by a plurality of users. It is possible to switch between the face image and the material image on the screen of each terminal to display only one of them, or to divide the display area and display the face image and the material image at the same time. Further, it is possible to display the image of one of a plurality of people on the full screen, or to display the image of a part or all of the users on a small screen. It is possible to specify any one or more of the plurality of users who participate in the online session using the terminal as the analysis target person. For example, the leader, facilitator, or administrator of an online session (hereinafter collectively referred to as the organizer) designates any user as the analysis target. Organizers of online sessions include, for example, instructors of online classes, chairs and facilitators of online conferences, and coaches of sessions for coaching purposes. The organizer of an online session is usually one of a plurality of users who participate in the online session, but may be another person who does not participate in the online session. In addition, all the participants may be the analysis target without designating the analysis target person. It is also possible for the leader, facilitator, or administrator of the online session (hereinafter collectively referred to as the organizer) to designate any user as the analysis target. Organizers of online sessions include, for example, instructors of online classes, chairs and facilitators of online conferences, and coaches of sessions for coaching purposes. The organizer of an online session is usually one of a plurality of users who participate in the online session, but may be another person who does not participate in the online session.

When a video session session is established between a plurality of terminals, the video session evaluation system according to the present embodiment displays at least a moving image acquired from the video session. The displayed moving image is acquired by the terminal, and at least the facial image contained in the moving image is identified for each predetermined frame unit. After that, the evaluation value for the identified facial image is calculated. The evaluation value is shared as necessary. In particular, in the present embodiment, the acquired moving image is stored in the terminal, analyzed and evaluated on the terminal, and the result is provided to the user of the terminal. Therefore, for example, even a video session containing personal information or a video session containing confidential information can be analyzed and evaluated without providing the video itself to an external evaluation organization or the like. Further, by providing only the evaluation result (evaluation value) to the external terminal as needed, the result can be visualized, cross analysis, or the like can be performed.

As shown in FIG. 1, the video session evaluation system according to the present embodiment has user terminals 10 and 20 having at least an input unit such as a camera unit and a microphone unit, a display unit such as a display unit, and an output unit such as a speaker. It also includes a video session service terminal 30 that provides bidirectional video sessions to user terminals 10 and 20, and an evaluation terminal 40 that performs a part of evaluation related to the video session.

<Hardware configuration example>
Each functional block, functional unit, and functional module described below can be configured by any of hardware, DSP (Digital Signal Processor), and software provided in a computer, for example. For example, when it is configured by software, it is actually configured to include a CPU, RAM, ROM, etc. of a computer, and is realized by operating a program stored in a recording medium such as RAM, ROM, a hard disk, or a semiconductor memory. The series of processes by the system and the terminal described in the present specification may be realized by using any software, hardware, and a combination of software and hardware. It is possible to create a computer program for realizing each function of the information sharing support device 10 according to the present embodiment and implement it on a PC or the like. It is also possible to provide a computer-readable recording medium in which such a computer program is stored. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like. Further, the above computer program may be distributed, for example, via a network without using a recording medium.

The evaluation terminal according to the present embodiment acquires a moving image from the video session service terminal, identifies at least the facial image contained in the moving image for each predetermined frame unit, and calculates the evaluation value for the facial image (). Details will be described later).

<How to get video>
As shown in FIG. 3, the video session service provided by the video session service terminal (hereinafter, may be simply referred to as “the service”) is bidirectionally imaged and voiced with respect to the user terminals 10 and 20. Communication is possible. This service displays a moving image acquired by the camera unit of the other user terminal on the display of the user terminal, and can output the sound acquired by the microphone unit of the other user terminal from the speaker. In addition, this service may record (record) moving images and audio (collectively referred to as "moving images, etc.") in a storage unit on at least one of the user terminals by either or both user terminals. It is configured to be possible. The recorded moving image information Vs (hereinafter referred to as "recording information") is cached in the user terminal that started recording and is recorded only locally in one of the user terminals. If necessary, the user can view the recorded information by himself / herself within the scope of using this service, share it with others, and so on.

<Function configuration example 1>
FIG. 4 is a block diagram showing a configuration example according to the present embodiment. As shown in FIG. 4, the video session evaluation system of the present embodiment is realized as a functional configuration of the user terminal 10. That is, the user terminal 10 includes a moving image acquisition unit 11, a biological reaction analysis unit 12, a peculiarity determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification unit 16 as its functions.

The moving image acquisition unit 11 acquires a moving image obtained by photographing a plurality of people (a plurality of users) with a camera provided in each terminal during an online session from each terminal. It does not matter whether the moving image acquired from each terminal is set to be displayed on the screen of each terminal. That is, the moving image acquisition unit 11 acquires the moving image from each terminal, including the moving image being displayed on each terminal and the moving image being hidden.

The biological reaction analysis unit 12 analyzes changes in the biological reaction of each of a plurality of persons based on the moving image (whether or not it is displayed on the screen) acquired by the moving image acquisition unit 11. In the present embodiment, the biological reaction analysis unit 12 separates the moving image acquired by the moving image acquisition unit 11 into a set of images (a collection of frame images) and a voice, and analyzes changes in the biological reaction from each.

For example, the biological reaction analysis unit 12 analyzes the user's face image using the frame image separated from the moving image acquired by the moving image acquisition unit 11, and thereby at least one of the facial expression, the line of sight, the pulse, and the movement of the face. Analyze changes in biological reactions related to one. In addition, the biological reaction analysis unit 12 analyzes changes in the biological reaction regarding at least one of the user's speech content and voice quality by analyzing the voice separated from the moving image acquired by the moving image acquisition unit 11.

When a person's emotions change, they appear as changes in biological reactions such as facial expressions, eyes, pulse, facial movements, speech content, and voice quality. In the present embodiment, changes in the user's emotions are analyzed by analyzing changes in the user's biological reaction. The emotion analyzed in this embodiment is, for example, the degree of comfort / discomfort. In the present embodiment, the biological reaction analysis unit 12 calculates the biological reaction index value reflecting the content of the change in the biological reaction by quantifying the change in the biological reaction according to a predetermined standard.

The analysis of changes in facial expressions is performed, for example, as follows. That is, for each frame image, a facial area is specified from the frame image, and the specified facial expressions are classified into a plurality of types according to an image analysis model trained in advance by machine learning. Then, based on the classification result, it is analyzed whether a positive facial expression change occurs between consecutive frame images, a negative facial expression change occurs, and how large the facial expression change occurs. The facial expression change index value according to the analysis result is output.

The analysis of the change in the line of sight is performed, for example, as follows. That is, for each frame image, the area of the eyes is specified from the frame image, and the orientation of both eyes is analyzed to analyze where the user is looking. For example, it analyzes whether the speaker's face being displayed, the shared material being displayed, or the outside of the screen is being viewed. In addition, it may be possible to analyze whether the movement of the line of sight is large or small, and whether the movement is frequent or infrequent. The change in the line of sight is also related to the degree of concentration of the user. The biological reaction analysis unit 12 outputs the line-of-sight change index value according to the analysis result of the line-of-sight change.

The analysis of the change in the pulse is performed, for example, as follows. That is, for each frame image, the face area is specified from the frame image. Then, using a trained image analysis model that captures the numerical value of the face color information (G in RGB), the change in the G color on the face surface is analyzed. By arranging the results along the time axis, a waveform showing the change in color information is formed, and the pulse is specified from this waveform. When a person is nervous, the pulse becomes faster, and when he / she feels calm, the pulse becomes slower. The biological reaction analysis unit 12 outputs a pulse change index value according to the analysis result of the pulse change.

The analysis of changes in facial movement is performed, for example, as follows. That is, for each frame image, the area of the face is specified from the frame image, and the orientation of the face is analyzed to analyze where the user is looking. For example, it analyzes whether the speaker's face being displayed, the shared material being displayed, or the outside of the screen is being viewed. In addition, it may be possible to analyze whether the movement of the face is large or small, and whether the movement is frequent or infrequent. The movement of the face and the movement of the line of sight may be combined and analyzed. For example, it may be possible to analyze whether the speaker's face being displayed is viewed straight, whether the speaker is viewed with an upper eye or a lower eye, or whether the speaker is viewed from an angle. The biological reaction analysis unit 12 outputs a face orientation change index value according to the analysis result of the face orientation change.

The content of the statement is analyzed, for example, as follows. That is, the biological reaction analysis unit 12 converts the voice into a character string by performing a known voice recognition process on the voice for a specified time (for example, a time of about 30 to 150 seconds), and performs morphological analysis of the character string. By doing so, words that are unnecessary for expressing conversation such as auxiliary words and acronyms are removed. Then, the remaining words are vectorized, and whether a positive emotional change is occurring, a negative emotional change is occurring, and how large the emotional change is occurring is analyzed, and the analysis result is used. Outputs the statement content index value.

The voice quality is analyzed, for example, as follows. That is, the biological reaction analysis unit 12 identifies the acoustic characteristics of the voice by performing a known voice analysis process on the voice for a specified time (for example, a time of about 30 to 150 seconds). Then, based on the acoustic characteristics, it is analyzed whether a positive voice quality change is occurring, a negative voice quality change is occurring, and how loud the voice quality change is occurring, and according to the analysis result. Outputs the voice quality change index value.

The biological reaction analysis unit 12 uses at least one of the facial expression change index value, the line-of-sight change index value, the pulse change index value, the face orientation change index value, the speech content index value, and the voice quality change index value calculated as described above. The biological reaction index value is calculated. For example, the biological reaction index value is calculated by weighting the facial expression change index value, the line-of-sight change index value, the pulse change index value, the face orientation change index value, the speech content index value, and the voice quality change index value.

The peculiarity determination unit 13 determines whether or not the change in the biological reaction analyzed for the analysis target person is specific to the change in the biological reaction analyzed for a person other than the analysis target person. In the present embodiment, the peculiarity determination unit 13 compares the changes in the biological reaction analyzed for the analysis target person with those of others based on the biological reaction index values calculated for each of the plurality of users by the biological reaction analysis unit 12. To determine whether it is specific or not.

For example, the peculiarity determination unit 13 calculates the variance of the biological reaction index value calculated for each of a plurality of persons by the biological reaction analysis unit 12, and compares the biological reaction index value calculated for the analysis target person with the variance. It is determined whether or not the change in the biological reaction analyzed for the person to be analyzed is specific compared to the other person.

The following three patterns can be considered as cases where the change in the biological reaction analyzed for the person to be analyzed is more specific than that of the other person. The first is the case where a particularly large change in the biological reaction has not occurred in the other person, but a relatively large change in the biological reaction has occurred in the person to be analyzed. The second is the case where a particularly large change in the biological reaction has not occurred in the subject to be analyzed, but a relatively large change in the biological reaction has occurred in the other person. The third is the case where a relatively large change in the biological reaction occurs in both the analysis target person and the other person, but the content of the change is different between the analysis target person and the other person.

The related event identification unit 14 identifies an event occurring with respect to at least one of the analysis subject, another person, and the environment when a change in the biological reaction determined to be specific by the peculiarity determination unit 13 occurs. .. For example, the related event identification unit 14 identifies the behavior of the analysis target person himself / herself from the moving image when a specific change in the biological reaction occurs for the analysis target person. In addition, the related event identification unit 14 identifies the words and actions of another person from the moving image when a specific change in the biological reaction occurs for the analysis target person. In addition, the related event identification unit 14 identifies the environment when a specific change in the biological reaction occurs for the analysis target person from the moving image. The environment is, for example, a shared material displayed on the screen, an environment reflected in the background of the person to be analyzed, and the like.

The clustering unit 15 includes changes in biological reactions determined to be specific by the peculiarity determination unit 13 (for example, one or a combination of eyes, pulse, facial movement, speech content, and voice quality) and the peculiarity. When the degree of correlation with the event (event specified by the related event identification unit 14) that occurs when a change in the biological reaction occurs is analyzed, and it is determined that the correlation is above a certain level. , Cluster the analysis target person or event based on the analysis result of the correlation.

For example, if a change in a specific biological reaction corresponds to a negative emotional change, and the event occurring when the specific change in the biological reaction occurs is also a negative event, a certain level The above correlation is detected. The clustering unit 15 clusters the analysis target person or the event in any of a plurality of pre-segmented classifications according to the content of the event, the degree of negativeness, the magnitude of the correlation, and the like.

Similarly, if a specific change in biological reaction corresponds to a positive emotional change, and the event occurring when the specific change in biological reaction occurs is also constant if it is a positive event. Correlation above the level is detected. The clustering unit 15 clusters the analysis target person or the event in any of a plurality of pre-segmented classifications according to the content of the event, the degree of positiveness, the magnitude of the correlation, and the like.

The analysis result notification unit 16 determines at least one of the changes in the biological reaction determined to be specific by the peculiarity determination unit 13, the event specified by the related event identification unit 14, and the classification clustered by the clustering unit 15. , Notify the designated person of the analysis target (analysis target person or the organizer of the online session).

For example, the analysis result notification unit 16 analyzes the analysis target as an event that occurs when a specific change in biological reaction occurs in the analysis target person (any of the above-mentioned three patterns; the same applies hereinafter). Notify the person to be analyzed of the person's own words and actions. As a result, the person to be analyzed can grasp that he / she has different emotions from others when he / she makes a certain word or action. At this time, the change of the specific biological reaction specified for the analysis target person may also be notified to the analysis target person. Further, the change in the biological reaction of the other person to be compared may be further notified to the analysis target person.

For example, the words and actions that the analysis target person performed without being particularly conscious of the usual emotions, or the feelings and actions received by others for the words and actions that the analysis target person specifically consciously performed with certain emotions. If the emotions held by the analysis target person are different from each other at the time, the analysis target person's own words and actions at that time are notified to the analysis target person. This makes it possible to discover words and behaviors that are well received by others and words and behaviors that are not well received by others, contrary to one's consciousness.

In addition, the analysis result notification unit 16 organizes the online session of the events occurring when the specific biological reaction of the analysis target person is different from that of others, together with the specific change of the biological reaction. Notify to. This allows the organizer of the online session to know what kind of phenomenon influences what kind of emotional change as a phenomenon peculiar to the designated analysis target person. Then, it becomes possible to take appropriate measures for the analysis target person according to the grasped contents.

In addition, the analysis result notification unit 16 notifies the organizer of the online session of the event occurring when the analysis target person has a specific change in biological reaction different from that of others or the clustering result of the analysis target person. do. As a result, the organizer of the online session can grasp the behavior tendency peculiar to the analysis target person and predict the behavior or state that may occur in the future, depending on which classification the specified analysis target person is clustered into. be able to. Then, it becomes possible to take appropriate measures for the analysis target person.

In the above embodiment, the biological reaction index value is calculated by quantifying the change in the biological reaction according to a predetermined standard, and the analysis target person is analyzed based on the biological reaction index value calculated for each of the plurality of persons. An example of determining whether or not a change in a biological reaction has been made is specific compared to another person has been described, but the present invention is not limited to this example. For example, it may be as follows.

That is, the biological reaction analysis unit 12 analyzes the movement of the line of sight for each of the plurality of people and generates a heat map showing the direction of the line of sight. In the peculiarity determination unit 13, the change in the biological reaction analyzed for the analysis target person is measured by the comparison between the heat map generated for the analysis target person by the biological reaction analysis unit 12 and the heat map generated for the other person. It is determined whether or not it is specific by comparing with the change in the biological reaction analyzed for.

As described above, in the present embodiment, the moving image of the video session is stored in the local storage of the user terminal 10, and the above-mentioned analysis is performed on the user terminal 10. Although it may depend on the machine specifications of the user terminal 10, it is possible to analyze the moving image information without providing it to the outside.

<Function configuration example 2>
As shown in FIG. 5, the video session evaluation system of the present embodiment may include a moving image acquisition unit 11, a biological reaction analysis unit 12, and a reaction information presentation unit 13a as functional configurations.

The reaction information presentation unit 13a presents information indicating changes in the biological reaction analyzed by the biological reaction analysis unit 12a, including the participants who are not displayed on the screen. For example, the reaction information presentation unit 13a presents information indicating changes in the biological reaction to the leader, facilitator, or manager of the online session (hereinafter collectively referred to as the organizer). Organizers of online sessions include, for example, instructors of online classes, chairs and facilitators of online conferences, and coaches of sessions for coaching purposes. The organizer of an online session is usually one of a plurality of users who participate in the online session, but may be another person who does not participate in the online session.

By doing so, the organizer of the online session can also grasp the state of the participants who are not displayed on the screen in the environment where the online session is held by a plurality of people.

<Functional configuration example 3>
FIG. 6 is a block diagram showing a configuration example according to the present embodiment. As shown in FIG. 6, in the video session evaluation system of the present embodiment, as a functional configuration, the same reference numerals may be given to the functions similar to those of the above-described first embodiment, and the description thereof may be omitted.

The system according to the present embodiment includes a camera unit for acquiring video of a video session, a microphone unit for acquiring audio, an analysis unit for analyzing and evaluating moving images, and information obtained by evaluating the acquired moving images. It includes an object generation unit that generates display objects (described later) based on the above, and a display unit that displays both a moving image of a video session and a display object during execution of the video session.

Similar to the above description, the analysis unit includes a moving image acquisition unit 11, a biological reaction analysis unit 12, a peculiarity determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification unit 16. The function of each element is as described above.

As shown in FIG. 7, the object generation unit has, if necessary, an object 50 indicating the recognized face portion based on the result of analyzing the moving image acquired from the video session by the analysis unit, and the above-mentioned object 50. The information 100 indicating the analyzed / evaluated content is superimposed and displayed on the moving image. When the faces of a plurality of people are moved in the moving image, the object 50 may identify and display the faces of all of the plurality of people.

Further, the object 50 is stopped by software in the application of the video session, for example, when the camera function of the video session is stopped (that is, the camera is not physically covered by the other party's terminal). However, if the other party's camera recognizes the other party's face, the object 50 or the object 100 may be displayed in the portion where the other party's face is located. This makes it possible for each other to confirm that the other party is in front of the terminal even if the camera function is turned off. In this case, for example, in a video session application, the information acquired from the camera may be hidden, while only the object 50 or the object 100 corresponding to the face recognized by the analysis unit may be displayed. Further, the video information acquired from the video session and the information recognized by the analysis unit may be divided into different display layers, and the layer related to the former information may be hidden.

When the object 50 or the object 100 has an area for displaying a plurality of moving images, the object 50 or the object 100 may be displayed only in all or a part of the area. For example, as shown in FIG. 8, it may be displayed only on the moving image on the guest side.

The embodiment of the invention described in the basic configuration example 1 to the basic configuration example 3 described above may be realized as a single device, or a plurality of devices (for example, a cloud server) partially or wholly connected by a network. It may be realized by such as. For example, the control unit 110 and the storage 130 of each terminal 10 may be realized by different servers connected to each other by a network. That is, the system includes user terminals 10 and 20, a video session service terminal 30 that provides bidirectional video sessions to user terminals 10 and 20, and an evaluation terminal 40 that evaluates video sessions. Variation combinations of the following configurations can be considered.
(1) All processing is performed only by the user terminal As shown in FIG. 9, a video session is performed (although a certain processing capacity is required) by performing the processing by the analysis unit on the terminal that is performing the video session. Analysis / evaluation results can be obtained at the same time as the time spent (in real time).
(2) Processing between the user terminal and the evaluation terminal As shown in FIG. 10, the evaluation terminal connected by a network or the like may be provided with an analysis unit. In this case, the moving image acquired by the user terminal is shared with the evaluation terminal at the same time as the video session or after the fact, and after being analyzed and evaluated by the analysis unit in the evaluation terminal, the information of the object 50 and the object 100 is obtained by the user. It is shared with or separately from the moving image data (that is, information including at least the analysis data) to the terminal and displayed on the display unit.

The following system is realized by using each of the above-mentioned functional configuration examples 1 to 3 or a combination thereof.

<Embodiment>
The moving image analysis system according to the embodiment of the present invention (hereinafter, simply referred to as “system”) photographs all of the participants or only a specific participant in an environment where an online session is held by a plurality of participants. The reaction of the participants is analyzed and analyzed based on the moving image obtained by the above. The analysis may be performed whether or not the participant is displayed on the screen during the online session.

As shown in FIG. 10, the system according to the present embodiment includes a moving image acquisition unit, an analysis unit, a text extraction unit, a word analysis unit, and a plot generation unit. The moving image acquisition unit acquires a moving image obtained by photographing a participant during an online session. The analysis unit analyzes changes in the biological reaction of the participants based on the moving images acquired by the moving image acquisition unit (see also FIGS. 3 to 5 and the like).

As shown in FIG. 11, the word analysis unit analyzes the words contained in the moving image and converts them into a predetermined document vector. The plot generator plots the converted document vector in a predetermined dimension together with the word. The example shown is an analysis of a "lecture" given by a lecturer and a student. As shown in the figure, the words "course", "pass", and "on the day" related to the academic test are plotted near the vertical axis as words used in the moving image, and near the horizontal axis in the moving image. The words "mathematics", "equation", and "solution" related to a specific subject (mathematics) academic test are plotted as the words used.

As shown in FIG. 12, for example, when the point of "course" is selected from the plot shown in FIG. 11, the moving image containing the word "course" is extracted and played (sequentially or selected above). Will be done. The moving image is reproduced from the frame containing the word.

As shown in FIG. 13, the document vector (pre-vector) included in the prior information prepared before the start of the online session is compared with the document vector (post-vector) of the moving image obtained in the online session. Further, an evaluation unit for evaluating the degree of overlap may be provided. As shown, the blackened points represent the pre-vector and the shaded points represent the post-vector.

The processes described with reference to the flow charts herein may not necessarily be performed in the order shown. Some processing steps may be performed in parallel. Further, additional processing steps may be adopted, and some processing steps may be omitted.

The embodiments described above may be combined as appropriate. In addition, the effects described herein are merely explanatory or exemplary and are not limited. That is, the techniques according to the present disclosure may have other effects apparent to those skilled in the art from the description herein, in addition to or in place of the above effects.

10, 20 User terminal 30 Video session service terminal 40 Evaluation terminal

Claims (2)

In an environment where an online session is held by a plurality of participants, regardless of whether or not the participants are displayed on the screen during the online session, the above is based on the moving image obtained by photographing the participants. It is a moving image analysis system that analyzes the reaction of participants.
A moving image acquisition unit that acquires a moving image obtained by photographing the participant during the online session, and a moving image acquisition unit.
An analysis unit that analyzes changes in biological reactions of the participants based on the moving images acquired by the moving image acquisition unit.
A word analysis unit that analyzes words contained in the moving image and converts them into a predetermined document vector.
A plot generator that plots the converted document vector in a given dimension with the word,
The document vector included in the prior information prepared before the start of the online session is compared with the document vector of the moving image of the online session to evaluate the degree of overlap.
Video analysis system. The moving image analysis system according to claim 1.
A reproduction unit for reproducing a moving image including a corresponding word when the plot is manipulated from a frame containing the word is further provided.
Video analysis system.

Images