WO2022064619A1 - Video meeting evaluation system and video meeting evaluation server - Google Patents

Video meeting evaluation system and video meeting evaluation server Download PDF

Info

Publication number
WO2022064619A1
WO2022064619A1 PCT/JP2020/036148 JP2020036148W WO2022064619A1 WO 2022064619 A1 WO2022064619 A1 WO 2022064619A1 JP 2020036148 W JP2020036148 W JP 2020036148W WO 2022064619 A1 WO2022064619 A1 WO 2022064619A1
Authority
WO
WIPO (PCT)
Prior art keywords
video meeting
evaluation
terminal
biological reaction
moving image
Prior art date
Application number
PCT/JP2020/036148
Other languages
French (fr)
Japanese (ja)
Inventor
渉三 神谷
Original Assignee
株式会社I’mbesideyou
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 株式会社I’mbesideyou filed Critical 株式会社I’mbesideyou
Priority to PCT/JP2020/036148 priority Critical patent/WO2022064619A1/en
Priority to JP2022515723A priority patent/JPWO2022064619A1/ja
Publication of WO2022064619A1 publication Critical patent/WO2022064619A1/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/76—Television signal recording
    • H04N5/765—Interface circuits between an apparatus for recording and another apparatus

Definitions

  • This disclosure relates to a video meeting evaluation system and a video meeting evaluation server.
  • Patent Document 1 a system for teaching some knowledge online or giving explanations is known (see, for example, Patent Document 1).
  • the method of measuring the effect by the above-mentioned questionnaire tends to be subjective, and it is insufficient as a method of objectively measuring the effect of the content of the video meeting.
  • an object of the present invention is to objectively evaluate the content of a video meeting in particular.
  • the present invention includes at least a video meeting service terminal that provides a video meeting to the first user terminal and the second user terminal and stores a moving image acquired in the video meeting, and an evaluation terminal that evaluates the video meeting.
  • a video meeting rating system The video meeting service terminal generates at least the authority information associated with the evaluation terminal by a permission operation from the first user terminal or the second user terminal.
  • the evaluation terminal acquires the moving image from the video meeting service terminal based on the authority information, identifies at least a facial image contained in the moving image for each predetermined frame unit, and evaluates the facial image. Calculate the value, Get a video meeting rating system.
  • the present disclosure comprises the following configurations.
  • Items 1 It includes at least a video meeting service terminal that provides a video meeting to the first user terminal and the second user terminal and stores a moving image acquired in the video meeting, and an evaluation terminal that evaluates the video meeting.
  • a video meeting rating system The video meeting service terminal generates at least the authority information associated with the evaluation terminal by a permission operation from the first user terminal or the second user terminal.
  • the evaluation terminal acquires the moving image from the video meeting service terminal based on the authority information, identifies at least a facial image contained in the moving image for each predetermined frame unit, and evaluates the facial image. Calculate the value, Video meeting rating system.
  • [Item 2] The video meeting evaluation system described in item 1
  • the evaluation terminal provides graph information in chronological order of the evaluation values.
  • [Item 3] The video meeting evaluation system according to item 1 or item 2.
  • the evaluation terminal calculates a plurality of evaluation values obtained by evaluating the face image from a plurality of different viewpoints.
  • [Item 4] The video meeting evaluation system according to any one of items 1 to 3.
  • the evaluation terminal calculates the evaluation value together with the sound included in the moving image.
  • [Item 5] The video meeting evaluation system according to any one of items 1 to 4.
  • the evaluation terminal calculates the evaluation value together with an object other than the face image contained in the moving image.
  • a video meeting is provided to at least the first user terminal and the second user terminal, and at least a moving image acquired by the first camera unit of the first user terminal or the second camera unit of the second user terminal is stored.
  • a video meeting evaluation server that is connected to a video meeting service terminal so that it can communicate with it.
  • the emotion analysis system of the present embodiment is unique in that, in an environment where a video meeting (hereinafter referred to as an online session including one-way and two-way) is held by a plurality of people, the analysis target person among the plurality of people is different from the others. It is a system that analyzes emotions (feelings that occur in the words and actions of oneself or others, such as comfort / discomfort or the degree thereof).
  • An online session is, for example, an online conference, an online class, an online chat, etc., in which terminals installed in multiple locations are connected to a server via a communication network such as the Internet, and moving images are transmitted between the terminals through the server. It is designed to be able to communicate.
  • the moving images handled in the online session include the face image and voice of the user who uses the terminal.
  • the moving image also includes an image such as a material shared and viewed by a plurality of users. It is possible to switch between the face image and the material image on the screen of each terminal to display only one of them, or to divide the display area and display the face image and the material image at the same time. Further, it is possible to display the image of one of a plurality of people on the full screen, or to display the image of a part or all of the users on a small screen.
  • the leader, facilitator, or administrator of the online session designates any user as the analysis target.
  • Organizers of online sessions include, for example, instructors of online classes, chairs and facilitators of online conferences, and coaches of sessions for coaching purposes.
  • the organizer of an online session is usually one of a plurality of users who participate in the online session, but may be another person who does not participate in the online session.
  • all the participants may be the analysis target without designating the analysis target person.
  • the leader, facilitator, or administrator of the online session (hereinafter collectively referred to as the organizer) to specify any user as the analysis target.
  • Organizers of online sessions include, for example, instructors of online classes, chairs and facilitators of online conferences, and coaches of sessions for coaching purposes.
  • the organizer of an online session is usually one of a plurality of users who participate in the online session, but may be another person who does not participate in the online session.
  • the video meeting evaluation system has user terminals 10 and 20 having at least an input unit such as a camera unit and a microphone unit, a display unit such as a display unit, and an output unit such as a speaker.
  • a video meeting service terminal 30 that provides bidirectional video meetings to user terminals 10 and 20 and an evaluation terminal 40 that evaluates video meetings are provided.
  • FIG. 2 is a diagram showing a hardware configuration example of a computer that realizes each of the terminals 10 to 40 according to the present embodiment.
  • the computer includes at least a control unit 11, a memory 12, a storage 13, a communication unit 14, an input / output unit 15, and the like. These are electrically connected to each other through the bus 16.
  • the control unit 11 is an arithmetic unit that controls the operation of each terminal as a whole, controls the transmission and reception of data between each element, and performs information processing necessary for application execution and authentication processing.
  • the control unit 11 is a processor such as a CPU, and executes each information processing by executing a program or the like stored in the storage 13 and expanded in the memory 12.
  • the memory 12 includes a main storage configured by a volatile storage device such as a DRAM and an auxiliary storage configured by a non-volatile storage device such as a flash memory or an HDD.
  • the memory 12 is used as a work area or the like of the control unit 11, and also stores a BIOS executed when the information sharing support device 10 is started, various setting information, and the like.
  • the storage 13 stores various programs such as application programs.
  • a database storing data used for each process may be built in the storage 13.
  • the video meeting service terminal 30 may record a moving image in an online session and store it in the storage 13.
  • the evaluation terminal 40 may acquire a moving image and store it in the storage 13 managed by the evaluation terminal 40 together with the analysis result (evaluation result).
  • the communication unit 14 connects the information sharing support device 10 to the network.
  • the communication unit 14 directly or network access points with an external device by, for example, a wired LAN, a wireless LAN, Wi-Fi (registered trademark), infrared communication, Bluetooth (registered trademark), short-range or non-contact communication, or the like. Communicate via.
  • the input / output unit 15 is, for example, an information input device such as a keyboard, a mouse, and a touch panel, and an output device such as a display.
  • the bus 16 is commonly connected to each of the above elements and transmits, for example, an address signal, a data signal, and various control signals.
  • the evaluation terminal acquires a moving image from the video meeting service terminal, identifies at least the facial image contained in the moving image for each predetermined frame unit, and calculates the evaluation value for the facial image. (Details will be described later).
  • the video meeting service provided by the video meeting service terminal (hereinafter, may be simply referred to as “the service”) is bidirectionally imaged and voiced with respect to the user terminals 10 and 20. Communication is possible.
  • This service displays a moving image acquired by the camera unit of the other user terminal on the display of the user terminal, and can output the sound acquired by the microphone unit of the other user terminal from the speaker.
  • this service is configured to be able to record (record) moving images and audio (collectively referred to as “moving images, etc.”) by either or both user terminals.
  • the recorded information Vs1 and Vs2 (hereinafter referred to as “recorded information”) are temporarily cached in the user terminal that started recording, and are either on the video meeting service terminal side or only locally on either user terminal, or. It will be recorded in both. The user can view the recorded information by himself / herself, share it with others, etc. within the scope of using this service.
  • the evaluation terminal 40 acquires the recorded information and performs analysis and evaluation as described later.
  • a download request may be made directly, or an access may be made via a predetermined URL.
  • the video meeting service terminal 30 is at least evaluated by a permission operation (authority information request) from at least the first user terminal 10 or the second user terminal 20.
  • a permission operation authority information request
  • the evaluation terminal 40 can access the moving image of the video meeting service terminal based on the authority information, and the moving image is acquired.
  • the evaluation terminal 40 evaluates the moving image acquired as described above by the following analysis.
  • FIG. 4 is a block diagram showing a configuration example according to the present embodiment.
  • the video meeting evaluation system of the present embodiment has a moving image acquisition unit 11, a biological reaction analysis unit 12, a peculiarity determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result as functional configurations.
  • a notification unit 16 is provided.
  • Each of the above functional blocks 11 to 16 can be configured by any of hardware, DSP (Digital Signal Processor), and software provided in the evaluation terminal 40, for example.
  • DSP Digital Signal Processor
  • each of the above functional blocks 11 to 16 is actually configured to include a computer CPU, RAM, ROM, etc., and is a program stored in a recording medium such as RAM, ROM, hard disk, or semiconductor memory. Is realized by the operation of.
  • the moving image acquisition unit 11 acquires a moving image obtained by shooting a plurality of people (multiple users) with a camera provided in each terminal during an online session. It does not matter whether the moving image acquired from each terminal is set to be displayed on the screen of each terminal. That is, the moving image acquisition unit 11 acquires the moving image from each terminal, including the moving image being displayed on each terminal and the moving image being hidden.
  • the biological reaction analysis unit 12 analyzes changes in the biological reaction of each of a plurality of persons based on the moving image (whether or not it is displayed on the screen) acquired by the moving image acquisition unit 11.
  • the biological reaction analysis unit 12 separates the moving image acquired by the moving image acquisition unit 11 into a set of images (a collection of frame images) and a voice, and analyzes changes in the biological reaction from each.
  • the biological reaction analysis unit 12 analyzes the user's face image using the frame image separated from the moving image acquired by the moving image acquisition unit 11, and thereby at least one of the facial expression, the line of sight, the pulse, and the movement of the face. Analyze changes in biological reactions related to one. Further, the biological reaction analysis unit 12 analyzes changes in the biological reaction regarding at least one of the user's speech content and voice quality by analyzing the voice separated from the moving image acquired by the moving image acquisition unit 11.
  • the biological reaction analysis unit 12 calculates a biological reaction index value that reflects the content of the change in the biological reaction by quantifying the change in the biological reaction according to a predetermined standard.
  • Analysis of changes in facial expressions is performed, for example, as follows. That is, for each frame image, a facial area is specified from the frame image, and the specified facial expressions are classified into a plurality of types according to an image analysis model trained in advance by machine learning. Then, based on the classification result, it is analyzed whether a positive facial expression change occurs between consecutive frame images, a negative facial expression change occurs, and how large the facial expression change occurs. The facial expression change index value according to the analysis result is output.
  • Analysis of changes in the line of sight is performed, for example, as follows. That is, for each frame image, the area of the eyes is specified from the frame image, and the orientation of both eyes is analyzed to analyze where the user is looking. For example, it analyzes whether the speaker's face being displayed, the shared material being displayed, or the outside of the screen is being viewed. In addition, it may be possible to analyze whether the movement of the line of sight is large or small, and whether the movement is frequent or infrequent. The change in the line of sight is also related to the degree of concentration of the user.
  • the biological reaction analysis unit 12 outputs the line-of-sight change index value according to the analysis result of the line-of-sight change.
  • Analysis of pulse changes is performed, for example, as follows. That is, for each frame image, the face area is specified from the frame image. Then, using a trained image analysis model that captures the numerical value of the face color information (G in RGB), the change in the G color on the face surface is analyzed. By arranging the results along the time axis, a waveform showing the change in color information is formed, and the pulse is specified from this waveform. When a person is nervous, the pulse becomes faster, and when he / she feels calm, the pulse becomes slower.
  • the biological reaction analysis unit 12 outputs a pulse change index value according to the analysis result of the pulse change.
  • Analysis of changes in facial movement is performed, for example, as follows. That is, for each frame image, the area of the face is specified from the frame image, and the orientation of the face is analyzed to analyze where the user is looking. For example, it analyzes whether the speaker's face being displayed, the shared material being displayed, or the outside of the screen is being viewed. In addition, it may be possible to analyze whether the movement of the face is large or small, and whether the movement is frequent or infrequent. The movement of the face and the movement of the line of sight may be combined and analyzed. For example, it may be possible to analyze whether the speaker's face being displayed is viewed straight, whether the speaker is viewed with an upper eye or a lower eye, or whether the speaker is viewed from an angle.
  • the biological reaction analysis unit 12 outputs a face orientation change index value according to the analysis result of the face orientation change.
  • the content of the statement is analyzed as follows, for example. That is, the biological reaction analysis unit 12 converts the voice into a character string by performing a known voice recognition process on the voice for a specified time (for example, a time of about 30 to 150 seconds), and morphologically analyzes the character string. By doing so, words unnecessary for expressing conversation such as particles and acronyms are removed. Then, the remaining words are vectorized, and whether a positive emotional change is occurring, a negative emotional change is occurring, and how large the emotional change is occurring is analyzed, and the analysis result is used. Outputs the statement content index value.
  • Voice quality analysis is performed as follows, for example. That is, the biological reaction analysis unit 12 identifies the acoustic characteristics of the voice by performing a known voice analysis process on the voice for a specified time (for example, a time of about 30 to 150 seconds). Then, based on the acoustic characteristics, it is analyzed whether a positive voice quality change is occurring, a negative voice quality change is occurring, and how loud the voice quality change is occurring, and according to the analysis result. Outputs the voice quality change index value.
  • the biological reaction analysis unit 12 uses at least one of the facial expression change index value, the line-of-sight change index value, the pulse change index value, the face orientation change index value, the speech content index value, and the voice quality change index value calculated as described above.
  • the biological reaction index value is calculated.
  • the biological reaction index value is calculated by weighting the facial expression change index value, the line-of-sight change index value, the pulse change index value, the face orientation change index value, the speech content index value, and the voice quality change index value.
  • the peculiarity determination unit 13 determines whether or not the change in the biological reaction analyzed for the analysis target person is specific to the change in the biological reaction analyzed for a person other than the analysis target person. In the present embodiment, the peculiarity determination unit 13 compares the changes in the biological reaction analyzed for the analysis target person with those of others based on the biological reaction index values calculated for each of the plurality of users by the biological reaction analysis unit 12. To determine whether it is specific or not.
  • the peculiarity determination unit 13 calculates the variance of the biological reaction index value calculated for each of a plurality of persons by the biological reaction analysis unit 12, and compares the variance with the biological reaction index value calculated for the analysis target person. It is determined whether or not the change in the biological reaction analyzed for the person to be analyzed is specific compared to the other person.
  • the following three patterns can be considered as cases where the changes in the biological reaction analyzed for the person to be analyzed are more specific than those of others.
  • the first is the case where a particularly large change in the biological reaction has not occurred in the other person, but a relatively large change in the biological reaction has occurred in the person to be analyzed.
  • the second is the case where a particularly large change in the biological reaction has not occurred in the analysis subject, but a relatively large change in the biological reaction has occurred in the other person.
  • the third is the case where a relatively large change in the biological reaction occurs in both the analysis target person and the other person, but the content of the change differs between the analysis target person and the other person.
  • the related event identification unit 14 identifies an event occurring with respect to at least one of the analysis subject, another person, and the environment when a change in the biological reaction determined to be specific by the peculiarity determination unit 13 occurs. .. For example, the related event identification unit 14 identifies the behavior of the analysis target person himself / herself from the moving image when a specific change in the biological reaction occurs for the analysis target person. In addition, the related event identification unit 14 identifies the words and actions of another person from the moving image when a specific change in the biological reaction occurs for the analysis target person. In addition, the related event identification unit 14 identifies the environment when a specific change in the biological reaction occurs for the analysis target person from the moving image. The environment is, for example, the shared material displayed on the screen, the one reflected in the background of the person to be analyzed, and the like.
  • the clustering unit 15 includes changes in biological reactions determined to be specific by the peculiarity determination unit 13 (for example, one or a combination of eyes, pulse, facial movement, speech content, and voice quality) and the peculiarity.
  • the degree of correlation with the event (event specified by the related event identification unit 14) that occurs when a change in the biological reaction occurs is analyzed, and it is determined that the correlation is above a certain level.
  • the clustering unit 15 clusters the analysis target person or the event in any of a plurality of pre-segmented classifications according to the content of the event, the degree of negativeness, the magnitude of the correlation, and the like.
  • the clustering unit 15 clusters the analysis target person or the event in any of a plurality of pre-segmented classifications according to the content of the event, the degree of positiveness, the magnitude of the correlation, and the like.
  • the analysis result notification unit 16 determines at least one of the changes in the biological reaction determined to be specific by the peculiarity determination unit 13, the event specified by the related event identification unit 14, and the classification clustered by the clustering unit 15. , Notify the designated person of the analysis target (analysis target person or the organizer of the online session).
  • the analysis result notification unit 16 analyzes the analysis target as an event that occurs when a specific change in biological reaction occurs in the analysis target person (any of the above-mentioned three patterns; the same applies hereinafter). Notify the person to be analyzed of the person's own words and actions. As a result, the person to be analyzed can grasp that he / she has different emotions from others when he / she makes a certain word or action. At this time, the change of the specific biological reaction specified for the analysis target person may also be notified to the analysis target person. Further, the change in the biological reaction of the other person to be compared may be further notified to the analysis target person.
  • the analysis result notification unit 16 is the organizer of the online session, in which the event occurring when the analysis target person undergoes a specific change in the biological reaction different from the others is described together with the specific change in the biological reaction. Notify to. This allows the organizer of the online session to know what kind of phenomenon influences what kind of emotional change as a phenomenon peculiar to the designated analysis target person. Then, it becomes possible to take appropriate measures for the analysis target person according to the grasped contents.
  • the analysis result notification unit 16 notifies the organizer of the online session of the event occurring when the analysis target person has a specific change in biological reaction different from that of others or the clustering result of the analysis target person. do.
  • the organizer of the online session can grasp the behavior tendency peculiar to the analysis target person and predict the behavior or state that may occur in the future, depending on which classification the specified analysis target person is clustered into. be able to. Then, it becomes possible to take appropriate measures for the analysis target person.
  • the biological reaction index value is calculated by quantifying the change in the biological reaction according to a predetermined standard, and the analysis target person is analyzed based on the biological reaction index value calculated for each of the plurality of persons.
  • An example of determining whether or not a change in a biological reaction has been made is specific compared to another person has been described, but the present invention is not limited to this example. For example, it may be as follows.
  • the biological reaction analysis unit 12 analyzes the movement of the line of sight for each of a plurality of people and generates a heat map showing the direction of the line of sight.
  • the change in the biological reaction analyzed for the analysis target person is measured by the comparison between the heat map generated for the analysis target person by the biological reaction analysis unit 12 and the heat map generated for the other person. It is determined whether or not it is specific by comparing with the change in the biological reaction analyzed for.
  • FIG. 5 is a block diagram showing a configuration example according to the present embodiment.
  • the video meeting evaluation system of the present embodiment includes a moving image acquisition unit 11, a biological reaction analysis unit 12, and a reaction information presentation unit 13a as functional configurations.
  • the reaction information presentation unit 13a presents information indicating changes in the biological reaction analyzed by the biological reaction analysis unit 12a, including participants who are not displayed on the screen.
  • the reaction information presentation unit 13a presents information indicating changes in the biological reaction to the leader, facilitator, or manager of the online session (hereinafter collectively referred to as the organizer).
  • Organizers of online sessions include, for example, instructors of online classes, chairs and facilitators of online conferences, and coaches of sessions for coaching purposes.
  • the organizer of an online session is usually one of a plurality of users who participate in the online session, but may be another person who does not participate in the online session.
  • the organizer of the online session can also grasp the state of the participants who are not displayed on the screen in the environment where the online session is held by multiple people.
  • the device described in the present specification may be realized as a single device, or may be realized by a plurality of devices (for example, a cloud server) which are partially or wholly connected by a network.
  • the control unit 11 and the storage 13 of the information sharing support device 10 may be realized by different servers connected to each other by a network.
  • the series of processes by the apparatus described in the present specification may be realized by using any of software, hardware, and a combination of software and hardware. It is possible to create a computer program for realizing each function of the information sharing support device 10 according to the present embodiment and implement it on a PC or the like. It is also possible to provide a computer-readable recording medium in which such a computer program is stored.
  • the recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like. Further, the above computer program may be distributed, for example, via a network without using a recording medium.
  • ⁇ Structure 1> A video meeting evaluation system that analyzes specific emotions of the person to be analyzed among the multiple people in an environment where online sessions are held by multiple people.
  • ⁇ Structure 2> A moving image acquisition unit that acquires moving images obtained by shooting the above multiple people during the online session, and a moving image acquisition unit. Based on the moving image acquired by the moving image acquisition unit, the biological reaction analysis unit that analyzes the change in the biological reaction for each of the plurality of persons, and the biological reaction analysis unit. It is provided with a peculiarity determination unit for determining whether or not the change in the biological reaction analyzed for the analysis target person is specific to the change in the biological reaction analyzed for others other than the analysis target person.
  • a video meeting rating system that features that.
  • the biological reaction analysis unit analyzes changes in the biological reaction related to at least one of facial expression, line of sight, pulse, and facial movement by analyzing the facial image in the moving image acquired by the moving image acquisition unit.
  • ⁇ Structure 4> The biological reaction analysis unit is characterized in that it analyzes changes in the biological reaction regarding at least one of the content of speech and voice quality by analyzing the voice in the moving image acquired by the moving image acquisition unit. Or the video meeting rating system described in 3.
  • the biological reaction analysis unit calculates the biological reaction index value by quantifying the change in the biological reaction according to a predetermined standard.
  • the change in the biological reaction analyzed for the analysis target person is different from the analysis target person based on the biological reaction index value calculated for each of the plurality of persons by the biological reaction analysis unit.
  • the video meeting evaluation system according to any one of configurations 2 to 4, wherein it is determined whether or not it is specific with respect to the change in the biological reaction analyzed for another person.
  • the peculiarity determination unit calculates the variance of the biological reaction index value calculated for each of the plurality of persons by the biological reaction analysis unit, and the biological reaction index value calculated for the person to be analyzed and the dispersion.
  • Configuration 5 Described in Configuration 5, characterized in that it is determined by comparison whether or not the change in the biological reaction analyzed for the subject to be analyzed is specific to the change in the biological reaction analyzed for the other person.
  • Video meeting rating system. ⁇ Structure 7>
  • the biological reaction analysis unit analyzes the movement of the line of sight for each of the plurality of people and generates a heat map showing the direction of the line of sight.
  • the peculiarity determination unit is a change in the biological reaction analyzed for the analysis target person by comparing the heat map generated for the analysis target person by the biological reaction analysis unit with the heat map generated for the other person.
  • the video meeting evaluation system according to the configuration 3 is characterized in that it determines whether or not it is specific with respect to the change in the biological reaction analyzed for the other person.
  • ⁇ Structure 8> A related event identification unit that identifies an event occurring with respect to at least one of the analysis target person, the other person, and the environment when a change in a biological reaction determined to be specific by the specificity determination unit occurs.
  • ⁇ Structure 9> The degree of correlation between the change in the biological reaction determined to be specific by the specific biological reaction unit and the event occurring when the specific biological reaction change occurs is analyzed, and the correlation is at a certain level.
  • ⁇ Structure 10> Analysis result notification to notify the analysis target person or the organizer of the online session of at least one of the change in the biological reaction determined to be specific by the specificity determination unit and the event specified by the related event identification unit.
  • ⁇ Structure 11> At least one of the changes in the biological reaction determined to be specific by the peculiarity determination unit, the event specified by the related event identification unit, and the classification clustered by the clustering unit can be analyzed by the person to be analyzed or the above.
  • ⁇ Structure 12> In an environment where an online session is held by multiple participants, the above is based on the moving image obtained by shooting the above participants regardless of whether or not the participants are displayed on the screen during the online session.
  • a reaction analysis system that analyzes the reactions of participants and presents the analysis results.
  • ⁇ Structure 13> A moving image acquisition unit that acquires a moving image obtained by photographing the participants during the online session, and a moving image acquisition unit. Based on the moving image acquired by the moving image acquisition unit, the biological reaction analysis unit that analyzes the change in the biological reaction of the participants, and the biological reaction analysis unit.
  • the item 12 comprises a reaction information presenting unit that presents information indicating changes in the biological reaction analyzed by the biological reaction analysis unit, including participants not displayed on the screen.
  • ⁇ Structure 14> The biological reaction analysis unit analyzes changes in the biological reaction related to at least one of facial expression, line of sight, pulse, and facial movement by analyzing the facial image in the moving image acquired by the moving image acquisition unit.
  • Item 13 The reaction analysis system according to Item 13.
  • ⁇ Structure 15> Item 13 characterized in that the biological reaction analysis unit analyzes changes in the biological reaction regarding at least one of the content of speech and voice quality by analyzing the voice in the moving image acquired by the moving image acquisition unit.
  • the reaction analysis system according to item 14 ⁇ Structure 16> The reaction analysis system according to item 13, wherein the biological reaction analysis unit analyzes where the participants who are not displayed on the screen are looking at the shared material displayed on the screen.
  • ⁇ Structure 17> The reaction analysis system according to item 13, wherein the biological reaction analysis unit analyzes at what timing during the online session a participant who is not displayed on the screen makes a voice.
  • ⁇ Structure 18> The reaction analysis system according to any one of items 13 to 17, wherein the reaction information presenting unit presents information indicating changes in the biological reaction to the organizer of the online session.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)

Abstract

[Problem] To evaluate a video meeting, by evaluating motion images obtained during the video meeting; and to be able to utilize the evaluation analysis results. [Solution] This video meeting evaluation system includes: a video meeting service terminal that provides a video meeting for at least a first user terminal and a second user terminal and stores motion images obtained during the video meeting; and an evaluation terminal that performs evaluation of the video meeting. The video meeting service terminal generates authority information associated to at least the evaluation terminal. The evaluation terminal: obtains the motion images from the video meeting service terminal on the basis of the authority information; and identifies at least a facial image included in the motion images, for each prescribed frame unit, and calculates an evaluation value for the facial images.

Description

ビデオミーティング評価システム及びビデオミーティング評価サーバVideo meeting evaluation system and video meeting evaluation server
　本開示は、ビデオミーティング評価システム及びビデオミーティング評価サーバに関する。 This disclosure relates to a video meeting evaluation system and a video meeting evaluation server.
　従来、オンラインで何らかの知識を教授したり、説明等を行うためのシステムが知られている（例えば、特許文献１参照）。 Conventionally, a system for teaching some knowledge online or giving explanations is known (see, for example, Patent Document 1).
　また、このようなオンラインで行われるビデオミーティングの効果測定の方法として、例えば、ビデオミーティング後にアンケート等を行う方法も提案されている（例えば、特許文献２参照）。 Further, as a method of measuring the effect of such an online video meeting, for example, a method of conducting a questionnaire or the like after the video meeting has been proposed (see, for example, Patent Document 2).
特開２０１９－５８６２５号公報Japanese Unexamined Patent Publication No. 2019-58625
　上述したアンケートによる効果測定の方法は、主観的になりがちであり、ビデオミーティングの内容に関して客観的な効果測定を行う方法としては不十分である。 The method of measuring the effect by the above-mentioned questionnaire tends to be subjective, and it is insufficient as a method of objectively measuring the effect of the content of the video meeting.
　また、第三者によってビデオミーティングを監視し、第三者による客観的な評価を取得する方法も考えられるが、手間と時間がかかりすぎて現実的ではない。 It is also possible to monitor the video meeting by a third party and obtain an objective evaluation by the third party, but it is not realistic because it takes too much time and effort.
　そこで、本発明は、ビデオミーティングの特に内容に関する評価を客観的に行うことを目的とする。 Therefore, an object of the present invention is to objectively evaluate the content of a video meeting in particular.
　本発明によれば、
　少なくとも前記第１ユーザ端末と前記第２ユーザ端末とにビデオミーティングを提供すると共に当該ビデオミーティングにおいて取得した動画像を記憶するビデオミーティングサービス端末と、前記ビデオミーティングに関する評価を行う評価端末と、を含む、ビデオミーティングの評価システムであって、
　前記ビデオミーティングサービス端末は、少なくとも前記第１ユーザ端末又は前記第２ユーザ端末からの許可操作によって、少なくとも前記評価端末に関連付けられた権限情報を生成し、
　前記評価端末は、前記権限情報に基づいて前記ビデオミーティングサービス端末から前記動画像を取得し、当該動画像内に含まれる少なくとも顔画像を所定のフレーム単位ごとに識別すると共に、前記顔画像に関する評価値を算出する、
ビデオミーティング評価システムが得られる。 According to the present invention
It includes at least a video meeting service terminal that provides a video meeting to the first user terminal and the second user terminal and stores a moving image acquired in the video meeting, and an evaluation terminal that evaluates the video meeting. , A video meeting rating system,
The video meeting service terminal generates at least the authority information associated with the evaluation terminal by a permission operation from the first user terminal or the second user terminal.
The evaluation terminal acquires the moving image from the video meeting service terminal based on the authority information, identifies at least a facial image contained in the moving image for each predetermined frame unit, and evaluates the facial image. Calculate the value,
Get a video meeting rating system.
　本開示によれば、ビデオミーティングの動画像を評価することにより、特に内容に関する評価を客観的に行うことができる。 According to this disclosure, by evaluating the moving image of the video meeting, it is possible to objectively evaluate the content in particular.
本実施の形態によるシステム全体図を示す図である。It is a figure which shows the whole system by this embodiment. 本実施の形態による端末の構成例を示す図である。It is a figure which shows the configuration example of the terminal by this embodiment. 本実施の形態による評価端末の機能ブロック図の一例である。This is an example of a functional block diagram of an evaluation terminal according to this embodiment. 本実施の形態による機能ブロック図である。It is a functional block diagram by this embodiment. 本実施の形態による機能ブロック図である。It is a functional block diagram by this embodiment.
　本開示の実施形態の内容を列記して説明する。本開示は、以下のような構成を備える。
［項目１］
　少なくとも前記第１ユーザ端末と前記第２ユーザ端末とにビデオミーティングを提供すると共に当該ビデオミーティングにおいて取得した動画像を記憶するビデオミーティングサービス端末と、前記ビデオミーティングに関する評価を行う評価端末と、を含む、ビデオミーティングの評価システムであって、
　前記ビデオミーティングサービス端末は、少なくとも前記第１ユーザ端末又は前記第２ユーザ端末からの許可操作によって、少なくとも前記評価端末に関連付けられた権限情報を生成し、
　前記評価端末は、前記権限情報に基づいて前記ビデオミーティングサービス端末から前記動画像を取得し、当該動画像内に含まれる少なくとも顔画像を所定のフレーム単位ごとに識別すると共に、前記顔画像に関する評価値を算出する、
ビデオミーティング評価システム。
［項目２］
　項目１に記載のビデオミーティング評価システムであって、
　前記評価端末は、前記評価値の時系列によるグラフ情報を提供する、
ビデオミーティング評価システム。
［項目３］
　項目１又は項目２に記載のビデオミーティング評価システムであって、
　前記評価端末は、前記顔画像を複数の異なる観点によって評価した複数の評価値を算出する、
ビデオミーティング評価システム。
［項目４］
　項目１乃至項目３のいずれかに記載のビデオミーティング評価システムであって、
　前記評価端末は、前記動画像に含まれる音声と共に前記評価値を算出する、
ビデオミーティング評価システム。
［項目５］
　項目１乃至項目４のいずれかに記載のビデオミーティング評価システムであって、
　前記評価端末は、前記動画像内に含まれる前記顔画像以外の対象物と共に前記評価値を算出する、
ビデオミーティング評価システム。
［項目６］
　少なくとも前記第１ユーザ端末と前記第２ユーザ端末とにビデオミーティングを提供すると共に少なくとも前記第１ユーザ端末の第１カメラ部又は前記第２ユーザ端末の第２カメラ部で取得した動画像を記憶するビデオミーティングサービス端末と通信可能に接続された、ビデオミーティング評価サーバであって、
　前記動画像を取得する手段と、
　当該動画像内に含まれる少なくとも顔画像を所定のフレーム単位ごとに識別する手段と、
　前記顔画像に関する評価値を算出する手段と、
を備える、ビデオミーティング評価サーバ。 The contents of the embodiments of the present disclosure will be listed and described. The present disclosure comprises the following configurations.
[Item 1]
It includes at least a video meeting service terminal that provides a video meeting to the first user terminal and the second user terminal and stores a moving image acquired in the video meeting, and an evaluation terminal that evaluates the video meeting. , A video meeting rating system,
The video meeting service terminal generates at least the authority information associated with the evaluation terminal by a permission operation from the first user terminal or the second user terminal.
The evaluation terminal acquires the moving image from the video meeting service terminal based on the authority information, identifies at least a facial image contained in the moving image for each predetermined frame unit, and evaluates the facial image. Calculate the value,
Video meeting rating system.
[Item 2]
The video meeting evaluation system described in item 1
The evaluation terminal provides graph information in chronological order of the evaluation values.
Video meeting rating system.
[Item 3]
The video meeting evaluation system according to item 1 or item 2.
The evaluation terminal calculates a plurality of evaluation values obtained by evaluating the face image from a plurality of different viewpoints.
Video meeting rating system.
[Item 4]
The video meeting evaluation system according to any one of items 1 to 3.
The evaluation terminal calculates the evaluation value together with the sound included in the moving image.
Video meeting rating system.
[Item 5]
The video meeting evaluation system according to any one of items 1 to 4.
The evaluation terminal calculates the evaluation value together with an object other than the face image contained in the moving image.
Video meeting rating system.
[Item 6]
A video meeting is provided to at least the first user terminal and the second user terminal, and at least a moving image acquired by the first camera unit of the first user terminal or the second camera unit of the second user terminal is stored. A video meeting evaluation server that is connected to a video meeting service terminal so that it can communicate with it.
The means for acquiring the moving image and
A means for identifying at least a facial image contained in the moving image for each predetermined frame unit,
A means for calculating an evaluation value for the face image and
A video meeting evaluation server.
　以下に添付図面を参照しながら、本開示の好適な実施の形態について詳細に説明する。なお、本明細書及び図面において、実質的に同一の機能構成を有する構成要素については、同一の符号を付することにより重複説明を省略する。 The preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings below. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.
＜基本機能＞ <Basic function>
　本実施形態の感情解析システムは、複数人でビデオミーティング（以下、一方向及び双方向含めてオンラインセッションという）が行われる環境において、当該複数人の中の解析対象者について他者とは異なる特異的な感情（自分または他人の言動に対して起こる気持ち。快・不快またはその程度など）を解析するシステムである。 The emotion analysis system of the present embodiment is unique in that, in an environment where a video meeting (hereinafter referred to as an online session including one-way and two-way) is held by a plurality of people, the analysis target person among the plurality of people is different from the others. It is a system that analyzes emotions (feelings that occur in the words and actions of oneself or others, such as comfort / discomfort or the degree thereof).
　オンラインセッションは、例えばオンライン会議、オンライン授業、オンラインチャットなどであり、複数の場所に設置された端末をインターネットなどの通信ネットワークを介してサーバに接続し、当該サーバを通じて複数の端末間で動画像をやり取りできるようにしたものである。 An online session is, for example, an online conference, an online class, an online chat, etc., in which terminals installed in multiple locations are connected to a server via a communication network such as the Internet, and moving images are transmitted between the terminals through the server. It is designed to be able to communicate.
　オンラインセッションで扱う動画像には、端末を使用するユーザの顔画像や音声が含まれる。また、動画像には、複数のユーザが共有して閲覧する資料などの画像も含まれる。各端末の画面上に顔画像と資料画像とを切り替えて何れか一方のみを表示させたり、表示領域を分けて顔画像と資料画像とを同時に表示させたりすることが可能である。また、複数人のうち１人の画像を全画面表示させたり、一部または全部のユーザの画像を小画面に分割して表示させたりすることが可能である。 The moving images handled in the online session include the face image and voice of the user who uses the terminal. In addition, the moving image also includes an image such as a material shared and viewed by a plurality of users. It is possible to switch between the face image and the material image on the screen of each terminal to display only one of them, or to divide the display area and display the face image and the material image at the same time. Further, it is possible to display the image of one of a plurality of people on the full screen, or to display the image of a part or all of the users on a small screen.
端末を使用してオンラインセッションに参加する複数のユーザのうち、何れか１人または複数人を解析対象者として指定することが可能である。例えば、オンラインセッションの主導者、進行者または管理者（以下、まとめて主催者という）が何れかのユーザを解析対象者として指定する。オンラインセッションの主催者は、例えばオンライン授業の講師、オンライン会議の議長やファシリテータ、コーチングを目的としたセッションのコーチなどである。オンラインセッションの主催者は、オンラインセッションに参加する複数のユーザの中の一人であるのが普通であるが、オンラインセッションに参加しない別人であってもよい。なお、解析対象者を指定せず全ての参加者を解析対象としてもよい。 It is possible to specify any one or more of the plurality of users who participate in the online session using the terminal as the analysis target person. For example, the leader, facilitator, or administrator of the online session (hereinafter collectively referred to as the organizer) designates any user as the analysis target. Organizers of online sessions include, for example, instructors of online classes, chairs and facilitators of online conferences, and coaches of sessions for coaching purposes. The organizer of an online session is usually one of a plurality of users who participate in the online session, but may be another person who does not participate in the online session. In addition, all the participants may be the analysis target without designating the analysis target person.
　また、オンラインセッションの主導者、進行者または管理者（以下、まとめて主催者という）が何れかのユーザを解析対象者として指定することも可能である。オンラインセッションの主催者は、例えばオンライン授業の講師、オンライン会議の議長やファシリテータ、コーチングを目的としたセッションのコーチなどである。オンラインセッションの主催者は、オンラインセッションに参加する複数のユーザの中の一人であるのが普通であるが、オンラインセッションに参加しない別人であってもよい。 It is also possible for the leader, facilitator, or administrator of the online session (hereinafter collectively referred to as the organizer) to specify any user as the analysis target. Organizers of online sessions include, for example, instructors of online classes, chairs and facilitators of online conferences, and coaches of sessions for coaching purposes. The organizer of an online session is usually one of a plurality of users who participate in the online session, but may be another person who does not participate in the online session.
　図１に示されるように、本実施の形態によるビデオミーティング評価システムは、少なくともカメラ部及びマイク部等の入力部と、ディスプレイ等の表示部とスピーカー等の出力部とを有するユーザ端末１０、２０と、ユーザ端末１０、２０に双方向のビデオミーティングを提供するビデオミーティングサービス端末３０と、ビデオミーティングに関する評価を行う評価端末４０とを備えている。 As shown in FIG. 1, the video meeting evaluation system according to the present embodiment has user terminals 10 and 20 having at least an input unit such as a camera unit and a microphone unit, a display unit such as a display unit, and an output unit such as a speaker. A video meeting service terminal 30 that provides bidirectional video meetings to user terminals 10 and 20 and an evaluation terminal 40 that evaluates video meetings are provided.
＜ハードウェア構成例＞
　図２は、本実施形態に係る各端末１０乃至４０を実現するコンピュータのハードウェア構成例を示す図である。コンピュータは、少なくとも、制御部１１、メモリ１２、ストレージ１３、通信部１４および入出力部１５等を備える。これらはバス１６を通じて相互に電気的に接続される。 <Hardware configuration example>
FIG. 2 is a diagram showing a hardware configuration example of a computer that realizes each of the terminals 10 to 40 according to the present embodiment. The computer includes at least a control unit 11, a memory 12, a storage 13, a communication unit 14, an input / output unit 15, and the like. These are electrically connected to each other through the bus 16.
　制御部１１は、各端末全体の動作を制御し、各要素間におけるデータの送受信の制御、及びアプリケーションの実行及び認証処理に必要な情報処理等を行う演算装置である。例えば制御部１１は、ＣＰＵ等のプロセッサであり、ストレージ１３に格納されメモリ１２に展開されたプログラム等を実行して各情報処理を実施する。 The control unit 11 is an arithmetic unit that controls the operation of each terminal as a whole, controls the transmission and reception of data between each element, and performs information processing necessary for application execution and authentication processing. For example, the control unit 11 is a processor such as a CPU, and executes each information processing by executing a program or the like stored in the storage 13 and expanded in the memory 12.
　メモリ１２は、ＤＲＡＭ等の揮発性記憶装置で構成される主記憶と、フラッシュメモリまたはＨＤＤ等の不揮発性記憶装置で構成される補助記憶と、を含む。メモリ１２は、制御部１１のワークエリア等として使用され、また、情報共有支援装置１０の起動時に実行されるＢＩＯＳ、及び各種設定情報等を格納する。 The memory 12 includes a main storage configured by a volatile storage device such as a DRAM and an auxiliary storage configured by a non-volatile storage device such as a flash memory or an HDD. The memory 12 is used as a work area or the like of the control unit 11, and also stores a BIOS executed when the information sharing support device 10 is started, various setting information, and the like.
　ストレージ１３は、アプリケーション・プログラム等の各種プログラムを格納する。各処理に用いられるデータを格納したデータベースがストレージ１３に構築されていてもよい。特にビデオミーティングサービス端末３０にはオンラインセッションにおける動画像を記録しストレージ１３に格納することとしてもよい。また、評価端末４０は、動画像を取得し、評価端末４０の管理するストレージ１３にその解析結果（評価結果）と共に格納することとしてもよい。 The storage 13 stores various programs such as application programs. A database storing data used for each process may be built in the storage 13. In particular, the video meeting service terminal 30 may record a moving image in an online session and store it in the storage 13. Further, the evaluation terminal 40 may acquire a moving image and store it in the storage 13 managed by the evaluation terminal 40 together with the analysis result (evaluation result).
　通信部１４は、情報共有支援装置１０をネットワークに接続する。通信部１４は、例えば、有線ＬＡＮ、無線ＬＡＮ、Ｗｉ－Ｆｉ（登録商標）、赤外線通信、Ｂｌｕｅｔｏｏｔｈ（登録商標）、近距離または非接触通信等の方式で、外部機器と直接またはネットワークアクセスポイントを介して通信する。 The communication unit 14 connects the information sharing support device 10 to the network. The communication unit 14 directly or network access points with an external device by, for example, a wired LAN, a wireless LAN, Wi-Fi (registered trademark), infrared communication, Bluetooth (registered trademark), short-range or non-contact communication, or the like. Communicate via.
　入出力部１５は、例えば、キーボード、マウス、タッチパネル等の情報入力機器、及びディスプレイ等の出力機器である。 The input / output unit 15 is, for example, an information input device such as a keyboard, a mouse, and a touch panel, and an output device such as a display.
　バス１６は、上記各要素に共通に接続され、例えば、アドレス信号、データ信号及び各種制御信号を伝達する。 The bus 16 is commonly connected to each of the above elements and transmits, for example, an address signal, a data signal, and various control signals.
　特に、本実施の形態による評価端末は、ビデオミーティングサービス端末から動画像を取得し、当該動画像内に含まれる少なくとも顔画像を所定のフレーム単位ごとに識別すると共に、顔画像に関する評価値を算出する（詳しくは後述する）。 In particular, the evaluation terminal according to the present embodiment acquires a moving image from the video meeting service terminal, identifies at least the facial image contained in the moving image for each predetermined frame unit, and calculates the evaluation value for the facial image. (Details will be described later).
＜動画の取得方法＞
　図３に示されるように、ビデオミーティングサービス端末が提供するビデオミーティングサービス（以下、単に「本サービス」と言うことがある」）は、ユーザ端末１０、２０に対して双方向に画像および音声によって通信が可能となるものである。本サービスは、ユーザ端末のディスプレイに相手のユーザ端末のカメラ部で取得した動画像を表示し、相手のユーザ端末のマイク部で取得した音声をスピーカーから出力可能となっている。 <How to get video>
As shown in FIG. 3, the video meeting service provided by the video meeting service terminal (hereinafter, may be simply referred to as “the service”) is bidirectionally imaged and voiced with respect to the user terminals 10 and 20. Communication is possible. This service displays a moving image acquired by the camera unit of the other user terminal on the display of the user terminal, and can output the sound acquired by the microphone unit of the other user terminal from the speaker.
　また、本サービスは双方の又はいずれかのユーザ端末によって、動画像及び音声（これらを合わせて「動画像等」という）を記録（レコーディング）することが可能に構成されている。記録された情報Ｖｓ１、Ｖｓ２（以下「記録情報」という）は、一時的には記録を開始したユーザ端末にキャッシュされつつ、ビデオミーティングサービス端末側か、またはいずれかのユーザ端末のローカルのみ、またはその両方に記録されることとなる。ユーザは、当該記録情報を本サービスの利用の範囲内で自分で視聴、他者に共有等行うことができる。 In addition, this service is configured to be able to record (record) moving images and audio (collectively referred to as "moving images, etc.") by either or both user terminals. The recorded information Vs1 and Vs2 (hereinafter referred to as "recorded information") are temporarily cached in the user terminal that started recording, and are either on the video meeting service terminal side or only locally on either user terminal, or. It will be recorded in both. The user can view the recorded information by himself / herself, share it with others, etc. within the scope of using this service.
　評価端末４０は、当該記録情報を取得して、後述するような分析及び評価を行う。記録情報の取得方法としては、例えば、直接にダウンロードリクエストを行うこととしてもよいし、所定のＵＲＬを経由してアクセスすることとしてもよい。 The evaluation terminal 40 acquires the recorded information and performs analysis and evaluation as described later. As a method of acquiring the recorded information, for example, a download request may be made directly, or an access may be made via a predetermined URL.
　特に、図３に示されるように、本実施の形態によれば、ビデオミーティングサービス端末３０は、少なくとも第１ユーザ端末１０又は第２ユーザ端末２０からの許可操作（権限情報リクエスト）によって、少なくとも評価端末４０に関連付けられた権限情報を生成する。評価端末４０は、権限情報に基づいてビデオミーティングサービス端末の当該動画像にアクセスが可能となり前記動画像が取得される。 In particular, as shown in FIG. 3, according to the present embodiment, the video meeting service terminal 30 is at least evaluated by a permission operation (authority information request) from at least the first user terminal 10 or the second user terminal 20. Generates the authority information associated with the terminal 40. The evaluation terminal 40 can access the moving image of the video meeting service terminal based on the authority information, and the moving image is acquired.
　評価端末４０は、以上のようにして取得した動画を以下のような分析によって評価を行う。 The evaluation terminal 40 evaluates the moving image acquired as described above by the following analysis.
＜実施例１＞
　以下、本発明の一実施形態を図面に基づいて説明する。図４は、本実施形態による構成例を示すブロック図である。図４に示すように、本実施形態のビデオミーティング評価システムは、機能構成として、動画像取得部１１、生体反応解析部１２、特異判定部１３、関連事象特定部１４、クラスタリング部１５および解析結果通知部１６を備えている。 <Example 1>
Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 4 is a block diagram showing a configuration example according to the present embodiment. As shown in FIG. 4, the video meeting evaluation system of the present embodiment has a moving image acquisition unit 11, a biological reaction analysis unit 12, a peculiarity determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result as functional configurations. A notification unit 16 is provided.
　上記各機能ブロック１１～１６は、例えば評価端末４０に備えられたハードウェア、ＤＳＰ（Digital Signal Processor）、ソフトウェアの何れによっても構成することが可能である。例えばソフトウェアによって構成する場合、上記各機能ブロック１１～１６は、実際にはコンピュータのＣＰＵ、ＲＡＭ、ＲＯＭなどを備えて構成され、ＲＡＭやＲＯＭ、ハードディスクまたは半導体メモリ等の記録媒体に記憶されたプログラムが動作することによって実現される。 Each of the above functional blocks 11 to 16 can be configured by any of hardware, DSP (Digital Signal Processor), and software provided in the evaluation terminal 40, for example. For example, when configured by software, each of the above functional blocks 11 to 16 is actually configured to include a computer CPU, RAM, ROM, etc., and is a program stored in a recording medium such as RAM, ROM, hard disk, or semiconductor memory. Is realized by the operation of.
　動画像取得部１１は、オンラインセッション中に各端末が備えるカメラにより複数人（複数のユーザ）を撮影することによって得られる動画像を各端末から取得する。各端末から取得する動画像は、各端末の画面上に表示されるように設定されているものか否かは問わない。すなわち、動画像取得部１１は、各端末に表示中の動画像および非表示中の動画像を含めて、動画像を各端末から取得する。 The moving image acquisition unit 11 acquires a moving image obtained by shooting a plurality of people (multiple users) with a camera provided in each terminal during an online session. It does not matter whether the moving image acquired from each terminal is set to be displayed on the screen of each terminal. That is, the moving image acquisition unit 11 acquires the moving image from each terminal, including the moving image being displayed on each terminal and the moving image being hidden.
　生体反応解析部１２は、動画像取得部１１により取得された動画像（画面上に表示中のものか否かは問わない）に基づいて、複数人のそれぞれについて生体反応の変化を解析する。本実施形態において生体反応解析部１２は、動画像取得部１１により取得された動画像を画像のセット（フレーム画像の集まり）と音声とに分離し、それぞれから生体反応の変化を解析する。 The biological reaction analysis unit 12 analyzes changes in the biological reaction of each of a plurality of persons based on the moving image (whether or not it is displayed on the screen) acquired by the moving image acquisition unit 11. In the present embodiment, the biological reaction analysis unit 12 separates the moving image acquired by the moving image acquisition unit 11 into a set of images (a collection of frame images) and a voice, and analyzes changes in the biological reaction from each.
　例えば、生体反応解析部１２は、動画像取得部１１により取得された動画像から分離したフレーム画像を用いてユーザの顔画像を解析することにより、表情、目線、脈拍、顔の動きの少なくとも１つに関する生体反応の変化を解析する。また、生体反応解析部１２は、動画像取得部１１により取得された動画像から分離した音声を解析することにより、ユーザの発言内容、声質の少なくとも１つに関する生体反応の変化を解析する。 For example, the biological reaction analysis unit 12 analyzes the user's face image using the frame image separated from the moving image acquired by the moving image acquisition unit 11, and thereby at least one of the facial expression, the line of sight, the pulse, and the movement of the face. Analyze changes in biological reactions related to one. Further, the biological reaction analysis unit 12 analyzes changes in the biological reaction regarding at least one of the user's speech content and voice quality by analyzing the voice separated from the moving image acquired by the moving image acquisition unit 11.
　人は感情が変化すると、それが表情、目線、脈拍、顔の動き、発言内容、声質などの生体反応の変化となって現れる。本実施形態では、ユーザの生体反応の変化を解析することを通じて、ユーザの感情の変化を解析する。本実施形態において解析する感情は、一例として、快／不快の程度である。本実施形態において生体反応解析部１２は、生体反応の変化を所定の基準に従って数値化することにより、生体反応の変化の内容を反映させた生体反応指標値を算出する。 When a person's emotions change, it appears as changes in biological reactions such as facial expressions, eyes, pulse, facial movements, speech content, and voice quality. In this embodiment, changes in the user's emotions are analyzed by analyzing changes in the user's biological reaction. The emotion analyzed in this embodiment is, for example, the degree of comfort / discomfort. In the present embodiment, the biological reaction analysis unit 12 calculates a biological reaction index value that reflects the content of the change in the biological reaction by quantifying the change in the biological reaction according to a predetermined standard.
　表情の変化の解析は、例えば以下のようにして行う。すなわち、フレーム画像ごとに、フレーム画像の中から顔の領域を特定し、事前に機械学習させた画像解析モデルに従って特定した顔の表情を複数に分類する。そして、その分類結果に基づいて、連続するフレーム画像間でポジティブな表情変化が起きているか、ネガティブな表情変化が起きているか、およびどの程度の大きさの表情変化が起きているかを解析し、その解析結果に応じた表情変化指標値を出力する。 Analysis of changes in facial expressions is performed, for example, as follows. That is, for each frame image, a facial area is specified from the frame image, and the specified facial expressions are classified into a plurality of types according to an image analysis model trained in advance by machine learning. Then, based on the classification result, it is analyzed whether a positive facial expression change occurs between consecutive frame images, a negative facial expression change occurs, and how large the facial expression change occurs. The facial expression change index value according to the analysis result is output.
　目線の変化の解析は、例えば以下のようにして行う。すなわち、フレーム画像ごとに、フレーム画像の中から目の領域を特定し、両目の向きを解析することにより、ユーザがどこを見ているかを解析する。例えば、表示中の話者の顔を見ているか、表示中の共有資料を見ているか、画面の外を見ているかなどを解析する。また、目線の動きが大きいか小さいか、動きの頻度が多いか少ないかなどを解析するようにしてもよい。目線の変化はユーザの集中度にも関連する。生体反応解析部１２は、目線の変化の解析結果に応じた目線変化指標値を出力する。 Analysis of changes in the line of sight is performed, for example, as follows. That is, for each frame image, the area of the eyes is specified from the frame image, and the orientation of both eyes is analyzed to analyze where the user is looking. For example, it analyzes whether the speaker's face being displayed, the shared material being displayed, or the outside of the screen is being viewed. In addition, it may be possible to analyze whether the movement of the line of sight is large or small, and whether the movement is frequent or infrequent. The change in the line of sight is also related to the degree of concentration of the user. The biological reaction analysis unit 12 outputs the line-of-sight change index value according to the analysis result of the line-of-sight change.
　脈拍の変化の解析は、例えば以下のようにして行う。すなわち、フレーム画像ごとに、フレーム画像の中から顔の領域を特定する。そして、顔の色情報（ＲＧＢのＧ）の数値を捉える学習済みの画像解析モデルを用いて、顔表面のＧ色の変化を解析する。その結果を時間軸に合わせて並べることによって色情報の変化を表した波形を形成し、この波形から脈拍を特定する。人は緊張すると脈拍が速くなり、気持ちが落ち着くと脈拍が遅くなる。生体反応解析部１２は、脈拍の変化の解析結果に応じた脈拍変化指標値を出力する。 Analysis of pulse changes is performed, for example, as follows. That is, for each frame image, the face area is specified from the frame image. Then, using a trained image analysis model that captures the numerical value of the face color information (G in RGB), the change in the G color on the face surface is analyzed. By arranging the results along the time axis, a waveform showing the change in color information is formed, and the pulse is specified from this waveform. When a person is nervous, the pulse becomes faster, and when he / she feels calm, the pulse becomes slower. The biological reaction analysis unit 12 outputs a pulse change index value according to the analysis result of the pulse change.
　顔の動きの変化の解析は、例えば以下のようにして行う。すなわち、フレーム画像ごとに、フレーム画像の中から顔の領域を特定し、顔の向きを解析することにより、ユーザがどこを見ているかを解析する。例えば、表示中の話者の顔を見ているか、表示中の共有資料を見ているか、画面の外を見ているかなどを解析する。また、顔の動きが大きいか小さいか、動きの頻度が多いか少ないかなどを解析するようにしてもよい。顔の動きと目線の動きとを合わせて解析するようにしてもよい。例えば、表示中の話者の顔をまっすぐ見ているか、上目遣いまたは下目使いに見ているか、斜めから見ているかなどを解析するようにしてもよい。生体反応解析部１２は、顔の向きの変化の解析結果に応じた顔向き変化指標値を出力する。 Analysis of changes in facial movement is performed, for example, as follows. That is, for each frame image, the area of the face is specified from the frame image, and the orientation of the face is analyzed to analyze where the user is looking. For example, it analyzes whether the speaker's face being displayed, the shared material being displayed, or the outside of the screen is being viewed. In addition, it may be possible to analyze whether the movement of the face is large or small, and whether the movement is frequent or infrequent. The movement of the face and the movement of the line of sight may be combined and analyzed. For example, it may be possible to analyze whether the speaker's face being displayed is viewed straight, whether the speaker is viewed with an upper eye or a lower eye, or whether the speaker is viewed from an angle. The biological reaction analysis unit 12 outputs a face orientation change index value according to the analysis result of the face orientation change.
　発言内容の解析は、例えば以下のようにして行う。すなわち、生体反応解析部１２は、指定した時間（例えば、３０～１５０秒程度の時間）の音声について公知の音声認識処理を行うことによって音声を文字列に変換し、当該文字列を形態素解析することにより、助詞、冠詞などの会話を表す上で不要なワードを取り除く。そして、残ったワードをベクトル化し、ポジティブな感情変化が起きているか、ネガティブな感情変化が起きているか、およびどの程度の大きさの感情変化が起きているかを解析し、その解析結果に応じた発言内容指標値を出力する。 The content of the statement is analyzed as follows, for example. That is, the biological reaction analysis unit 12 converts the voice into a character string by performing a known voice recognition process on the voice for a specified time (for example, a time of about 30 to 150 seconds), and morphologically analyzes the character string. By doing so, words unnecessary for expressing conversation such as particles and acronyms are removed. Then, the remaining words are vectorized, and whether a positive emotional change is occurring, a negative emotional change is occurring, and how large the emotional change is occurring is analyzed, and the analysis result is used. Outputs the statement content index value.
　声質の解析は、例えば以下のようにして行う。すなわち、生体反応解析部１２は、指定した時間（例えば、３０～１５０秒程度の時間）の音声について公知の音声解析処理を行うことによって音声の音響的特徴を特定する。そして、その音響的特徴に基づいて、ポジティブな声質変化が起きているか、ネガティブな声質変化が起きているか、およびどの程度の大きさの声質変化が起きているかを解析し、その解析結果に応じた声質変化指標値を出力する。 Voice quality analysis is performed as follows, for example. That is, the biological reaction analysis unit 12 identifies the acoustic characteristics of the voice by performing a known voice analysis process on the voice for a specified time (for example, a time of about 30 to 150 seconds). Then, based on the acoustic characteristics, it is analyzed whether a positive voice quality change is occurring, a negative voice quality change is occurring, and how loud the voice quality change is occurring, and according to the analysis result. Outputs the voice quality change index value.
　生体反応解析部１２は、以上のようにして算出した表情変化指標値、目線変化指標値、脈拍変化指標値、顔向き変化指標値、発言内容指標値、声質変化指標値の少なくとも１つを用いて生体反応指標値を算出する。例えば、表情変化指標値、目線変化指標値、脈拍変化指標値、顔向き変化指標値、発言内容指標値および声質変化指標値を重み付け計算することにより、生体反応指標値を算出する。 The biological reaction analysis unit 12 uses at least one of the facial expression change index value, the line-of-sight change index value, the pulse change index value, the face orientation change index value, the speech content index value, and the voice quality change index value calculated as described above. The biological reaction index value is calculated. For example, the biological reaction index value is calculated by weighting the facial expression change index value, the line-of-sight change index value, the pulse change index value, the face orientation change index value, the speech content index value, and the voice quality change index value.
　特異判定部１３は、解析対象者について解析された生体反応の変化が、解析対象者以外の他者について解析された生体反応の変化と比べて特異的か否かを判定する。本実施形態において、特異判定部１３は、生体反応解析部１２により複数のユーザのそれぞれについて算出された生体反応指標値に基づいて、解析対象者について解析された生体反応の変化が他者と比べて特異的か否かを判定する。 The peculiarity determination unit 13 determines whether or not the change in the biological reaction analyzed for the analysis target person is specific to the change in the biological reaction analyzed for a person other than the analysis target person. In the present embodiment, the peculiarity determination unit 13 compares the changes in the biological reaction analyzed for the analysis target person with those of others based on the biological reaction index values calculated for each of the plurality of users by the biological reaction analysis unit 12. To determine whether it is specific or not.
　例えば、特異判定部１３は、生体反応解析部１２により複数人のそれぞれについて算出された生体反応指標値の分散を算出し、解析対象者について算出された生体反応指標値と分散との対比により、解析対象者について解析された生体反応の変化が他者と比べて特異的か否かを判定する。 For example, the peculiarity determination unit 13 calculates the variance of the biological reaction index value calculated for each of a plurality of persons by the biological reaction analysis unit 12, and compares the variance with the biological reaction index value calculated for the analysis target person. It is determined whether or not the change in the biological reaction analyzed for the person to be analyzed is specific compared to the other person.
　解析対象者について解析された生体反応の変化が他者と比べて特異的である場合として、次の３パターンが考えられる。１つ目は、他者については特に大きな生体反応の変化が起きていないが、解析対象者について比較的大きな生体反応の変化が起きた場合である。２つ目は、解析対象者については特に大きな生体反応の変化が起きていないが、他者について比較的大きな生体反応の変化が起きた場合である。３つ目は、解析対象者についても他者についても比較的大きな生体反応の変化が起きているが、変化の内容が解析対象者と他者とで異なる場合である。 The following three patterns can be considered as cases where the changes in the biological reaction analyzed for the person to be analyzed are more specific than those of others. The first is the case where a particularly large change in the biological reaction has not occurred in the other person, but a relatively large change in the biological reaction has occurred in the person to be analyzed. The second is the case where a particularly large change in the biological reaction has not occurred in the analysis subject, but a relatively large change in the biological reaction has occurred in the other person. The third is the case where a relatively large change in the biological reaction occurs in both the analysis target person and the other person, but the content of the change differs between the analysis target person and the other person.
　関連事象特定部１４は、特異判定部１３により特異的であると判定された生体反応の変化が起きたときに解析対象者、他者および環境の少なくとも１つに関して発生している事象を特定する。例えば、関連事象特定部１４は、解析対象者について特異的な生体反応の変化が起きたときにおける解析対象者自身の言動を動画像から特定する。また、関連事象特定部１４は、解析対象者について特異的な生体反応の変化が起きたときにおける他者の言動を動画像から特定する。また、関連事象特定部１４は、解析対象者について特異的な生体反応の変化が起きたときにおける環境を動画像から特定する。環境は、例えば画面に表示中の共有資料、解析対象者の背景に写っているものなどである。 The related event identification unit 14 identifies an event occurring with respect to at least one of the analysis subject, another person, and the environment when a change in the biological reaction determined to be specific by the peculiarity determination unit 13 occurs. .. For example, the related event identification unit 14 identifies the behavior of the analysis target person himself / herself from the moving image when a specific change in the biological reaction occurs for the analysis target person. In addition, the related event identification unit 14 identifies the words and actions of another person from the moving image when a specific change in the biological reaction occurs for the analysis target person. In addition, the related event identification unit 14 identifies the environment when a specific change in the biological reaction occurs for the analysis target person from the moving image. The environment is, for example, the shared material displayed on the screen, the one reflected in the background of the person to be analyzed, and the like.
　クラスタリング部１５は、特異判定部１３により特異的であると判定された生体反応の変化（例えば、目線、脈拍、顔の動き、発言内容、声質のうち１つまたは複数の組み合わせ）と、当該特異的な生体反応の変化が起きたときに発生している事象（関連事象特定部１４により特定された事象）との相関の程度を解析し、相関が一定レベル以上であると判定された場合に、その相関の解析結果に基づいて解析対象者または事象をクラスタリングする。 The clustering unit 15 includes changes in biological reactions determined to be specific by the peculiarity determination unit 13 (for example, one or a combination of eyes, pulse, facial movement, speech content, and voice quality) and the peculiarity. When the degree of correlation with the event (event specified by the related event identification unit 14) that occurs when a change in the biological reaction occurs is analyzed, and it is determined that the correlation is above a certain level. , Cluster the analysis target person or event based on the analysis result of the correlation.
　例えば、特異的な生体反応の変化がネガティブな感情変化に相当するものであり、当該特異的な生体反応の変化が起きたときに発生している事象もネガティブな事象である場合には一定レベル以上の相関が検出される。クラスタリング部１５は、その事象の内容やネガティブな度合い、相関の大きさなどに応じて、あらかじめセグメント化した複数の分類の何れかに解析対象者または事象をクラスタリングする。 For example, if a specific change in biological reaction corresponds to a negative emotional change, and the event occurring when the specific change in biological reaction occurs is also a negative event, a certain level. The above correlation is detected. The clustering unit 15 clusters the analysis target person or the event in any of a plurality of pre-segmented classifications according to the content of the event, the degree of negativeness, the magnitude of the correlation, and the like.
　同様に、特異的な生体反応の変化がポジティブな感情変化に相当するものであり、当該特異的な生体反応の変化が起きたときに発生している事象もポジティブな事象である場合には一定レベル以上の相関が検出される。クラスタリング部１５は、その事象の内容やポジティブな度合い、相関の大きさなどに応じて、あらかじめセグメント化した複数の分類の何れかに解析対象者または事象をクラスタリングする。 Similarly, if a specific change in biological reaction corresponds to a positive emotional change, and the event occurring when the specific change in biological reaction occurs is also constant if it is a positive event. Correlation above the level is detected. The clustering unit 15 clusters the analysis target person or the event in any of a plurality of pre-segmented classifications according to the content of the event, the degree of positiveness, the magnitude of the correlation, and the like.
　解析結果通知部１６は、特異判定部１３により特異的であると判定された生体反応の変化、関連事象特定部１４により特定された事象、およびクラスタリング部１５によりクラスタリングされた分類の少なくとも１つを、解析対象者の指定者（解析対象者またはオンラインセッションの主催者）に通知する。 The analysis result notification unit 16 determines at least one of the changes in the biological reaction determined to be specific by the peculiarity determination unit 13, the event specified by the related event identification unit 14, and the classification clustered by the clustering unit 15. , Notify the designated person of the analysis target (analysis target person or the organizer of the online session).
　例えば、解析結果通知部１６は、解析対象者について他者とは異なる特異的な生体反応の変化が起きたとき（上述した３パターンの何れか。以下同様）に発生している事象として解析対象者自身の言動を解析対象者自身に通知する。これにより、解析対象者は、自分がある言動を行ったときに他者とは違う感情を持っていることを把握することができる。このとき、解析対象者について特定された特異的な生体反応の変化も併せて解析対象者に通知するようにしてもよい。さらに、対比される他者の生体反応の変化を更に解析対象者に通知するようにしてもよい。 For example, the analysis result notification unit 16 analyzes the analysis target as an event that occurs when a specific change in biological reaction occurs in the analysis target person (any of the above-mentioned three patterns; the same applies hereinafter). Notify the person to be analyzed of the person's own words and actions. As a result, the person to be analyzed can grasp that he / she has different emotions from others when he / she makes a certain word or action. At this time, the change of the specific biological reaction specified for the analysis target person may also be notified to the analysis target person. Further, the change in the biological reaction of the other person to be compared may be further notified to the analysis target person.
　例えば、解析対象者が普段どおりの感情で特に意識せずに行った言動、または、解析対象者がある感情を伴って特に意識して行った言動に対して他者が受けた感情と、言動の際に解析対象者自身が抱いていた感情とが相違している場合に、そのときの解析対象者自身の言動が解析対象者に通知される。これにより、自分の意識に反して他者の受けが良い言動や他者の受けが良くない言動などを発見することも可能である。 For example, the words and actions that the analysis target person performed without being particularly conscious of the usual emotions, or the feelings and actions received by others for the words and actions that the analysis target person specifically consciously performed with certain emotions. If the emotions held by the analysis target person are different from each other at the time, the analysis target person's own words and actions at that time are notified to the analysis target person. This makes it possible to discover words and behaviors that are well received by others and words and behaviors that are not well received by others, contrary to one's consciousness.
　また、解析結果通知部１６は、解析対象者について他者とは異なる特異的な生体反応の変化が起きたときに発生している事象を、特異的な生体反応の変化と共にオンラインセッションの主催者に通知する。これにより、オンラインセッションの主催者は、指定した解析対象者に特有の現象として、どのような事象がどのような感情の変化に影響を与えているのかを知ることができる。そして、その把握した内容に応じて適切な処置を解析対象者に対して行うことが可能となる。 In addition, the analysis result notification unit 16 is the organizer of the online session, in which the event occurring when the analysis target person undergoes a specific change in the biological reaction different from the others is described together with the specific change in the biological reaction. Notify to. This allows the organizer of the online session to know what kind of phenomenon influences what kind of emotional change as a phenomenon peculiar to the designated analysis target person. Then, it becomes possible to take appropriate measures for the analysis target person according to the grasped contents.
　また、解析結果通知部１６は、解析対象者について他者とは異なる特異的な生体反応の変化が起きたときに発生している事象または解析対象者のクラスタリング結果をオンラインセッションの主催者に通知する。これにより、オンラインセッションの主催者は、指定した解析対象者がどの分類にクラスタリングされたかによって、解析対象者に特有の行動の傾向を把握したり、今後起こり得る行動や状態などを予測したりすることができる。そして、それに対して適切な処置を解析対象者に対して行うことが可能となる。 In addition, the analysis result notification unit 16 notifies the organizer of the online session of the event occurring when the analysis target person has a specific change in biological reaction different from that of others or the clustering result of the analysis target person. do. As a result, the organizer of the online session can grasp the behavior tendency peculiar to the analysis target person and predict the behavior or state that may occur in the future, depending on which classification the specified analysis target person is clustered into. be able to. Then, it becomes possible to take appropriate measures for the analysis target person.
　なお、上記実施形態では、生体反応の変化を所定の基準に従って数値化することによって生体反応指標値を算出し、複数人のそれぞれについて算出された生体反応指標値に基づいて、解析対象者について解析された生体反応の変化が他者と比べて特異的か否かを判定する例について説明したが、この例に限定されない。例えば、以下のようにしてもよい。 In the above embodiment, the biological reaction index value is calculated by quantifying the change in the biological reaction according to a predetermined standard, and the analysis target person is analyzed based on the biological reaction index value calculated for each of the plurality of persons. An example of determining whether or not a change in a biological reaction has been made is specific compared to another person has been described, but the present invention is not limited to this example. For example, it may be as follows.
　すなわち、生体反応解析部１２は、複数人のそれぞれについて目線の動きを解析して目線の方向を示すヒートマップを生成する。特異判定部１３は、生体反応解析部１２により解析対象者について生成されたヒートマップと他者について生成されたヒートマップとの対比により、解析対象者について解析された生体反応の変化が、他者について解析された生体反応の変化と比べて特異的か否かを判定する。 That is, the biological reaction analysis unit 12 analyzes the movement of the line of sight for each of a plurality of people and generates a heat map showing the direction of the line of sight. In the peculiarity determination unit 13, the change in the biological reaction analyzed for the analysis target person is measured by the comparison between the heat map generated for the analysis target person by the biological reaction analysis unit 12 and the heat map generated for the other person. It is determined whether or not it is specific by comparing with the change in the biological reaction analyzed for.
＜実施例２＞
　以下、本発明の実施例２に基づいて説明する。図５は、本実施形態による構成例を示すブロック図である。図１に示すように、本実施形態のビデオミーティング評価システムは、機能構成として、動画像取得部１１、生体反応解析部１２および反応情報提示部１３ａを備えている。 <Example 2>
Hereinafter, description will be made based on Example 2 of the present invention. FIG. 5 is a block diagram showing a configuration example according to the present embodiment. As shown in FIG. 1, the video meeting evaluation system of the present embodiment includes a moving image acquisition unit 11, a biological reaction analysis unit 12, and a reaction information presentation unit 13a as functional configurations.
　反応情報提示部１３ａは、画面に表示されていない参加者を含めて生体反応解析部１２ａにより解析された生体反応の変化を示す情報を提示する。例えば、反応情報提示部１３ａは、生体反応の変化を示す情報をオンラインセッションの主導者、進行者または管理者（以下、まとめて主催者という）に提示する。オンラインセッションの主催者は、例えばオンライン授業の講師、オンライン会議の議長やファシリテータ、コーチングを目的としたセッションのコーチなどである。オンラインセッションの主催者は、オンラインセッションに参加する複数のユーザの中の一人であるのが普通であるが、オンラインセッションに参加しない別人であってもよい。 The reaction information presentation unit 13a presents information indicating changes in the biological reaction analyzed by the biological reaction analysis unit 12a, including participants who are not displayed on the screen. For example, the reaction information presentation unit 13a presents information indicating changes in the biological reaction to the leader, facilitator, or manager of the online session (hereinafter collectively referred to as the organizer). Organizers of online sessions include, for example, instructors of online classes, chairs and facilitators of online conferences, and coaches of sessions for coaching purposes. The organizer of an online session is usually one of a plurality of users who participate in the online session, but may be another person who does not participate in the online session.
　このようにすることにより、オンラインセッションの主催者は、複数人でオンラインセッションが行われる環境において、画面に表示されていない参加者の様子も把握することができる。 By doing so, the organizer of the online session can also grasp the state of the participants who are not displayed on the screen in the environment where the online session is held by multiple people.
　以上、添付図面を参照しながら本開示の好適な実施形態について詳細に説明したが、本開示の技術的範囲はかかる例に限定されない。本開示の技術分野における通常の知識を有する者であれば、請求の範囲に記載された技術的思想の範疇内において、各種の変更例または修正例に想到し得ることは明らかであり、これらについても、当然に本開示の技術的範囲に属するものと了解される。 Although the preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that anyone with ordinary knowledge in the technical field of the present disclosure may come up with various modifications or modifications within the scope of the technical ideas set forth in the claims. Is, of course, understood to belong to the technical scope of the present disclosure.
　本明細書において説明した装置は、単独の装置として実現されてもよく、一部または全部がネットワークで接続された複数の装置（例えばクラウドサーバ）等により実現されてもよい。例えば、情報共有支援装置１０の制御部１１およびストレージ１３は、互いにネットワークで接続された異なるサーバにより実現されてもよい。 The device described in the present specification may be realized as a single device, or may be realized by a plurality of devices (for example, a cloud server) which are partially or wholly connected by a network. For example, the control unit 11 and the storage 13 of the information sharing support device 10 may be realized by different servers connected to each other by a network.
　本明細書において説明した装置による一連の処理は、ソフトウェア、ハードウェア、及びソフトウェアとハードウェアとの組合せのいずれを用いて実現されてもよい。本実施形態に係る情報共有支援装置１０の各機能を実現するためのコンピュータプログラムを作製し、ＰＣ等に実装することが可能である。また、このようなコンピュータプログラムが格納された、コンピュータで読み取り可能な記録媒体も提供することが可能である。記録媒体は、例えば、磁気ディスク、光ディスク、光磁気ディスク、フラッシュメモリ等である。また、上記のコンピュータプログラムは、記録媒体を用いずに、例えばネットワークを介して配信されてもよい。 The series of processes by the apparatus described in the present specification may be realized by using any of software, hardware, and a combination of software and hardware. It is possible to create a computer program for realizing each function of the information sharing support device 10 according to the present embodiment and implement it on a PC or the like. It is also possible to provide a computer-readable recording medium in which such a computer program is stored. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like. Further, the above computer program may be distributed, for example, via a network without using a recording medium.
　また、本明細書においてフローチャート図を用いて説明した処理は、必ずしも図示された順序で実行されなくてもよい。いくつかの処理ステップは、並列的に実行されてもよい。また、追加的な処理ステップが採用されてもよく、一部の処理ステップが省略されてもよい。 Further, the processes described in the present specification using the flowchart diagram do not necessarily have to be executed in the order shown in the figure. Some processing steps may be performed in parallel. Further, additional processing steps may be adopted, and some processing steps may be omitted.
　また、本明細書に記載された効果は、あくまで説明的または例示的なものであって限定的ではない。つまり、本開示に係る技術は、上記の効果とともに、または上記の効果に代えて、本明細書の記載から当業者には明らかな他の効果を奏しうる。 Further, the effects described in the present specification are merely explanatory or exemplary and are not limited. That is, the technique according to the present disclosure may exert other effects apparent to those skilled in the art from the description of the present specification, in addition to or in place of the above effects.
　以下の構成も本発明に含み得る。
＜構成１＞
　複数人でオンラインセッションが行われる環境において、当該複数人の中の解析対象者について他者とは異なる特異的な感情を解析するビデオミーティング評価システム。
＜構成２＞
　上記オンラインセッション中に上記複数人を撮影することによって得られる動画像を取得する動画像取得部と、
　上記動画像取得部により取得された動画像に基づいて、上記複数人のそれぞれについて生体反応の変化を解析する生体反応解析部と、
　上記解析対象者について解析された上記生体反応の変化が、上記解析対象者以外の他者について解析された上記生体反応の変化と比べて特異的か否かを判定する特異判定部とを備えた
ことを特徴とするビデオミーティング評価システム。
＜構成３＞
　上記生体反応解析部は、上記動画像取得部により取得された動画像にける顔画像を解析することにより、表情、目線、脈拍、顔の動きの少なくとも１つに関する生体反応の変化を解析することを特徴とする構成２に記載のビデオミーティング評価システム。
＜構成４＞
　上記生体反応解析部は、上記動画像取得部により取得された動画像にける音声を解析することにより、発言内容、声質の少なくとも１つに関する生体反応の変化を解析することを特徴とする構成２または３に記載のビデオミーティング評価システム。
＜構成５＞
　上記生体反応解析部は、上記生体反応の変化を所定の基準に従って数値化することによって生体反応指標値を算出し、
　上記特異判定部は、上記生体反応解析部により上記複数人のそれぞれについて算出された上記生体反応指標値に基づいて、上記解析対象者について解析された上記生体反応の変化が、上記解析対象者以外の他者について解析された上記生体反応の変化と比べて特異的か否かを判定する
ことを特徴とする構成２～４の何れか１項に記載のビデオミーティング評価システム。
＜構成６＞
　上記特異判定部は、上記生体反応解析部により上記複数人のそれぞれについて算出された上記生体反応指標値の分散を算出し、上記解析対象者について算出された上記生体反応指標値と上記分散との対比により、上記解析対象者について解析された上記生体反応の変化が、上記他者について解析された上記生体反応の変化と比べて特異的か否かを判定することを特徴とする構成５に記載のビデオミーティング評価システム。
＜構成７＞
　上記生体反応解析部は、上記複数人のそれぞれについて上記目線の動きを解析して目線の方向を示すヒートマップを生成し、
　上記特異判定部は、上記生体反応解析部により上記解析対象者について生成されたヒートマップと上記他者について生成されたヒートマップとの対比により、上記解析対象者について解析された上記生体反応の変化が、上記他者について解析された上記生体反応の変化と比べて特異的か否かを判定する
ことを特徴とする構成３に記載のビデオミーティング評価システム。
＜構成８＞
　上記特異判定部により特異的であると判定された生体反応の変化が起きたときに上記解析対象者、上記他者および環境の少なくとも１つに関して発生している事象を特定する関連事象特定部を更に備えたことを特徴とする構成２～７の何れか１項に記載のビデオミーティング評価システム。
＜構成９＞
　上記特異判定部により特異的であると判定された生体反応の変化と、当該特異的な生体反応の変化が起きたときに発生している事象との相関の程度を解析し、相関が一定レベル以上であると判定された場合に、その相関の解析結果に基づいて上記解析対象者または上記事象をクラスタリングするクラスタリング部を更に備えたことを特徴とする構成８に記載のビデオミーティング評価システム。
＜構成１０＞
　上記特異判定部により特異的であると判定された生体反応の変化および上記関連事象特定部により特定された事象の少なくとも一方を、上記解析対象者または上記オンラインセッションの主催者に通知する解析結果通知部を更に備えたことを特徴とする構成８に記載のビデオミーティング評価システム。
＜構成１１＞
　上記特異判定部により特異的であると判定された生体反応の変化、上記関連事象特定部により特定された事象、および上記クラスタリング部によりクラスタリングされた分類の少なくとも１つを、上記解析対象者または上記オンラインセッションの主催者に通知する解析結果通知部を更に備えたことを特徴とする請求項９に記載のビデオミーティング評価システム。
＜構成１２＞
　複数人の参加者でオンラインセッションが行われる環境において、オンラインセッション中に参加者が画面に表示されているか否かによらず、上記参加者を撮影することによって得られる動画像をもとに上記参加者の反応を解析し、その解析結果を提示する反応解析システム。
＜構成１３＞
　上記オンラインセッション中に上記参加者を撮影することによって得られる動画像を取得する動画像取得部と、
　上記動画像取得部により取得された動画像に基づいて、上記参加者について生体反応の変化を解析する生体反応解析部と、
　上記画面に表示されていない参加者を含めて上記生体反応解析部により解析された上記生体反応の変化を示す情報を提示する反応情報提示部とを備えた
ことを特徴とする項目１２に記載の反応解析システム。
＜構成１４＞
　上記生体反応解析部は、上記動画像取得部により取得された動画像にける顔画像を解析することにより、表情、目線、脈拍、顔の動きの少なくとも１つに関する生体反応の変化を解析することを特徴とする項目１３に記載の反応解析システム。
＜構成１５＞
　上記生体反応解析部は、上記動画像取得部により取得された動画像にける音声を解析することにより、発言内容、声質の少なくとも１つに関する生体反応の変化を解析することを特徴とする項目１３又は項目１４に記載の反応解析システム。
＜構成１６＞
　上記生体反応解析部は、上記画面に表示されていない参加者が、上記画面に表示されている共有資料のどこを見ているかを解析することを特徴とする項目１３に記載の反応解析システム。
＜構成１７＞
　上記生体反応解析部は、上記画面に表示されていない参加者が、上記オンラインセッション中のどのタイミングで声を出したかを解析することを特徴とする項目１３に記載の反応解析システム。
＜構成１８＞
　上記反応情報提示部は、上記生体反応の変化を示す情報を上記オンラインセッションの主催者に提示することを特徴とする項目１３乃至項目１７の何れか１項に記載の反応解析システム。 The following configurations may also be included in the present invention.
<Structure 1>
A video meeting evaluation system that analyzes specific emotions of the person to be analyzed among the multiple people in an environment where online sessions are held by multiple people.
<Structure 2>
A moving image acquisition unit that acquires moving images obtained by shooting the above multiple people during the online session, and a moving image acquisition unit.
Based on the moving image acquired by the moving image acquisition unit, the biological reaction analysis unit that analyzes the change in the biological reaction for each of the plurality of persons, and the biological reaction analysis unit.
It is provided with a peculiarity determination unit for determining whether or not the change in the biological reaction analyzed for the analysis target person is specific to the change in the biological reaction analyzed for others other than the analysis target person. A video meeting rating system that features that.
<Structure 3>
The biological reaction analysis unit analyzes changes in the biological reaction related to at least one of facial expression, line of sight, pulse, and facial movement by analyzing the facial image in the moving image acquired by the moving image acquisition unit. The video meeting evaluation system according to the configuration 2, which comprises.
<Structure 4>
The biological reaction analysis unit is characterized in that it analyzes changes in the biological reaction regarding at least one of the content of speech and voice quality by analyzing the voice in the moving image acquired by the moving image acquisition unit. Or the video meeting rating system described in 3.
<Structure 5>
The biological reaction analysis unit calculates the biological reaction index value by quantifying the change in the biological reaction according to a predetermined standard.
In the peculiarity determination unit, the change in the biological reaction analyzed for the analysis target person is different from the analysis target person based on the biological reaction index value calculated for each of the plurality of persons by the biological reaction analysis unit. The video meeting evaluation system according to any one of configurations 2 to 4, wherein it is determined whether or not it is specific with respect to the change in the biological reaction analyzed for another person.
<Structure 6>
The peculiarity determination unit calculates the variance of the biological reaction index value calculated for each of the plurality of persons by the biological reaction analysis unit, and the biological reaction index value calculated for the person to be analyzed and the dispersion. Described in Configuration 5, characterized in that it is determined by comparison whether or not the change in the biological reaction analyzed for the subject to be analyzed is specific to the change in the biological reaction analyzed for the other person. Video meeting rating system.
<Structure 7>
The biological reaction analysis unit analyzes the movement of the line of sight for each of the plurality of people and generates a heat map showing the direction of the line of sight.
The peculiarity determination unit is a change in the biological reaction analyzed for the analysis target person by comparing the heat map generated for the analysis target person by the biological reaction analysis unit with the heat map generated for the other person. However, the video meeting evaluation system according to the configuration 3 is characterized in that it determines whether or not it is specific with respect to the change in the biological reaction analyzed for the other person.
<Structure 8>
A related event identification unit that identifies an event occurring with respect to at least one of the analysis target person, the other person, and the environment when a change in a biological reaction determined to be specific by the specificity determination unit occurs. The video meeting evaluation system according to any one of configurations 2 to 7, further comprising.
<Structure 9>
The degree of correlation between the change in the biological reaction determined to be specific by the specific biological reaction unit and the event occurring when the specific biological reaction change occurs is analyzed, and the correlation is at a certain level. The video meeting evaluation system according to configuration 8, further comprising a clustering unit for clustering the analysis target person or the event based on the analysis result of the correlation when it is determined to be the above.
<Structure 10>
Analysis result notification to notify the analysis target person or the organizer of the online session of at least one of the change in the biological reaction determined to be specific by the specificity determination unit and the event specified by the related event identification unit. The video meeting evaluation system according to configuration 8, further comprising a section.
<Structure 11>
At least one of the changes in the biological reaction determined to be specific by the peculiarity determination unit, the event specified by the related event identification unit, and the classification clustered by the clustering unit can be analyzed by the person to be analyzed or the above. The video meeting evaluation system according to claim 9, further comprising an analysis result notification unit for notifying the organizer of the online session.
<Structure 12>
In an environment where an online session is held by multiple participants, the above is based on the moving image obtained by shooting the above participants regardless of whether or not the participants are displayed on the screen during the online session. A reaction analysis system that analyzes the reactions of participants and presents the analysis results.
<Structure 13>
A moving image acquisition unit that acquires a moving image obtained by photographing the participants during the online session, and a moving image acquisition unit.
Based on the moving image acquired by the moving image acquisition unit, the biological reaction analysis unit that analyzes the change in the biological reaction of the participants, and the biological reaction analysis unit.
Item 12. The item 12 comprises a reaction information presenting unit that presents information indicating changes in the biological reaction analyzed by the biological reaction analysis unit, including participants not displayed on the screen. Reaction analysis system.
<Structure 14>
The biological reaction analysis unit analyzes changes in the biological reaction related to at least one of facial expression, line of sight, pulse, and facial movement by analyzing the facial image in the moving image acquired by the moving image acquisition unit. Item 13. The reaction analysis system according to Item 13.
<Structure 15>
Item 13 characterized in that the biological reaction analysis unit analyzes changes in the biological reaction regarding at least one of the content of speech and voice quality by analyzing the voice in the moving image acquired by the moving image acquisition unit. Alternatively, the reaction analysis system according to item 14.
<Structure 16>
The reaction analysis system according to item 13, wherein the biological reaction analysis unit analyzes where the participants who are not displayed on the screen are looking at the shared material displayed on the screen.
<Structure 17>
The reaction analysis system according to item 13, wherein the biological reaction analysis unit analyzes at what timing during the online session a participant who is not displayed on the screen makes a voice.
<Structure 18>
The reaction analysis system according to any one of items 13 to 17, wherein the reaction information presenting unit presents information indicating changes in the biological reaction to the organizer of the online session.
　１０、２０　　　ユーザ端末
　３０　　　ビデオミーティングサービス端末
　４０　　　評価端末

  10, 20 User terminal 30 Video meeting service terminal 40 Evaluation terminal

Claims (5)

  1. 　少なくとも前記第１ユーザ端末と前記第２ユーザ端末とにビデオミーティングを提供すると共に当該ビデオミーティングにおいて取得した動画像を記憶するビデオミーティングサービス端末と、前記ビデオミーティングに関する評価を行う評価端末と、を含む、ビデオミーティングの評価システムであって、
    　前記ビデオミーティングサービス端末は、少なくとも前記第１ユーザ端末又は前記第２ユーザ端末からの許可操作によって、少なくとも前記評価端末に関連付けられた権限情報を生成し、
    　前記評価端末は、前記権限情報に基づいて前記ビデオミーティングサービス端末から前記動画像を取得し、当該動画像内に含まれる少なくとも顔画像を所定のフレーム単位ごとに識別すると共に、前記顔画像に関する評価値を算出する、
    ビデオミーティング評価システム。     It includes at least a video meeting service terminal that provides a video meeting to the first user terminal and the second user terminal and stores a moving image acquired in the video meeting, and an evaluation terminal that evaluates the video meeting. , A video meeting rating system,
    The video meeting service terminal generates at least the authority information associated with the evaluation terminal by a permission operation from the first user terminal or the second user terminal.
    The evaluation terminal acquires the moving image from the video meeting service terminal based on the authority information, identifies at least a facial image contained in the moving image for each predetermined frame unit, and evaluates the facial image. Calculate the value,
    Video meeting rating system.
  2. 　請求項１に記載のビデオミーティング評価システムであって、
    　前記評価端末は、前記評価値の時系列によるグラフ情報を提供する、
    ビデオミーティング評価システム。     The video meeting evaluation system according to claim 1.
    The evaluation terminal provides graph information in chronological order of the evaluation values.
    Video meeting rating system.
  3. 　請求項１又は請求項２に記載のビデオミーティング評価システムであって、
    　前記評価端末は、前記顔画像を複数の異なる観点によって評価した複数の評価値を算出する、
    ビデオミーティング評価システム。     The video meeting evaluation system according to claim 1 or 2.
    The evaluation terminal calculates a plurality of evaluation values obtained by evaluating the face image from a plurality of different viewpoints.
    Video meeting rating system.
  4. 　請求項１乃至請求項３のいずれかに記載のビデオミーティング評価システムであって、
    　前記評価端末は、前記動画像に含まれる音声と共に前記評価値を算出する、
    ビデオミーティング評価システム。     The video meeting evaluation system according to any one of claims 1 to 3.
    The evaluation terminal calculates the evaluation value together with the sound included in the moving image.
    Video meeting rating system.
  5. 　請求項１乃至請求項４のいずれかに記載のビデオミーティング評価システムであって、
    　前記評価端末は、前記動画像内に含まれる前記顔画像以外の対象物と共に前記評価値を算出する、
    ビデオミーティング評価システム。

          The video meeting evaluation system according to any one of claims 1 to 4.
    The evaluation terminal calculates the evaluation value together with an object other than the face image contained in the moving image.
    Video meeting rating system.
PCT/JP2020/036148 2020-09-24 2020-09-24 Video meeting evaluation system and video meeting evaluation server WO2022064619A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/JP2020/036148 WO2022064619A1 (en) 2020-09-24 2020-09-24 Video meeting evaluation system and video meeting evaluation server
JP2022515723A JPWO2022064619A1 (en) 2020-09-24 2020-09-24

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2020/036148 WO2022064619A1 (en) 2020-09-24 2020-09-24 Video meeting evaluation system and video meeting evaluation server

Publications (1)

Publication Number Publication Date
WO2022064619A1 true WO2022064619A1 (en) 2022-03-31

Family

ID=80844606

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/036148 WO2022064619A1 (en) 2020-09-24 2020-09-24 Video meeting evaluation system and video meeting evaluation server

Country Status (2)

Country Link
JP (1) JPWO2022064619A1 (en)
WO (1) WO2022064619A1 (en)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014504460A (en) * 2010-09-30 2014-02-20 アフェクティヴァ，インコーポレイテッド Emotion data measurement for web-enabled applications
JP2015075908A (en) * 2013-10-09 2015-04-20 日本電信電話株式会社 Emotion information display control device, method of the same, and program
JP2018068618A (en) * 2016-10-28 2018-05-10 株式会社東芝 Emotion estimating device, emotion estimating method, emotion estimating program, and emotion counting system
JP2018121752A (en) * 2017-01-30 2018-08-09 国立大学法人 東京大学 Image analysis apparatus, image analysis method and image analysis program
JP2019058625A (en) * 2017-09-26 2019-04-18 株式会社エモスタ Emotion reading device and emotion analysis method
JP2019148852A (en) * 2018-02-26 2019-09-05 京セラドキュメントソリューションズ株式会社 System for determining degree of understanding and program for determining degree of understanding
JP2020109578A (en) * 2019-01-07 2020-07-16 本田技研工業株式会社 Information processing device and program

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014504460A (en) * 2010-09-30 2014-02-20 アフェクティヴァ，インコーポレイテッド Emotion data measurement for web-enabled applications
JP2015075908A (en) * 2013-10-09 2015-04-20 日本電信電話株式会社 Emotion information display control device, method of the same, and program
JP2018068618A (en) * 2016-10-28 2018-05-10 株式会社東芝 Emotion estimating device, emotion estimating method, emotion estimating program, and emotion counting system
JP2018121752A (en) * 2017-01-30 2018-08-09 国立大学法人 東京大学 Image analysis apparatus, image analysis method and image analysis program
JP2019058625A (en) * 2017-09-26 2019-04-18 株式会社エモスタ Emotion reading device and emotion analysis method
JP2019148852A (en) * 2018-02-26 2019-09-05 京セラドキュメントソリューションズ株式会社 System for determining degree of understanding and program for determining degree of understanding
JP2020109578A (en) * 2019-01-07 2020-07-16 本田技研工業株式会社 Information processing device and program

Also Published As

Publication number Publication date
JPWO2022064619A1 (en) 2022-03-31

Similar Documents

Publication Publication Date Title
JP7120693B1 (en) Video image analysis system
WO2022064621A1 (en) Video meeting evaluation system and video meeting evaluation server
WO2022024194A1 (en) Emotion analysis system
WO2022064619A1 (en) Video meeting evaluation system and video meeting evaluation server
WO2022064617A1 (en) Video meeting evaluation system and video meeting evaluation server
WO2022064618A1 (en) Video meeting evaluation system and video meeting evaluation server
WO2022064620A1 (en) Video meeting evaluation system and video meeting evaluation server
WO2022074785A1 (en) Video meeting evaluation terminal, video meeting evaluation system, and video meeting evaluation program
WO2022113248A1 (en) Video meeting evaluation terminal and video meeting evaluation method
WO2022137502A1 (en) Video meeting evaluation terminal, video meeting evaluation system, and video meeting evaluation program
JP7471683B2 (en) Reaction notification system
JP7465012B2 (en) Video meeting evaluation terminal, video meeting evaluation system and video meeting evaluation program
JP7388768B2 (en) Video analysis program
WO2022145043A1 (en) Video meeting evaluation terminal, video meeting evaluation system, and video meeting evaluation program
WO2022145042A1 (en) Video meeting evaluation terminal, video meeting evaluation system, and video meeting evaluation program
JP7138998B1 (en) VIDEO SESSION EVALUATION TERMINAL, VIDEO SESSION EVALUATION SYSTEM AND VIDEO SESSION EVALUATION PROGRAM
WO2022145041A1 (en) Video meeting evaluation terminal, video meeting evaluation system, and video meeting evaluation program
WO2022145038A1 (en) Video meeting evaluation terminal, video meeting evaluation system and video meeting evaluation program
WO2022145039A1 (en) Video meeting evaluation terminal, video meeting evaluation system and video meeting evaluation program
WO2023032058A1 (en) Video session evaluation terminal, video session evaluation system, and video session evaluation program
JP7100938B1 (en) Video analysis program
JP7121436B1 (en) Video analysis program
WO2022254497A1 (en) Video analysis system
JP7121433B1 (en) Video analysis program
WO2022201266A1 (en) Video analysis program

Legal Events

Date Code Title Description
ENP Entry into the national phase

Ref document number: 2022515723

Country of ref document: JP

Kind code of ref document: A

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20955213

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20955213

Country of ref document: EP

Kind code of ref document: A1