WO2022201383A1 - 動画像分析プログラム - Google Patents
動画像分析プログラム Download PDFInfo
- Publication number
- WO2022201383A1 WO2022201383A1 PCT/JP2021/012360 JP2021012360W WO2022201383A1 WO 2022201383 A1 WO2022201383 A1 WO 2022201383A1 JP 2021012360 W JP2021012360 W JP 2021012360W WO 2022201383 A1 WO2022201383 A1 WO 2022201383A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- unit
- moving image
- analysis
- biological reaction
- evaluation
- Prior art date
Links
- 238000004458 analytical method Methods 0.000 title claims abstract description 90
- 238000006243 chemical reaction Methods 0.000 claims abstract description 88
- 238000011156 evaluation Methods 0.000 claims abstract description 46
- 230000008859 change Effects 0.000 claims description 67
- 238000010191 image analysis Methods 0.000 claims description 26
- 238000004891 communication Methods 0.000 abstract description 12
- 230000008921 facial expression Effects 0.000 description 16
- 230000008451 emotion Effects 0.000 description 14
- 238000010586 diagram Methods 0.000 description 11
- 230000006399 behavior Effects 0.000 description 9
- 238000012545 processing Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 8
- 238000000034 method Methods 0.000 description 8
- 230000002996 emotional effect Effects 0.000 description 7
- 230000001815 facial effect Effects 0.000 description 7
- 230000000694 effects Effects 0.000 description 5
- 239000000463 material Substances 0.000 description 4
- 238000004590 computer program Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 210000000887 face Anatomy 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 208000035473 Communicable disease Diseases 0.000 description 1
- 230000003542 behavioural effect Effects 0.000 description 1
- 230000008512 biological response Effects 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 238000010219 correlation analysis Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000004424 eye movement Effects 0.000 description 1
- 238000003703 image analysis method Methods 0.000 description 1
- 230000004630 mental health Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 230000001151 other effect Effects 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/44—Event detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
- G06V40/176—Dynamic expression
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
- H04N7/15—Conference systems
Definitions
- the present invention provides moving images obtained by photographing participants in an environment where an online session is held by a plurality of participants, regardless of whether or not the participants are displayed on the screen during the online session. It relates to a moving image analysis system that analyzes participants' reactions to and.
- Patent Document 1 A technique for analyzing the emotions others receive in response to a speaker's remarks (see Patent Document 1, for example).
- Patent Document 2 There is also known a technique for analyzing changes in facial expressions of a subject in chronological order over a long period of time and estimating the emotions held during that period (see, for example, Patent Literature 2).
- Patent Documents 3 to 5 Furthermore, there are known techniques for identifying factors that have the greatest influence on changes in emotions (see Patent Documents 3 to 5, for example).
- Patent Document 6 a technology that compares the subject's usual facial expression with the current facial expression and issues an alert when the facial expression is dark.
- Patent Documents 7 to 9 There is also known a technique for determining the degree of emotion of a subject by comparing the subject's normal (expressionless) facial expression with the current facial expression (for example, Patent Documents 7 to 9). reference). Furthermore, techniques for analyzing the emotions of an organization and the atmosphere within a group that individuals feel are also known (see, for example, Patent Documents 10 and 11).
- the purpose of the present invention is to objectively evaluate these communications in order to conduct more efficient communication in situations where online communication is the main focus, such as meetings and lectures.
- a moving image analysis system for analyzing reactions of participants, a moving image acquisition unit that acquires a moving image obtained by shooting the participant during the online session; an analysis unit that analyzes changes in biological reactions of the participant based on the moving image acquired by the moving image acquisition unit; a target reading unit that reads target information related to the analysis result; An evaluation unit that compares and evaluates the read target information and the analysis results of the participant, A moving image analysis system is obtained.
- exchanged communication can be objectively evaluated in order to conduct more efficient communication in situations where online communication is the main activity.
- FIG. 1 is an example of a functional block diagram of an evaluation terminal according to an embodiment of the present invention
- FIG. 3 is a diagram showing functional configuration example 1 of the evaluation terminal according to the embodiment of the present invention
- FIG. 8 is a diagram showing functional configuration example 2 of the evaluation terminal according to the embodiment of the present invention
- FIG. 10 is a diagram showing a functional configuration example 3 of the evaluation terminal according to the embodiment of the present invention
- 7 is a screen display example according to the functional configuration example 3 of FIG. 6.
- FIG. FIG. 7 is another screen display example according to the functional configuration example 3 of FIG. 6.
- FIG. FIG. 12 is a diagram showing another configuration of functional configuration example 3 of the evaluation terminal according to the embodiment of the present invention
- FIG. 12 is a diagram showing another configuration of functional configuration example 3 of the evaluation terminal according to the embodiment of the present invention.
- 1 is a functional block diagram of a system according to embodiments of the invention;
- FIG. 5 is a comparison diagram of analysis results (emotion graphs) according to the embodiment of the present invention; It is an example of a display of the comparison result in embodiment of this invention.
- the contents of the embodiments of the present disclosure are listed and described.
- the present disclosure has the following configurations.
- Items 1 In an environment where an online session is held by a plurality of participants, regardless of whether or not the participants are displayed on the screen during the online session, based on the moving image obtained by shooting the participants
- a moving image analysis system for analyzing reactions of participants a moving image acquisition unit that acquires a moving image obtained by shooting the participant during the online session; an analysis unit that analyzes changes in biological reactions of the participant based on the moving image acquired by the moving image acquisition unit; a target reading unit that reads target information related to the analysis result;
- An evaluation unit that compares and evaluates the read target information and the analysis results of the participant, Video image analysis system.
- the moving image analysis system according to item 1 The target information is a target value for changes in the biological reaction set in time series, The evaluation unit evaluates whether the change in the biological reaction of the participant satisfies the target value, Video image analysis system.
- the moving image analysis system according to item 2 As a result of the evaluation, when the trend of the change in the biological reaction of the participant in a predetermined period deviates from the target value by a predetermined threshold or more, a notification unit that notifies the expert terminal is further provided.
- Video image analysis system The moving image analysis system according to any one of items 1 to 3, further comprising an evaluation providing unit that provides the participant with the result of the evaluation; Video image analysis system.
- a moving image analysis apparatus having the configuration of the moving image analysis system according to any one of items 1 to 4.
- a moving image analysis program that causes a moving image analyzing apparatus to function as a moving image analyzing system according to any one of items 1 to 4.
- a moving image analysis method for executing the configuration of the moving image analysis system according to any one of items 1 to 4 as steps.
- a video session in an environment where a video session (hereinafter referred to as an online session including one-way and two-way sessions) is held by a plurality of people, the person to be analyzed among the plurality of people is different from the others. It is a system that analyzes and evaluates specific emotions (feelings that occur in response to one's own or others' words and actions. pleasant/unpleasant, or their degree).
- Online sessions are, for example, online meetings, online classes, online chats, etc. Terminals installed in multiple locations are connected to a server via a communication network such as the Internet, and moving images are transmitted between multiple terminals through the server. It's made to be interactable.
- Moving images handled in online sessions include facial images and voices of users using terminals.
- Moving images also include images such as materials that are shared and viewed by a plurality of users. It is possible to switch between the face image and the document image on the screen of each terminal to display only one of them, or to divide the display area and display the face image and the document image at the same time. In addition, it is possible to display the image of one user out of a plurality of users on the full screen, or divide the images of some or all of the users into small screens and display them. It is possible to designate one or a plurality of users among a plurality of users participating in an online session using terminals as analysis subjects.
- an online session leader, moderator, or manager designates any user as an analysis subject.
- Hosts of online sessions are, for example, instructors of online classes, chairpersons and facilitators of online meetings, coaches of sessions for coaching purposes, and the like.
- An online session host is typically one of the users participating in the online session, but may be another person who does not participate in the online session. It should be noted that all participants may be subject to analysis without specifying the person to be analyzed.
- an online session leader, moderator, or administrator hereinafter collectively referred to as the organizer to designate any user as an analysis subject.
- Hosts of online sessions are, for example, instructors of online classes, chairpersons and facilitators of online meetings, coaches of sessions for coaching purposes, and the like.
- An online session host is typically one of the users participating in the online session, but may be another person who does not participate in the online session.
- the video session evaluation system displays at least moving images obtained from a video session established between a plurality of terminals.
- the displayed moving image is acquired by the terminal, and at least a face image included in the moving image is identified for each predetermined frame unit. An evaluation value for the identified face image is then calculated.
- the evaluation value is shared as necessary.
- the acquired moving image is stored in the terminal, analyzed and evaluated on the terminal, and the result is provided to the user of the terminal. Therefore, for example, even a video session containing personal information or a video session containing confidential information can be analyzed and evaluated without providing the moving image itself to an external evaluation agency or the like.
- the evaluation result evaluation value
- the video session evaluation system includes user terminals 10 and 20 each having at least an input unit such as a camera unit and a microphone unit, a display unit such as a display, and an output unit such as a speaker. , a video session service terminal 30 for providing an interactive video session to the user terminals 10, 20, and an evaluation terminal 40 for performing part of the evaluation of the video session.
- Each functional block, functional unit, and functional module described below can be configured by any of hardware, DSP (Digital Signal Processor), and software provided in a computer, for example.
- DSP Digital Signal Processor
- a computer CPU random access memory
- RAM random access memory
- ROM read-only memory
- a series of processes by the systems and terminals described herein may be implemented using software, hardware, or a combination of software and hardware. It is possible to create a computer program for realizing each function of the information sharing support device 10 according to the present embodiment and implement it in a PC or the like. It is also possible to provide a computer-readable recording medium storing such a computer program.
- the recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like.
- the above computer program may be distributed, for example, via a network without using a recording medium.
- the evaluation terminal acquires a moving image from a video session service terminal, identifies at least a face image included in the moving image for each predetermined frame unit, and calculates an evaluation value for the face image ( will be described in detail later).
- the video session service provided by the video session service terminal (hereinafter sometimes simply referred to as "this service") provides user terminals 10 and 20 with two-way images and voice. Communication is possible.
- this service a moving image captured by the camera of the other user's terminal is displayed on the display of the user's terminal, and audio captured by the microphone of the other's user's terminal can be output from the speaker.
- this service allows both or either of the user terminals to record moving images and sounds (collectively referred to as "moving images, etc.") in the storage unit of at least one of the user terminals. configured as possible.
- the recorded moving image information Vs (hereinafter referred to as “recorded information”) is cached in the user terminal that started recording and is locally recorded only in one of the user terminals. If necessary, the user can view the recorded information by himself or share it with others within the scope of using this service.
- FIG. 4 is a block diagram showing a configuration example according to this embodiment.
- the video session evaluation system of this embodiment is implemented as a functional configuration of the user terminal 10.
- the user terminal 10 has, as its functions, a moving image acquisition unit 11, a biological reaction analysis unit 12, a peculiar determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification unit 16.
- the moving image acquisition unit 11 acquires from each terminal a moving image obtained by photographing a plurality of people (a plurality of users) with a camera provided in each terminal during an online session. It does not matter whether the moving image acquired from each terminal is set to be displayed on the screen of each terminal. That is, the moving image acquisition unit 11 acquires moving images from each terminal, including moving images being displayed and moving images not being displayed on each terminal.
- the biological reaction analysis unit 12 analyzes changes in the biological reaction of each of a plurality of people based on the moving images (whether or not they are being displayed on the screen) acquired by the moving image acquiring unit 11.
- the biological reaction analysis unit 12 separates the moving image acquired by the moving image acquisition unit 11 into a set of images (collection of frame images) and voice, and analyzes changes in the biological reaction from each.
- the biological reaction analysis unit 12 analyzes the user's facial image using a frame image separated from the moving image acquired by the moving image acquisition unit 11 to obtain at least one of facial expression, gaze, pulse, and facial movement. Analyze changes in biological reactions related to Further, the biological reaction analysis unit 12 analyzes the voice separated from the moving image acquired by the moving image acquisition unit 11 to analyze changes in the biological reaction related to at least one of the user's utterance content and voice quality.
- the biological reaction analysis unit 12 calculates a biological reaction index value reflecting the change in biological reaction by quantifying the change in biological reaction according to a predetermined standard.
- the analysis of changes in facial expressions is performed as follows. That is, for each frame image, a facial region is identified from the frame image, and the identified facial expressions are classified into a plurality of types according to an image analysis model machine-learned in advance. Then, based on the classification results, it analyzes whether positive facial expression changes occur between consecutive frame images, whether negative facial expression changes occur, and to what extent the facial expression changes occur, A facial expression change index value corresponding to the analysis result is output.
- the analysis of changes in line of sight is performed as follows. That is, for each frame image, the eye region is specified in the frame image, and the orientation of both eyes is analyzed to analyze where the user is looking. For example, it analyzes whether the user is looking at the face of the speaker being displayed, whether the user is looking at the shared material being displayed, or whether the user is looking outside the screen. Also, it may be analyzed whether the eye movement is large or small, or whether the movement is frequent or infrequent. A change in line of sight is also related to the user's degree of concentration.
- the biological reaction analysis unit 12 outputs a line-of-sight change index value according to the analysis result of the line-of-sight change.
- the analysis of pulse changes is performed, for example, as follows. That is, for each frame image, the face area is specified in the frame image. Then, using a trained image analysis model that captures numerical values of face color information (G of RGB), changes in the G color of the face surface are analyzed. By arranging the results along the time axis, a waveform representing changes in color information is formed, and the pulse is identified from this waveform. When a person is tense, the pulse speeds up, and when the person is calm, the pulse slows down. The biological reaction analysis unit 12 outputs a pulse change index value according to the analysis result of the pulse change.
- G of RGB face color information
- analysis of changes in facial movement is performed as follows. That is, for each frame image, the face area is specified in the frame image, and the direction of the face is analyzed to analyze where the user is looking. For example, it analyzes whether the user is looking at the face of the speaker being displayed, whether the user is looking at the shared material being displayed, or whether the user is looking outside the screen. Further, it may be analyzed whether the movement of the face is large or small, or whether the movement is frequent or infrequent. The movement of the face and the movement of the line of sight may be analyzed together. For example, it may be analyzed whether the face of the speaker being displayed is viewed straight, whether the face is viewed with upward or downward gaze, or whether the face is viewed obliquely.
- the biological reaction analysis unit 12 outputs a face orientation change index value according to the analysis result of the face orientation change.
- Analysis of the contents of the statement is performed, for example, as follows. That is, the biological reaction analysis unit 12 converts the voice into a character string by performing known voice recognition processing on the voice for a specified time (for example, about 30 to 150 seconds), and morphologically analyzes the character string. By doing so, words such as particles and articles that are unnecessary for expressing conversation are removed. Then, vectorize the remaining words, analyze whether a positive emotional change has occurred, whether a negative emotional change has occurred, and to what extent the emotional change has occurred. Outputs the statement content index value.
- Voice quality analysis is performed, for example, as follows. That is, the biological reaction analysis unit 12 identifies the acoustic features of the voice by performing known voice analysis processing on the voice for a specified time (for example, about 30 to 150 seconds). Then, based on the acoustic features, it analyzes whether a positive change in voice quality has occurred, whether a negative change in voice quality has occurred, and to what extent the change in voice quality has occurred, and according to the analysis results, output the voice quality change index value.
- a specified time for example, about 30 to 150 seconds
- the biological reaction analysis unit 12 uses at least one of the facial expression change index value, eye line change index value, pulse change index value, face direction change index value, statement content index value, and voice quality change index value calculated as described above. to calculate the biological reaction index value.
- the biological reaction index value is calculated by weighting the facial expression change index value, eye line change index value, pulse change index value, face direction change index value, statement content index value, and voice quality change index value.
- the peculiarity determination unit 13 determines whether or not the change in the analyzed biological reaction of the person to be analyzed is more specific than the change in the analyzed biological reaction of the person other than the person to be analyzed. In the present embodiment, the peculiarity determination unit 13 compares changes in the biological reaction of the person to be analyzed with those of others based on the biological reaction index values calculated for each of the plurality of users by the biological reaction analysis unit 12. is specific or not.
- the peculiar determination unit 13 calculates the variance of the biological reaction index values calculated for each of the plurality of persons by the biological reaction analysis unit 12, and compares the biological reaction index values calculated for the analysis subject with the variance, It is determined whether or not the change in the analyzed biological reaction of the person to be analyzed is specific compared to others.
- the following three patterns are conceivable as cases where the changes in biological reactions analyzed for the subject of analysis are more specific than those of others.
- the first is a case where a relatively large change in biological reaction occurs in the subject of analysis, although no particularly large change in biological reaction has occurred in the other person.
- the second is a case where a particularly large change in biological reaction has not occurred in the subject of analysis, but a relatively large change in biological reaction has occurred in the other person.
- the third is a case where a relatively large change in biological reaction occurs in both the subject of analysis and the other person, but the content of the change differs between the subject of analysis and the other person.
- the related event identification unit 14 identifies an event occurring in relation to at least one of the person to be analyzed, the other person, and the environment when the change in the biological reaction determined to be peculiar by the peculiarity determination unit 13 occurs. .
- the related event identification unit 14 identifies from the moving image the speech and behavior of the person to be analyzed when a specific change in biological reaction occurs in the person to be analyzed.
- the related event identifying unit 14 identifies, from the moving image, the speech and behavior of the other person when a specific change in the biological reaction of the person to be analyzed occurs.
- the related event identification unit 14 identifies from the moving image the environment in which a specific change in the biological reaction of the person to be analyzed occurs.
- the environment is, for example, the shared material being displayed on the screen, the background image of the person to be analyzed, and the like.
- the clustering unit 15 clusters the change in the biological reaction determined to be specific by the peculiarity determination unit 13 (for example, one or a combination of eye gaze, pulse, facial movement, statement content, and voice quality), and the peculiarity Analyzing the degree of correlation with an event (event identified by the related event identification unit 14) that occurs when a change in biological reaction occurs, and if it is determined that the correlation is at a certain level or more , to cluster the subjects or events based on the correlation analysis results.
- the peculiarity determination unit 13 for example, one or a combination of eye gaze, pulse, facial movement, statement content, and voice quality
- the clustering unit 15 clusters the person to be analyzed or the event into one of a plurality of pre-segmented categories according to the content of the event, the degree of negativity, the magnitude of the correlation, and the like.
- the clustering unit 15 clusters the person to be analyzed or the event into one of a plurality of pre-segmented classifications according to the content of the event, the degree of positivity, the degree of correlation, and the like.
- the analysis result notification unit 16 reports at least one of the changes in the biological reaction determined to be specific by the peculiar determination unit 13, the event identified by the related event identification unit 14, and the classification clustered by the clustering unit 15. , to notify the designator of the subject of analysis (the subject of analysis or the organizer of the online session).
- the analysis result notification unit 16 recognizes that when a change in a specific biological reaction that is different from that of the other person occurs in the person to be analyzed (one of the three patterns described above; the same applies hereinafter), the analysis target is Notifies the person to be analyzed of his/her own behavior. This allows the person to be analyzed to understand that he/she has a different feeling from others when he or she performs a certain behavior. At this time, the person to be analyzed may also be notified of the change in the specific biological reaction identified for the person to be analyzed. Furthermore, the person to be analyzed may be further notified of the change in the biological reaction of the other person to be compared.
- the words and deeds of the person to be analyzed performed without being particularly conscious of their usual emotions, or the words and deeds of the person to be analyzed consciously accompanied by certain emotions, and the emotions and behaviors that others received
- the emotion held by the person to be analyzed is different from the feeling held by the person to be analyzed at the time
- the person to be analyzed is notified of the speech and behavior of the person to be analyzed at that time.
- the analysis result notification unit 16 notifies the organizer of the online session of the event occurring when the person to be analyzed undergoes a specific change in biological reaction that is different from that of the other person, together with the change in the specific biological reaction. to notify.
- the organizer of the online session can know what kind of event affects what kind of emotional change as a phenomenon specific to the specified analysis subject. Then, it becomes possible to perform appropriate treatment on the person to be analyzed according to the grasped contents.
- the analysis result notification unit 16 notifies the organizer of the online session of the event occurring when a specific change in biological reaction occurs in the analysis subject, which is different from that of others, or the clustering result of the analysis subject. do.
- online session organizers can grasp behavioral tendencies peculiar to analysis subjects and predict possible future behaviors and situations, depending on which classification the specified analysis subjects have been clustered into. be able to. Then, it becomes possible to take appropriate measures for the person to be analyzed.
- the biological reaction index value is calculated by quantifying the change in biological reaction according to a predetermined standard, and the analysis subject is analyzed based on the biological reaction index value calculated for each of the plurality of people.
- the biological reaction analysis unit 12 analyzes the movement of the line of sight for each of a plurality of people and generates a heat map indicating the direction of the line of sight.
- the peculiar determination unit 13 compares the heat map generated for the person to be analyzed by the biological reaction analysis unit 12 with the heat map generated for the other person, so that the change in the biological reaction analyzed for the person to be analyzed It is determined whether it is specific compared with the change in biological response analyzed for.
- moving images of a video session are stored in the local storage of the user terminal 10, and the above analysis is performed on the user terminal 10.
- the machine specs of the user terminal 10 it is possible to analyze the moving image information without providing it to the outside.
- the video session evaluation system of this embodiment may include a moving image acquisition unit 11, a biological reaction analysis unit 12, and a reaction information presentation unit 13a as functional configurations.
- the reaction information presentation unit 13a presents information indicating changes in biological reactions analyzed by the biological reaction analysis unit 12a, including participants not displayed on the screen.
- the reaction information presenting unit 13a presents information indicating changes in biological reactions to an online session leader, moderator, or administrator (hereinafter collectively referred to as the organizer).
- Hosts of online sessions are, for example, instructors of online classes, chairpersons and facilitators of online meetings, coaches of sessions for coaching purposes, and the like.
- An online session host is typically one of the users participating in the online session, but may be another person who does not participate in the online session.
- the organizer of the online session can also grasp the state of the participants who are not displayed on the screen in an environment where the online session is held by multiple people.
- FIG. 6 is a block diagram showing a configuration example according to this embodiment. As shown in FIG. 6, in the video session evaluation system of the present embodiment, functions similar to those of the above-described first embodiment are given the same reference numerals, and explanations thereof may be omitted.
- the system includes a camera unit that acquires images of a video session, a microphone unit that acquires audio, an analysis unit that analyzes and evaluates moving images, and information obtained by evaluating the acquired moving images.
- an object generator for generating a display object (described below) based on the display; and a display for displaying both the moving image of the video session and the display object during execution of the video session.
- the analysis unit includes the moving image acquisition unit 11, the biological reaction analysis unit 12, the peculiar determination unit 13, the related event identification unit 14, the clustering unit 15, and the analysis result notification unit 16, as described above.
- the function of each element is as described above.
- the object generation unit generates an object 50 representing the recognized face part and the above-mentioned Information 100 indicating the content of the analysis/evaluation performed is superimposed on the moving image and displayed.
- the object 50 may identify and display all faces of a plurality of persons when the faces of the plurality of persons are moved in the moving image.
- the object 50 is, for example, when the camera function of the video session is stopped at the other party's terminal (that is, it is stopped by software within the application of the video session instead of physically covering the camera). If the other party's face is recognized by the other party's camera, the object 50 or the object 100 may be displayed in the part where the other party's face is located. This makes it possible for both parties to confirm that the other party is in front of the terminal even if the camera function is turned off. In this case, for example, in a video session application, the information obtained from the camera may be hidden while only the object 50 or object 100 corresponding to the face recognized by the analysis unit is displayed. Also, the video information acquired from the video session and the information recognized by the analysis unit may be divided into different display layers, and the layer relating to the former information may be hidden.
- the objects 50 and 100 may be displayed in all areas or only in some areas. For example, as shown in FIG. 8, it may be displayed only on the moving image on the guest side.
- the embodiments of the invention described in Basic Configuration Example 1 to Basic Configuration Example 3 described above may be implemented as a single device, or a plurality of devices (for example, cloud servers) partially or entirely connected by a network. and the like.
- the control unit 110 and the storage 130 of each terminal 10 may be realized by different servers connected to each other via a network. That is, the system includes user terminals 10, 20, a video session service terminal 30 for providing an interactive video session to the user terminals 10, 20, and an evaluation terminal 40 for evaluating the video session, Variation combinations of the following configurations are conceivable. (1) Processing everything only on the user terminal As shown in FIG. 9, by performing the processing by the analysis unit on the terminal that is performing the video session (although a certain processing capacity is required), the video session can be performed.
- an analysis unit may be provided in an evaluation terminal connected via a network or the like.
- the moving images acquired by the user terminal are shared with the evaluation terminal at the same time as or after the video session, and are analyzed and evaluated by the analysis unit in the evaluation terminal.
- the moving image data that is, information including at least analysis data
- a moving image analysis system (hereinafter simply referred to as "system") according to an embodiment of the present invention captures all participants or only a specific participant in an environment where an online session is held with a plurality of participants. Participants' reactions are analyzed based on the moving images obtained by this process. The analysis may occur whether or not participants are on screen during the online session.
- the system according to the present embodiment for example, automatically traces through online sessions the effect measurement of treatment for participants whose mental health has deteriorated due to receiving new stress. Uses include: Conversely, it can also be used to confirm whether or not there is a goal of emotional expression that should be considered as an ideal one and one is growing toward it.
- the system includes a moving image acquisition unit that acquires moving image data, an analysis unit, a target reading unit, and an evaluation unit.
- a moving image acquisition unit acquires a moving image obtained by photographing a participant during an online session.
- the analysis unit analyzes changes in the biological reaction of the participant based on the moving image acquired by the moving image acquiring unit (see also FIGS. 3 to 5, etc.).
- the target reading unit reads target information related to analysis results.
- the target information is a target value of change in the participant's biological reaction set in chronological order.
- the target value is a chronological order of analytical values (analytical values corresponding to anger, fear, expectation, surprise, joy, sadness, trust, disgust, etc.) from a given point of view (for example, the solid line graph in FIG. 11). good too.
- the evaluation unit evaluates whether the analysis result by the analysis unit is in line with the target information and provides it to the participants. For example, the progress can be visually understood by writing the history of the past analysis results together with the goal as shown in the figure.
- a notification unit that performs notification may be further provided.
- Video image analysis system if the trend of the change in the participant's biological reaction during a predetermined period (see Fig. 11) deviates from the target value by a predetermined threshold or more, A notification unit that performs notification may be further provided.
- Video image analysis system if the trend of the change in the participant's biological reaction during a predetermined period (see Fig. 11) deviates from the target value by a predetermined threshold or more, A notification unit that performs notification may be further provided.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Ophthalmology & Optometry (AREA)
- Signal Processing (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
Abstract
Description
複数人の参加者でオンラインセッションが行われる環境において、オンラインセッション中に参加者が画面に表示されているか否かによらず、前記参加者を撮影することによって得られる動画像をもとに前記参加者の反応を解析する動画像分析システムであって、
前記オンラインセッション中に前記参加者を撮影することによって得られる動画像を取得する動画像取得部と、
前記動画像取得部により取得された動画像に基づいて、前記参加者について生体反応の変化を解析する解析部と、
前記解析結果に関する目標情報を読み込む目標読込部と、
読み込んだ前記目標情報と、前記参加者の解析結果とを比較して評価する評価部と、を備える、
動画像分析システム
が得られる。
[項目1]
複数人の参加者でオンラインセッションが行われる環境において、オンラインセッション中に参加者が画面に表示されているか否かによらず、前記参加者を撮影することによって得られる動画像をもとに前記参加者の反応を解析する動画像分析システムであって、
前記オンラインセッション中に前記参加者を撮影することによって得られる動画像を取得する動画像取得部と、
前記動画像取得部により取得された動画像に基づいて、前記参加者について生体反応の変化を解析する解析部と、
前記解析結果に関する目標情報を読み込む目標読込部と、
読み込んだ前記目標情報と、前記参加者の解析結果とを比較して評価する評価部と、を備える、
動画像分析システム。
[項目2]
項目1に記載の動画像分析システムであって、
前記目標情報は、時系列に設定された前記生体反応の変化の目標値であり、
前記評価部は、前記参加者の前記生体反応の変化が、前記目標値を満たしているか否かを評価する、
動画像分析システム。
[項目3]
項目2に記載の動画像分析システムであって、
前記評価の結果、所定期間における前記参加者の生体反応の変化のトレンドが、前記目標値から所定の閾値以上乖離した場合に、専門家端末へ通知を行う通知部を更に備える、
動画像分析システム。
[項目4]
項目1乃至項目3のいずれかに記載の動画像分析システムであって、
前記評価の結果を前記参加者に提供する評価提供部を更に備える、
動画像分析システム。
[項目5]
項目1乃至項目4のいずれかに記載の動画像分析システムの構成を備えた動画像分析装置。
[項目6]
項目1乃至項目4いずれかに記載の動画像分析システムの構成を動画像分析装置に機能させる動画像分析プログラム。
[項目7]
項目1乃至項目4のいずれかに記載の動画像分析システムの構成をステップとして実行する動画像分析方法。
本実施形態のビデオセッション評価システムは、複数人でビデオセッション(以下、一方向及び双方向含めてオンラインセッションという)が行われる環境において、当該複数人の中の解析対象者について他者とは異なる特異的な感情(自分または他人の言動に対して起こる気持ち。快・不快またはその程度など)を解析し評価するシステムである。オンラインセッションは、例えばオンライン会議、オンライン授業、オンラインチャットなどであり、複数の場所に設置された端末をインターネットなどの通信ネットワークを介してサーバに接続し、当該サーバを通じて複数の端末間で動画像をやり取りできるようにしたものである。オンラインセッションで扱う動画像には、端末を使用するユーザの顔画像や音声が含まれる。また、動画像には、複数のユーザが共有して閲覧する資料などの画像も含まれる。各端末の画面上に顔画像と資料画像とを切り替えて何れか一方のみを表示させたり、表示領域を分けて顔画像と資料画像とを同時に表示させたりすることが可能である。また、複数人のうち1人の画像を全画面表示させたり、一部または全部のユーザの画像を小画面に分割して表示させたりすることが可能である。端末を使用してオンラインセッションに参加する複数のユーザのうち、何れか1人または複数人を解析対象者として指定することが可能である。例えば、オンラインセッションの主導者、進行者または管理者(以下、まとめて主催者という)が何れかのユーザを解析対象者として指定する。オンラインセッションの主催者は、例えばオンライン授業の講師、オンライン会議の議長やファシリテータ、コーチングを目的としたセッションのコーチなどである。オンラインセッションの主催者は、オンラインセッションに参加する複数のユーザの中の一人であるのが普通であるが、オンラインセッションに参加しない別人であってもよい。なお、解析対象者を指定せず全ての参加者を解析対象としてもよい。また、オンラインセッションの主導者、進行者または管理者(以下、まとめて主催者という)が何れかのユーザを解析対象者として指定することも可能である。オンラインセッションの主催者は、例えばオンライン授業の講師、オンライン会議の議長やファシリテータ、コーチングを目的としたセッションのコーチなどである。オンラインセッションの主催者は、オンラインセッションに参加する複数のユーザの中の一人であるのが普通であるが、オンラインセッションに参加しない別人であってもよい。
以下に説明する各機能ブロック、機能単位、機能モジュールは、例えばコンピュータに備えられたハードウェア、DSP(Digital Signal Processor)、ソフトウェアの何れによっても構成することが可能である。例えばソフトウェアによって構成する場合、実際にはコンピュータのCPU、RAM、ROMなどを備えて構成され、RAMやROM、ハードディスクまたは半導体メモリ等の記録媒体に記憶されたプログラムが動作することによって実現される。本明細書において説明するシステム及び端末による一連の処理は、ソフトウェア、ハードウェア、及びソフトウェアとハードウェアとの組合せのいずれを用いて実現されてもよい。本実施形態に係る情報共有支援装置10の各機能を実現するためのコンピュータプログラムを作製し、PC等に実装することが可能である。また、このようなコンピュータプログラムが格納された、コンピュータで読み取り可能な記録媒体も提供することが可能である。記録媒体は、例えば、磁気ディスク、光ディスク、光磁気ディスク、フラッシュメモリ等である。また、上記のコンピュータプログラムは、記録媒体を用いずに、例えばネットワークを介して配信されてもよい。
図3に示されるように、ビデオセッションサービス端末が提供するビデオセッションサービス(以下、単に「本サービス」と言うことがある」)は、ユーザ端末10、20に対して双方向に画像および音声によって通信が可能となるものである。本サービスは、ユーザ端末のディスプレイに相手のユーザ端末のカメラ部で取得した動画像を表示し、相手のユーザ端末のマイク部で取得した音声をスピーカーから出力可能となっている。また、本サービスは双方の又はいずれかのユーザ端末によって、動画像及び音声(これらを合わせて「動画像等」という)を少なくともいずれかのユーザ端末上の記憶部に記録(レコーディング)することが可能に構成されている。記録された動画像情報Vs(以下「記録情報」という)は、記録を開始したユーザ端末にキャッシュされつついずれかのユーザ端末のローカルのみに記録されることとなる。ユーザは、必要があれば当該記録情報を本サービスの利用の範囲内で自分で視聴、他者に共有等行うこともできる。
図4は、本実施形態による構成例を示すブロック図である。図4に示すように、本実施形態のビデオセッション評価システムは、ユーザ端末10が有する機能構成として実現される。すなわち、ユーザ端末10はその機能として、動画像取得部11、生体反応解析部12、特異判定部13、関連事象特定部14、クラスタリング部15および解析結果通知部16を備えている。
図5に示すように、本実施形態のビデオセッション評価システムは、機能構成として、動画像取得部11、生体反応解析部12および反応情報提示部13aを備えていてもよい。
図6は、本実施形態による構成例を示すブロック図である。図6に示すように、本実施形態のビデオセッション評価システムは、機能構成として、上述した実施の形態1と類似する機能については同一つの参照符号を付して説明を省略することがある。
(1)すべてをユーザ端末のみで処理
図9に示されるように、解析部による処理をビデオセッションを行っている端末で行うことにより、(一定の処理能力は必要なものの)ビデオセッションを行っている時間と同時に(リアルタイムに)分析・評価結果を得ることができる。
(2)ユーザ端末と評価端末とで処理
図10に示されるように、ネットワーク等で接続された評価端末に解析部を備えさせることとしてもよい。この場合、ユーザ端末で取得された動画像は、ビデオセッションと同時に又は事後的に評価端末に共有され、評価端末における解析部によって分析・評価されたのちに、オブジェクト50及びオブジェクト100の情報がユーザ端末に動画像データと共に又は別に(即ち、少なくとも解析データを含む情報が)共有され表示部に表示される。
本発明の実施の形態による動画像分析システム(以下、単に「システム」という)は、複数人の参加者でオンラインセッションが行われる環境において、当該参加者の全員又は特定の参加者のみを撮影することによって得られる動画像をもとに参加者の反応を解析・分析するものである。分析は、オンラインセッション中に参加者が画面に表示されているか否かによらず行われるものとしてもよい。
動画像分析システム。
30 ビデオセッションサービス端末
40 評価端末
Claims (4)
- 複数人の参加者でオンラインセッションが行われる環境において、オンラインセッション中に参加者が画面に表示されているか否かによらず、前記参加者を撮影することによって得られる動画像をもとに前記参加者の反応を解析する動画像分析システムであって、
前記オンラインセッション中に前記参加者を撮影することによって得られる動画像を取得する動画像取得部と、
前記動画像取得部により取得された動画像に基づいて、前記参加者について生体反応の変化を解析する解析部と、
前記解析結果に関する目標情報を読み込む目標読込部と、
読み込んだ前記目標情報と、前記参加者の解析結果とを比較して評価する評価部と、を備える、
動画像分析システム。 - 請求項1に記載の動画像分析システムであって、
前記目標情報は、時系列に設定された前記生体反応の変化の目標値であり、
前記評価部は、前記参加者の前記生体反応の変化が、前記目標値を満たしているか否かを評価する、
動画像分析システム。 - 請求項2に記載の動画像分析システムであって、
前記評価の結果、所定期間における前記参加者の生体反応の変化のトレンドが、前記目標値から所定の閾値以上乖離した場合に、専門家端末へ通知を行う通知部を更に備える、
動画像分析システム。 - 請求項1乃至請求項3のいずれかに記載の動画像分析システムであって、
前記評価の結果を前記参加者に提供する評価提供部を更に備える、
動画像分析システム。
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US18/552,384 US11935329B2 (en) | 2021-03-24 | 2021-03-24 | Video analysis program |
PCT/JP2021/012360 WO2022201383A1 (ja) | 2021-03-24 | 2021-03-24 | 動画像分析プログラム |
JP2021572316A JP7152817B1 (ja) | 2021-03-24 | 2021-03-24 | 動画像分析プログラム |
JP2022013829A JP7152819B1 (ja) | 2021-03-24 | 2022-02-01 | 動画像分析プログラム |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/JP2021/012360 WO2022201383A1 (ja) | 2021-03-24 | 2021-03-24 | 動画像分析プログラム |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2022201383A1 true WO2022201383A1 (ja) | 2022-09-29 |
Family
ID=83396646
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2021/012360 WO2022201383A1 (ja) | 2021-03-24 | 2021-03-24 | 動画像分析プログラム |
Country Status (3)
Country | Link |
---|---|
US (1) | US11935329B2 (ja) |
JP (1) | JP7152817B1 (ja) |
WO (1) | WO2022201383A1 (ja) |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2015186127A (ja) * | 2014-03-25 | 2015-10-22 | ブラザー工業株式会社 | プログラム及びサーバ装置 |
JP2016149063A (ja) * | 2015-02-13 | 2016-08-18 | オムロン株式会社 | 感情推定装置及び感情推定方法 |
JP2019148852A (ja) * | 2018-02-26 | 2019-09-05 | 京セラドキュメントソリューションズ株式会社 | 理解度判定システムおよび理解度判定プログラム |
JP2021027917A (ja) * | 2019-08-09 | 2021-02-25 | 美津濃株式会社 | 情報処理装置、情報処理システム、および機械学習装置 |
Family Cites Families (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP3201355B2 (ja) | 1998-08-28 | 2001-08-20 | 日本電気株式会社 | 感情解析システム |
JP2011154665A (ja) | 2010-01-25 | 2011-08-11 | Opn Corp | 表情変化分析システム |
JP2011186521A (ja) | 2010-03-04 | 2011-09-22 | Nec Corp | 感情推定装置および感情推定方法 |
JP5300795B2 (ja) | 2010-06-28 | 2013-09-25 | 日本電信電話株式会社 | 顔表情増幅装置、表情認識装置、顔表情増幅方法、表情認識方法、及びプログラム |
US9026476B2 (en) * | 2011-05-09 | 2015-05-05 | Anurag Bist | System and method for personalized media rating and related emotional profile analytics |
JP5917841B2 (ja) | 2011-06-15 | 2016-05-18 | 日産自動車株式会社 | 気分判定装置及び気分判定装置の作動方法 |
CN106462915A (zh) | 2014-05-13 | 2017-02-22 | 有限会社赛尔雷伯斯 | 情绪和气氛的数据输入、显示和分析装置 |
WO2016178329A1 (ja) | 2015-05-07 | 2016-11-10 | ソニー株式会社 | 情報処理システム、制御方法、および記憶媒体 |
CN106562792B (zh) | 2015-10-08 | 2021-08-06 | 松下电器(美国)知识产权公司 | 信息提示装置的控制方法和信息提示装置 |
JP6801459B2 (ja) | 2017-01-10 | 2020-12-16 | 日本電気株式会社 | 情報処理装置、いじめ発見方法、情報処理システム、および、コンピュータプログラム |
JP2019058625A (ja) | 2017-09-26 | 2019-04-18 | 株式会社エモスタ | 感情読み取り装置及び感情解析方法 |
JP2020048610A (ja) | 2018-09-21 | 2020-04-02 | 富士ゼロックス株式会社 | 状態評価システム |
JP2020086559A (ja) | 2018-11-16 | 2020-06-04 | 具亨 西川 | 感情分析システム |
JP7379907B2 (ja) * | 2019-07-30 | 2023-11-15 | 株式会社リコー | 情報処理装置、情報処理プログラム、情報処理システム、情報処理方法 |
-
2021
- 2021-03-24 WO PCT/JP2021/012360 patent/WO2022201383A1/ja active Application Filing
- 2021-03-24 US US18/552,384 patent/US11935329B2/en active Active
- 2021-03-24 JP JP2021572316A patent/JP7152817B1/ja active Active
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2015186127A (ja) * | 2014-03-25 | 2015-10-22 | ブラザー工業株式会社 | プログラム及びサーバ装置 |
JP2016149063A (ja) * | 2015-02-13 | 2016-08-18 | オムロン株式会社 | 感情推定装置及び感情推定方法 |
JP2019148852A (ja) * | 2018-02-26 | 2019-09-05 | 京セラドキュメントソリューションズ株式会社 | 理解度判定システムおよび理解度判定プログラム |
JP2021027917A (ja) * | 2019-08-09 | 2021-02-25 | 美津濃株式会社 | 情報処理装置、情報処理システム、および機械学習装置 |
Also Published As
Publication number | Publication date |
---|---|
US11935329B2 (en) | 2024-03-19 |
JP7152817B1 (ja) | 2022-10-13 |
US20240062586A1 (en) | 2024-02-22 |
JPWO2022201383A1 (ja) | 2022-09-29 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2022230156A1 (ja) | 動画像分析システム | |
WO2022201383A1 (ja) | 動画像分析プログラム | |
JP7152819B1 (ja) | 動画像分析プログラム | |
JP7121433B1 (ja) | 動画像分析プログラム | |
WO2022201269A1 (ja) | 動画像分析プログラム | |
WO2022230050A1 (ja) | 動画像分析システム | |
WO2022201265A1 (ja) | 動画像分析プログラム | |
WO2022230065A1 (ja) | 動画像分析システム | |
JP7121436B1 (ja) | 動画像分析プログラム | |
WO2022254489A1 (ja) | 動画像分析システム | |
WO2022230070A1 (ja) | 動画像分析システム | |
WO2022201274A1 (ja) | 動画像分析プログラム | |
WO2022201275A1 (ja) | 動画像分析プログラム | |
WO2022201268A1 (ja) | 動画像分析プログラム | |
WO2022230049A1 (ja) | 動画像分析システム | |
WO2022230137A1 (ja) | 動画像分析システム | |
WO2022201384A1 (ja) | 動画像分析プログラム | |
WO2022264217A1 (ja) | 動画像分析システム | |
WO2022201270A1 (ja) | 動画像分析プログラム | |
WO2022201264A1 (ja) | 動画像分析プログラム | |
WO2022254492A1 (ja) | 動画像分析システム | |
WO2022230051A1 (ja) | 動画像分析システム | |
WO2022201271A1 (ja) | 動画像分析プログラム | |
WO2022264218A1 (ja) | 動画像分析システム | |
WO2022254490A1 (ja) | 動画像分析システム |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
ENP | Entry into the national phase |
Ref document number: 2021572316 Country of ref document: JP Kind code of ref document: A |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21932993 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 18552384 Country of ref document: US |
|
WWE | Wipo information: entry into national phase |
Ref document number: 202327068365 Country of ref document: IN |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 21932993 Country of ref document: EP Kind code of ref document: A1 |