WO2022249461A1 - Système d'analyse vidéo - Google Patents

Système d'analyse vidéo Download PDF

Info

Publication number
WO2022249461A1
WO2022249461A1 PCT/JP2021/020464 JP2021020464W WO2022249461A1 WO 2022249461 A1 WO2022249461 A1 WO 2022249461A1 JP 2021020464 W JP2021020464 W JP 2021020464W WO 2022249461 A1 WO2022249461 A1 WO 2022249461A1
Authority
WO
WIPO (PCT)
Prior art keywords
information
analysis
unit
moving image
user
Prior art date
Application number
PCT/JP2021/020464
Other languages
English (en)
Japanese (ja)
Inventor
渉三 神谷
Original Assignee
株式会社I’mbesideyou
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 株式会社I’mbesideyou filed Critical 株式会社I’mbesideyou
Priority to PCT/JP2021/020464 priority Critical patent/WO2022249461A1/fr
Priority to JP2023523920A priority patent/JPWO2022249461A1/ja
Publication of WO2022249461A1 publication Critical patent/WO2022249461A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/14—Systems for two-way working
    • H04N7/15—Conference systems

Definitions

  • the present invention relates to a moving image analysis system that analyzes participants' biological reactions based on moving images obtained from online sessions conducted by multiple participants.
  • Patent Document 1 A technique for analyzing the emotions others receive in response to a speaker's remarks (see Patent Document 1, for example).
  • Patent Document 2 For example.
  • Patent Document 3 A technique for chronologically analyzing changes in a subject's facial expression over a long period of time and estimating the emotions held during that period.
  • Patent Documents 3 to 5 There are also known techniques for identifying the factors that most affected changes in emotions (see Patent Documents 3 to 5, for example).
  • Patent Documents 3 to 5 There is also known a technique that compares the subject's usual facial expression with the current facial expression and issues an alert when the facial expression is dark (see Patent Document 6, for example).
  • Patent Document 6 There is also known a technique for determining the degree of emotion of a subject by comparing the subject's normal (expressionless) facial expression with the current facial expression (see, for example, Patent Documents 7 to 9).
  • Techniques for analyzing the emotions of an organization and the atmosphere felt by individuals within a group are also known (see Patent Documents 10 and 11, for example).
  • the purpose of the present invention is to objectively evaluate these communications in order to conduct more efficient communication in situations where online communication is the main focus, such as meetings and lectures.
  • a moving image analysis system a moving image acquisition unit that acquires a moving image obtained by photographing the user during the online session for each of the plurality of users; an analysis unit that analyzes changes in biological reactions of the user based on the moving image acquired by the moving image acquisition unit; a character information specifying unit that specifies character information having an attribute corresponding to information related to the analysis result by the analysis unit; an output unit that outputs the identified character information;
  • a moving image analysis system is obtained.
  • exchanged communication can be objectively evaluated in order to conduct more efficient communication in situations where online communication is the main activity.
  • FIG. 1 is an example of a functional block diagram of an evaluation terminal according to an embodiment of the present invention
  • FIG. 3 is a diagram showing functional configuration example 1 of the evaluation terminal according to the embodiment of the present invention
  • FIG. 8 is a diagram showing functional configuration example 2 of the evaluation terminal according to the embodiment of the present invention
  • FIG. 10 is a diagram showing a functional configuration example 3 of the evaluation terminal according to the embodiment of the present invention
  • 7 is a screen display example according to the functional configuration example 3 of FIG. 6.
  • FIG. FIG. 7 is another screen display example according to the functional configuration example 3 of FIG. 6.
  • FIG. FIG. 12 is a diagram showing another configuration of functional configuration example 3 of the evaluation terminal according to the embodiment of the present invention
  • FIG. 12 is a diagram showing another configuration of functional configuration example 3 of the evaluation terminal according to the embodiment of the present invention. It is a figure showing an example of composition of a system concerning this embodiment. It is a figure showing an example of functional composition of a system concerning this embodiment.
  • FIG. 10 is a diagram showing an example of a list of analysis result data to which character information is added; It is a figure which shows an example of the data output by an output part.
  • FIG. 5 is a diagram showing an example of a display mode of a screen displayed on the evaluator terminal by the output unit according to the present embodiment; 4 is a flow chart showing an example of the flow of processing by the system according to the present embodiment;
  • the contents of the embodiments of the present disclosure are listed and described.
  • the present disclosure has the following configurations. (Item 1) In an environment where an online session is held by a plurality of users, the reaction of the user is analyzed based on a moving image obtained by photographing the user regardless of whether or not the user is displayed on a screen during the online session.
  • a moving image analysis system a moving image acquisition unit that acquires a moving image obtained by photographing the user during the online session for each of the plurality of users; an analysis unit that analyzes changes in biological reactions of the user based on the moving image acquired by the moving image acquisition unit; a character information specifying unit that specifies character information having an attribute corresponding to information related to the analysis result by the analysis unit; an output unit that outputs the identified character information;
  • a moving image analysis system (Item 2) The moving image analysis system according to item 1, The character information includes character object information, The output unit outputs object information of the character. Video image analysis system.
  • the moving image analysis system according to item 2 The output unit changes the output mode of the object information of the character according to the information on the change in the biological reaction analyzed by the analysis unit.
  • Video image analysis system. (Item 4) The moving image analysis system according to any one of items 1 to 3, The character information identifying unit identifies the character information based on attributes of the user who is the target of the analysis.
  • Video image analysis system. (Item 5) The moving image analysis system according to any one of items 1 to 4, The output unit does not output information of the user who is the target of the analysis. Video image analysis system.
  • the moving image analysis system outputs the character information together with information related to the analysis result to a terminal of another user different from the user who is the target of the analysis, further comprising a feedback information acquisition unit that acquires feedback information on the information related to the analysis result input to the terminal of the other user who acquired the character information, The moving image analysis system, wherein the output unit outputs a notification based on the feedback information acquired by the feedback information acquisition unit to the terminal of the user linked to the character information.
  • a video session in an environment where a video session (hereinafter referred to as an online session including one-way and two-way sessions) is held by a plurality of people, the person to be analyzed among the plurality of people is different from the others. It is a system that analyzes and evaluates specific emotions (feelings that occur in response to one's own or others' words and actions. pleasant/unpleasant, or their degree).
  • Online sessions are, for example, online meetings, online classes, online chats, etc. Terminals installed in multiple locations are connected to a server via a communication network such as the Internet, and moving images are transmitted between multiple terminals through the server. It's made to be interactable.
  • Moving images handled in online sessions include facial images and voices of users using terminals.
  • Moving images also include images such as materials that are shared and viewed by a plurality of users. It is possible to switch between the face image and the document image on the screen of each terminal to display only one of them, or to divide the display area and display the face image and the document image at the same time. In addition, it is possible to display the image of one user out of a plurality of users on the full screen, or divide the images of some or all of the users into small screens and display them. It is possible to designate one or a plurality of users among a plurality of users participating in an online session using terminals as analysis subjects.
  • an online session leader, moderator, or manager designates any user as an analysis subject.
  • Hosts of online sessions are, for example, instructors of online classes, chairpersons and facilitators of online meetings, coaches of sessions for coaching purposes, and the like.
  • An online session host is typically one of the users participating in the online session, but may be another person who does not participate in the online session. It should be noted that all participants may be subject to analysis without specifying the person to be analyzed.
  • an online session leader, moderator, or administrator hereinafter collectively referred to as the organizer to designate any user as an analysis subject.
  • Hosts of online sessions are, for example, instructors of online classes, chairpersons and facilitators of online meetings, coaches of sessions for coaching purposes, and the like.
  • An online session host is typically one of the users participating in the online session, but may be another person who does not participate in the online session.
  • the video session evaluation system displays at least moving images obtained from a video session established between a plurality of terminals.
  • the displayed moving image is acquired by the terminal, and at least a face image included in the moving image is identified for each predetermined frame unit. An evaluation value for the identified face image is then calculated.
  • the evaluation value is shared as necessary.
  • the acquired moving image is stored in the terminal, analyzed and evaluated on the terminal, and the result is provided to the user of the terminal. Therefore, for example, even a video session containing personal information or a video session containing confidential information can be analyzed and evaluated without providing the moving image itself to an external evaluation agency or the like.
  • the evaluation result evaluation value
  • the video session evaluation system includes user terminals 10 and 20 each having at least an input unit such as a camera unit and a microphone unit, a display unit such as a display, and an output unit such as a speaker. , a video session service terminal 30 for providing an interactive video session to the user terminals 10, 20, and an evaluation terminal 40 for performing part of the evaluation of the video session.
  • Each functional block, functional unit, and functional module described below can be configured by any of hardware, DSP (Digital Signal Processor), and software provided in a computer, for example.
  • DSP Digital Signal Processor
  • a computer CPU random access memory
  • RAM random access memory
  • ROM read-only memory
  • a series of processes by the systems and terminals described herein may be implemented using software, hardware, or a combination of software and hardware. It is possible to create a computer program for realizing each function of the information sharing support device 10 according to the present embodiment and implement it in a PC or the like. It is also possible to provide a computer-readable recording medium storing such a computer program.
  • the recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like.
  • the above computer program may be distributed, for example, via a network without using a recording medium.
  • the evaluation terminal acquires a moving image from a video session service terminal, identifies at least a face image included in the moving image for each predetermined frame unit, and calculates an evaluation value for the face image ( will be described in detail later).
  • the video session service provided by the video session service terminal (hereinafter sometimes simply referred to as "this service") provides user terminals 10 and 20 with two-way images and voice. Communication is possible.
  • this service a moving image captured by the camera of the other user's terminal is displayed on the display of the user's terminal, and audio captured by the microphone of the other's user's terminal can be output from the speaker.
  • this service allows both or either of the user terminals to record moving images and sounds (collectively referred to as "moving images, etc.") in the storage unit of at least one of the user terminals. configured as possible.
  • the recorded moving image information Vs (hereinafter referred to as “recorded information”) is cached in the user terminal that started recording and is locally recorded only in one of the user terminals. If necessary, the user can view the recorded information by himself or share it with others within the scope of using this service.
  • FIG. 4 is a block diagram showing a configuration example according to this embodiment.
  • the video session evaluation system of this embodiment is implemented as a functional configuration of the user terminal 10.
  • the user terminal 10 has, as its functions, a moving image acquisition unit 11, a biological reaction analysis unit 12, a peculiar determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification unit 16.
  • the moving image acquisition unit 11 acquires from each terminal a moving image obtained by photographing a plurality of people (a plurality of users) with a camera provided in each terminal during an online session. It does not matter whether the moving image acquired from each terminal is set to be displayed on the screen of each terminal. That is, the moving image acquisition unit 11 acquires moving images from each terminal, including moving images being displayed and moving images not being displayed on each terminal.
  • the biological reaction analysis unit 12 analyzes changes in the biological reaction of each of a plurality of people based on the moving images (whether or not they are being displayed on the screen) acquired by the moving image acquiring unit 11.
  • the biological reaction analysis unit 12 separates the moving image acquired by the moving image acquisition unit 11 into a set of images (collection of frame images) and voice, and analyzes changes in the biological reaction from each.
  • the biological reaction analysis unit 12 analyzes the user's facial image using a frame image separated from the moving image acquired by the moving image acquisition unit 11 to obtain at least one of facial expression, gaze, pulse, and facial movement. Analyze changes in biological reactions related to Further, the biological reaction analysis unit 12 analyzes the voice separated from the moving image acquired by the moving image acquisition unit 11 to analyze changes in the biological reaction related to at least one of the user's utterance content and voice quality.
  • the biological reaction analysis unit 12 calculates a biological reaction index value reflecting the change in biological reaction by quantifying the change in biological reaction according to a predetermined standard.
  • the analysis of changes in facial expressions is performed as follows. That is, for each frame image, a facial region is identified from the frame image, and the identified facial expressions are classified into a plurality of types according to an image analysis model machine-learned in advance. Then, based on the classification results, it analyzes whether positive facial expression changes occur between consecutive frame images, whether negative facial expression changes occur, and to what extent the facial expression changes occur, A facial expression change index value corresponding to the analysis result is output.
  • the analysis of changes in line of sight is performed as follows. That is, for each frame image, the eye region is specified in the frame image, and the orientation of both eyes is analyzed to analyze where the user is looking. For example, it analyzes whether the user is looking at the face of the speaker being displayed, whether the user is looking at the shared material being displayed, or whether the user is looking outside the screen. Also, it may be analyzed whether the eye movement is large or small, or whether the movement is frequent or infrequent. A change in line of sight is also related to the user's degree of concentration.
  • the biological reaction analysis unit 12 outputs a line-of-sight change index value according to the analysis result of the line-of-sight change.
  • the analysis of pulse changes is performed, for example, as follows. That is, for each frame image, the face area is specified in the frame image. Then, using a trained image analysis model that captures numerical values of face color information (G of RGB), changes in the G color of the face surface are analyzed. By arranging the results along the time axis, a waveform representing changes in color information is formed, and the pulse is identified from this waveform. When a person is tense, the pulse speeds up, and when the person is calm, the pulse slows down. The biological reaction analysis unit 12 outputs a pulse change index value according to the analysis result of the pulse change.
  • G of RGB face color information
  • analysis of changes in facial movement is performed as follows. That is, for each frame image, the face area is specified in the frame image, and the direction of the face is analyzed to analyze where the user is looking. For example, it analyzes whether the user is looking at the face of the speaker being displayed, whether the user is looking at the shared material being displayed, or whether the user is looking outside the screen. Further, it may be analyzed whether the movement of the face is large or small, or whether the movement is frequent or infrequent. The movement of the face and the movement of the line of sight may be analyzed together. For example, it may be analyzed whether the face of the speaker being displayed is viewed straight, whether the face is viewed with upward or downward gaze, or whether the face is viewed obliquely.
  • the biological reaction analysis unit 12 outputs a face orientation change index value according to the analysis result of the face orientation change.
  • the biological reaction analysis unit 12 converts the voice into a character string by performing known voice recognition processing on the voice for a specified time (for example, about 30 to 150 seconds), and morphologically analyzes the character string. By doing so, words such as particles and articles that are unnecessary for expressing conversation are removed. Then, vectorize the remaining words, analyze whether a positive emotional change has occurred, whether a negative emotional change has occurred, and to what extent the emotional change has occurred. Outputs the utterance content index value.
  • Voice quality analysis is performed, for example, as follows. That is, the biological reaction analysis unit 12 identifies the acoustic features of the voice by performing known voice analysis processing on the voice for a specified time (for example, about 30 to 150 seconds). Then, based on the acoustic features, it analyzes whether a positive change in voice quality has occurred, whether a negative change in voice quality has occurred, and to what extent the change in voice quality has occurred, and according to the analysis results, output the voice quality change index value.
  • a specified time for example, about 30 to 150 seconds
  • the biological reaction analysis unit 12 uses at least one of the facial expression change index value, eye line change index value, pulse change index value, face direction change index value, statement content index value, and voice quality change index value calculated as described above. to calculate the biological reaction index value.
  • the biological reaction index value is calculated by weighting the facial expression change index value, eye line change index value, pulse change index value, face direction change index value, statement content index value, and voice quality change index value.
  • the peculiarity determination unit 13 determines whether or not the change in the analyzed biological reaction of the person to be analyzed is more specific than the change in the analyzed biological reaction of the person other than the person to be analyzed. In the present embodiment, the peculiarity determination unit 13 compares changes in the biological reaction of the person to be analyzed with those of others based on the biological reaction index values calculated for each of the plurality of users by the biological reaction analysis unit 12. is specific or not.
  • the peculiar determination unit 13 calculates the variance of the biological reaction index values calculated for each of the plurality of persons by the biological reaction analysis unit 12, and compares the biological reaction index values calculated for the analysis subject with the variance, It is determined whether or not the change in the analyzed biological reaction of the person to be analyzed is specific compared to others.
  • the following three patterns are conceivable as cases where the changes in biological reactions analyzed for the subject of analysis are more specific than those of others.
  • the first is a case where a relatively large change in biological reaction occurs in the subject of analysis, although no particularly large change in biological reaction has occurred in the other person.
  • the second is a case where a particularly large change in biological reaction has not occurred in the subject of analysis, but a relatively large change in biological reaction has occurred in the other person.
  • the third is a case where a relatively large change in biological reaction occurs in both the subject of analysis and the other person, but the content of the change differs between the subject of analysis and the other person.
  • the related event identification unit 14 identifies an event occurring in relation to at least one of the person to be analyzed, the other person, and the environment when the change in the biological reaction determined to be peculiar by the peculiarity determination unit 13 occurs. .
  • the related event identification unit 14 identifies from the moving image the speech and behavior of the person to be analyzed when a specific change in biological reaction occurs in the person to be analyzed.
  • the related event identifying unit 14 identifies, from the moving image, the speech and behavior of the other person when a specific change in the biological reaction of the person to be analyzed occurs.
  • the related event identification unit 14 identifies from the moving image the environment in which a specific change in the biological reaction of the person to be analyzed occurs.
  • the environment is, for example, the shared material being displayed on the screen, the background image of the person to be analyzed, and the like.
  • the clustering unit 15 clusters the change in the biological reaction determined to be specific by the peculiarity determination unit 13 (for example, one or a combination of eye gaze, pulse, facial movement, statement content, and voice quality), and the peculiarity Analyzing the degree of correlation with an event (event identified by the related event identification unit 14) that occurs when a change in biological reaction occurs, and if it is determined that the correlation is at a certain level or more , to cluster the subjects or events based on the correlation analysis results.
  • the peculiarity determination unit 13 for example, one or a combination of eye gaze, pulse, facial movement, statement content, and voice quality
  • the clustering unit 15 clusters the person to be analyzed or the event into one of a plurality of pre-segmented categories according to the content of the event, the degree of negativity, the magnitude of the correlation, and the like.
  • the clustering unit 15 clusters the person to be analyzed or the event into one of a plurality of pre-segmented classifications according to the content of the event, the degree of positivity, the degree of correlation, and the like.
  • the analysis result notification unit 16 reports at least one of the changes in the biological reaction determined to be specific by the peculiar determination unit 13, the event identified by the related event identification unit 14, and the classification clustered by the clustering unit 15. , to notify the designator of the subject of analysis (the subject of analysis or the organizer of the online session).
  • the analysis result notification unit 16 recognizes that when a change in a specific biological reaction that is different from that of the other person occurs in the person to be analyzed (one of the three patterns described above; the same applies hereinafter), the analysis target is Notifies the person to be analyzed of his/her own behavior. This allows the person to be analyzed to understand that he/she has a different feeling from others when he or she performs a certain behavior. At this time, the person to be analyzed may also be notified of the change in the specific biological reaction identified for the person to be analyzed. Furthermore, the person to be analyzed may be further notified of the change in the biological reaction of the other person to be compared.
  • the words and deeds of the person to be analyzed performed without being particularly conscious of their usual emotions, or the words and deeds of the person to be analyzed consciously accompanied by certain emotions, and the emotions and behaviors that others received
  • the emotion held by the person to be analyzed is different from the feeling held by the person to be analyzed at the time
  • the person to be analyzed is notified of the speech and behavior of the person to be analyzed at that time.
  • the analysis result notification unit 16 notifies the organizer of the online session of the event occurring when the person to be analyzed undergoes a specific change in biological reaction that is different from that of the other person, together with the change in the specific biological reaction. to notify.
  • the organizer of the online session can know what kind of event affects what kind of emotional change as a phenomenon specific to the specified analysis subject. Then, it becomes possible to perform appropriate treatment on the person to be analyzed according to the grasped contents.
  • the analysis result notification unit 16 notifies the organizer of the online session of the event occurring when a specific change in biological reaction occurs in the analysis subject, which is different from that of others, or the clustering result of the analysis subject. do.
  • online session organizers can grasp behavioral tendencies peculiar to analysis subjects and predict possible future behaviors and situations, depending on which classification the specified analysis subjects have been clustered into. be able to. Then, it becomes possible to take appropriate measures for the person to be analyzed.
  • the biological reaction index value is calculated by quantifying the change in biological reaction according to a predetermined standard, and the analysis subject is analyzed based on the biological reaction index value calculated for each of the plurality of people.
  • the biological reaction analysis unit 12 analyzes the movement of the line of sight for each of a plurality of people and generates a heat map indicating the direction of the line of sight.
  • the peculiar determination unit 13 compares the heat map generated for the person to be analyzed by the biological reaction analysis unit 12 with the heat map generated for the other person, so that the change in the biological reaction analyzed for the person to be analyzed It is determined whether it is specific compared with the change in biological response analyzed for.
  • moving images of a video session are stored in the local storage of the user terminal 10, and the above analysis is performed on the user terminal 10.
  • the machine specs of the user terminal 10 it is possible to analyze the moving image information without providing it to the outside.
  • the video session evaluation system of this embodiment may include a moving image acquisition unit 11, a biological reaction analysis unit 12, and a reaction information presentation unit 13a as functional configurations.
  • the reaction information presentation unit 13a presents information indicating changes in biological reactions analyzed by the biological reaction analysis unit 12a, including participants not displayed on the screen.
  • the reaction information presenting unit 13a presents information indicating changes in biological reactions to an online session leader, moderator, or administrator (hereinafter collectively referred to as the organizer).
  • Hosts of online sessions are, for example, instructors of online classes, chairpersons and facilitators of online meetings, coaches of sessions for coaching purposes, and the like.
  • An online session host is typically one of the users participating in the online session, but may be another person who does not participate in the online session.
  • the organizer of the online session can also grasp the state of the participants who are not displayed on the screen in an environment where the online session is held by multiple people.
  • FIG. 6 is a block diagram showing a configuration example according to this embodiment. As shown in FIG. 6, in the video session evaluation system of the present embodiment, functions similar to those of the above-described first embodiment are given the same reference numerals, and explanations thereof may be omitted.
  • the system includes a camera unit that acquires images of a video session, a microphone unit that acquires audio, an analysis unit that analyzes and evaluates moving images, and information obtained by evaluating the acquired moving images.
  • an object generator for generating a display object (described below) based on the display; and a display for displaying both the moving image of the video session and the display object during execution of the video session.
  • the analysis unit includes the moving image acquisition unit 11, the biological reaction analysis unit 12, the peculiar determination unit 13, the related event identification unit 14, the clustering unit 15, and the analysis result notification unit 16, as described above.
  • the function of each element is as described above.
  • the object generation unit generates an object 50 representing the recognized face part and the above-mentioned Information 100 indicating the content of the analysis/evaluation performed is superimposed on the moving image and displayed.
  • the object 50 may identify and display all faces of a plurality of persons when the faces of the plurality of persons are moved in the moving image.
  • the object 50 is, for example, when the camera function of the video session is stopped at the other party's terminal (that is, it is stopped by software within the application of the video session instead of physically covering the camera). If the other party's face is recognized by the other party's camera, the object 50 or the object 100 may be displayed in the part where the other party's face is located. This makes it possible for both parties to confirm that the other party is in front of the terminal even if the camera function is turned off. In this case, for example, in a video session application, the information obtained from the camera may be hidden while only the object 50 or object 100 corresponding to the face recognized by the analysis unit is displayed. Also, the video information acquired from the video session and the information recognized by the analysis unit may be divided into different display layers, and the layer relating to the former information may be hidden.
  • the objects 50 and 100 may be displayed in all areas or only in some areas. For example, as shown in FIG. 8, it may be displayed only on the moving image on the guest side.
  • the embodiments of the invention described in Basic Configuration Example 1 to Basic Configuration Example 3 described above may be implemented as a single device, or a plurality of devices (for example, cloud servers) partially or entirely connected by a network. and the like.
  • the control unit 110 and the storage 130 of each terminal 10 may be realized by different servers connected to each other via a network. That is, the system includes user terminals 10, 20, a video session service terminal 30 for providing an interactive video session to the user terminals 10, 20, and an evaluation terminal 40 for evaluating the video session, Variation combinations of the following configurations are conceivable. (1) Processing everything only on the user terminal As shown in FIG. 8, by performing the processing by the analysis unit on the terminal that is performing the video session (although a certain processing capacity is required), the video session can be performed.
  • an analysis unit may be provided in an evaluation terminal connected via a network or the like.
  • the moving images acquired by the user terminal are shared with the evaluation terminal at the same time as or after the video session, and are analyzed and evaluated by the analysis unit in the evaluation terminal.
  • the moving image data that is, information including at least analysis data
  • a moving image analysis system (hereinafter simply referred to as "system") according to an embodiment of the present disclosure shoots all participants or only a specific participant in an environment where an online session is held with a plurality of participants. Participants' reactions are analyzed based on the moving images obtained by this process. The analysis may occur whether or not participants are on screen during the online session. For example, the system according to the present embodiment statistically analyzes and outputs contents such as the amount and frequency of communication between users and their feelings at that time by analyzing moving images.
  • Such analysis results are associated with information (user information) on the person subject to analysis. Therefore, for example, when a third party (e.g., the analysis subject's superior, the person being evaluated, etc.) views the analysis results of the analysis subject, information about the analysis subject's inner state is also disclosed to the third party. Therefore, privacy may not be sufficiently protected. On the other hand, if a third party cannot view the results of such analysis, it is difficult to grasp what kind of communication took place.
  • a third party e.g., the analysis subject's superior, the person being evaluated, etc.
  • FIG. 10 is a diagram showing an example of the system configuration according to this embodiment.
  • an evaluator terminal 50 is a terminal for the evaluator to view analysis results of reactions of at least one of the users 10 and 20 during online sessions with the users 10 and 20, for example.
  • FIG. 11 is a diagram showing an example of the functional configuration of the system according to this embodiment.
  • the system shown in FIG. 11 includes an analysis result DB 21, a character information specifying section 22, an output section 23, and a feedback information acquisition section .
  • the analysis result DB 21 can be realized by the above-described storage medium or the like.
  • the character information specifying unit 22, the output unit 23, and the feedback information acquisition unit 24 read a program stored in a storage medium or the like provided in the evaluation terminal 40 or the like into a memory or the like and are executed by a processor such as a CPU. It can be realized by These functional units are preferably provided in an information processing device separate from the evaluator terminal 50 . For example, these functional units may be provided in the evaluation terminal 40 . With such a configuration, as will be described later, it is possible to prevent the evaluator terminal 50 from accessing the user information.
  • the analysis result DB 21 is, for example, a database that stores analysis result data obtained by the various functional units described above.
  • the analysis result data includes a session number specifying an online session to be analyzed. and an analysis number specifying the analysis result. , a user ID (specific information) for specifying the person to be analyzed, an address of analysis data, and the like.
  • the address of analysis data is information that can indicate the storage location of analysis data stored in, for example, the analysis result DB 21 or another storage terminal.
  • the analysis result data may include other information related to the analysis.
  • a user ID is an example of user information.
  • the analysis result data includes analysis information obtained as a result of analysis of movement on a moving image caused by the user, input information generated by the user's input to the user terminal, etc. May be included as information.
  • the character information specifying unit 22 may have a function of specifying character information having attributes corresponding to information related to analysis results.
  • Character information means information about characters having predetermined personalities, dispositions, etc., such as animals, cartoons, and animations. Each character information has an attribute. Attributes can include, for example, personality, temperament, age, gender, race, and other characteristics unique to the character. Character information may also include character object information (for example, the character's face, whole body, objects symbolizing the character, etc.).
  • the character information specifying unit 22 determines that the user has a character with a quiet personality attribute. can be identified.
  • the characters to be specified are characters of a common type or work. This makes it easier for evaluators to better understand the context of the session, as the character's characteristics are more clearly defined from the context based on common types and works.
  • FIG. 12 is a diagram showing an example of analysis result data.
  • the analysis result data 1001 is session number. 1011, analysis no. 1012 , user ID 1013 , analytical data 1014 and character information 1015 .
  • Character information 1015 is added by the character information specifying unit 22 to the analysis result data stored in the analysis result DB 21 .
  • the output unit 23 may have a function of outputting the analysis result data to which the character information stored in the analysis result DB 21 is added to the evaluator terminal 50 .
  • the output unit 23 may exclude the user information from the analysis result data and output the data to the evaluator terminal 50 .
  • FIG. 13 is a diagram showing an example of analysis result data output by the output unit 23.
  • the output analysis result data 1002 includes session numbers. 1021, analysis no. 1022 , analysis data 1023 and character information 1024 . That is, the analysis result data 1002 does not include user information regarding the person to be analyzed.
  • the evaluator terminal 50 outputs the data of the analysis result, it does not have information about which person to be analyzed provided the analysis result. Therefore, the evaluator can prevent feedback due to bias without knowing information about the subject of analysis or based on such information.
  • the mode of output to the evaluator terminal 50 by the output unit 23 is not particularly limited.
  • the output unit 23 may control the display device of the evaluator terminal 50 to display the analysis results in the form of a dashboard or a list.
  • FIG. 14 is a diagram showing an example of an output mode to the evaluator terminal 50 by the output unit 23 according to this embodiment.
  • a character object 1101 is displayed for each user as information of the online session to be analyzed.
  • the displayed information about the user may be information that does not specify the person to be analyzed.
  • each character's object 1101 can be displayed as an avatar that shows what the participants are like in the session.
  • a lion, a mouse, a raccoon dog and a wolf can each be displayed.
  • the character object 1101 may be displayed instead of the moving image of each participant in the moving image review area included in the screen 1100 displayed on the evaluator terminal 50, for example. Further, such a character object 1101 may be displayed in association with a graph indicating analysis results included in the screen 1100, for example.
  • the output unit 23 may output only the character information instead of the analysis result data.
  • the output unit 23 may output character information corresponding to each participant in one session. This allows the evaluator to grasp the status of each participant in the session without displaying detailed analysis results.
  • the output mode of the object information of the displayed character may be changed according to the change in the biological reaction of the analysis result data.
  • the output unit 23 may change the expression, appearance, etc. of the character displayed by the object, or may change the tone of voice in the case of voice.
  • the output unit 23 may change the displayed character itself.
  • the output unit 23 may have a function of outputting a notification based on the feedback information acquired from the evaluator terminal 50 by the feedback information acquisition unit 24, which will be described later, to the user terminals 10, 20, etc. of the person to be analyzed.
  • the feedback information acquired by the feedback information acquisition unit 24 is associated with the analysis result data described later together with the user ID. Therefore, the output unit 23 can output a notification based on the feedback information to the person to be analyzed.
  • the notification here may be, for example, notification of the feedback information itself, or notification of information such as points to be improved or evaluation points based on the feedback information.
  • the feedback information acquisition unit 24 may have a function of acquiring feedback information on analysis result data input to the evaluator terminal 50 .
  • Such feedback information may include, for example, any information such as texts and annotations regarding feedback to the analysis subject regarding the analysis result data.
  • Such feedback information may be associated with the time in the moving image to be analyzed that may be included in the analysis data, the image displayed in the moving image, or the like.
  • notification may or may not include the information of the evaluator who gave the feedback.
  • the feedback information acquisition unit 24 may associate the acquired feedback information with the analysis result data stored in the analysis target DB 21 .
  • the feedback information acquisition unit 24 obtains the analysis number associated with the acquired feedback information. etc., the same analysis no. can be specified and feedback information can be added to the analysis result data.
  • the output unit 23 can output a notification based on the feedback information to the person to be analyzed corresponding to the user ID.
  • FIG. 15 is a flowchart showing an example of the flow of processing by the system according to this embodiment.
  • the character information specifying unit 22 specifies character information based on the analysis result data acquired from the analysis result DB 21, and assigns the character information to the analysis result data (step S101).
  • the analysis result data acquired from the analysis result DB 21 may be, for example, analysis result data for each user who participated in one or more sessions.
  • the output unit 23 outputs the analysis result data to which the character information is added to the evaluator terminal 50 (S103).
  • the evaluator terminal 50 displays the acquired character information (step S105). At this time, analysis data and the like may be displayed on the evaluator terminal 50 . Also, feedback information for such display may be input to the evaluator terminal 50 .
  • the feedback information acquisition unit 24 (of the evaluation terminal 40) may acquire the feedback information and add the feedback information to the corresponding analysis result data. . Then, the output unit 23 may output, to the user terminal 10 (20) of the person to be analyzed, feedback information included in the analysis result data associated with the user ID of the person to be analyzed.
  • the character information corresponding to the analysis result of the user is output to the evaluator terminal 50 .
  • the evaluator who receives this information can intuitively grasp the situation of the person to be analyzed in the session without specifying the person to be analyzed. This allows the evaluator to intuitively grasp the communication situation while protecting the privacy of the person to be analyzed more reliably. Therefore, it is possible to make a more objective evaluation of the content of the moving images of the video session.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)

Abstract

Le problème décrit par la présente invention concerne l'évaluation objective des communications en ligne, qui sont devenues courantes, afin d'effectuer une communication plus efficace. La solution selon l'invention porte sur un système qui comprend : une unité d'acquisition vidéo qui acquiert une vidéo obtenue en filmant chacun d'une pluralité d'utilisateurs pendant une session en ligne; une unité d'analyse qui analyse les changements dans les réactions biologiques des utilisateurs sur la base de la vidéo acquise par l'unité d'acquisition vidéo; une unité de spécification d'informations de caractères qui spécifie des informations de caractères ayant des attributs correspondant aux informations liées aux résultats d'analyse de l'unité d'analyse; et une unité de sortie qui sort les informations de caractères spécifiées.
PCT/JP2021/020464 2021-05-28 2021-05-28 Système d'analyse vidéo WO2022249461A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/JP2021/020464 WO2022249461A1 (fr) 2021-05-28 2021-05-28 Système d'analyse vidéo
JP2023523920A JPWO2022249461A1 (fr) 2021-05-28 2021-05-28

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2021/020464 WO2022249461A1 (fr) 2021-05-28 2021-05-28 Système d'analyse vidéo

Publications (1)

Publication Number Publication Date
WO2022249461A1 true WO2022249461A1 (fr) 2022-12-01

Family

ID=84228521

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/020464 WO2022249461A1 (fr) 2021-05-28 2021-05-28 Système d'analyse vidéo

Country Status (2)

Country Link
JP (1) JPWO2022249461A1 (fr)
WO (1) WO2022249461A1 (fr)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003016475A (ja) * 2001-07-04 2003-01-17 Oki Electric Ind Co Ltd 画像コミュニケーション機能付き情報端末装置および画像配信システム
JP2009543611A (ja) * 2006-07-12 2009-12-10 メディカル サイバーワールド、インコーポレイテッド コンピュータ化医療訓練システム
JP2015186127A (ja) * 2014-03-25 2015-10-22 ブラザー工業株式会社 プログラム及びサーバ装置
JP2018505462A (ja) * 2014-12-11 2018-02-22 インテル コーポレイション アバター選択機構

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003016475A (ja) * 2001-07-04 2003-01-17 Oki Electric Ind Co Ltd 画像コミュニケーション機能付き情報端末装置および画像配信システム
JP2009543611A (ja) * 2006-07-12 2009-12-10 メディカル サイバーワールド、インコーポレイテッド コンピュータ化医療訓練システム
JP2015186127A (ja) * 2014-03-25 2015-10-22 ブラザー工業株式会社 プログラム及びサーバ装置
JP2018505462A (ja) * 2014-12-11 2018-02-22 インテル コーポレイション アバター選択機構

Also Published As

Publication number Publication date
JPWO2022249461A1 (fr) 2022-12-01

Similar Documents

Publication Publication Date Title
WO2022230156A1 (fr) Système d'analyse vidéo
WO2022249461A1 (fr) Système d'analyse vidéo
WO2022249460A1 (fr) Système d'analyse vidéo
JP7152819B1 (ja) 動画像分析プログラム
JP7197955B1 (ja) ビデオミーティング評価端末
JP7121436B1 (ja) 動画像分析プログラム
JP7121433B1 (ja) 動画像分析プログラム
JP7152817B1 (ja) 動画像分析プログラム
WO2022269802A1 (fr) Système d'analyse vidéo
WO2022254489A1 (fr) Système d'analyse vidéo
WO2022254494A1 (fr) Système d'analyse vidéo
WO2022249462A1 (fr) Système d'analyse vidéo
WO2022254495A1 (fr) Système d'analyse vidéo
JP7121439B1 (ja) 動画像分析システム
WO2022230070A1 (fr) Système d'analyse vidéo
WO2022230155A1 (fr) Système d'analyse vidéo
WO2022230049A1 (fr) Système d'analyse vidéo
WO2022201269A1 (fr) Programme d'analyse vidéo
WO2022264221A1 (fr) Système d'analyse vidéo
WO2022269801A1 (fr) Système d'analyse vidéo
WO2022201265A1 (fr) Programme d'analyse vidéo
WO2022230138A1 (fr) Système d'analyse vidéo
WO2022230065A1 (fr) Système d'analyse vidéo
WO2022230050A1 (fr) Système d'analyse vidéo
WO2022269800A1 (fr) Système d'analyse vidéo

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21943105

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023523920

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21943105

Country of ref document: EP

Kind code of ref document: A1