WO2022024354A1

WO2022024354A1 - Reaction analysis system

Info

Publication number: WO2022024354A1
Application number: PCT/JP2020/029467
Authority: WO
Inventors: 渉三神谷
Original assignee: 株式会社I’mbesideyou
Priority date: 2020-07-31
Filing date: 2020-07-31
Publication date: 2022-02-03
Also published as: JP7242114B2; WO2022025200A1; JP2023075197A; JPWO2022025200A1

Abstract

The present invention comprises: a moving image acquisition unit 11 that acquires a moving image obtained by photographing a subject to be analyzed during an online session, in each of a plurality of online sessions; a biological reaction analysis unit 12 that analyzes a change in the biological reaction of the subject to be analyzed on the basis of the moving image acquired by the moving image acquisition unit 11; and an anomaly assessment unit 13 that assesses whether an analyzed biological reaction change in the subject to be analyzed during one online session is anomalous in comparison with an analyzed biological reaction change in the subject to be analyzed during an online session earlier in time than the one online session, wherein an anomalous reaction in the subject to be analyzed different from previous reactions is analyzed on the basis of the moving image obtained by photographing the subject to be analyzed in each of the plurality of online sessions in an environment in which the online sessions are conducted with a plurality of persons.

Description

Reaction analysis system

The present invention relates to a reaction analysis system.

Conventionally, a technique for analyzing emotions received by others in response to a speaker's remark is known (see, for example, Patent Document 1). Further, there is also known a technique of analyzing changes in a subject's facial expression over a long period of time in a time series and estimating the emotions held during that period (see, for example, Patent Document 2). Further, a technique for identifying an element that has the greatest influence on emotional changes is also known (see, for example, Patent Documents 3 to 5). Furthermore, there is also known a technique of comparing a subject's normal facial expression with the current facial expression and issuing an alert when the facial expression is dark (see, for example, Patent Document 6).

Japanese Unexamined Patent Publication No. 2019-58625 Japanese Unexamined Patent Publication No. 2016-149063 Japanese Unexamined Patent Publication No. 2020-86559 Japanese Unexamined Patent Publication No. 2000-76421 Japanese Unexamined Patent Publication No. 2017-201499 Japanese Unexamined Patent Publication No. 2018-112831

An object of the present invention is to analyze a specific reaction different from the previous one for a participant in an environment where an online session is held by a plurality of people, and to make it possible to utilize the analysis result.

In order to solve the above-mentioned problems, the reaction analysis system of the present invention is based on a moving image obtained by photographing a participant for each of a plurality of online sessions in an environment where an online session is held by a plurality of people. , Analyze different specific reactions for participants.

According to the present invention configured as described above, in an environment where an online session is held by a plurality of people, it is possible to analyze a specific reaction different from the previous one for the participants and utilize the analysis result.

It is a block diagram which shows the functional structure example of the reaction analysis system by this embodiment.

The reaction analysis system of the present embodiment is different from the previous one for the participants based on the moving images obtained by photographing the participants for each of a plurality of online sessions in an environment where an online session is held by a plurality of people. It is a system that analyzes specific reactions. In this embodiment, the biological reaction of the participants is analyzed. In addition, the emotions of the participants (feelings that occur in the words and actions of oneself or others, such as comfort / discomfort or the degree thereof) may be analyzed through the analysis of biological reactions.

An online session is, for example, an online conference, an online class, an online chat, etc., in which terminals installed in multiple locations are connected to a server via a communication network such as the Internet, and moving images are transmitted between the terminals through the server. It is designed to be able to communicate. It is preferable that the target online session in the present embodiment is performed multiple times with the same participant or almost the same participant.

The moving image handled in the online session includes the face image and voice of the user who uses the terminal. In addition, the moving image also includes an image such as a material shared and viewed by a plurality of users. It is possible to switch between the face image and the material image on the screen of each terminal to display only one of them, or to divide the display area and display the face image and the material image at the same time. Further, it is possible to display the image of one of a plurality of people on the full screen, or to display the image of a part or all of the users on a small screen.

It is possible to specify any one or more of the multiple users who participate in the online session using the terminal as the analysis target. For example, the leader, facilitator, or administrator of the online session (hereinafter collectively referred to as the organizer) designates any user as the analysis target. Organizers of online sessions include, for example, instructors of online classes, chairs and facilitators of online conferences, and coaches of sessions for coaching purposes. The organizer of an online session is usually one of a plurality of users who participate in the online session, but may be another person who does not participate in the online session. In addition, all the participants may be the analysis target without designating the analysis target person.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a functional configuration example of the reaction analysis system according to the present embodiment. As shown in FIG. 1, the reaction analysis system of the present embodiment has a moving image acquisition unit 11, a biological reaction analysis unit 12, a peculiarity determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification as functional configurations. The part 16 is provided.

Each of the above functional blocks 11 to 16 can be configured by any of hardware, DSP (Digital Signal Processor), and software provided in the server device, for example. For example, when configured by software, each of the above functional blocks 11 to 16 is actually configured to include a computer CPU, RAM, ROM, etc., and is a program stored in a recording medium such as RAM, ROM, a hard disk, or a semiconductor memory. Is realized by the operation of.

The moving image acquisition unit 11 acquires a moving image obtained by shooting a plurality of people (multiple users) with a camera provided in each terminal during the online session for each of a plurality of online sessions. It does not matter whether the moving image acquired from each terminal is set to be displayed on the screen of each terminal. That is, the moving image acquisition unit 11 acquires the moving image from each terminal, including the moving image being displayed on each terminal and the moving image being hidden.

The biological reaction analysis unit 12 analyzes changes in the biological reaction of the designated analysis target person based on the moving image acquired by the moving image acquisition unit 11. It should be noted that the change in the biological reaction may be analyzed for each of a plurality of persons including others other than the analysis target person. In the present embodiment, the biological reaction analysis unit 12 separates the moving image acquired by the moving image acquisition unit 11 into a set of images (a collection of frame images) and a voice, and analyzes changes in the biological reaction from each.

For example, the biological reaction analysis unit 12 analyzes the user's face image using the frame image separated from the moving image acquired by the moving image acquisition unit 11, and thereby at least one of the facial expression, the line of sight, the pulse, and the movement of the face. Analyze changes in biological reactions related to one. Further, the biological reaction analysis unit 12 analyzes changes in the biological reaction regarding at least one of the user's speech content and voice quality by analyzing the voice separated from the moving image acquired by the moving image acquisition unit 11.

When a person's emotions change, it appears as changes in biological reactions such as facial expressions, eyes, pulse, facial movements, speech content, and voice quality. In this embodiment, changes in biological reactions caused by changes in the user's emotions are analyzed. Further, the change in the user's emotion may be analyzed by analyzing the change in the biological reaction. The emotion analyzed in this case is, for example, the degree of comfort / discomfort. In the present embodiment, the biological reaction analysis unit 12 calculates a biological reaction index value that reflects the content of the change in the biological reaction by quantifying the change in the biological reaction according to a predetermined standard.

Analysis of changes in facial expressions is performed, for example, as follows. That is, for each frame image, a facial area is specified from the frame image, and the specified facial expressions are classified into a plurality of types according to an image analysis model trained in advance by machine learning. Then, based on the classification result, it is analyzed whether a positive facial expression change occurs between consecutive frame images, a negative facial expression change occurs, and how large the facial expression change occurs. The facial expression change index value according to the analysis result is output.

Analysis of changes in the line of sight is performed, for example, as follows. That is, for each frame image, the area of the eyes is specified from the frame image, and the orientation of both eyes is analyzed to analyze where the user is looking. For example, it analyzes whether the speaker's face being displayed, the shared material being displayed, or the outside of the screen is being viewed. In addition, it may be possible to analyze whether the movement of the line of sight is large or small, and whether the movement is frequent or infrequent. The change in the line of sight is also related to the degree of concentration of the user. The biological reaction analysis unit 12 outputs the line-of-sight change index value according to the analysis result of the line-of-sight change.

Analysis of pulse changes is performed, for example, as follows. That is, for each frame image, the face area is specified from the frame image. Then, using a trained image analysis model that captures the numerical value of the face color information (G in RGB), the change in the G color on the face surface is analyzed. By arranging the results along the time axis, a waveform showing the change in color information is formed, and the pulse is specified from this waveform. When a person is nervous, the pulse becomes faster, and when he / she feels calm, the pulse becomes slower. The biological reaction analysis unit 12 outputs a pulse change index value according to the analysis result of the pulse change.

Analysis of changes in facial movement is performed, for example, as follows. That is, for each frame image, the area of the face is specified from the frame image, and the orientation of the face is analyzed to analyze where the user is looking. For example, it analyzes whether the speaker's face being displayed, the shared material being displayed, or the outside of the screen is being viewed. In addition, it may be possible to analyze whether the movement of the face is large or small, and whether the movement is frequent or infrequent. The movement of the face and the movement of the line of sight may be combined and analyzed. For example, it may be possible to analyze whether the speaker's face being displayed is viewed straight, whether the speaker is viewed with an upper eye or a lower eye, or whether the speaker is viewed from an angle. The biological reaction analysis unit 12 outputs a face orientation change index value according to the analysis result of the face orientation change.

The content of the statement is analyzed as follows, for example. That is, the biological reaction analysis unit 12 converts the voice into a character string by performing a known voice recognition process on the voice for a specified time (for example, a time of about 30 to 150 seconds), and morphologically analyzes the character string. By doing so, words unnecessary for expressing conversation such as particles and acronyms are removed. Then, the remaining words are vectorized, and whether a positive emotional change is occurring, a negative emotional change is occurring, and how large the emotional change is occurring is analyzed, and the analysis result is used. Outputs the statement content index value.

Voice quality analysis is performed as follows, for example. That is, the biological reaction analysis unit 12 identifies the acoustic characteristics of the voice by performing a known voice analysis process on the voice for a specified time (for example, a time of about 30 to 150 seconds). Then, based on the acoustic characteristics, it is analyzed whether a positive voice quality change is occurring, a negative voice quality change is occurring, and how loud the voice quality change is occurring, and according to the analysis result. Outputs the voice quality change index value.

The biological reaction analysis unit 12 uses at least one of the facial expression change index value, the line-of-sight change index value, the pulse change index value, the face orientation change index value, the speech content index value, and the voice quality change index value calculated as described above. The biological reaction index value is calculated. For example, the biological reaction index value is calculated by weighting the facial expression change index value, the line-of-sight change index value, the pulse change index value, the face orientation change index value, the speech content index value, and the voice quality change index value.

In the peculiarity determination unit 13, the change in the biological reaction analyzed for the analysis target person in one online session is the change in the biological reaction analyzed for the analysis target person in the online session prior to the one online session. Judge whether it is specific or not by comparison. In the present embodiment, the peculiarity determination unit 13 is based on the biological reaction index value calculated for the analysis target person by the biological reaction analysis unit 12, and the change in the biological reaction analyzed for the analysis target person is more specific than before. Judge whether or not.

For example, the peculiarity determination unit 13 calculates the variance of the biological reaction index value calculated for each of a plurality of online sessions for the analysis target person by the biological reaction analysis unit 12, and the biological reaction index value calculated for this online session. By contrasting with the variance, it is determined whether or not the change in the biological reaction analyzed for the subject to be analyzed this time is more specific than before.

The related event identification unit 14 identifies an event occurring with respect to at least one of the analysis subject, another person, and the environment when a change in the biological reaction determined to be specific by the peculiarity determination unit 13 occurs. .. For example, the related event identification unit 14 identifies the behavior of the analysis target person himself / herself from the moving image when a specific change in the biological reaction occurs for the analysis target person. In addition, the related event identification unit 14 identifies the behavior of another person from the moving image when a specific change in the biological reaction occurs for the analysis target person. In addition, the related event identification unit 14 identifies the environment when a specific change in the biological reaction occurs for the analysis target person from the moving image. The environment is, for example, a shared material displayed on the screen, an environment reflected in the background of the person to be analyzed, and the like. By identifying such an event, it is possible to capture an event that may have influenced the change in the reaction of the analysis subject.

The clustering unit 15 is the content of the change in the biological reaction determined to be specific by the peculiarity determination unit 13 (for example, one or a combination of the line of sight, the pulse, the movement of the face, the content of speech, and the voice quality) and the past. The pattern of change in biological response is clustered based on the magnitude of change from. This makes it possible to analyze and present patterns of behaviors that may occur in the future for the analysis target person. For example, it is possible to understand that the subject to be analyzed may withdraw from the online session when the degree of smile gradually decreases and negative biological reactions continue for a predetermined number of times. You may want to give an alert to let you know.

The analysis result notification unit 16 is at least one of a change in a biological reaction determined to be specific by the peculiarity determination unit 13, an event specified by the related event identification unit 14, and a change pattern clustered by the clustering unit 15. Is notified to the designated person (the organizer of the online session) of the analysis target. The organizer of the online session may be notified of the future behavior pattern of the analysis target person predicted from the clustered change pattern, a predetermined alert based on the future behavior pattern, and the like.

This allows the organizer of the online session to know what kind of event affects what kind of change in biological reaction for the specified analysis target person. Then, it becomes possible to take appropriate measures for the analysis target person according to the grasped contents. In addition, the organizer of the online session grasps the behavior tendency of the specified analysis target person and predicts the behavior and state that may occur in the future, depending on what kind of change pattern the specified analysis target person is clustered into. be able to. Then, it becomes possible to take appropriate measures for the analysis target person.

It should be noted that the above embodiments are merely examples of the embodiment of the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from its gist or its main features.

11 Moving image acquisition unit 12 Biological reaction analysis unit 13 Singularity determination unit 14 Related event identification unit 15 Clustering unit 16 Analysis result notification unit

Claims

Reaction analysis that analyzes a specific reaction different from before for a participant based on a moving image obtained by taking a picture of a participant for each of multiple online sessions in an environment where an online session is held by multiple people. system.
For each of the plurality of online sessions, a moving image acquisition unit that acquires a moving image obtained by photographing the participants during the online session, and a moving image acquisition unit.
Based on the moving image acquired by the moving image acquisition unit, the biological reaction analysis unit that analyzes changes in the biological reaction of the participants, and the biological reaction analysis unit.
The changes in the biological response analyzed for the participant for one online session are specific compared to the changes in the biological response analyzed for the participant for the online session prior to the online session in time. The reaction analysis system according to claim 1, further comprising a peculiarity determination unit for determining whether or not the condition is present.
The biological reaction analysis unit analyzes changes in the biological reaction related to at least one of facial expression, line of sight, pulse, and facial movement by analyzing the facial image in the moving image acquired by the moving image acquisition unit. 2. The reaction analysis system according to claim 2.
The claim is characterized in that the biological reaction analysis unit analyzes changes in the biological reaction relating to at least one of the content of speech and voice quality by analyzing the voice in the moving image acquired by the moving image acquisition unit. The reaction analysis system according to 2 or 3.
The biological reaction analysis unit calculates the biological reaction index value by quantifying the change in the biological reaction according to a predetermined standard.
In the peculiarity determination unit, based on the biological reaction index value calculated for the participant by the biological reaction analysis unit, the change in the biological reaction analyzed for the participant with respect to the online session is the above-mentioned one. Any one of claims 2 to 4, wherein it is determined whether or not the online session prior to the online session is specific to the change in the biological reaction analyzed for the participants. The reaction analysis system described in.
A relationship that identifies an event occurring with respect to at least one of the analysis target person, another person other than the analysis target person, and the environment when a change in a biological reaction determined to be specific by the specificity determination unit occurs. The reaction analysis system according to any one of claims 2 to 5, further comprising an event identification unit.
It is characterized by further including a clustering unit for clustering the change pattern of the biological reaction based on the content of the change of the biological reaction determined to be specific by the specificity determination unit and the magnitude of the change from the past. The reaction analysis system according to any one of claims 2 to 6.
Further provided with an analysis result notification unit for notifying the organizer of the online session of at least one of the changes in the biological reaction determined to be specific by the peculiarity determination unit and the event specified by the related event identification unit. The emotion analysis system according to claim 6.
Further provided with an analysis result notification unit for notifying the organizer of the online session of at least one of the changes in the biological reaction determined to be specific by the peculiarity determination unit and the change patterns clustered by the clustering unit. The emotion analysis system according to claim 7.