JP2019148849A - System for determining degree of understanding and program for determining degree of understanding - Google Patents

Abstract

To provide a system for determining the degree of understanding and a program for determining the degree of understanding that can accurately determine a listener's degree of understanding on the basis of the facial expression of the listener.SOLUTION: A system for determining the degree of understanding comprises: means for analyzing feeling that analyzes a listener's feeling in S51 on the basis of the listener's facial expression photographed by a camera; and means for analyzing the degree of understanding that analyzes the listener's degree of understanding in S53 to S56 on the basis of the feeling analyzed by the means for analyzing feeling. The means for analyzing the degree of understanding determines the listener's degree of understanding on the basis of the feeling of "disgust" among the feelings analyzed by the means for analyzing feeling.SELECTED DRAWING: Figure 9

Description

  The present invention relates to an understanding level determination system and an understanding level determination program for determining an understanding level of a listener.

  As a conventional understanding level determination system, there is known a system for determining a listener's level of understanding from movement information such as a facial expression of a listener photographed by a camera (see, for example, Patent Document 1).

Japanese Patent Laid-Open No. 2006-177483

  However, in the understanding level determination system described in Patent Document 1, since the specific method for determining the level of understanding of the listener based on the facial expression of the listener is unknown, the level of understanding of the listener is determined with high accuracy. There is a problem that it is unclear whether or not it can be done.

  Accordingly, an object of the present invention is to provide an understanding level determination system and an understanding level determination program that can determine the level of understanding of a listener with high accuracy based on the facial expression of the listener.

  An understanding level determination system according to the present invention includes an emotion analysis unit that analyzes an emotion of the listener based on a facial expression of the listener photographed by a camera, and an understanding of the listener based on the emotion that is analyzed by the emotion analysis unit. Comprehension level analysis means for analyzing the degree, wherein the understanding level analysis means determines the level of understanding of the listener based on an emotion of “disgust” among the emotions analyzed by the emotion analysis means. To do.

  With this configuration, the understanding level determination system of the present invention determines the level of understanding of the listener based on the emotion of “disgust” that correlates with the level of understanding of the listener among the emotions analyzed based on the facial expression of the listener. Therefore, it is possible to determine the degree of understanding of the listener with high accuracy based on the facial expression of the listener.

  In the understanding level determination system of the present invention, the understanding level analysis means displays the determined level of understanding on a monitor, and the understanding level determination system is a remote conference in which the camera and the monitor are installed at a remote location. It may be a system.

  Remote meetings, that is, meetings between people who are far away from each other, have less information that can be obtained visually than a face-to-face meeting, ie, a meeting between people who are in the same place. It is difficult for non-verbal communication such as facial expressions and gestures to be transmitted to the speaker, and it is difficult for the speaker to determine the level of understanding of the listener with high accuracy. However, since the understanding level determination system of the present invention determines the listener's level of understanding based on the facial expression of the listener taken by the camera, the listener's level of understanding is determined with high accuracy even in a remote conference system. be able to.

  The understanding level determination program of the present invention includes an emotion analysis unit that analyzes an emotion of the listener based on a facial expression of the listener photographed by a camera, and an understanding of the listener based on the emotion that is analyzed by the emotion analysis unit. An understanding level analysis means for analyzing the degree of the sound, and the understanding level analysis means determines the level of understanding of the listener based on an emotion of “disgust” among the emotions analyzed by the emotion analysis means. It is characterized by.

  With this configuration, the computer that executes the understanding level determination program of the present invention enables the understanding level of the listener based on the emotion of “disgust” that correlates with the understanding level of the listener among the emotions analyzed based on the facial expression of the listener. Therefore, it is possible to determine the degree of understanding of the listener with high accuracy based on the facial expression of the listener.

  The understanding level determination system and the understanding level determination program of the present invention can determine the listener's understanding level with high accuracy based on the facial expression of the listener.

It is a top view which shows the environment of the experiment for a development conducted in order to develop the understanding level determination system which concerns on one embodiment of this invention. (A) It is the graph which showed the intensity | strength of the feeling of "disgust" for every time among the results of the experiment for development performed in the environment shown in FIG. (B) It is the graph which showed the intensity | strength of the feeling of "anger" for every time among the results of the experiment for development performed in the environment shown in FIG. (C) It is the graph which showed the intensity | strength of the feeling of "sadness" for every time among the results of the experiment for development performed in the environment shown in FIG. (D) It is the graph which showed the intensity | strength of the feeling of "contempt" for every time among the results of the experiment for development performed in the environment shown in FIG. (A) It is the graph which showed the intensity | strength of the feeling of "joy" for every time among the results of the experiment for development performed in the environment shown in FIG. (B) It is the graph which showed the intensity | strength of the feeling of "fear" for every time among the results of the experiment for development performed in the environment shown in FIG. (C) It is the graph which showed the intensity | strength of the feeling of "surprise" among the results of the experiment for development performed in the environment shown in FIG. 1 for every time. (A) It is the graph which remove | excluded the data of the scene 1 and the scene 5 from the graph shown to Fig.2 (a). (B) It is the graph which showed the intensity | strength of the feeling of "disgust" for every time among the results of the experiment for development with respect to the test subject different from the test subject of the data shown to Fig.4 (a). (A) It is the figure which clustered the intensity | strength of the feeling of "disgust" among the results of the experiment for a development with respect to the test subject of the data shown to Fig.4 (a). (B) It is the figure which clustered the intensity | strength of the feeling of "disgust" among the results of the experiment for a development with respect to the test subject of the data shown in FIG.4 (b). The figure which shows the ratio of the scene which contains the data contained in the cluster 4, and the scene which does not contain the data contained in the cluster 4 for every understanding degree which the test subject answered in the development experiment performed in the environment shown in FIG. is there. 1 is a block diagram of an understanding level determination system according to an embodiment of the present invention. FIG. 8 is a block diagram of the computer shown in FIG. 7. It is a flowchart of operation | movement of the computer shown in FIG. 8 in the case of displaying on a monitor the understanding level of the listener image | photographed with a camera. It is a block diagram of the understanding level determination system of a structure different from the understanding level determination system shown in FIG.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

  First, an experiment (hereinafter referred to as “development experiment”) performed to develop the understanding level determination system according to the present embodiment will be described.

  The inventor of the present application conducted a development experiment described below in order to confirm the relationship between the facial expression of the listener and the level of understanding of the listener.

1. Contents of Development Experiment FIG. 1 is a top view showing the environment of the development experiment.

  As shown in FIG. 1, the development experiment is performed by two persons, that is, a subject 11 and an explanation staff 12 who reads a specific explanation from the subject 11. In the development experiment, the subject 11 and the instructor 12 are seated facing each other across the table 13 with a distance of about 2 m. Then, the facial expression of the subject 11 while the explanation staff 12 is reading the explanation is recorded by the camera 14 arranged on the table 13 at a distance of about 1 m from the subject 11.

  As the explanatory text used in the development experiment, a philosophy field subject to which it was difficult to make a difference in knowledge among multiple subjects was adopted. The explanatory text used in the development experiment is divided into six scenes from scene 1 to scene 6, and the difficulty levels of scene 2 and scene 3 are scene 1, scene 4, scene 5 and scene 6, respectively. It is higher than each difficulty level.

The procedure for the development experiment is as follows.
(1) The instructor explains the contents of the experiment and the precautions to the subject.
(2) The instructor explains the flow of the experiment using scene 1.
(3) Q & A about the experiment (4) The instructor explains the scene 2.
(5) The test subject answers the level of understanding of scene 2 in five levels from 1 to 5 (the larger the numerical value, the higher the level of understanding).
(6) The above (4) and (5) are repeated in the order of scene 3, scene 4, scene 5, and scene 6.

  The development experiment was conducted on each of 10 subjects under the above conditions.

2. Analysis of the result of the development experiment The explanation is read out by the explanation staff by analyzing the data of the moving image acquired by the camera 14 in the development experiment using the affdex manufactured by effectiva, which is facial expression analysis software. The emotion of the subject in the middle was acquired. affdex determines the intensity of the subject's emotion from the facial expression of the subject for each of the seven emotions of “disgust”, “anger”, “sadness”, “contempt”, “joy”, “fear” and “surprise”. A value of 0.00 to 100.00 can be acquired every 1/30 seconds.

  2 (a), FIG. 2 (b), FIG. 2 (c), FIG. 2 (d), FIG. 3 (a), FIG. 3 (b), and FIG. 3 (c) are the results of development experiments. These are graphs showing the intensity of emotions of “disgust”, “anger”, “sadness”, “contempt”, “joy”, “fear”, and “surprise” for each hour.

  The graphs shown in FIGS. 2 and 3 are graphs showing the strength of each of the seven emotions of a specific subject for each time. In each graph, the data indicated by a bold line represents the degree of understanding answered by the subject subject as a reference, and is not directly related to the numerical value on the vertical axis indicating the strength of emotion. 2 and FIG. 3, the subject subject answered that the understanding level of the scene 1 is 3, the understanding level of the scenes 2 and 3 is 2, and the understanding level of the scenes 4 and 5 is 4. The answer is that the degree of understanding of the scene 6 is 5.

  As shown in FIG. 2 and FIG. 3, among the seven emotional intensities of “disgust”, “anger”, “sadness”, “contempt”, “joy”, “fear” and “surprise”, “disgust” There was a relationship between the intensity of emotion and the level of comprehension that the subjects answered. That is, in the graph shown in FIG. 2A, the intensity of the feeling of “disgust” is high in the scene 2 and the scene 3 with the low understanding level answered by the subject. The data shown in FIG. 2 and FIG. 3 are data of one subject, but the tendency that the intensity of the feeling of “disgust” increases in the scene where the level of understanding answered by the subject is low appears for all subjects. .

  FIG. 4A is a graph obtained by removing the data of the scene 1 and the scene 5 from the graph shown in FIG. FIG. 4B is a graph showing the intensity of the feeling of “hate” among the results of the development experiment for a subject different from the subject subject to the data shown in FIG.

  The graph shown in FIG. 4B excludes the data of the scene 1 and the scene 5 similarly to the graph shown in FIG.

  As shown in FIG. 4, there are individual differences in the intensity of emotion. Therefore, when the threshold value of the emotion strength for determining that the understanding level is low is a fixed numerical value that does not depend on the subject, the understanding level cannot be determined appropriately. Therefore, the threshold needs to be acquired for each subject. Clustering exists as a method for acquiring a threshold for each subject.

  5 (a) and 5 (b), respectively, cluster the intensity of emotions of “disgust” among the results of the development experiment for the test subject of the data shown in FIG. 4 (a) and FIG. 4 (b). FIG.

  As shown in FIG. 5, by clustering the intensity of “hate” emotion for each subject, the influence of individual differences in emotion intensity can be suppressed. Specifically, the k-means method was used as a clustering method. In FIG. 5, each one of the circles indicates the emotion intensity every 1/30 seconds.

  When performing clustering, the subject's emotional intensity is normally obtained from the facial expression of the subject using facial expression analysis software, such as when the subject is speaking or when the subject is touching the subject's own face. If not, the data was removed because it would interfere with proper clustering. Also, if the subject is not yet used to the experiment, the facial expression of the subject may not be natural. Therefore, when performing clustering, the data of scene 1, which is a scene at a time when the subject is not yet used to the experiment, is removed because it hinders appropriate clustering. In addition, scene 5 has a smaller number of data than other scenes. Therefore, when performing clustering, the data of scene 5 is removed because it hinders proper clustering.

  In the experiment, the test subject answered only one of the understanding levels from 2 to 5 among the 5 levels of understanding from 1 to 5. Therefore, when performing clustering, the number of clusters is four. The cluster with the lowest average value included in the cluster is defined as cluster 1, the cluster with the second lowest average value included in the cluster is defined as cluster 2, and the cluster with the lowest average value included in the cluster is defined as cluster 3. Is the cluster 3, and the cluster with the highest average value of the numbers contained in the cluster is the cluster 4.

  FIG. 6 is a diagram illustrating a ratio of a scene including data included in the cluster 4 and a scene not including data included in the cluster 4 for each understanding level answered by the subject in the development experiment.

  As shown in FIG. 6, among the scenes for which the test subject answered 2 as the degree of understanding, 67% of the scenes included data included in cluster 4, and 33% of the scenes that did not include data included in cluster 4 It is. Of the scenes in which the test subject has answered 3 as an understanding level, 33% of the scenes include data included in the cluster 4, and 67% of the scenes do not include data included in the cluster 4. Of the scenes in which the test subject has answered 4 as the degree of understanding, 14% of the scenes include data included in cluster 4, and 86% do not include data included in cluster 4. Of the scenes in which the subject answered 5 as the degree of understanding, 17% of the scenes include data included in the cluster 4, and 83% do not include data included in the cluster 4.

  Here, it is determined that a scene with a comprehension level of 1 or 2 answered by the subject is “not understood” and a scene with a comprehension level of 3, 4 or 5 answered by the test subject is “understood”. judge. Then, it is determined that the scene including the data included in the cluster 4 is “not understood”, and the scene not including the data included in the cluster 4 is determined as “understanding”. Of the scenes determined as “not understood” based on the degree of understanding that the test subject answered, the scenes determined as “not understood” based on the intensity of the emotion of “disgust”, that is, in the cluster 4 The ratio of the scene including the included data was 67%. Of the scenes determined as “understanding” based on the degree of understanding that the test subject answered, the scenes determined as “understanding” based on the intensity of the emotion of “disgust”, that is, in cluster 4 The ratio of scenes that do not include included data was 79%. Based on the degree of comprehension that the test subjects answered, they did not understand or understood, and they did not understand or understood based on the emotional intensity of disgust. The coincidence rate with the determination of “was 75%.

  Next, the configuration of the understanding level determination system according to the present embodiment will be described.

  FIG. 7 is a block diagram of the understanding level determination system 20 according to the present embodiment.

  As shown in FIG. 7, the understanding level determination system 20 includes a computer 30, a camera 35, a microphone 36, a monitor 37, and a speaker 38 connected to the computer 30, and a computer 40 that is arranged away from the computer 30. A camera 45, a microphone 46, a monitor 47, and a speaker 48 connected to the computer 40 are provided. The computer 30 and the computer 40 can communicate with each other via a network 21 such as the Internet.

  In the understanding level determination system 20, the computer 30 transmits the video acquired by the camera 35 and the audio acquired by the microphone 36 to the computer 40. Then, the computer 40 displays the video transmitted from the computer 30 on the monitor 47 and outputs the audio transmitted from the computer 30 from the speaker 48. Similarly, the computer 40 transmits the video acquired by the camera 45 and the audio acquired by the microphone 46 to the computer 30. The computer 30 displays the video transmitted from the computer 40 on the monitor 37 and outputs the audio transmitted from the computer 40 from the speaker 38. That is, the comprehension level determination system 20 is a remote conference system that enables a conference between people who are separated from each other.

  FIG. 8 is a block diagram of the computer 30.

  As shown in FIG. 8, the computer 30 includes an operation unit 31 that is an input device such as a keyboard and a mouse through which various operations are input, and a network such as a LAN (Local Area Network) and the Internet, or a network. A communication unit 32 that is a communication device that directly communicates with an external device via a wired or wireless connection without using a network, and a nonvolatile storage device such as a semiconductor memory or an HDD (Hard Disk Drive) that stores various types of information And a control unit 34 that controls the entire computer 30.

  The storage unit 33 stores an understanding level determination program 33a for determining the level of understanding of the listener. The understanding level determination program 33a may be installed in the computer 30 of the computer 30, for example, or from an external storage medium such as a USB (Universal Serial Bus) memory, a CD (Compact Disk), a DVD (Digital Versatile Disk), etc. It may be additionally installed on the computer 30 or may be additionally installed on the computer 30 from the network.

  The control unit 34 includes, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory) storing programs and various data, and a RAM (Random Access Memory) used as a work area of the CPU. Yes. The CPU executes a program stored in the ROM or the storage unit 33.

  The control unit 34 executes the understanding level determination program 33a to analyze the emotion of the listener based on the facial expression of the listener, and the emotion analysis unit 34a based on the emotion analyzed by the emotion analysis unit 34a. An understanding level analysis means 34b for analyzing the understanding level is realized.

  The configuration of the computer 40 is the same as the configuration of the computer 30.

  Next, the operation of the computer 30 when the understanding level of the listener photographed by the camera 45 is displayed on the monitor 37 will be described.

  FIG. 9 is a flowchart of the operation of the computer 30 when the understanding level of the listener photographed by the camera 45 is displayed on the monitor 37.

  When the computer 30 receives the moving image file of the video transmitted from the computer 40 (hereinafter referred to as “target moving image file”), the computer 30 executes the operation shown in FIG.

  As shown in FIG. 9, the emotion analysis means 34a acquires the emotion of the listener by analyzing the target moving image file using facial expression analysis software such as FACS (Facial Action Coding System) (S51). Here, the facial expression analysis software may be software that analyzes emotions using facial feature points. For example, affdex manufactured by Affectiva may be employed.

  Next, the emotion analysis unit 34a generates time-series data indicating the intensity of the “disgust” emotion among the emotions acquired in S51 for each hour (S52). That is, the emotion analysis means 34a generates time series data similar to the time series data shown in FIG.

  Next, the understanding level analysis unit 34b divides the time series data generated in S52 by a specific break in time (S53). Here, for example, when the speaker designates a break in the speaker's own presentation material, the break designated by the speaker in the presentation material may be adopted. A specific time such as every second may be employed.

  The understanding level analysis means 34b determines the intensity of the emotion of the listener from the facial expression of the listener, such as when the listener is speaking or after the operation of S53, or when the listener is touching the listener's own face. Data that cannot be normally acquired by the analysis software is removed from the time-series data divided by specific delimiters in S53 (S54). Here, the understanding level analysis unit 34b can recognize that the listener is speaking and that the listener is touching the face of the listener by image processing on the target moving image file.

  The understanding level analysis unit 34b performs clustering on the data of the time series data generated in S54 after the processing in S54 as in the example shown in FIG. 5 (S55). Here, the understanding level analysis unit 34b performs clustering using, for example, a k-average method. In addition, the understanding level analysis unit 34b may determine the number of clusters using, for example, an elbow method, or may use a specific number such as the number instructed by the user as the number of clusters.

  After the process of S55, the understanding level analysis unit 34b determines whether it is “I don't understand” or “I understand” for each of the sections divided in S53 (S56). Here, the comprehension level analysis unit 34b determines that the section including the data included in the cluster having the highest average value of the numerical values included in the cluster classified in S55 among the sections divided in S53, Is determined. On the other hand, the comprehension level analysis unit 34b determines that the section that does not include the data included in the cluster having the highest average value of the numerical values included in the clusters classified in S55 among the sections divided in S53, Is determined.

  The understanding level analyzing unit 34b displays the determination result in S56 on the monitor 37 after the process of S56 (S57). For example, the understanding level analysis unit 34b displays that it does not understand with a pictogram representing a feeling of sadness for the section determined as “not understood” in S56, and “understands” in S56. For the section determined to be, the understanding is displayed with a pictograph or the like representing a calm feeling.

  The understanding level analysis unit 34b ends the operation shown in FIG. 9 after the process of S57.

  In the above, the operation of the computer 30 when the understanding level of the listener photographed by the camera 45 is displayed on the monitor 37 has been described. The same applies to the operation of the computer 40 when the understanding level of the listener photographed by the camera 35 is displayed on the monitor 47.

  As described above, the understanding level determination system 20 determines the listener's level of understanding based on the emotion of “disgust” that correlates with the level of understanding of the listener among the emotions analyzed in S51 based on the facial expression of the listener. Since it determines (S56), a listener's comprehension degree can be determined with high precision based on a facial expression of a listener.

  Remote meetings, that is, meetings between people who are far away from each other, have less information that can be obtained visually than a face-to-face meeting, ie, a meeting between people who are in the same place. It is difficult for non-verbal communication such as facial expressions and gestures to be transmitted to the speaker, and it is difficult for the speaker to determine the level of understanding of the listener with high accuracy. However, since the understanding level determination system 20 determines the listener's level of understanding based on the facial expression of the listener photographed by the camera 45, the listener's level of understanding can be determined with high accuracy even in a remote conference system. Can do. Therefore, in the understanding level determination system 20, when the listener does not understand, the speaker can lower the level of the utterance content or re-explain the result, thereby improving the accuracy of information transmission. be able to.

  The remote conference system has been described above. However, the understanding level determination system may not be a remote conference system. For example, the understanding level determination system may be as shown in FIG.

  FIG. 10 is a block diagram of an understanding level determination system 60 having a different configuration from the understanding level determination system 20 shown in FIG.

  As shown in FIG. 10, the understanding level determination system 60 includes a computer 30, a camera 35 and a monitor 37 connected to the computer 30. The camera 35 is arranged at a position where the listener can be photographed. The monitor 37 is arranged at a position where the screen can be visually recognized by the speaker.

20 Understanding determination system (remote conference system)
30 Computer 33a Comprehension Determining Program 34a Emotion Analysis Means 34b Comprehension Analyzing Means 35 Camera 37 Monitor 40 Computer 45 Camera 47 Monitor 60 Understanding Degree Determination System

Claims (3)

Emotion analysis means for analyzing the emotion of the listener based on the facial expression of the listener photographed by the camera;
Comprehension level analysis means for analyzing the level of understanding of the listener based on the emotion analyzed by the emotion analysis means,
The comprehension level analysis system characterized in that the comprehension level analysis unit determines the comprehension level of the listener based on an emotion of "disgust" among the emotions analyzed by the emotion analysis unit. The understanding level analysis means displays the determined level of understanding on a monitor,
The understanding level determination system according to claim 1, wherein the understanding level determination system is a remote conference system installed at a location where the camera and the monitor are separated from each other. Emotion analysis means for analyzing the emotion of the listener based on the facial expression of the listener photographed by the camera;
An understanding level analysis means for analyzing the understanding level of the listener based on the emotion analyzed by the emotion analysis means;
The understanding level analysis unit determines the understanding level of the listener based on an emotion of “disgust” among the emotions analyzed by the emotion analysis unit.