JP2020123884A - Concentration evaluation program, device, and method - Google Patents

Abstract

To appropriately evaluate the degree of concentration of a user on watching a video even when the user's face does not face a screen in many cases.SOLUTION: When multiple users watch the same content, many users take notes at the same timing (t1 and t2) in an important portion in terms of content, and users who are not concentrated may delay their reaction, and do not take notes, and therefore, the movements of the heads of the target users who are watching a video are detected, the degree of correlation to the movements of the heads of multiple users is calculated, and the degree of concentration of the target user for watching the video is evaluated on the basis of the calculated degree of correlations.SELECTED DRAWING: Figure 1

Description

The disclosed technology relates to a concentration degree evaluation program, a concentration degree evaluation device, and a concentration degree evaluation method.

With the development of communication technology, various services for delivering high-quality video are increasing. For example, as a learning content, a video in which a lecture such as a study cram school is recorded is distributed, and a student attends the lecture by viewing the video. In this type of attendance, it is difficult for the instructor to watch the attendance of each student, and it is difficult for the instructor to grasp the degree of concentration of the student in the lecture.

Therefore, it is possible to grasp the degree of concentration by the facial expressions of the students who watch the lecture video. For example, there is a technique for identifying facial expressions by using a feature obtained by acquiring an image of a user who is watching a moving image, cutting out a face area, and extracting from the cut face area by a box filter or a Gabor filter, and a machine-learned classifier. Proposed. In this technique, the degree of concentration is estimated using the change in the identified facial expression.

Further, as a technique for evaluating the situation of a user in a lecture, an information processing system for calculating an evaluation of an object by an observer who observes the object has been proposed. This system detects the smile of the observer who observes the object, detects the degree to which the face of the observer faces substantially downward, and evaluates the observer according to the degree of smile and the degree of downward. calculate.

In addition, a technique has been proposed in which the degree of concentration of a user is determined based on the fluctuation range of the vertical movement of the head of the user during a conference, the number of vertical movements of the head, the period, and the like.

Further, there has been proposed an attention target determination device that calculates the direction of the attention target based on the surrounding conditions, the face orientation, the degree of interlocking between the attention target and the face orientation, and the attention target determination rule. This device evaluates that the interlocking property is high when the timing of the change in the surrounding conditions and the timing of the change in the face direction match.

International Publication No. 2017/018012 JP, 2010-141843, A JP, 2009-26190, A

Jacob Whitehill, Zewelanji Serpell, Yi-Ching Lin, Aysha Foster, Javier R. Movellan, "The Faces of Engagement: Automatic Recognition of Student Engagement from Facial Expressions", IEEE Transactions on Affective Computing, Volume 5, 2014

For example, when watching a lecture video as described above, when the user takes notes or looks at the material at hand that is distributed separately from the video, the user has the opportunity to look down, that is, the face should be right on the screen. There are many opportunities not to meet.

The prior art that estimates the degree of concentration based on changes in facial expressions is based on the assumption that the screen is always viewed, and if there are many occasions where the face is not directly facing the screen, as described above, There are many times when it is impossible to estimate.

In addition, with this conventional technology, the face is not directly facing the screen while the user is concentrating on the lecture itself while taking notes or looking at the material at hand Cannot be identified. Therefore, there is a possibility that the degree of concentration cannot be estimated, or that it is estimated that the person is not concentrated.

Further, with the conventional technology that evaluates the situation of the user based on the movement of the head or the like, it is not possible to evaluate the degree of concentration of the user with respect to viewing a moving image.

As one aspect, the disclosed technique aims to appropriately evaluate the degree of concentration of a user with respect to viewing a moving image, even when the user's face often does not face the screen.

As one aspect, the disclosed technique detects the movement of the head of the target user who is watching a moving image, and detects the head of each of the plurality of users detected when the plurality of users watch the moving image. The degree of correlation of the movement of the target user's head with respect to the movement is calculated. Then, the disclosed technology evaluates the degree of concentration of the target user with respect to viewing of the moving image, based on the calculated degree of correlation.

As one aspect, there is an effect that it is possible to appropriately evaluate the degree of concentration of a user with respect to watching a moving image even when there are many occasions when the user's face is not directly facing the screen.

It is a figure for demonstrating the relationship between the timing of head movement, and the degree of concentration. It is a functional block diagram of a concentration degree evaluation device concerning a 1st embodiment. It is a figure which shows an example of the average pattern database (DB) in 1st Embodiment. It is a figure for demonstrating the synchrony of the motion of a head. It is a figure which shows an example of evaluation value DB. It is a block diagram showing a schematic structure of a computer which functions as a concentration degree evaluation device. It is a flow chart which shows an example of generation processing. It is a flow chart which shows an example of evaluation processing in a 1st embodiment. It is a functional block diagram of a concentration degree evaluation device concerning a 2nd and 3rd embodiment. It is a figure which shows an example of importance DB. It is a figure for demonstrating the average of the importance degree in a fixed time. It is a figure for demonstrating calculation of the evaluation value using an importance degree and a correlation degree. It is a flow chart which shows an example of evaluation processing in a 2nd embodiment. It is a figure for explaining clustering of a pattern of a plurality of users. It is a figure for explaining clustering of a pattern of a plurality of users. It is a figure which shows an example of the average pattern DB in 3rd Embodiment. It is a flow chart which shows an example of evaluation processing in a 3rd embodiment.

Hereinafter, an example of an embodiment according to the disclosed technology will be described with reference to the drawings. In each of the following embodiments, a case will be described as an example in which the degree of concentration of a user with respect to viewing a lecture video as learning content is evaluated.

First, before describing the details of each embodiment, the operation principle common to each embodiment will be described.

During viewing of a lecture video, the user often faces downward to take notes. The timing of taking notes and memos is the same for a plurality of users. As shown in FIG. 1, when the same content is viewed by a plurality of users, a large number of users take notes at the same timing (t1 and t2 in FIG. 1) in an important part in terms of content. .. On the other hand, a user who is not concentrated may have a delayed response or may not take notes.

Therefore, in each of the following embodiments, the degree of concentration is evaluated by grasping the timing of taking a note by the movement of the user's head and comparing the timing with the timings of a plurality of users.

Hereinafter, each embodiment will be described in detail.

<First Embodiment>
As shown in FIG. 2, a user image, which is moving image data of a user watching a lecture moving image, is input to the concentration evaluation apparatus 10 according to the first embodiment. The user image is captured by, for example, a camera installed above the display or the like where the lecture moving image is reproduced. The user image is associated with time information that is synchronized with the elapsed time from the start point of the lecture video that was viewed. The degree-of-concentration evaluation device 10 evaluates the degree of concentration of the user based on the user image and outputs the evaluation result.

As shown in FIG. 2, the concentration degree evaluation device 10 functionally includes a detection unit 12, a generation unit 14, a calculation unit 16, and an evaluation unit 18. An average pattern database (DB) 22 and an evaluation value DB 24 are stored in a predetermined storage area of the concentration degree evaluation device 10.

The detection unit 12 detects the movement of the user's head from the user image input to the concentration degree evaluation device 10. Specifically, the detection unit 12 detects the orientation of the user's head from each frame of the user image corresponding to each time of the predetermined time interval, and detects the temporal change in the orientation of the head for a certain time. It is detected as a pattern indicating the movement of the head. The frame for detecting the orientation of the head may be every frame of the user image or every predetermined number of frames. As a method of detecting the orientation of the head, a conventionally known method can be used.

For example, the detection unit 12 may detect the orientation of the user's head as an angle in the vertical direction with respect to the orientation facing the screen, or the detected angle may be detected as the head facing the screen. The direction of the part may be 1, and the downward direction of the head may be 0 for binarization.

The detection unit 12 transfers the pattern detected from each user image for each of the plurality of users to the generation unit 14. In addition, the detection unit 12 transfers the pattern detected from the user image of the target user whose concentration degree is evaluated to the calculation unit 16.

The generation unit 14 generates an average pattern from the patterns of a plurality of users passed from the detection unit 12. Specifically, the generation unit 14 generates an average pattern by averaging values (angles or binarized values) indicating the orientation of the head detected at the same time in a plurality of patterns. When a binarized value is used as the value indicating the orientation of the head, the closer the continuous value obtained as the average value is to 1, the more the user's face is directly facing the screen, and 0 The closer to, the more users are pointing down.

The generation unit 14 stores the generated average pattern in the average pattern DB 22. FIG. 3 shows an example of the average pattern DB 22. In the example of FIG. 3, a binarized value is used as the value indicating the head direction in the average pattern DB 22, and a continuous value that is the average is “average head direction”, which indicates each time. It is stored in association with "time information". As the time information, the elapsed time from the start point of the lecture moving image synchronized with the time information of the user image may be used, or the frame number of the frame corresponding to that time may be used.

The calculation unit 16 calculates the degree of correlation of the movement of the head of the target user with respect to the movement of the head of each of the plurality of users. Specifically, the calculating unit 16 calculates the degree of correlation indicating the synchronism between the pattern of the target user passed from the detecting unit 12 and the average pattern stored in the average pattern DB 22.

FIG. 4 shows a diagram for explaining the synchronism of the movement of the head. As shown in FIG. 4, when there is a small difference (eg, A in FIG. 4) at which the change of the pattern starts, such as where the head is lowered, the two patterns are synchronized. It can be said that they are doing. On the other hand, when there is a large difference in the timing at which the pattern starts to change with respect to the patterns of a plurality of users (for example, BA in FIG. 4), this pattern is not synchronized with the patterns of a plurality of users. I can say.

For example, the calculation unit 16 reads a part of the average pattern corresponding to the time information of the pattern of the target user and the time pattern from the average pattern DB 22, and the mutual relationship between the pattern of the target user of the constant time and the average pattern. The number can be calculated as the degree of correlation. The degree of correlation is not limited to the cross-correlation coefficient, and may be any evaluation index that increases as the synchronism between the pattern of the target user and the average pattern increases.

The calculation unit 16 transfers the calculated correlation degree to the evaluation unit 18.

The evaluation unit 18 evaluates the degree of concentration of the target user with respect to viewing and listening to the lecture moving image, based on the degree of correlation passed from the calculation unit 16. The evaluation unit 18 evaluates that the higher the degree of correlation, the higher the degree of concentration of the target user.

For example, the evaluation unit 18 normalizes the correlation degree passed from the calculation unit 16 with a value of 0 to 1 and refers to the evaluation value DB 24 as shown in FIG. 5 to derive an evaluation value corresponding to the correlation degree. To do.

The evaluation unit 18 outputs, as the evaluation result of the degree of concentration, a set of evaluation values based on the degree of correlation derived at regular time intervals for the time from the start to the end of the user image.

The degree-of-concentration evaluation device 10 can be realized by, for example, a computer 40 shown in FIG. The computer 40 includes a CPU (Central Processing Unit) 41, a memory 42 as a temporary storage area, and a non-volatile storage unit 43. The computer 40 also includes an input/output device 44 such as an input unit and a display unit, and an R/W (Read/Write) unit 45 that controls reading and writing of data from and to the storage medium 49. The computer 40 also includes a communication I/F 46 connected to a network such as the Internet. The CPU 41, the memory 42, the storage unit 43, the input/output device 44, the R/W unit 45, and the communication I/F 46 are connected to each other via a bus 47.

The storage unit 43 can be realized by an HDD (Hard Disk Drive), an SSD (Solid State Drive), a flash memory, or the like. The storage unit 43 serving as a storage medium stores a concentration degree evaluation program 50 for causing the computer 40 to function as the concentration degree evaluation device 10. The concentration evaluation program 50 has a detection process 52, a generation process 54, a calculation process 56, and an evaluation process 58. In addition, the storage unit 43 has an information storage area 60 in which information constituting each of the average pattern DB 22 and the evaluation value DB 24 is stored.

The CPU 41 reads the concentration degree evaluation program 50 from the storage unit 43, expands it in the memory 42, and sequentially executes the processes of the concentration degree evaluation program 50. The CPU 41 operates as the detection unit 12 illustrated in FIG. 2 by executing the detection process 52. Further, the CPU 41 operates as the generation unit 14 illustrated in FIG. 2 by executing the generation process 54. Further, the CPU 41 operates as the calculation unit 16 shown in FIG. 2 by executing the calculation process 56. Further, the CPU 41 operates as the evaluation unit 18 shown in FIG. 2 by executing the evaluation process 58. Further, the CPU 41 reads information from the information storage area 60 and loads each of the average pattern DB 22 and the evaluation value DB 24 in the memory 42. As a result, the computer 40 that executes the concentration degree evaluation program 50 functions as the concentration degree evaluation device 10. The CPU 41 that executes the program is hardware.

The function realized by the concentration evaluation program 50 can also be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit) or the like.

Next, the operation of the concentration degree evaluation device 10 according to the first embodiment will be described. When the user images of the plurality of users are input to the concentration degree evaluation device 10, the generation process shown in FIG. 7 is executed in the concentration degree evaluation device 10. In the state where the generation process is executed and the average pattern is stored in the average pattern DB 22, the user image of the target user who views the same lecture moving image as the lecture moving image corresponding to the stored average pattern is input to the concentration degree evaluation device 10. To be done. Then, in the degree-of-concentration evaluation device 10, the evaluation process shown in FIG. 8 is executed.

Hereinafter, each of the generation processing and the evaluation processing will be described in detail. The generation processing and the evaluation processing are examples of the concentration degree evaluation method of the disclosed technology.

First, the generation process will be described.

In step S12 of FIG. 7, the detection unit 12 detects the orientation of the user's head from each frame corresponding to each time of the predetermined time interval in each of the user images of each of the plurality of users, and detects the head. The temporal change in the direction of is detected as a pattern indicating the movement of the head. The detection unit 12 transfers the pattern detected from each user image for each of the plurality of users to the generation unit 14.

Next, in step S14, the generation unit 14 averages the values (angles or binarized values) indicating the orientation of the head detected at the same time in the plurality of patterns transferred from the detection unit 12. By doing so, an average pattern is generated.

Next, in step S16, the generation unit 14 stores the generated average pattern in the average pattern DB 22, and the generation process ends.

Next, the evaluation process will be described.

In step S22 of FIG. 8, the detection unit 12 acquires each frame corresponding to each time of the predetermined time interval from the input user image of the target user, detects the orientation of the head, and corresponds to the frame. It is recorded in a predetermined recording area in association with the time information to be recorded.

Next, in step S24, the detection unit 12 determines whether or not the orientation of the head has been recorded for a certain period of time. If it has been recorded for a certain period of time, the process proceeds to step S26, and if it has not been recorded, the process returns to step S22.

In step S26, the detection unit 12 detects the orientation of the head recorded for a certain period of time as a pattern of the target user, and transfers it to the calculation unit 16. Then, the calculation unit 16 reads, from the average pattern DB 22, a portion of the average pattern in which the time information corresponds to the pattern for a certain time of the target user. Then, the calculation unit 16 calculates the degree of correlation between the pattern of the target user for a certain period of time and the average pattern. The calculation unit 16 transfers the calculated correlation degree to the evaluation unit 18.

Next, in step S28, the evaluation unit 18 normalizes the correlation degree passed from the calculation unit 16 with a value of 0 to 1, and refers to the evaluation value DB 24 as shown in FIG. The evaluation value to be derived is derived. The evaluation unit 18 associates the time information indicating a certain time with the derived evaluation value, and temporarily records them in a predetermined recording area.

Next, in step S30, the detection unit 12 determines whether or not the input user image has reached the end position, that is, whether or not the lecture moving image in which the user image and the time information are synchronized has ended. If the lecture moving image has ended, the process proceeds to step S32, and if not finished, the process returns to step S22.

In step S32, the evaluation unit 18 outputs a set of evaluation values based on the degree of correlation, which is temporarily stored in a predetermined recording area and is derived at regular time intervals, as the evaluation result of the degree of concentration, and the evaluation process ends.

As described above, the concentration degree evaluation device according to the first embodiment detects a pattern indicating the movement of the head of the target user who is viewing the lecture video. The degree-of-concentration evaluation device calculates a correlation degree between an average pattern obtained by averaging patterns of a plurality of users detected when a plurality of users watch the same lecture video and a pattern of a target user, and calculates the correlation degree. Based on, we evaluate the degree of concentration of the target user for viewing the lecture video. As described above, when the user concentrates on watching the video, many users use the same technique of lowering their heads, and even when there are many occasions when the user's face is not directly facing the screen, It is possible to appropriately evaluate the degree of concentration of the user with respect to watching the video.

<Second Embodiment>
Next, a second embodiment will be described. In the concentration degree evaluation device according to the second embodiment, the same parts as those of the concentration degree evaluation device 10 according to the first embodiment are designated by the same reference numerals and detailed description thereof will be omitted.

The concentration evaluation device 210 according to the second embodiment functionally includes a detection unit 12, a generation unit 14, a calculation unit 16, and an evaluation unit 218, as shown in FIG. 9. Further, an average pattern DB 22 and an importance DB 26 are stored in a predetermined storage area of the concentration evaluation device 210.

As described above, the user often looks down to take notes while watching the lecture video. The timing of taking this note is the same for a plurality of users. This feature becomes more prominent where the content of the lecture video is more important.

Therefore, the evaluation unit 218 evaluates that the higher the degree of importance of the content of the lecture video and the higher the degree of correlation with the average pattern, the higher the degree of concentration of the target user.

Specifically, the evaluation unit 218 refers to the importance DB 26 and acquires the importance of each section of the lecture video.

FIG. 10 shows an example of the importance DB 26. In the example of FIG. 10, for each “section”, “time information” indicating the section and “importance” are stored in association with each other. If the content is a lecture video, for example, a section that contains an important word in the words spoken by the instructor and a section that includes changes in audio information such as sound pressure at that time are highly important sections. Can be defined as Also, a section that includes a location where the teacher's face looks at the camera, draws a picture, or puts a mark on a part of the board is defined as a section with high importance. Good.

Also, at important points during the lecture, many users who are concentrated on the lecture video move their heads up and down by taking notes and other actions. Therefore, a section including a portion where a predetermined number of users' heads are raised and lowered may be defined as a section with high importance.

Based on the above definition of importance, the importance of each section is defined in advance by being manually defined by the content creator or automatically defined by using image processing and audio processing. , Are stored in the above-mentioned importance DB 26. The division of the sections stored in the importance DB 26 may be evenly divided at predetermined time intervals, or may be divided into indefinite lengths according to the timing at which the importance changes.

When the evaluation unit 218 evaluates the degree of concentration based on the user image of the target user for a fixed time (hereinafter, this fixed time is referred to as “fixed time S”), each of the sections included in the fixed time S of the lecture video. Is acquired from the importance DB 26. The evaluation unit 218 calculates the average of the acquired degrees of importance, and as shown in FIG. 11, when the average of the degrees of importance at the constant time S is equal to or greater than a predetermined threshold value, the degree of concentration at the constant time S is evaluated. To do.

The evaluation of the degree of concentration is not performed for a certain period S when the average of the degrees of importance is less than the threshold value. It should be noted that, in a certain time S where the average importance level is less than a threshold value, that is, in a place where the importance level is low, the user does not spend much time doing his/her head down to take notes, and the user's face is on the camera. It is thought that the time to face is long. Therefore, for a certain period of time S in which the average importance degree is less than the threshold value, the concentration degree may be evaluated by using an existing method such as evaluating the concentration degree using a facial expression.

The evaluation unit 218 calculates the evaluation value V using the importance of each of the sections included in the constant time S when the importance is equal to or higher than the threshold value. For example, as illustrated in FIG. 12, the evaluation unit 218 calculates the importance degree ω _i of each of the sections Si (i=1, 2,..., N) included in the constant time S by the calculation unit 16. The evaluation value V is calculated using the correlation degree a _{i of} each section. The evaluation value V can be, for example, a value obtained by dividing the inner product of the importance degree ω _i and the correlation degree a _i by the number of sections n included in the constant time S, as shown below.

V=(Σ _{(i=1 to n)} ω _i ·a _i )/n

The evaluation unit 218 outputs, as the evaluation result of the degree of concentration, a set of evaluation values V based on the degree of importance and the degree of correlation calculated for each constant time S for the time from the start to the end of the user image.

The degree-of-concentration evaluation device 210 can be realized by the computer 40 shown in FIG. 6, for example. The storage unit 43 of the computer 40 stores a concentration degree evaluation program 250 for causing the computer 40 to function as the concentration degree evaluation device 210. The concentration evaluation program 250 has a detection process 52, a generation process 54, a calculation process 56, and an evaluation process 258. In addition, the storage unit 43 has an information storage area 60 in which information configuring each of the average pattern DB 22 and the importance DB 26 is stored.

The CPU 41 reads the concentration degree evaluation program 250 from the storage unit 43, expands it in the memory 42, and sequentially executes the processes of the concentration degree evaluation program 250. The CPU 41 operates as the evaluation unit 218 shown in FIG. 9 by executing the evaluation process 258. Further, the CPU 41 reads information from the information storage area 60 and loads each of the average pattern DB 22 and the importance DB 26 in the memory 42. Other processes are the same as those of the concentration degree evaluation program 50 according to the first embodiment. As a result, the computer 40 that executes the concentration degree evaluation program 250 functions as the concentration degree evaluation device 210.

The function realized by the concentration evaluation program 250 can be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC or the like.

Next, the operation of the concentration degree evaluation device 210 according to the second embodiment will be described. In the second embodiment, the evaluation process is different from that in the first embodiment. Therefore, the evaluation process in the second embodiment will be described with reference to FIG. The same processes as those of the evaluation process (FIG. 8) in the first embodiment are designated by the same reference numerals and detailed description thereof will be omitted.

After the steps S22 and S24 of FIG. 13 are performed, when the head orientation for a certain time S is detected as the pattern of the target user, the process proceeds to the next step S32.

In step S32, the evaluation unit 218 acquires the importance ω _i of each section Si included in the constant time S of the lecture video from the importance DB 26 and calculates the average of the importance. Then, the evaluation unit 218 determines whether or not the average of the degrees of importance in the constant time S is equal to or more than a predetermined threshold value. If the average importance level is equal to or higher than the threshold value, the process proceeds to step S34, and if it is less than the threshold value, the process proceeds to step S36.

In step S36, it is determined that the degree of concentration is not evaluated for the certain time S, and the process proceeds to step S20.

In step S34, the calculation unit 16 divides the fixed time S into sections Si (i=1, 2,..., N) corresponding to the sections stored in the importance DB 26.

Next, in step S226, the calculation unit 16 reads, from the average pattern DB 22, an average pattern portion in which the time information corresponds to the pattern indicating the orientation of the head of the target user for the fixed time S. Then, the calculation unit 16 calculates the degree of correlation a _i between the pattern of the target user and the average pattern for each section Si within the fixed time S.

Next, in step S228, the evaluation unit 218 obtains the importance omega _i of each section Si contained in a predetermined time S lecture video from importance DB 26, and severity omega _i of each section Si, the step An evaluation value V is calculated using a _i calculated in S226. The evaluation unit 218 associates the time information indicating the constant time S with the calculated evaluation value V and temporarily records them in a predetermined recording area.

Hereinafter, similarly to the first embodiment, steps S20 and S22 are executed and the evaluation process ends.

As described above, the concentration degree evaluation apparatus according to the second embodiment detects the pattern indicating the movement of the head of the target user who is viewing the lecture video. The degree-of-concentration evaluation apparatus calculates a degree of correlation between an average pattern obtained by averaging patterns of a plurality of users detected when a plurality of users watch the same lecture moving image and a pattern of a target user. Then, the degree-of-concentration evaluation device evaluates the degree of concentration of the target user with respect to the viewing of the lecture video based on the degree of importance indicating the content importance of the lecture video and the calculated degree of correlation. As described above, when the user concentrates on watching a moving image, many users use the fact that their heads are lowered at the same timing, and that the feature is prominently shown at an important part of the content. This makes it possible to appropriately evaluate the degree of concentration of the user with respect to watching a moving image even when there are many occasions when the user's face is not directly facing the screen.

<Third Embodiment>
Next, a third embodiment will be described. In the concentration degree evaluation device according to the third embodiment, the same parts as those of the concentration degree evaluation device 210 according to the second embodiment are designated by the same reference numerals and detailed description thereof will be omitted.

As shown in FIG. 9, the concentration evaluation device 310 according to the third embodiment functionally includes a detection unit 12, a generation unit 314, a calculation unit 316, and an evaluation unit 318. Further, the average pattern DB 322 and the importance DB 26 are stored in a predetermined storage area of the concentration evaluation device 310.

As described above, the user often turns downward while taking a lecture video, for example, to take a note, and the timing of taking the note is the same for a plurality of users. However, the timing and the number of times to lower the head for taking notes have different tendencies for each user. Therefore, in the third embodiment, when the degree of concentration of the target user is evaluated in consideration of the difference in the tendency of the movement of the head for each user, the head of the user group having the same tendency as the target user. By comparing with the movement of, the timing at which the target user is likely to take a note is appropriately set.

The generation unit 314 classifies the patterns indicating the movements of the heads of the plurality of users into a plurality of clusters, and generates an average pattern for each cluster. Specifically, the generation unit 314 classifies each pattern of the plurality of users into a plurality of clusters according to the number of times the head is lowered and the timing when the head is lowered.

For example, the generation unit 314 calculates the number of times the head is lowered and the index indicating the timing shift for each of the patterns indicating the movements of the heads of the plurality of users. The index indicating the timing shift can be, for example, the number of times that α% (for example, 60%) of users have not lowered their heads while they are lowering their heads. Then, as shown in FIG. 14, the generation unit 314 clusters each pattern by a method such as K-means using the normalized values of the index indicating the number of times the head is lowered and the timing shift. To do. In the case of clustering by K-means, k may be around 3. Also, when there is only one element (pattern) in the cluster, the generation unit 314 combines the cluster with the nearest cluster.

In the present embodiment, a case will be described in which each pattern is clustered using an index indicating the number of times the head is lowered and the timing shift, but clustering using either one or clustering using another index may be performed. You can go.

However, as shown in FIG. 15C, in the case of clustering by the number of times the head is lowered, patterns with different head movement tendencies may be classified into the same cluster. For example, as shown by D in FIG. 15, when clustering is performed also by using the timing shift of lowering the head, patterns belonging to different clusters (patterns indicated by black circles and patterns indicated by dot circles) are the same cluster. are categorized.

Further, in the case where clustering is performed also using the shift in the timing of lowering the head, the patterns that can be classified (the pattern indicated by the white circles and the pattern indicated by the check circles) are lowered as shown by E in FIG. Sometimes it may be difficult to classify only by the number of times.

By clustering each pattern using an index indicating the number of times the head is lowered and the timing shift as in the present embodiment, the tendency of the head movement for each user can be more appropriately classified.

The generation unit 314 averages the patterns belonging to each cluster and generates an average pattern for each cluster. The average pattern generation method is the same as that in the first embodiment.

Here, if a pattern that is largely deviated from the center of the cluster is classified as the same cluster, a value that is largely deviated when the average pattern is generated from the patterns belonging to the cluster is affected. Therefore, in order to reduce the influence of the pattern greatly deviated from the cluster center and generate the average pattern, the generation unit 314 uses the pattern p*σ away from the cluster center when generating the average pattern for the cluster. Exclude. Note that p is a coefficient and σ is the variance of the cluster. The value of p is set to a larger value in a cluster in which the number of times of lowering the head is larger, considering that the timing is more likely to be shifted as the number of times of lowering the head is increased.

The generation unit 314 stores, in the average pattern DB 322, information on the patterns belonging to each cluster and the generated average pattern for each cluster. FIG. 16 shows an example of the average pattern DB 322. In the example of FIG. 16, the average pattern DB 322 includes a cluster-pattern table 322A in which information on patterns belonging to each cluster is stored, and an average pattern table 322B in which average patterns for each cluster are stored. In the cluster-pattern table 322A, "cluster" which is identification information of each cluster and "pattern" belonging to each cluster are stored in association with each other.

The calculation unit 316 calculates the degree of correlation between the average pattern generated from the patterns belonging to the cluster into which the patterns indicating the movement of the head of the target user are classified, and the pattern of the target user.

Specifically, the calculation unit 316 selects an arbitrary number of samples for each cluster from the cluster-pattern table 322A of the average pattern DB 322, and calculates the distance between the pattern of the target user and the sample. The calculation unit 316 uses the calculated distance to determine the closest cluster in K-NN (K-Nearest Neighbor), for example.

The cluster determination is not limited to the above method. For example, instead of the cluster-pattern table 322A, information about cluster centers and variances, which is information for identifying each cluster, is stored, and based on the pattern of the target user and the distance between each cluster center, the target The cluster to which the user's pattern belongs may be determined.

In addition, the calculation unit 316 evaluates information indicating that there is no corresponding cluster when the pattern of the target user is more than a predetermined distance (for example, p*σ) from any cluster center. Notify the section 318.

Similar to the evaluation unit 218 in the second embodiment, the evaluation unit 318 calculates the evaluation value V for each section Si using the correlation degree a _i with the average pattern and the importance degree ω _i .

Further, when the calculation unit 316 notifies the evaluation unit 318 of the pattern of the target user in a certain section Si that the corresponding cluster does not exist, the evaluation unit 318 sets the evaluation value V of the degree of concentration of the section Si. Set to 0.

The concentration degree evaluation apparatus 310 can be realized by, for example, the computer 40 shown in FIG. The storage unit 43 of the computer 40 stores a concentration degree evaluation program 350 for causing the computer 40 to function as the concentration degree evaluation device 310. The concentration evaluation program 350 has a detection process 52, a generation process 354, a calculation process 356, and an evaluation process 358. In addition, the storage unit 43 has an information storage area 60 in which information configuring each of the average pattern DB 322 and the importance DB 26 is stored.

The CPU 41 reads the concentration degree evaluation program 350 from the storage unit 43, expands it in the memory 42, and sequentially executes the processes of the concentration degree evaluation program 350. The CPU 41 operates as the detection unit 12 illustrated in FIG. 9 by executing the detection process 52. Further, the CPU 41 operates as the generation unit 314 illustrated in FIG. 9 by executing the generation process 354. Further, the CPU 41 operates as the calculation unit 316 shown in FIG. 9 by executing the calculation process 356. Further, the CPU 41 operates as the evaluation unit 318 shown in FIG. 9 by executing the evaluation process 358. Further, the CPU 41 reads information from the information storage area 60 and loads each of the average pattern DB 322 and the importance DB 26 in the memory 42. As a result, the computer 40 that executes the concentration degree evaluation program 350 functions as the concentration degree evaluation device 310.

The function realized by the concentration evaluation program 350 can be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC or the like.

Next, the operation of the concentration degree evaluation device 310 according to the third embodiment will be described. In the third embodiment, the evaluation process is different from that in the first and second embodiments, and therefore the evaluation process in the third embodiment will be described with reference to FIG. The same processes as the evaluation process (FIG. 8) in the first embodiment (FIG. 8) or the evaluation process in the second embodiment (FIG. 13) are denoted by the same reference numerals and detailed description thereof will be omitted.

Through steps S22 to S32 of FIG. 17, the pattern of the target user for the fixed time S is detected, and it is determined that the average of the importance ω _i of the section Si included in the fixed time S of the lecture video is equal to or more than the threshold. Then, the process proceeds to the next step S42.

In step S42, the calculation unit 316 selects an arbitrary number of samples for each cluster from the cluster-pattern table 322A of the average pattern DB 322, and calculates the distance between the pattern of the target user and the sample. Then, the calculation unit 316 uses the calculated distance to determine the closest cluster in, for example, K-NN (K-Nearest Neighbor).

Next, in step S44, the calculation unit 316 determines whether or not the pattern of the target user is away from any of the cluster centers by a predetermined distance (for example, the above p*σ) or more. Determine if the pattern is an outlier. If the pattern of the target user is an outlier, the process proceeds to step S46, and if it is not an outlier, the process proceeds to step S34.

In step S46, the calculation unit 316 notifies the evaluation unit 318 of information indicating that there is no corresponding cluster for the pattern of the target user. Then, the evaluation unit 318 sets the evaluation value V of the concentration degree for the section Si to 0 and records it in a predetermined recording area, and the process proceeds to step S20.

In step S34, the calculation unit 316 divides the fixed time S into sections Si (i=1, 2,..., N) corresponding to the sections stored in the importance DB 26.

Next, in step S326, the calculation unit 316 acquires the average pattern of the clusters determined in step S42 from the average pattern table 322B of the average pattern DB 322, and the correlation degree a _i with the pattern of the target user for each section Si. calculate.

Hereinafter, similarly to the second embodiment, steps S228, S20, and S22 are executed, and the evaluation process ends.

As described above, the concentration degree evaluation apparatus according to the third embodiment detects a pattern indicating the movement of the head of the target user who is viewing the lecture video. The degree-of-concentration evaluation apparatus clusters each pattern of a plurality of users detected when a plurality of users watch the same lecture moving image, and generates an average pattern by averaging the patterns belonging to each cluster. Then, the concentration degree evaluation device calculates the degree of correlation between the pattern of the target user and the average pattern of the cluster to which the pattern belongs. Then, the degree-of-concentration evaluation device evaluates the degree of concentration of the target user with respect to the viewing of the lecture video based on the degree of importance indicating the content importance of the lecture video and the calculated degree of correlation. This makes it possible to use the degree of correlation with the average pattern of the user group having similar head movements for the evaluation of the degree of concentration. Therefore, in addition to the effect of the second embodiment, The degree of concentration can be evaluated more appropriately in consideration of the tendency of movement.

The evaluation result of the degree of concentration obtained by each of the above embodiments may be managed in association with the identification information of the lecture video. For example, the concentration evaluation device transmits the evaluation result of the concentration to the content distribution server that distributes the lecture video, and the content distribution server stores the received evaluation result in association with the identification information of the lecture video. be able to. As a result, the content creator, the lecturer who gives a lecture, and the like can use the information on the evaluation result of the degree of concentration as feedback information.

The evaluation result is not limited to the evaluation value derived from the evaluation value DB shown in FIG. 5 and the evaluation value calculated from the correlation degree and the importance degree. For example, when the derived or calculated evaluation value is equal to or more than a predetermined threshold value, the evaluation result of “concentrated” may be output, and when the derived evaluation value is less than the threshold value, “not concentrated” may be output.

Further, in the third embodiment, the pattern of the target user may be appropriately added to the corresponding cluster, and clustering may be performed again at an arbitrary timing.

Further, when clustering patterns of a plurality of users, the pattern may be a pattern in the entire content (in the example of the above-described embodiment, the lecture moving image) or may be a pattern for each section Si. In the latter case, a cluster result as shown in FIG. 14 is generated for each section Si.

Further, in the second and third embodiments, the case where the importance is defined for each section in advance has been described, but the importance may be defined for each time of the lecture video. In this case, the average of the degrees of importance in the constant time S may be calculated as the average of the degrees of importance in each time included in the constant time. When calculating the evaluation value based on the importance and the correlation, the evaluation value should be calculated by the inner product of the average of the importance of each time included in each section and the correlation of the section. Good.

Each of the above-described embodiments can be applied to a case where a plurality of users simultaneously watch a lecture video in the same place, but the present invention is not limited to this case and can be applied to a case where each user individually watches a lecture video. it can.

Further, in the above-described embodiment, the aspect in which the concentration evaluation program is stored (installed) in the storage unit in advance has been described, but the present invention is not limited to this. The program according to the disclosed technology can be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, or a USB memory.

With respect to each of the above embodiments, the following supplementary notes are further disclosed.

(Appendix 1)
Detects the head movement of the target user who is watching the video,
Calculating a degree of correlation of the movement of the head of the target user with respect to the movement of the head of each of the plurality of users detected when the plurality of users watch the moving image;
A degree-of-concentration evaluation program for causing a computer to execute a process including evaluating the degree of concentration of the target user with respect to viewing of the moving image based on the calculated degree of correlation.

(Appendix 2)
The concentration degree evaluation program according to appendix 1, which detects a temporal change in the orientation of the head as the movement of the head.

(Appendix 3)
3. The degree-of-concentration evaluation program according to Supplementary Note 1 or Supplementary Note 2, which calculates a degree of correlation of head movements of the target user with respect to an average of head movements of the plurality of users.

(Appendix 4)
Classifying each head movement of the plurality of users into a plurality of clusters,
The correlation degree of the movement of the head of the target user with respect to the movement of the head of the user belonging to the cluster in which the movement of the head of the target user is classified is calculated. Concentration evaluation program.

(Appendix 5)
5. The degree-of-concentration evaluation program according to attachment 4, which classifies the movements of the heads of the plurality of users into the plurality of clusters according to the number of times the head is lowered and the timing of the head fall.

(Appendix 6)
Among the movements of the head of the user classified into the same cluster, the correlation degree of the movement of the head of the target user with respect to the average of the movement of the head of the user within a predetermined distance from the center of the cluster, The concentration degree evaluation program according to supplementary note 4 or supplementary note 5, which is calculated.

(Appendix 7)
Supplementary note 4 to supplementary note 6 wherein when the movement of the head of the target user is away from the center of any cluster by a predetermined distance or more, the target user is evaluated as not concentrating on viewing the moving image. The concentration degree evaluation program according to any one of 1.

(Appendix 8)
8. The degree-of-concentration evaluation program according to any one of notes 1 to 7, wherein the higher the degree of correlation, the higher the degree of concentration of the target user.

(Appendix 9)
9. The degree-of-concentration evaluation program according to any one of appendices 1 to 8, wherein the higher the degree of importance of the content of the moving image and the higher the degree of correlation, the higher the degree of concentration of the target user.

(Appendix 10)
The concentration evaluation program according to any one of appendices 1 to 9, which manages the evaluation result of the concentration in association with the identification information of the moving image.

(Appendix 11)
11. The concentration degree evaluation program according to any one of appendices 1 to 10, wherein each of the target user and the plurality of users detects the movement of the head when viewing the moving image individually.

(Appendix 12)
A detection unit that detects the movement of the head of the target user who is watching the video;
A calculating unit that calculates a degree of correlation of head movements of the target user with respect to head movements of the plurality of users detected when the plurality of users watch the moving image,
An evaluation unit that evaluates the degree of concentration of the target user with respect to viewing of the moving image, based on the calculated correlation degree;
Concentration degree evaluation device including.

(Appendix 13)
The concentration degree evaluation device according to attachment 12, wherein the detection unit detects a temporal change in the orientation of the head as the movement of the head.

(Appendix 14)
The concentration degree evaluation device according to supplementary note 12 or supplementary note 13, wherein the calculation unit calculates the degree of correlation of the movement of the head of the target user with respect to the average of the movements of the heads of the plurality of users.

(Appendix 15)
A generator for classifying the head movements of the plurality of users into a plurality of clusters,
The calculation unit calculates the degree of correlation of the movement of the head of the target user with respect to the movement of the head of the user who belongs to the cluster into which the movement of the head of the target user is classified. The degree-of-concentration evaluation device according to item 1.

(Appendix 16)
16. The concentration degree evaluation device according to appendix 15, wherein the generation unit classifies the movements of the heads of the plurality of users into the plurality of clusters in accordance with the number of times the heads are lowered and the timing at which the heads are lowered.

(Appendix 17)
Among the movements of the heads of the users classified into the same cluster, the calculation unit calculates the head movement of the target user with respect to the average movement of the heads of the users within a predetermined distance from the center of the cluster. 17. The degree-of-concentration evaluation device according to supplementary note 15 or supplementary note 16 for calculating the correlation degree of movement.

(Appendix 18)
The evaluation unit evaluates that the target user is not concentrated on viewing the video when the movement of the head of the target user is more than a predetermined distance from the center of any cluster. The concentration degree evaluation device according to any one of appendices 12 to 17.

(Appendix 19)
Detects the head movement of the target user who is watching the video,
Calculating a degree of correlation of the movement of the head of the target user with respect to the movement of the head of each of the plurality of users detected when the plurality of users watch the moving image;
A degree-of-concentration evaluation method in which a computer executes a process including evaluating a degree of concentration of the target user with respect to viewing of the moving image based on the calculated degree of correlation.

(Appendix 20)
Detects the head movement of the target user who is watching the video,
Calculating a degree of correlation of the movement of the head of the target user with respect to the movement of the head of each of the plurality of users detected when the plurality of users watch the moving image;
A storage medium that stores a concentration degree evaluation program for causing a computer to execute processing including evaluating the degree of concentration of the target user with respect to viewing of the moving image based on the calculated correlation degree.

10, 210, 310 Concentration degree evaluation device 12 Detection unit 14, 314 Generation unit 16, 316 Calculation unit 18, 218, 318 Evaluation unit 22, 322 Average pattern DB
24 Evaluation value DB
26 Importance DB
40 computer 41 CPU
42 memory 43 storage unit 49 storage medium 50, 250, 350 concentration evaluation program

Claims (12)

Detects the movement of the head of the target user who is watching the video,
Calculating a degree of correlation of the movement of the head of the target user with respect to the movement of the head of each of the plurality of users detected when the plurality of users watch the moving image;
A degree-of-concentration evaluation program for causing a computer to execute a process including evaluating the degree of concentration of the target user with respect to viewing of the moving image based on the calculated degree of correlation. The concentration degree evaluation program according to claim 1, wherein a temporal change in the orientation of the head is detected as the movement of the head. The concentration degree evaluation program according to claim 1, wherein a degree of correlation of head movements of the target user is calculated with respect to an average of head movements of the plurality of users. Classifying each head movement of the plurality of users into a plurality of clusters,
The correlation degree of the movement of the head of the target user with respect to the movement of the head of the user who belongs to the cluster in which the movement of the head of the target user is classified is calculated according to any one of claims 1 to 3. Concentration evaluation program described. The degree-of-concentration evaluation program according to claim 4, wherein the movements of the heads of the plurality of users are classified into the plurality of clusters according to the number of times the head is lowered and the timing when the head is lowered. Among the movements of the head of the user classified into the same cluster, the correlation degree of the movement of the head of the target user with respect to the average of the movement of the head of the user within a predetermined distance from the center of the cluster, The concentration degree evaluation program according to claim 4 or 5, which is calculated. 5. When the movement of the head of the target user is away from a center of any cluster by a predetermined distance or more, it is evaluated that the target user is not concentrated on viewing the moving image. Item 7. The concentration evaluation program according to any one of items 6. The concentration degree evaluation program according to any one of claims 1 to 7, wherein the higher the degree of correlation, the higher the degree of concentration of the target user. The concentration degree evaluation program according to any one of claims 1 to 8, wherein the higher the degree of importance of the content of the moving image and the higher the degree of correlation, the higher the degree of concentration of the target user. The concentration degree evaluation program according to any one of claims 1 to 9, wherein the evaluation result of the degree of concentration is managed in association with the identification information of the moving image. A detection unit that detects the movement of the head of the target user who is watching the video;
A calculating unit that calculates a degree of correlation of head movements of the target user with respect to head movements of the plurality of users detected when the plurality of users watch the moving image,
An evaluation unit that evaluates the degree of concentration of the target user with respect to viewing of the moving image, based on the calculated correlation degree;
Concentration degree evaluation device including. Detects the head movement of the target user who is watching the video,
Calculating a degree of correlation of the movement of the head of the target user with respect to the movement of the head of each of the plurality of users detected when the plurality of users watch the moving image;
A degree-of-concentration evaluation method in which a computer executes a process including evaluating a degree of concentration of the target user with respect to viewing of the moving image based on the calculated degree of correlation.