JP2019215731A - Concentration evaluation program, device, and method - Google Patents

Abstract

To appropriately evaluate a degree of concentration on viewing a moving image even when a user's line of sight deviates from a screen.SOLUTION: A detection unit 12 detects a line of sight position to a screen of a user viewing a moving image, and an evaluation unit 14 evaluates a degree of concentration of the user viewing the moving image based on a first line of sight position just before the line of sight position to the screen is no longer detected, and a second line of sight position when the line of sight to the screen is detected again after the line of sight to the screen is no longer detected.SELECTED DRAWING: Figure 2

Description

  The disclosed technology relates to a concentration evaluation program, a concentration evaluation device, and a concentration evaluation method.

  With the development of communication technology, various services for delivering high-quality moving images have been increasing. For example, as a learning content, a video recording a lecture such as a learning cram school is distributed, and a student attends the lecture by watching the video. In such a attending format, it is difficult for the instructor to monitor the attending status of each student, and it is difficult to grasp the degree of concentration of the student on the lecture.

  Thus, for example, it is conceivable to grasp the degree of concentration based on the expression of the student who claims the lecture video. However, the expression does not greatly change depending on the degree of concentration, and it is difficult to use the expression as an index of the degree of concentration. In such a situation, a technique for estimating the degree of concentration in viewing a moving image using the line of sight of a user viewing the moving image has been proposed.

  For example, an evaluation method for more accurately grasping the degree of concentration of a user on a display object has been proposed. In this method, a first point of interest of a user in a first period in which at least a first object is displayed on a screen is specified, and after the first period, a user in a second period in which at least a second object is displayed in the screen. Is specified. Then, the degree of approximation between the direction of the first vector from the reference point in the first object to the reference point in the second object and the direction of the second vector from the first point of gaze to the second point of gaze is calculated.

JP 2016-224142 A

  For example, when viewing a lecture movie as described above, there are many opportunities for the student's line of sight to be off the screen, such as when the student takes notes or looks at materials at hand distributed separately from the movie.

  In the related art, the degree of concentration is evaluated on the assumption that the line of sight exists on the screen. For this reason, when there are many opportunities for the line of sight to deviate from the screen, as in the case of viewing the lecture video, there is a possibility that the degree of concentration cannot be evaluated or that there is a lot of time to evaluate that the user is not concentrated. There is.

  Even while taking notes or looking at materials at hand, the student is likely to concentrate on the lecture, and the prior art cannot evaluate the degree of concentration in such a case appropriately.

  As one aspect, the disclosed technology aims to appropriately evaluate the degree of concentration for viewing a moving image even when the user's line of sight is off the screen.

  As one aspect, the disclosed technology detects a gaze position with respect to a screen of a user who is viewing a moving image. Then, based on the first line-of-sight position and the second line-of-sight position, the degree of concentration of the user in viewing the moving image is evaluated. The first gaze position is a gaze position immediately before the gaze position on the screen is no longer detected. The second line-of-sight position is a line-of-sight position when the line-of-sight position on the screen is no longer detected and the line-of-sight position on the screen is detected again.

  As one aspect, there is an effect that even when the user's line of sight is off the screen, the degree of concentration on viewing a moving image can be appropriately evaluated.

It is a block diagram showing a schematic structure of a concentration evaluation system concerning a 1st and 2nd embodiment. It is a functional block diagram of a concentration evaluation device concerning a 1st and 2nd embodiment. It is a figure for demonstrating the characteristic of the gaze which comes off a screen and returns to a screen again. FIG. 7 is a diagram for explaining the characteristics of the line of sight when the user is not concentrated on watching a moving image. It is a figure for demonstrating the characteristic of the gaze which goes out of a screen and returns to a screen again. It is a figure showing an example of a look position table. It is a figure showing an example of an attention area database in a 1st embodiment. FIG. 11 is a block diagram illustrating a schematic configuration of a computer that functions as a concentration evaluation apparatus according to the first and second embodiments. It is a flowchart which shows an example of the concentration evaluation process in 1st Embodiment. It is a flowchart which shows an example of an evaluation value a calculation process. It is a figure for explaining calculation of evaluation value b1 when an attention area is a figure. It is a figure for explaining calculation of evaluation value b1 when an attention area is a text. It is a figure showing an example of an attention area database in a 2nd embodiment. It is a flow chart which shows an example of concentration degree evaluation processing in a 2nd embodiment. It is a flow chart which shows an example of evaluation value b1 calculation processing. FIG. 3 is a functional block diagram of the concentration evaluation apparatus including a setting unit.

  Hereinafter, an example of an embodiment according to the disclosed technology will be described in detail with reference to the drawings. In each of the following embodiments, a case will be described as an example in which the degree of concentration of a user on viewing a lecture moving image as learning content is evaluated.

<First Embodiment>
As shown in FIG. 1, the concentration evaluation system 100 according to the first embodiment includes a concentration evaluation device 10 and a gaze measurement device 30.

  The eye gaze measuring device 30 is configured to include a visible light camera and a control unit, and is installed at a position where an area in front of the display device 36 can be photographed, such as an upper end or a lower end of the display device 36 that displays a moving image. . The eye gaze measuring device 30 measures the user's gaze direction based on an image of the user's eye region facing the screen, which is included in the image captured by the visible light camera. For example, the line-of-sight measuring device 30 measures the line-of-sight direction based on the positional relationship between a reference point (eye corner, corneal reflection point, etc.) and a moving point (iris, pupil, etc.) in the eye area. The gaze measurement device 30 outputs gaze data indicating the measured gaze direction.

  As shown in FIG. 2, the line-of-sight data output from the line-of-sight measuring device 30 is input to the concentration evaluation device 10. The degree-of-concentration evaluation device 10 detects the position of the user's line of sight with respect to the screen based on the input line-of-sight data, and outputs an evaluation result that evaluates the degree of concentration of the user in watching the moving image based on the detected line-of-sight position.

  Here, as described above, when viewing a lecture moving image as learning content, the user's line of sight may be off the screen due to the user taking notes or viewing materials at hand. In this embodiment, the degree of concentration can be appropriately evaluated even when the user's line of sight is off the screen. Hereinafter, the reason will be described.

  If the user's line of sight goes off the screen by taking notes or viewing materials at hand while the user is concentrating on the lecture, the line of sight that returns to the screen again is as shown in FIG. It was confirmed that there were features as shown.

  Specifically, from the state where the line of sight of the user 40 is on the screen 38 displayed on the display device 36 as shown at A in FIG. 3, the line of sight of the user 40 is displayed as shown at B in FIG. Is removed. Then, after that, when the line of sight of the user 40 returns to the screen 38 again, as shown in C in FIG. This is for confirming what has been written, such as when taking notes.

  Thereafter, as shown by D in FIG. 3, the line of sight moves to the part currently being explained, or as shown by E in FIG. 3, the line of sight moves to the face of the speaker (instructor). Or The line of sight may return from the state B in FIG. 3 to the state D or E in FIG. 3 without passing through the state C in FIG.

  When the degree of concentration is reduced, as shown by F in FIG. 4, before the line of sight deviates from the screen 38, the line of sight is at a position deviating from the speaker's face or the part currently being explained. There is a feature. Also, as shown by G in FIG. 4, when the line of sight deviating from the screen 38 returns to the screen 38 again, the line of sight before the line of sight deviates, the part currently being explained, or the speaker Even when the gaze does not return to any of the faces, this is a feature of the gaze when the degree of concentration is low.

  Further, the above-described features are the same even when the lecture screen is only slides, that is, when no speaker appears on the screen 38, as shown in FIG. From the state where the line of sight is located at the position indicated by H in FIG. 5, the line of sight of the user 40 deviates from the screen 38 as indicated by I in FIG. 5, and returns to the screen 38 as indicated by J in FIG. I will come. In this case, when the user 40 concentrates on watching the moving image, the line of sight returns to the line of sight position before the line of sight deviates, or to the part currently being explained.

  In the first embodiment, the degree of concentration when the line of sight deviates from the screen is evaluated using the characteristics of the line of sight as described above.

  As shown in FIG. 2, the concentration evaluation apparatus 10 functionally includes a detection unit 12 and an evaluation unit 14. An attention area database (DB) 20 is stored in a predetermined storage area of the concentration evaluation apparatus 10.

  The detection unit 12 acquires the line-of-sight data input to the concentration evaluation apparatus 10 and, based on the line-of-sight direction of the user 40 indicated by the line-of-sight data, determines the position of the line of sight of the user , On the screen 38. The detecting unit 12 sequentially records the detected line-of-sight position in a line-of-sight position table 22 as shown in FIG. 6, for example, in association with a frame number which is identification information of a frame in which the line-of-sight position is detected. In FIG. 6, when the “line of sight” is blank (in FIG. 6, indicated by “−”), no line of sight with respect to the screen 38 was detected in that frame, that is, the line of sight of the user 40 38. Hereinafter, the frame with the frame number i is referred to as “frame i”.

  The evaluation unit 14 detects the first line-of-sight position, which is the line-of-sight position immediately before the line-of-sight position on the screen 38 is no longer detected, and the line-of-sight position on the screen 38, and detects the line-of-sight position on the screen 38 again. The second line-of-sight position at the time of being performed is specified. Then, the evaluation unit 14 evaluates the degree of concentration of the user 40 in viewing the moving image based on the first line-of-sight position and the second line-of-sight position. Specifically, the evaluation unit 14 evaluates the degree of concentration based on the distance between the first gaze position and the second gaze position and the relationship between the first gaze position and the second gaze position and the attention area.

  Here, the attention area is information such as characters and figures included in a blackboard or a slide in the video, a portion where a change in the speaker's movement has occurred, a person's face displayed in the video, or a change in display. Is a part where the utterance has occurred, a part highly relevant to the utterance content in the voice information, and the like. These attention areas are manually set by the creator of the moving image, or are automatically set by using an image processing technique, and information of the set attention areas is stored in the attention area DB 20.

  FIG. 7 shows an example of the attention area DB 20. In the example of FIG. 7, information (for example, coordinates of a diagonal corner of a rectangular region of interest) for specifying a region of interest is stored as a “region of interest” for each frame specified by a “frame number”.

  More specifically, the evaluation unit 14 specifies, as the first line-of-sight position, the line-of-sight position in the frame immediately before the frame in which the line-of-sight position is blank in the line-of-sight position table 22. Further, the evaluation unit 14 specifies, as the second line-of-sight position, the line-of-sight position in the frame in which the coordinates indicating the line-of-sight position have been recorded from the blank state in the line-of-sight position table 22. In the example of FIG. 6, the evaluation unit 14 specifies the gaze position “(x2, y2)” in the frame 2 as the first gaze position and the gaze position “(xn, yn)” in the frame n as the second gaze position. .

  In addition, the evaluation unit 14 calculates an evaluation value a for evaluating the degree of concentration. At this time, the evaluation unit 14 calculates the evaluation value a so that the evaluation value a increases as the distance between the first line-of-sight position and the second line-of-sight position decreases. Further, when each of the first line-of-sight position and the second line-of-sight position is included in the predetermined region of interest, the evaluation unit 14 includes each of the first line-of-sight position and the second line-of-sight position in the predetermined region of interest. The calculation is performed such that the evaluation value a becomes higher than the case where there is no evaluation value a.

  Hereinafter, an example of calculation of the evaluation value a will be described. Here, a case where the evaluation value a of 0 to s (s is an arbitrary constant satisfying s> 0, for example, 100) will be described.

  When the first line-of-sight position is not included in the attention area, the evaluation unit 14 calculates the evaluation value a = s.

  When the second line-of-sight position is detected in the same region of interest as the region of interest including the first line-of-sight position, the evaluation unit 14 determines the distance d1 between the first line-of-sight position and the second line-of-sight position. , As the evaluation value a ′ before normalization.

  When the second line-of-sight position is detected in another region of interest different from the region of interest including the first line-of-sight position, the evaluation unit 14 determines the center position of the region of interest including the second line-of-sight position. The distance d2 between the position and the second line-of-sight position is calculated as an evaluation value a 'before normalization.

  In addition, when the first line-of-sight position is included in the region of interest but the second line-of-sight position is not included in the region of interest, the evaluation unit 14 calculates the evaluation value a ′ before normalization as a ′ = It is calculated as d1 + p * d3. Here, d3 is the shortest distance between the outer periphery of the attention area including the first line-of-sight position and the second line-of-sight position, and p is an arbitrary constant of p> 0 (for example, 0.6). The meaning of the evaluation value a ′ is obtained by adding a term “p * d3” representing a penalty for the fact that the second gaze position is out of the attention area to the distance d1 between the first gaze position and the second gaze position. It is.

  The evaluation unit 14 normalizes the evaluation value a 'before normalization from 0 to s to obtain an evaluation value a. For example, assuming that the maximum value that can be taken as the evaluation value a 'before normalization is a'max, the evaluation unit 14 normalizes by a = s * a' / a'max.

  The evaluation unit 14 sets the evaluation value a as the current degree of concentration V. In this case, the concentration V indicates that the smaller the value, the higher the concentration. The evaluation unit 14 compares the degree of concentration V with a threshold T (for example, T = s × 0.8) indicating whether or not the user 40 is concentrated on watching the moving image. The evaluation unit 14 outputs an evaluation result indicating that the user is concentrated when V> T, and outputs an evaluation result indicating that the user is not concentrated when V ≦ T.

  The concentration evaluation device 10 can be realized by, for example, a computer 50 shown in FIG. The computer 50 includes a CPU (Central Processing Unit) 51, a memory 52 as a temporary storage area, and a nonvolatile storage unit 53. In addition, the computer 50 includes an input / output I / F (Interface) 54 to which the eye gaze measuring device 30, the input device, the display device, and the like are connected, and an R / W (Read / Write) that controls reading and writing of data from and to the storage medium 59. Write) unit 55. The computer 50 includes a communication I / F 56 connected to a network such as the Internet. The CPU 51, the memory 52, the storage unit 53, the input / output I / F 54, the R / W unit 55, and the communication I / F 56 are connected to each other via a bus 57.

  The storage unit 53 can be realized by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or the like. The storage unit 53 serving as a storage medium stores a concentration evaluation program 60 for causing the computer 50 to function as the concentration evaluation device 10. The concentration evaluation program 60 has a detection process 62 and an evaluation process 64. The storage unit 53 has an information storage area 70 in which information configuring the attention area DB 20 is stored.

  The CPU 51 reads the concentration evaluation program 60 from the storage unit 53 and expands it in the memory 52, and sequentially executes the processes of the concentration evaluation program 60. The CPU 51 operates as the detection unit 12 illustrated in FIG. 2 by executing the detection process 62. The CPU 51 operates as the evaluation unit 14 illustrated in FIG. 2 by executing the evaluation process 64. The CPU 51 creates the line-of-sight position table 22 in the memory 52 when the detection process 62 is executed. Further, the CPU 51 reads information from the information storage area 70 and expands the attention area DB 20 in the memory 52. Thus, the computer 50 that has executed the concentration degree evaluation program 60 functions as the concentration degree evaluation device 10. Note that the CPU 51 that executes the program is hardware.

  The function realized by the concentration evaluation program 60 can be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit).

  Next, an operation of the concentration evaluation system 100 according to the first embodiment will be described.

  When the concentration evaluation system 100 is started, the gaze measurement device 30 starts measuring the gaze direction of the user 40, and gaze data indicating the measured gaze direction is sent from the gaze measurement device 30 to the concentration evaluation device 10. Will be entered. Then, the concentration evaluation apparatus 10 executes the concentration evaluation processing shown in FIG. The concentration evaluation process is an example of the concentration evaluation method of the disclosed technology.

  In step S12 of the concentration level evaluation process illustrated in FIG. 9, the detection unit 12 acquires the line-of-sight data input to the concentration level evaluation device 10. Then, the detection unit 12 detects the gaze position of the user 40 with respect to the screen 38 of the currently displayed frame as a coordinate position on the screen 38 based on the gaze direction of the user 40 indicated by the gaze data. Further, the detection unit 12 records the detected line-of-sight position in the line-of-sight position table 22 as shown in FIG. 6, for example, in association with the frame number of the frame in which the line-of-sight position is detected. In addition, when the gaze position with respect to the screen 38 is not detected, the detection unit 12 sets the “gaze position” in the gaze position table 22 to be blank.

  Next, in step S14, the evaluation unit 14 determines whether or not the line of sight of the user 40 has deviated from the screen 38. This determination can be made based on whether the “line-of-sight position” one frame before is not blank in the line-of-sight position table 22 and the “line-of-sight position” of the current frame is blank in step S12. If the user's line of sight does not deviate from the screen 38, the process returns to step S12. When the line of sight of the user 40 deviates from the screen 38, the evaluation unit 14 specifies the “line of sight” one frame before as the first line of sight, and the process proceeds to step S16.

  In step S16, the detection unit 12 detects the gaze position of the user 40 with respect to the screen 38 of the currently displayed frame and records it in the gaze position table 22, as in step S12.

  Next, in step S18, the evaluation unit 14 determines whether or not the line of sight of the user 40 has returned to the screen 38. This determination can be made based on whether the “line-of-sight position” one frame before in the line-of-sight position table 22 is blank, and whether the coordinates have been recorded in the “line-of-sight position” of the current frame in step S16. If the line of sight of the user 40 has not returned to the screen 38, the process returns to step S16. When the line of sight of the user 40 returns to the screen 38, the evaluation unit 14 specifies the “line of sight” of the current frame as the second line of sight, and the process proceeds to step S20.

  In step S20, an evaluation value a calculation process shown in FIG. 10 is executed.

  In step S22 of the evaluation value a calculation process illustrated in FIG. 10, the evaluation unit 14 determines whether or not the first line-of-sight position specified in step S14 is included in the attention area with reference to the attention area DB 20. . Specifically, the first line-of-sight position detected from the frame 2 is (x2, y2), and the attention area DB20 shown in FIG. 7 is used. In this case, the evaluation unit 14 determines whether or not the attention area of the frame 2 specified by “(x11, y11), (x12, y12)” includes the first line-of-sight position (x2, y2). . If the first line-of-sight position is not included in the attention area, the process proceeds to step S24, and if it is included, the process proceeds to step S26.

  In step S24, the evaluation unit 14 calculates the evaluation value a = s, and the processing returns to the concentration evaluation processing (FIG. 9).

  In step S26, the evaluation unit 14 determines whether the second line-of-sight position specified in step S18 is detected in the same region of interest as the region of interest including the first line-of-sight position determined in step S22. Is determined. The relationship between the second line-of-sight position and the attention area can be determined in the same manner as in step S22. If the second line-of-sight position is detected in the same region of interest as the first line-of-sight position, the process proceeds to step S28, and the second line-of-sight position is not detected in the same region of interest as the first line-of-sight position. In this case, the process proceeds to step S30.

  In step S28, the evaluation unit 14 calculates the distance d1 between the first line-of-sight position and the second line-of-sight position as the evaluation value a 'before normalization, and the process proceeds to step S36.

  In step S30, the evaluation unit 14 determines whether the second line-of-sight position is detected in another region of interest different from the region of interest including the first line-of-sight position. If the second line-of-sight position is included in another region of interest, the process proceeds to step S32; otherwise, the process proceeds to step S34.

  In step S32, the evaluation unit 14 calculates a distance d2 between the center position of the attention area including the second line-of-sight position and the second line-of-sight position as an evaluation value a 'before normalization, and the process proceeds to step S36. I do.

  In step S34, the evaluation unit 14 uses the distance d1 between the first line-of-sight position and the second line-of-sight position and the shortest distance d3 between the outer periphery of the attention area including the first line-of-sight position and the second line-of-sight position, The evaluation value a ′ before normalization is calculated as a ′ = d1 + p * d3, and the process proceeds to step S36.

  In step S36, the evaluation unit 14 normalizes the evaluation value a 'before normalization calculated in step S28, S32, or S34 from 0 to s to obtain an evaluation value a. Then, the process returns to the concentration evaluation process (FIG. 9).

  Next, in step S40 of the concentration evaluation process shown in FIG. 9, the evaluation unit 14 sets the evaluation value a to the current concentration V, and sets a threshold T indicating whether or not the user 40 is concentrated on watching the moving image. Compare. The evaluation unit 14 outputs an evaluation result indicating that the user is concentrated when V> T, and outputs an evaluation result indicating that the user is not concentrated when V ≦ T.

  Next, in step S42, the detection unit 12 determines whether the moving image has ended. For example, when the gaze direction is measured by the gaze measurement device 30 while the moving image is being reproduced, it is determined that the moving image has ended when gaze data is no longer input from the gaze measurement device 30. it can. If the moving image has not ended, the process returns to step S12, and if the moving image has ended, the concentration evaluation process ends.

  As described above, according to the concentration evaluation system according to the first embodiment, the concentration evaluation device performs the first gaze position immediately before the line of sight deviates from the screen and the second line of sight when the line of sight returns to the screen. The degree of concentration is evaluated based on the distance from the line of sight. The concentration evaluation device evaluates the concentration based on the relationship between each of the first line-of-sight position and the second line-of-sight position and the attention area. Accordingly, even when the user's line of sight is off the screen, the degree of concentration on viewing a moving image can be appropriately evaluated.

Second Embodiment
Next, a second embodiment will be described. In the concentration evaluation system according to the second embodiment, the same parts as those in the concentration evaluation system 100 according to the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

  As shown in FIG. 1, the concentration evaluation system 200 according to the second embodiment includes a concentration evaluation device 210 and a gaze measurement device 30.

  Functionally, the concentration evaluation device 210 includes a detection unit 12 and an evaluation unit 214, as shown in FIG. In addition, an attention area DB 220 is stored in a predetermined storage area of the concentration evaluation apparatus 210.

  In the second embodiment, in addition to the features described in the first embodiment, utilizing the features of the pattern of the movement of the line of sight before and after the line of sight departs from the screen 38 and after the line of sight returns to the screen 38, Evaluation of the degree of concentration when the line of sight deviates from the screen 38 will be described. The gaze movement patterns before the gaze deviates from the screen 38 and after the gaze returns to the screen 38 have the following characteristics.

  When taking a note on a diagram or text displayed on a blackboard or a slide, for example, to check the attention area, the gaze is focused on the diagram as a movement of the gaze before the gaze is removed from the screen 38. The feature is that the movement of the line of sight in the reading direction is stabilized in a certain direction. Further, as the movement of the line of sight after returning the line of sight to the screen 38, if the user is not concentrated on watching a moving image, the line of sight moves out of the attention area and moves greatly.

  Therefore, the evaluation unit 214 determines the evaluation value b1 for evaluating the degree of concentration, and determines the movement of the gaze at a predetermined time before the gaze position with respect to the screen 38 is no longer detected and the movement of checking the attention area. An evaluation value b1 that becomes higher as the degree of coincidence with the pattern becomes higher is calculated.

  Hereinafter, an example of calculation of the evaluation value b1 will be described. Here, a case where the evaluation value b1 of 0 to r (r is an arbitrary constant of r> 0, for example, 100) will be described.

  In the case where the attention area is a diagram, the evaluation unit 214 sets the evaluation value b1 such that the evaluation value b1 becomes higher as the overlapping range of the attention area and the range in which the eyes move in a predetermined time before the gaze position on the screen 38 is no longer detected. calculate. This captures the characteristic of moving the line of sight to the entire figure when checking the figure.

  For example, when the center position (xm, ym) of the movement of the line of sight for M seconds before the line of sight deviates from the screen 38 is within the attention area, the evaluation unit 214 determines the center position (xm, ym) of the attention area and M The distance d4 from the center of the line of sight movement in seconds is calculated as the evaluation value b1 'before normalization.

  Further, as shown in FIG. 11, the evaluation unit 214 calculates the center position (xm, ym) of the movement of the line of sight for M seconds before the line of sight deviates from the screen 38, and the variance in the x direction of the line of sight for M seconds. Calculate σx and variance σy in the y direction. Then, the evaluation unit 214 specifies a rectangular region having a length of 2σy, a width of 2σx, and a center (xm, ym) as a region in which the line of sight has moved, and determines a common region between the region of interest and the region in which the line of sight has moved in the region of interest. Is calculated as the evaluation value b1 ′ before normalization.

  Further, when the attention area is a sentence, the evaluation unit 214 increases as the difference between the variance in the vertical direction and the variance in the horizontal direction of the gaze movement for a predetermined time before the gaze position on the screen 38 is not detected is increased. Next, the evaluation value b1 is calculated. This captures the feature of moving the line of sight along the direction of the sentence when checking the sentence.

  For example, when the center position (xm, ym) of the movement of the line of sight for M seconds before the line of sight deviates from the screen 38 is within the attention area, the evaluation unit 214 determines the movement of the line of sight for every predetermined unit time during M seconds. A scan path, which is a vector to be indicated, is specified. As shown by K in FIG. 12, when a sentence is checked, an approximate scan path appears along the attention area of the sentence. That is, the variance of the movement of the line of sight in the direction along the sentence increases, and the variance of the movement of the line of sight in the direction orthogonal to the direction along the sentence decreases. On the other hand, as shown by L in FIG. 12, when the text is not confirmed, the scan path appears regardless of the attention area of the text. That is, the difference between the vertical variance and the horizontal variance of the line-of-sight movement is reduced.

  Thus, the evaluation unit 214 calculates all combinations of scan paths included in the M seconds before the line of sight deviates from the screen 38 or the cosine similarity between adjacent scan paths, and calculates the reciprocal of the average of the calculated cosine similarities. Is calculated as the evaluation value b1 ′ before normalization.

  Further, the evaluation unit 214 calculates the reciprocal of the difference between the variance σx in the x direction and the variance σy in the y direction of the gaze position for M seconds before the gaze deviates from the screen 38 as the evaluation value b1 ′ before normalization. May be. When the average of the cosine similarities of the scan paths is 0 or the difference in variance is 0, the evaluation unit 214 may set the evaluation value b1 'before normalization to b1' = r.

  As described above, in the second embodiment, the method of calculating the evaluation value b1 differs depending on the type of the attention area. Therefore, for example, as shown in FIG. 13, information of the type of the attention area is also added to the attention area DB 220, and the evaluation unit 214 refers to this information to determine whether the attention area is a figure, a sentence, or the like. May be determined.

  When the center position (xm, ym) of the movement of the line of sight for M seconds before the line of sight deviates from the screen 38 is not within the attention area, the evaluation unit 214 sets the evaluation value b1 to b1 = r. When the attention area including the center position (xm, ym) of the movement of the line of sight for M seconds before the line of sight deviates from the screen 38 is neither a figure nor a text, the evaluation unit 214 sets the evaluation value b1 to b1 = Q × r (q is an arbitrary constant of 0 <q <1, for example, 0.5). The evaluation unit 214 normalizes the evaluation value b 'before normalization from 0 to r to obtain an evaluation value b1.

  In addition, the evaluation unit 214 evaluates the evaluation value b2 for evaluating the degree of concentration, the evaluation value b2 being higher as the variance of the line-of-sight position detected during a predetermined time after the detection of the second line-of-sight position is smaller. calculate.

  For example, the evaluation unit 214 calculates the variance σx in the x direction and the variance σy in the y direction of the gaze position for N seconds after the detection of the second gaze position, and calculates the evaluation value b2 as b2 = σx + σy. I do.

  The evaluation unit 214 uses the evaluation value a calculated in the same manner as in the first embodiment, the evaluation value b1 calculated as described above, and the evaluation value b2, for example, as shown in the following equation (1). The degree V is calculated.

  Note that φ and ωi are weight constants, which are defined in advance. For example, it can be defined as φ = 0.5, ω1 = 0.3, ω2 = 0.002.

  The evaluation unit 214 outputs an evaluation result obtained by comparing the calculated degree of concentration V with the threshold T, as in the first embodiment.

  The concentration evaluation apparatus 210 can be realized by, for example, the computer 50 shown in FIG. The storage unit 53 of the computer 50 stores a concentration evaluation program 260 for causing the computer 50 to function as the concentration evaluation device 210. The concentration evaluation program 260 has a detection process 62 and an evaluation process 264. The storage unit 53 has an information storage area 70 in which information configuring the attention area DB 220 is stored.

  The CPU 51 reads out the concentration evaluation program 260 from the storage unit 53 and expands it in the memory 52, and sequentially executes the processes of the concentration evaluation program 260. The CPU 51 operates as the detection unit 12 illustrated in FIG. 2 by executing the detection process 62. The CPU 51 operates as the evaluation unit 214 illustrated in FIG. 2 by executing the evaluation process 264. The CPU 51 creates the line-of-sight position table 22 in the memory 52 when the detection process 62 is executed. Further, the CPU 51 reads information from the information storage area 70 and expands the attention area DB 220 in the memory 52. Thus, the computer 50 that has executed the concentration degree evaluation program 260 functions as the concentration degree evaluation device 210.

  The function realized by the concentration evaluation program 260 can be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC or the like.

  Next, an operation of the concentration evaluation system 200 according to the second embodiment will be described.

  In the second embodiment, the concentration evaluation apparatus 210 executes the concentration evaluation processing shown in FIG. The concentration evaluation process is an example of the concentration evaluation method of the disclosed technology. In the concentration evaluation process shown in FIG. 14, the same reference numerals are given to the same processes as the concentration evaluation process (FIG. 9) in the first embodiment, and the detailed description will be omitted.

  After step S14, in the next step S50, an evaluation value b1 calculation process shown in FIG. 15 is executed.

  In step S52 of the evaluation value b1 calculation process illustrated in FIG. 15, the evaluation unit 214 determines whether or not the user 40 is looking at the attention area before the line of sight is removed from the screen 38. Specifically, the evaluation unit 214 determines the center position (xm, ym) of the movement of the line of sight for M seconds before the line of sight deviates from the screen 38 as the region of interest included in the current frame in the region of interest DB 220. It is determined whether or not it is included in the range. If the user 40 has not seen the region of interest, the process proceeds to step S54, and if the user 40 has seen the region of interest, the process proceeds to step S56.

  In step S54, the evaluation unit 214 calculates the evaluation value b1 as b1 = r, and the processing returns to the concentration evaluation processing (FIG. 14).

  In step S56, the evaluation unit 214 specifies the type of the attention area determined to have been viewed by the user 40 in step S52 with reference to the attention area DB 220, and determines whether the identified type of the attention area is a diagram. I do. If the region of interest is a diagram, the process proceeds to step S58; if the region of interest is not a diagram, the process proceeds to step S60.

  In step S58, the evaluation unit 214 calculates the distance d4 between the center position of the attention area and the center position of the line of sight movement for M seconds before the line of sight deviates from the screen 38 as the evaluation value b1 'before normalization. I do. Alternatively, the evaluation unit 214 determines the reciprocal of the ratio of the common area of the area in which the line of sight has moved for M seconds before the line of sight deviates from the screen 38 and the attention area in the attention area by the evaluation value b1 ′ before normalization. May be calculated as Then, the process proceeds to step S66.

  In step S60, the evaluation unit 214 determines whether the type of the attention area determined to have been viewed by the user 40 in step S52 is a sentence. If the attention area is text, the process proceeds to step S62. If the attention area is not text, the process proceeds to step S64.

  In step S62, the evaluation unit 214 calculates all combinations of scan paths included in M seconds before the line of sight deviates from the screen 38, or the cosine similarity between adjacent scan paths. Then, the evaluation unit 214 calculates the reciprocal of the average of the calculated cosine similarities as the evaluation value b1 'before normalization. Alternatively, the evaluation unit 214 calculates the reciprocal of the difference between the variance σx in the x direction and the variance σy in the y direction of the gaze position for M seconds before the gaze deviates from the screen 38 as the evaluation value b1 ′ before normalization. May be. Then, the process proceeds to step S66.

  In step S64, the evaluation unit 214 calculates the evaluation value b1 as b1 = q × r, and the processing returns to the concentration evaluation processing (FIG. 14).

  In step S66, the evaluation unit 214 normalizes the evaluation value b1 'before normalization calculated in step S58 or S62 from 0 to r to obtain an evaluation value b1. Then, the process returns to the concentration evaluation process (FIG. 14).

  Returning to the concentration evaluation process shown in FIG. 14, the process goes to the next step S70 via steps S16, S18, and S20. In step S70, the evaluation unit 214 calculates the variance σx in the x direction and the variance σy in the y direction of the gaze position for N seconds after the detection of the second gaze position, and calculates the evaluation value b2 as b2 = σx + σy Is calculated.

  Next, in step S240, the evaluation unit 214 uses the evaluation value a calculated in step S20, the evaluation value b1 calculated in step S50, and the evaluation value b2 calculated in step S70, for example, (1) The degree of concentration V shown in the equation is calculated. Then, the evaluation unit 214 compares the degree of concentration V with the threshold value T, and outputs an evaluation result indicating that the user is concentrated when V> T, and outputs the evaluation result when V ≦ T. Outputs the evaluation result indicating that there is none.

  As described above, according to the concentration evaluation system according to the second embodiment, the concentration evaluation device determines the first gaze position immediately before the line of sight deviates from the screen and the second line of sight when the line of sight returns to the screen. The degree of concentration is evaluated based on the distance from the line of sight. The concentration evaluation device evaluates the concentration based on the relationship between each of the first line-of-sight position and the second line-of-sight position and the attention area. Further, the degree of concentration is evaluated based on the pattern of the movement of the gaze before the gaze deviates from the screen and after returning to the screen. Accordingly, even when the user's line of sight is off the screen, the degree of concentration on viewing a moving image can be appropriately evaluated.

  In each of the above embodiments, while the user's line of sight is on the screen, the degree of concentration may be evaluated by a conventionally known method using the line of sight with respect to the screen.

  Further, in each of the above embodiments, a case has been described in which the information of the attention area manually set by the creator of the moving image or automatically set using the image processing technique is stored in the attention area DB. , But is not limited to this.

  For example, it is conceivable that the gaze of many users who concentrate and watch is gathered in an important area of interest in a moving image. Therefore, as shown in FIG. 16, the concentration evaluation apparatus 310 is configured to include the detection unit 12, the evaluation unit 314, and the setting unit 16. Then, the evaluation unit 314 stores, in the evaluation result DB 24, the evaluation result of the degree of concentration for each of the plurality of users for each frame. Then, the setting unit 16 refers to the evaluation result DB 24, specifies a predetermined number or more of users 40 with a high degree of concentration, and specifies an area where the gaze positions of those users 40 are gathered from the gaze position table 22. . Then, the setting unit 16 may set the specified region as a region of interest and store the region in the region of interest DB 20 (or 220).

  Further, in each of the above embodiments, for simplicity of description, a case where one attention area exists in one frame has been described as an example. However, a plurality of attention areas may be included in one frame.

  Further, in each of the above-described embodiments, the case where the calculated degree of concentration is compared with the threshold value to output whether or not concentration has occurred as an evaluation result has been described. However, the present invention is not limited to this. For example, the calculated degree of concentration may be output as it is, or an evaluation result obtained by converting the calculated degree of concentration into a plurality of levels of evaluation (for example, three levels of high, medium, and low) may be output.

  Further, in the above embodiment, the mode in which the concentration evaluation program is stored (installed) in the storage unit in advance has been described, but the present invention is not limited to this. The program according to the disclosed technology can be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, and a USB memory.

  Regarding each of the above embodiments, the following supplementary notes are further disclosed.

(Appendix 1)
Detects the gaze position with respect to the screen of the user watching the video,
A first line-of-sight position that is a line-of-sight position immediately before the line-of-sight position is no longer detected, and a second line-of-sight position when the line-of-sight position for the screen is no longer detected and the line-of-sight position for the screen is detected again. A concentration evaluation program for causing a computer to execute a process including evaluating a degree of concentration of the user with respect to the video viewing based on the video.

(Appendix 2)
2. The concentration evaluation program according to claim 1, wherein the smaller the distance between the first line-of-sight position and the second line-of-sight position, the higher the degree of concentration.

(Appendix 3)
When each of the first line-of-sight position and the second line-of-sight position is included in a predetermined region of interest, compared to the case where each of the first line-of-sight position and the second line-of-sight position is not included in the region of interest, 3. The concentration evaluation program according to Supplementary Note 1 or 2, wherein the degree of concentration is highly evaluated.

(Appendix 4)
The higher the degree of coincidence between the movement of the gaze at a predetermined time before the gaze position on the screen is no longer detected and the predetermined pattern indicating the movement of checking the attention area, the higher the concentration is evaluated. 3. The concentration evaluation program according to any one of 3.

(Appendix 5)
5. The concentration evaluation program according to claim 4, wherein when the attention area is a diagram, the degree of concentration is evaluated to be higher as the range in which the line of sight moves during the predetermined time and the overlap area of the attention area are larger.

(Appendix 6)
5. The concentration evaluation program according to claim 4, wherein, when the attention area is a sentence, the degree of concentration is evaluated to be higher as the difference between the vertical variance and the horizontal variance of the line of sight movement in the predetermined time is larger.

(Appendix 7)
The concentration evaluation program according to any one of Supplementary notes 1 to 6, wherein the smaller the variance of the line-of-sight positions detected during a predetermined time after the detection of the second line-of-sight position, the higher the degree of concentration.

(Appendix 8)
8. The concentration evaluation program according to any one of Supplementary Notes 1 to 7, wherein an attention area is set based on a line of sight when the degree of concentration is evaluated to be high.

(Appendix 9)
Any one of Supplementary notes 1 to 8, wherein the display device on which the screen is displayed, or the gaze direction with respect to the screen is detected based on a gaze direction measured by a gaze measurement device attached to the user and measuring a gaze direction. Or the concentration evaluation program according to item 1.

(Appendix 10)
The concentration evaluation program according to any one of Supplementary Notes 1 to 9, wherein the line-of-sight position is detected as a coordinate value on the screen.

(Appendix 11)
A detection unit that detects a gaze position with respect to a screen of a user who is watching a video,
A first line-of-sight position that is a line-of-sight position immediately before the line-of-sight position on the screen is no longer detected, and a second line-of-sight position when the line-of-sight position on the screen is no longer detected and the line-of-sight position on the screen is detected again An evaluation unit that evaluates the degree of concentration of the user on the video viewing based on the
Concentration evaluation device including.

(Appendix 12)
12. The concentration evaluation apparatus according to claim 11, wherein the evaluation unit evaluates the concentration higher as the distance between the first line-of-sight position and the second line-of-sight position is smaller.

(Appendix 13)
The evaluation unit is configured such that, when each of the first line-of-sight position and the second line-of-sight position is included in a predetermined region of interest, each of the first line-of-sight position and the second line-of-sight position is included in the region of interest. 13. The concentration evaluation device according to appendix 11 or 12, wherein the degree of concentration is evaluated to be higher than in a case where there is no concentration.

(Appendix 14)
The evaluation unit evaluates the degree of concentration higher as the degree of coincidence between the movement of the line of sight at a predetermined time before the line of sight position on the screen is no longer detected and a predetermined pattern indicating the movement of confirming the attention area is higher. 14. The concentration evaluation apparatus according to any one of Supplementary Notes 11 to 13.

(Appendix 15)
15. The concentration evaluation apparatus according to claim 14, wherein, when the region of interest is a diagram, the degree of concentration increases as the range of overlap between the range of movement of the line of sight at the predetermined time and the region of interest increases. .

(Appendix 16)
The concentration according to Supplementary Note 14, wherein the evaluation unit evaluates the degree of concentration higher as the difference between the variance in the vertical direction and the variance in the horizontal direction of the line-of-sight movement at the predetermined time is larger when the attention area is a sentence. Degree evaluation device.

(Appendix 17)
The evaluation unit according to any one of Supplementary Note 11 to Supplementary Note 16, which evaluates the degree of concentration higher as the variance of the line-of-sight position detected at a predetermined time after the second line-of-sight position is detected is smaller. Concentration evaluation device.

(Appendix 18)
18. The degree-of-concentration evaluation apparatus according to any one of appendices 11 to 17, further comprising a setting unit configured to set a region of interest based on a gaze position when the degree of concentration is evaluated to be high.

(Appendix 19)
Detects the gaze position with respect to the screen of the user watching the video,
A first line-of-sight position that is a line-of-sight position immediately before the line-of-sight position on the screen is no longer detected, and a second line-of-sight position when the line-of-sight position on the screen is no longer detected and the line-of-sight position on the screen is detected again A computer-executable process for evaluating the degree of concentration of the user based on the video viewing based on the method.

(Appendix 20)
Detects the gaze position with respect to the screen of the user watching the video,
A first line-of-sight position that is a line-of-sight position immediately before the line-of-sight position is no longer detected, and a second line-of-sight position when the line-of-sight position for the screen is no longer detected and the line-of-sight position for the screen is detected again. A storage medium storing a concentration evaluation program for causing a computer to execute a process including evaluating the user's concentration with respect to the moving image viewing based on the video.

10, 210, 310 Concentration evaluation device 12 Detectors 14, 214, 314 Evaluator 16 Setting units 20, 220 Attention area DB
22 gaze position table 30 gaze measurement device 36 display device 38 screen 40 user 50 computer 51 CPU
52 memory 53 storage unit 59 storage medium 60, 260 concentration evaluation program 100, 200 concentration evaluation system

Claims (10)

Detects the gaze position with respect to the screen of the user watching the video,
A first line-of-sight position that is a line-of-sight position immediately before the line-of-sight position on the screen is no longer detected, and a second line-of-sight position when the line-of-sight position on the screen is no longer detected and the line-of-sight position on the screen is detected again A concentration evaluation program for causing a computer to execute a process including evaluating a degree of concentration of the user with respect to the video viewing based on the video.   The concentration degree evaluation program according to claim 1, wherein the concentration degree is evaluated to be higher as the distance between the first line-of-sight position and the second line-of-sight position is smaller.   When each of the first line-of-sight position and the second line-of-sight position is included in a predetermined region of interest, compared to the case where each of the first line-of-sight position and the second line-of-sight position is not included in the region of interest, The concentration evaluation program according to claim 1, wherein the concentration is evaluated to be high.   The concentration level is evaluated to be higher as the degree of coincidence between the movement of the line of sight at a predetermined time before the line of sight position on the screen is no longer detected and a predetermined pattern indicating the movement of confirming the attention area is higher. The concentration evaluation program according to claim 3.   The concentration evaluation program according to claim 4, wherein, when the attention area is a diagram, the degree of concentration is evaluated to be higher as the range in which the line of sight moves during the predetermined time and the overlap area of the attention area are larger.   The concentration evaluation program according to claim 4, wherein when the attention area is a sentence, the degree of concentration is evaluated to be higher as the difference between the vertical variance and the horizontal variance of the line of sight movement in the predetermined time is larger.   The degree of concentration evaluation according to any one of claims 1 to 6, wherein the degree of concentration is evaluated to be higher as the variance of the line-of-sight positions detected at a predetermined time after the detection of the second line-of-sight position is smaller. program.   The concentration evaluation program according to any one of claims 1 to 7, wherein the attention area is set based on a line of sight when the concentration is evaluated to be high. A detection unit that detects a gaze position with respect to a screen of a user who is watching a video,
A first line-of-sight position that is a line-of-sight position immediately before the line-of-sight position is no longer detected, and a second line-of-sight position when the line-of-sight position for the screen is no longer detected and the line-of-sight position for the screen is detected again. An evaluation unit that evaluates the degree of concentration of the user on the video viewing based on the
Concentration evaluation device including. Detects the gaze position with respect to the screen of the user watching the video,
A first line-of-sight position that is a line-of-sight position immediately before the line-of-sight position is no longer detected, and a second line-of-sight position when the line-of-sight position for the screen is no longer detected and the line-of-sight position for the screen is detected again. A concentration evaluation method in which a computer executes a process including evaluating a degree of concentration of the user with respect to the video viewing based on the video.