JP6583996B2 - Video evaluation apparatus and program - Google Patents

Description

  The present invention relates to a video evaluation device and a program.

  Video evaluation mainly uses indirect data such as audience rating, or uses a follow-up survey such as a questionnaire, which is a direct indicator of the evaluation of video content that changes over time. Absent.

  There is an example of measuring and analyzing the line of sight during video viewing as an index for video evaluation considering temporal variation. The movement of the line of sight is closely related to the amount of information transmitted in the video and the impression of the video after viewing, and is considered to be effective as an index for evaluating the production effect during video production. As an evaluation of the video using the movement of the line of sight, there is a method of evaluating the effect of presentation and expression from the movement of the gazing point and the distribution of the gazing point of a large number of viewers (for example, see Patent Document 1).

  In addition, a model has been proposed in which the feature amount of an image is analyzed based on a human visual information processing model, and a focused region (saliency map) is estimated (see, for example, Non-Patent Documents 1 to 3). Furthermore, a method for evaluating a video by comparing a saliency map of the video with a line-of-sight distribution has been proposed (see, for example, Patent Document 2).

JP 2007-310454 A Japanese Patent No. 5306940

L. Itti, 2 others, "A Model of Saliency-Based Visual Attention for Rapid Scene Analysis", IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 1998, VOL. 20, NO.11, p.1254-1259 O. Le Meur, 2 others, “Predicting visual fixations on video based on low-level visual features”, Vision Research 47, 2007, p.2483-2498 Neil D. B. Bruce, 1 other, "Saliency, attention, and visual search: An information theoretic approach", Journal of Vision, 2009, 9 (3), p.1-24

In the above-described conventional technology, it is impossible to quantitatively evaluate the degree to which the viewer is attracted by the characteristics of the video.
For example, in the technique of Patent Document 1, the influence of the attractiveness of the video content itself is not taken into consideration. For this reason, it is impossible to specify a factor such as whether it is due to the context of the video or that the video is guided to a highly prominent part, and there is a problem in evaluating the effect of the video expression used in the production. In addition, the model obtained by the techniques of Non-Patent Documents 1 to 3 measures the line of sight when actually watching the video, and evaluates the validity of the model from the correlation with the distribution of gazing points and the stopping time. However, the video itself has not been evaluated. The technique of Patent Document 2 has an accuracy problem because the temporal characteristics of eye movement (eye movement delay time) are not taken into consideration when comparing the line-of-sight distribution and the saliency map.

  The present invention has been made in view of such circumstances, and provides a video evaluation apparatus and a program that can accurately evaluate the degree to which a viewer is attracted by the characteristics of a video.

One aspect of the present invention provides a saccade extraction unit that identifies the frame from the start to the end of saccade based on the moving speed of the viewpoint obtained from line-of-sight data indicating the viewpoint during video viewing for each frame, Gaze area extraction that identifies the frame that is watched from the frame from the end of saccade to the start of the next saccade based on the moving speed, and extracts the gaze area based on the viewpoint in the identified frame A window function calculation unit that calculates an area window function that represents a weighting factor in the gaze area based on the distribution of the viewpoint in the frame from which the gaze area is obtained, and the frame when the saccade is ended The latency, which is the delay time from when the gaze target is determined internally until the movement of the line of sight starts, is combined with the time required for saccade. Was said frame back by time duration number of frames, the target frame extracting unit that extracts as a target frame, for each pixel on the basis of a map which calculates the value of the feature amount of each pixel of the target frame or image for each block of Using a value obtained by applying the area window function to the portion of the saliency map corresponding to the gaze area, the saliency map is calculated by calculating a saliency map in which the degree of saliency is quantified. And an attractiveness calculating unit that calculates an attractiveness level that is a representative value.
According to the present invention, the video evaluation device obtains the movement speed of the line of sight from the line-of-sight data indicating the viewpoint for each frame when the viewer views the video, and from the start to the end of the saccade based on the obtained movement speed. Identify the frame. Furthermore, the video evaluation device identifies a frame that the viewer has watched from the end of the saccade to the start of the next saccade based on the moving speed of the viewpoint, and determines the gaze area based on the position of the viewpoint in the frame. When extracted, an area window function is calculated based on the viewpoint distribution in the gaze area. The video evaluation device calculates a saliency map of the target frame that goes back by the number of frames for the latency and the time required for the saccade from the frame where the saccade is completed. The degree of attraction is calculated using the value to which the function is applied.
As a result, the video evaluation apparatus can accurately evaluate the degree to which the viewer is attracted by the characteristics of the video with a quantitative value in consideration of the temporal characteristics of the eye movement.

One aspect of the present invention is the above-described video evaluation device, wherein the window function calculation unit is a distribution in which a two-dimensional mixed normal distribution is applied to the viewpoint included in the gaze region, within a certain distance from the viewpoint. A distribution obtained by normalizing a certain area after adding up all the viewpoints included in the gaze area, or a distribution having a uniform value within a certain distance from the center of gravity of the viewpoint included in the gaze area The area window function is calculated.
According to the present invention, the video evaluation apparatus calculates an area window function such that the coefficient value applied to the saliency map of the target frame becomes higher as the position of the gaze area closer to the part where the viewer is particularly gazing. can do.
As a result, the video evaluation apparatus increases the weight of the value of the saliency map for the portion of the frame image that attracts the viewer's attention so that the viewer can obtain a high degree of attraction. It is possible to accurately evaluate the degree of being attracted by.

One aspect of the present invention is the above-described video evaluation device, in which, for each target, the target of the target is measured based on viewpoint data measured when the position of the target is temporally switched and presented. A delay time that is a difference from a time when presentation is started to a time when saccade to the target is started is calculated, and the latency is calculated by averaging the delay times calculated for the plurality of targets. And a latency extraction unit.
According to the present invention, the video evaluation device calculates a delay time, which is a difference from the presentation of the index to the time when the saccade is started, for a plurality of indices, based on the viewpoint data when the personal calibration is performed. Then, the latency is calculated by averaging the calculated delay times.
Thereby, the video evaluation apparatus can specify the frame when the gaze target is determined internally using the latency specific to each viewer, and can use it for calculating the degree of attraction. Therefore, the video evaluation apparatus can accurately evaluate the degree to which the viewer is attracted by the characteristics of the video.

One aspect of the present invention is the above-described video evaluation device, in which the gaze region extraction unit calculates a distribution in two-dimensional coordinates based on the viewpoint in the frame that has been watched, and the value of the calculated distribution is greater than or equal to a threshold value This area is extracted as a gaze area.
According to the present invention, the video evaluation device extracts a gaze region based on the position of the viewpoint in the frame gazeed after the saccade.
Thereby, the video evaluation apparatus can obtain a portion in which the saccade is induced in the frame image as a gaze area.

According to one aspect of the present invention, a saccade extracting unit that identifies a frame from the start to the end of a saccade based on a moving speed of the viewpoint obtained from line-of-sight data indicating a viewpoint during video viewing for each frame. Then, based on the moving speed, the frame that is watched from the frame from the end of saccade to the start of the next saccade is identified, and the gaze area is extracted based on the viewpoint in the identified frame Gaze area extraction means, window function calculation means for calculating an area window function representing a weighting factor in the gaze area based on the distribution of the viewpoint in the frame from which the gaze area was obtained, and when saccade is completed Latency and sucker, which is a delay time from the state in which the gaze target is internally determined until the movement of the line of sight starts from the frame. The frame traced back by the number of frame time and total time fraction of that required for de, based the target frame extracting means for extracting a target frame, the map calculated value of the feature amount of each pixel of the target frame pixel A saliency map calculating means for calculating a saliency map in which the degree of saliency is quantified for each block or image block, and a value obtained by applying the area window function to a portion of the saliency map corresponding to the gaze area And a program for functioning as an image evaluation apparatus including an attractiveness calculating unit that calculates an attractiveness that is a value that quantitatively represents attractiveness.

  According to the present invention, it is possible to accurately evaluate the degree to which a viewer is attracted by the characteristics of an image.

It is a functional block diagram which shows the structure of the image | video evaluation system by embodiment of this invention. It is a figure which shows the data structure of the gaze data by the embodiment. It is a figure which shows the position of the visual target shown on a screen at the time of personal calibration by the embodiment. It is a figure which shows the example of the time change of the optotype shown on a screen at the time of the personal calibration by the same embodiment. It is the figure which showed typically the timing of the change of a viewpoint when a visual target changes at the time of the personal calibration by the same embodiment. It is a processing flow which shows the attractiveness calculation process of the image | video evaluation apparatus by the embodiment. It is a figure which shows the extraction example of saccade and gaze by the embodiment. It is a figure which shows the determination method of the target frame by the embodiment. It is a figure which shows the example of the saccade and area window function for every scene by the embodiment. It is a figure which shows the example of the evaluation result data by the same embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
In the embodiment of the present invention, the eye movement of the viewer at the time of video viewing is measured, and attention is paid to a saccade (jumping eye movement) component in which the position of the gazing point changes greatly. The video evaluation device calculates the level of saliency around the end point of the saccade at the time of determination of the internal (mental) gaze target, and the index of the attractiveness that caused the saccade (hereinafter, “attractiveness”). Described as “degree”). The degree of attractiveness is a value that quantitatively represents the attractiveness. The video evaluation device calculates the degree of attraction for each scene of the video and the entire video from the degree of attraction calculated for all saccades obtained during the video viewing, and sets it as an evaluation value for the attractiveness of the video. The scene is from cut to cut.

  FIG. 1 is a functional block diagram showing a configuration of a video evaluation system 10 according to an embodiment of the present invention, and only functional blocks related to the present embodiment are extracted and shown. The video evaluation system 10 includes a video storage unit 1, a video playback unit 2, a line-of-sight measurement unit 3, a line-of-sight data storage unit 4, and a video evaluation device 5.

The video storage unit 1 stores video data. The video data stored in the video storage unit 1 includes video data used for latency extraction and video data to be evaluated. Saccade means moving the viewpoint quickly by moving the eyeball, and latency is the delay from when the viewer determines the target of gaze internally until saccade starts on that target. It's time.
The video playback unit 2 presents the video data stored in the video storage unit 1 to the viewer. That is, the video reproduction unit 2 reproduces the video data and displays an image of each frame of the video data on a display or a screen (hereinafter referred to as “screen”).
The line-of-sight measurement unit 3 measures eye movements when the viewer is viewing the video data, and the video reproduction unit 2 displays a frame image on the screen, and the user is turning the line of sight on the screen. Get the viewpoint that is the position of. The viewpoint is represented, for example, by coordinates when the vertical and horizontal directions of the screen are the X axis and the Y axis, respectively.
The line-of-sight data accumulating unit 4 accumulates line-of-sight data indicating the viewpoint on the screen obtained by the line-of-sight measurement unit 3 during video viewing for each frame.

The video evaluation device 5 can be realized by a computer device. The video evaluation device 5 includes a latency extraction unit 51, a saccade extraction unit 52, a gaze region extraction unit 53, a window function calculation unit 54, a target region extraction unit 55, a target frame extraction unit 56, a saliency map calculation unit 57, and an attraction A degree calculation unit 58 and a storage unit 59 are provided.
The latency extraction unit 51 uses the line-of-sight data obtained at the time of personal calibration performed prior to the video evaluation process to extract a latency that is a delay time of the viewer's eye movement. The saccade extraction unit 52 extracts a saccade component using line-of-sight data when the viewer views the video data to be evaluated. The gaze area extraction unit 53 extracts a gaze area based on the viewpoint from the frame including the saccade end point to the frame where the next saccade starts. The window function calculation unit 54 calculates an area window function representing a weighting coefficient in the gaze area based on the distribution of viewpoints in the gaze area. The target area extraction unit 55 extracts, for each saccade, a gaze area including a saccade end point as a target area. The target frame extraction unit 56 extracts, as a target frame, a frame that is temporally retroactive to the saccade latency frame extracted by the latency extraction unit 51 from the saccade start frame. The start frame of saccade is a frame that goes back from the end frame of saccade by the time frame required for saccade. That is, the target frame corresponds to a frame that is back from the frame including the saccade end point by the number of frames corresponding to the sum of the latency and the time required for saccade. The saliency map calculation unit 57 calculates a saliency map of the target frame. The attractiveness calculating unit 58 calculates a value representing the attractiveness by applying an area window function to a portion corresponding to the target area of the saliency map. The degree of attractiveness calculation unit 58 normalizes the value representing the attractiveness by the area window function, and calculates the degree of attractiveness. The storage unit 59 stores various data such as the degree of attraction obtained for each saccade. The storage unit 59 temporarily stores data used by each unit for processing.

  FIG. 2 is a diagram illustrating an example of line-of-sight data. As shown in the drawing, the line-of-sight data is data in which a frame ID (hereinafter referred to as “FID”), a time code, and viewpoint position information are associated with each other. The FID is identification information that identifies a frame of video data. The time code indicates the playback time of the frame specified by the FID. The viewpoint position information is coordinates representing a viewpoint that is a position on the screen at which the user's line of sight was directed when the image of the frame specified by the FID was presented.

Next, the operation of the video evaluation system 10 will be described.
First, the video evaluation system 10 performs personal calibration for extracting the latency prior to video evaluation.

  FIG. 3 is a diagram illustrating the position of the visual target presented on the screen during personal calibration. In the figure, the vertical position has three types of right, center, and left, and the horizontal position has three types of up, center, and bottom. The combination is different.

  FIG. 4 is a diagram illustrating an example of a temporal change in the target presented on the screen during personal calibration. The screen shown in the figure is displayed when the video playback unit 2 plays back the video data for latency extraction. In the figure, the target D5, the target D1, the target D2, the target D3, the target D6, the target D9, the target D8, the target D7, the target D4, and the target D5 are presented in this order. Each target is presented for about 2 seconds, and switches to the next target. The viewer is required to gaze at these targets, and the viewpoint changes as the targets are switched.

  FIG. 5 is a diagram schematically showing the timing of changing the viewpoint when the target changes during personal calibration. In the figure, in order to pay attention to a change in the vertical direction of the line of sight, a case where the target D1, the target D2, the target D3, the target D1, and the target D3 are displayed in this order is shown. Below the displayed images of each time-series frame, coordinate changes in the vertical direction of the viewpoint are shown. In general, the viewpoint moves with a substantially constant delay (latency) from when the gaze target is determined internally. The time when the gaze target is determined internally is the moment when the viewer is willing to pay attention to the target after the target is presented. For example, when a visual target is displayed at time t1, the viewer internally determines the visual target as a gaze target at time t2. The viewer starts moving the line of sight toward the gaze target (target) at time t3 after the latent time. Thereby, saccade is started. And saccade is complete | finished at the time t4 when the movement of the eyes | visual_axis to the gaze target (target) was complete | finished.

In personal calibration, only one point of the target is displayed in the video, so that the saliency is very high. Therefore, the display time of the target (t1) can be regarded as the time (t2) when the target of gaze is determined internally. Therefore, the latency is calculated as the time from the target display to the saccade start time (for example, time t3−time t1). Therefore, from this delay time (latency 1, latency 2, latency 3, latency 4, ...) obtained when switching and presenting each target by personal calibration, The number of frames corresponding to the average latency (hereinafter referred to as “latency equivalent number of frames”) is calculated and used in the target frame extraction process described later.
In the actual video, various gaze objects are displayed simultaneously in the video. Therefore, the moment when a certain subject is displayed in the video image does not necessarily coincide with the time determined internally as a gaze target.

The operation at the time of personal calibration in the video evaluation system 10 will be described.
The video playback unit 2 reads and plays back the latency extraction video data stored in the video storage unit 1 and presents a screen on which the displayed target changes in about 2 seconds. The line-of-sight measurement unit 3 measures the eye movement when the viewer is gazing at the target, and sets the FID and time code of the frame image at the time of measurement and the viewpoint position information of the viewpoint obtained by the measurement. The line-of-sight data is recorded in the line-of-sight data storage unit 4. The latency extraction unit 51 of the video evaluation device 5 reads the recorded line-of-sight data from the line-of-sight data storage unit 4. The latency extraction unit 51 calculates the moving speed (for example, angular velocity) of the viewpoint based on the viewpoint position information in each time code indicated by the line-of-sight data. The latency extraction unit 51 determines that the viewpoint is saccade when the moving speed of the viewpoint is equal to or higher than a predetermined speed. The latency extraction unit 51 calculates, for each target, a delay time that is a difference from the time when the presentation of the target is started to the time when the saccade to the target is started. The latency extraction unit 51 calculates the average of the delay times of all the targets and sets the latency. The latency extraction unit 51 further calculates a latency equivalent frame number that is the number of frames corresponding to the latency.

Next, an operation during video evaluation in the video evaluation system 10 will be described.
The video playback unit 2 reads and plays back video content, which is video data to be evaluated, stored in the video storage unit 1. The line-of-sight measurement unit 3 measures eye movements when the viewer is viewing the video content, and sets the FID and time code of the frame image at the time of measurement and the viewpoint position information of the viewpoint obtained by the measurement. The line-of-sight data is recorded in the line-of-sight data storage unit 4.

FIG. 6 is a processing flow showing the attractiveness calculation processing by the video evaluation device 5.
In the video storage unit 1, information indicating the correspondence between a scene of video content and the FID or playback time of the scene is registered in advance.

The saccade extraction unit 52 of the video evaluation device 5 reads the line-of-sight data generated for the video content from the line-of-sight data storage unit 4. The saccade extracting unit 52 calculates the moving speed (for example, angular velocity) of the viewpoint based on the viewpoint position information in each time code indicated by the line-of-sight data. The saccade extraction unit 52 extracts each time zone in which the calculated moving speed is equal to or higher than a predetermined speed (for example, 5 deg / s) as a time zone when the saccade is generated (step S105). The frames at the start time and end time of the time zone in which saccade occurs are the frames at the saccade start time and saccade end time, respectively. Here, the saccade that appears i-th (i is an integer equal to or greater than 1) is referred to as saccade #i. The saccade extraction unit 52 assigns each saccade a saccade ID for identifying the saccade.
In addition, the predetermined speed used at the time of personal calibration and in step S105 is the same, and is set in advance based on experience values.

The gaze region extraction unit 53 identifies a frame in which the viewpoint movement occurs at a predetermined speed or less from the frames in the time zone that are not extracted as saccades. The gaze area extraction unit 53 extracts a gaze area based on the viewpoint in the identified frame (step S110).
Specifically, the gaze area extraction unit 53 identifies a frame in which the viewpoint movement occurs at a predetermined speed or less between the frame at the end of the saccade and the frame at the start of the next saccade. The viewpoint position information of the frame is read from the line-of-sight data. The gaze area extraction unit 53 obtains a normal distribution in the two-dimensional coordinates of the X axis and the Y axis from the coordinates indicated by the read viewpoint position information, and sets an area in which the value in the normal distribution is a threshold value or more as the gaze area. Hereinafter, a gaze area extracted from a frame between the frame at the end of saccade #i and the frame at the start of saccade # (i + 1) will be referred to as gaze area #i.
The predetermined speed and threshold used in step S110 are set in advance based on experience values.

  For each gaze area, the window function calculation unit 54 calculates the distribution of weights for the gaze area from the distribution of viewpoints in the frame from which the gaze area is obtained, and obtains the area window function (step S115). Hereinafter, the area window function obtained from the gaze area #i is referred to as area window function #i.

FIG. 7 is a diagram illustrating an extraction example of saccade and gaze.
For example, it is assumed that saccade # 1 is extracted in the frames of FID “3” to FID “5” and saccade # 2 is extracted in the frames of FID “12” to FID “15”. Then, assume that frames of FID “6” to FID “11” are watched between saccade # 1 and saccade # 2, and gaze area # 1 is obtained from the viewpoint of these frames. In this case, the area window function # 1 is calculated from the viewpoint distribution of the frames of FID “6” to FID “11” in the gaze area # 1.

  The window function calculation unit 54 calculates the distribution of weights for the gaze area from the distribution of viewpoints, for example, as in the following (1) to (3), and sets it as the area window function.

(1) Distribution obtained by fitting a two-dimensional mixed normal distribution to viewpoints included in the gaze area.
(2) A distribution obtained by adding an area within a certain distance from each viewpoint to all the viewpoints included in the gaze area, and then normalizing the distribution. Examples of distances include observed measurement errors.
(3) A distribution having a uniform value within a certain distance from the center of gravity of each viewpoint included in the gaze area.

  The window function calculation unit 54 normalizes the distribution calculated by any one of the above (1) to (3) so that the total sum of the weights becomes 1, thereby obtaining an area window function.

  The target area extraction unit 55 selects one unselected saccade #i from the saccades extracted by the saccade extraction unit 52 in step S105 (step S120). For example, the target area extraction unit 55 selects unselected saccades in ascending order of i. The target area extraction unit 55 extracts the gaze area #i calculated by the gaze area extraction unit 53 as a target area (step S125).

  The target frame extraction unit 56 remarkably displays a frame that traces back the number of frames required for the saccade from the frame at the end of saccade #i and the number of frames corresponding to the latency obtained by the latency extraction unit 51. The target frame for calculating the sex map is determined (step S130). The target frame is a frame when the gaze target is determined internally. A frame that goes back the number of frames required for saccade from the frame at the end of saccade #i corresponds to the frame at the start of saccade #i. Therefore, the target frame is a frame that is retroactive by the number of frames corresponding to the latency from the frame at the start of saccade #i.

  FIG. 8 is a diagram illustrating a method for determining a target frame. This figure shows an example in which the target frame of saccade # 1 shown in FIG. 7 is determined. The target frame extraction unit 56 identifies the frame of FID “03” at the start of saccade as a frame that is retroactive from the number of frames required for saccade from the frame of FID “05” at the end of saccade. Further, the target frame extraction unit 56 determines the FID “01” frame that is temporally retroactive from the frame of the FID “03” at the start of saccade by the number of frames corresponding to the latency “2” as the target frame. The target frame extraction unit 56 outputs the FID of the target frame to the saliency map calculation unit 57. This is because it is considered that there is an image that attracts the viewer's eyes in the area including the viewpoint when the saccade is completed.

  In FIG. 6, the saliency map calculation unit 57 reads out the target frame, which is a frame image specified by the FID output from the target frame extraction unit 56 in step S130, from the video content stored in the video storage unit 1. . The saliency map calculation unit 57 calculates a saliency map of the read target frame (step S135). For calculation of the saliency map, for example, Non-Patent Documents 1 to 3 can be used, but any other saliency map calculation method may be used.

  For example, the saliency map is calculated as follows. First, for each type of feature quantity (color, luminance value, motion vector, etc.), a map is generated that calculates the feature quantity value for each pixel of the input image, and the scale of the input image is set to 1/2, 1/4. , 1/8,..., A feature value map is generated. Then, after creating a feature map that emphasizes the difference in the value of the feature value by calculating the map difference between the scales for each type of feature value, the feature map created for each type of feature value is linearly combined, A saliency map is calculated by quantifying the degree of saliency for each image or each block of an image.

  The attractiveness calculation unit 58 extracts a saliency map of a portion corresponding to the target region (gaze region #i) from the saliency map calculated by the saliency map calculation unit 57. The attractiveness calculation unit 58 calculates the weighted sum of the saliency map by adding the value of each pixel obtained by multiplying the saliency map of the target area by the weighting coefficient indicated by the area window function #i, and A value representing the sex (step S140). The degree of attractiveness calculation unit 58 normalizes the value calculated in step S140 by the sum of the area window functions to obtain the degree of attractiveness (step S145). The attractiveness calculating unit 58 stores the scene ID of the scene including the saccade #i, the saccade ID of the saccade #i, the FID of the start frame and the FID of the end frame of the saccade #i in association with each other. Write to part 59. The scene ID of the scene including saccade #i is read from the video storage unit 1.

The target area extraction unit 55 determines whether or not the processing has been completed for all saccades extracted by the saccade extraction unit 52 in step S105 (step S150). When it is determined that there is a saccade that has not been processed (step S150: NO), the target area extraction unit 55 adds 1 to the current value of i and repeats the processing from step S120.
Then, when the target area extraction unit 55 determines that the process has been completed for all saccades (step S150: YES), the process ends.

  By performing the above-described degree of attraction processing, a time-series vector relating to the degree of attraction is obtained. The attraction level calculation unit 58 calculates the average value of the attraction level for each scene or for the entire video content and sets it as the evaluation value for each scene or for the entire video content. The attractiveness calculating unit 58 generates evaluation result data indicating the calculated evaluation value for each scene or for the entire video, and writes the result into the storage unit 59 for output.

  FIG. 9 is a diagram illustrating an example of the saccade and the area window function for each scene. This figure shows four saccades with saccade IDs “01” to “04” in the scene 1 and an area window function used for calculating the degree of attraction when these saccades are selected. Similarly, saccades are obtained for scenes 2, 3 and so on, and the degree of attraction is calculated.

  FIG. 10 is a diagram illustrating an example of evaluation result data. The evaluation results shown in the figure are the scene ID of each saccade, the saccade ID, the FID of the start frame, the FID of the end frame, the degree of enticing, the degree of enticing by scene, and the degree of enticing of the entire video content. Show.

In the above-described embodiment, for example, the line-of-sight measurement unit 3 is an eye movement measurement device using a high-speed camera such as 240 fps (frame per second) or 500 fbs, and the video reproduction unit 2 has a high frame rate such as 60 fps or 120 fbs. This is suitable for a video display device. In the case of an eye movement measurement device using a high-speed camera, as shown in FIG. 5, it is possible to observe the saccade start time t3 and the saccade end time t4 separately.
However, in general, the speed of saccade is very fast, and the required time may be less than one frame (33 milliseconds) by a normal video camera with a 30 fbs frame rate. Further, in the case of calibration, the distance of the saccade is shortened because the target is moved to a relatively close place to promote eye movement. For this reason, it is considered that the eye movement measuring device that measures with a normal video camera cannot easily capture the saccade start time t3 and the saccade end time t4 separately. That is, time t3≈time t4. Therefore, the latency extraction unit 51 calculates, for each target, a delay time that is a difference from the time when the presentation of the target is started to the time when the saccade to the target ends (or the time when the target is started). To do. This delay time is a time including the latency and the time required for saccade. The latency extraction unit 51 calculates the average of the delay times of all the targets, and calculates the number of frames corresponding to the calculated average of the delay times. Then, in step S130, the target frame extraction unit 56 internally determines a target to be watched for a frame that is backed by the number of frames corresponding to the average delay time from the frame at the end of saccade #i. The target frame that is a frame is determined.

  According to the embodiment described above, the viewer can view the video from the feature amount of the line of sight at the time of viewing the video content (gaze point distribution, moving speed, etc.) and the saliency map for estimating a highly attractive area in the image. The degree of expression attracted by the saliency of the expression can be quantified to evaluate the quality of the video. As a result, it can be used as an objective evaluation value for various effects attempted during video production.

  In addition, the above-mentioned video evaluation apparatus 5 has a computer system inside. The operation process of the video evaluation apparatus 5 is stored in a computer-readable recording medium in the form of a program, and the above processing is performed by the computer system reading and executing this program. The computer system here includes a CPU, various memories, an OS, and hardware such as peripheral devices.

Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.
The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

DESCRIPTION OF SYMBOLS 1 Image | video storage part 2 Image | video reproduction | regeneration part 3 Eye-gaze measurement part 4 Eye-gaze data storage part 5 Image | video evaluation apparatus 10 Image | video evaluation system 51 Latency extraction part 52 Saccade extraction part 53 Gaze area extraction part 54 Window function calculation part 55 Target area extraction part 56 Target frame extraction unit 57 Saliency map calculation unit 58 Attraction level calculation unit 59 Storage unit

Claims (5)

A saccade extraction unit that identifies the frame from the start to the end of saccade based on the moving speed of the viewpoint obtained from the line-of-sight data indicating the viewpoint during video viewing for each frame;
A gaze area that identifies the frame that has been watched from the frame from the end of saccade to the start of the next saccade based on the moving speed, and extracts a gaze area based on the viewpoint in the identified frame An extractor;
A window function calculation unit that calculates an area window function representing a weighting coefficient in the gaze area based on the distribution of the viewpoint in the frame from which the gaze area is obtained;
From the frame at the end of saccade, from the state in which the target of gaze is determined internally to the start of movement of the line of sight, the latency is the sum of the time required for saccade and the time required for saccade A target frame extraction unit that extracts the frame that is traced back by the number of frames as a target frame;
A saliency map calculating unit that calculates a saliency map in which the degree of saliency is quantified for each pixel or each block of an image based on a map that calculates a value of a feature amount for each pixel of the target frame;
Using a value obtained by applying the area window function to the portion of the saliency map corresponding to the gaze area, and an attractiveness calculating unit that calculates an attractiveness that is a value that quantitatively represents attractiveness;
A video evaluation apparatus comprising: The window function calculating unit adds a distribution obtained by applying a two-dimensional mixed normal distribution to the viewpoint included in the gaze area, and an area within a certain distance from the viewpoint for all the viewpoints included in the gaze area. The area window function is calculated by a distribution normalized after matching, or a distribution with a uniform value within a certain distance from the center of gravity of the viewpoint included in the gaze area,
The video evaluation apparatus according to claim 1, wherein: Based on the viewpoint data measured when the target position is switched and displayed in time, saccade to the target starts from the time when the target presentation starts for each target. A latency extraction unit that calculates a delay time that is a difference up to the determined time, and calculates the latency by averaging the delay times calculated for the plurality of targets.
The video evaluation apparatus according to claim 1, wherein the video evaluation apparatus is a video evaluation apparatus. The gaze area extraction unit calculates a distribution in two-dimensional coordinates based on the viewpoint in the frame that has been watched, and extracts an area where the calculated distribution value is a threshold value or more as a gaze area.
The video evaluation apparatus according to claim 1, wherein the video evaluation apparatus is a video evaluation apparatus. Computer
Saccade extraction means for identifying the frame from the start to the end of saccade based on the moving speed of the viewpoint obtained from the line-of-sight data indicating the viewpoint during video viewing for each frame;
A gaze area that identifies the frame that has been watched from the frame from the end of saccade to the start of the next saccade based on the moving speed, and extracts a gaze area based on the viewpoint in the identified frame Extraction means;
Window function calculating means for calculating an area window function representing a weighting factor in the gaze area based on the distribution of the viewpoint in the frame from which the gaze area is obtained;
From the frame at the end of saccade, from the state in which the target of gaze is determined internally to the start of movement of the line of sight, the latency is the sum of the time required for saccade and the time required for saccade Target frame extraction means for extracting the frame that is traced back by the number of frames as a target frame;
Saliency map calculating means for calculating a saliency map in which the degree of saliency is quantified for each pixel or each block of an image based on a map for which a value of a feature amount for each pixel of the target frame is calculated;
Using a value obtained by applying the area window function to the portion of the saliency map corresponding to the gaze area, and an attraction degree calculating means for calculating an attraction degree that is a value that quantitatively represents attraction.
A program for causing a video evaluation apparatus to function.