TWI769755B - An eye movement index analysis method for moving stimulating objects on areas of interest - Google Patents

Abstract

This invention relate to an eye movement index analysis method for moving stimulating objects on areas of interest. This is capturing and recording a viewable range of user by a visual image capturing device, to generate a process video file with moving stimulating objects. And an eye tracker device detects the location and action of user’s eye synchronously, to generate an eye movement data file. Then, a processing device executes time and space synchronization with areas of interest (AOI) set by researcher and users’ eye movement, and to generate an eye tracking index file.

Description

An eye movement index analysis method for a region of interest of a dynamic stimulus

The present invention relates to an eye movement analysis method, especially an eye movement behavior analysis method that can be used to analyze a user's movable area of interest.

In the past, in order to understand the user's attention distribution and the learner's cognitive process, usually through self-report questionnaires, or through interviews or thinking aloud, and then the researchers conducted sentence analysis, but these process analysis methods are easy to use. Subjects were unable to reflect their inner true cognitive process due to memory bias or to conform to social expectations.

In order to obtain more objective behavioral data, in recent years, eye movement monitoring technology provides natural and instant measurements to explore topics such as cognition, emotion, motivation, etc. Therefore, eye tracking technology has been widely used in various fields . Among them, eye tracking is a method of using optical instruments to track and record eye movements. By observing and analyzing the trajectory of eye movements, it is possible to study the distribution of human visual perception or attention, and then infer its cognitive process. Mayer (2010) mentioned in the special issue of Learning and Instruction "Multimedia Research and Eye-tracking Data": "Eye-tracking data provide researchers with an When learning research tools, will you be able to understand what teaching material in which block (Where) works? When does it work? How long has it happened? It is more important to understand how (How) works to show the inner process of reading in more detail".

Although there have been quite a few studies using eye trackers to explore learners' attention distribution when reading text or pictures, if they are learning from dynamic videos or dynamic systems, due to dynamic stimuli and static The text or pictures are different and will not be fixed in the same position for a long time. Therefore, with the conventional analysis technology, it is impossible to analyze the eye movement of the dynamic video or dynamic system to generate the eye movement index data of the learners. Therefore, , how to provide and analyze the eye movement behavior of the learners in the movable area of interest, and obtain the eye movement index data of the subject's eyes falling into the area of interest, will be the focus and focus of this case.

One of the objectives of the present invention is to provide recording and integration of the eye movement trajectory, so as to analyze and obtain the eye movement index data of the subject's eye falling into the region of interest.

An eye movement index analysis method for a region of interest of a dynamic stimulus of the present invention is an eye movement index analysis system for a region of interest of a dynamic stimulus for analyzing the range of a subject's visual field for at least one stab The attention distribution of stimuli, the eye movement index analysis system includes a stimulus recording device, an eye movement detection device, a processing device and a display device, and the processing device is signal-connected to the The stimulus recording device, the eye movement detection device and the display device, the processing device has a stimuli and a region of interest overlay playback module, a region of interest setting module, and an image space-time synchronization module, one Time synchronization module and an eye movement index analysis module in a region of interest, wherein the eye movement analysis method includes: a) using the stimulus recording module to record the visual field content browsed by the subject B) utilize described eye movement detection device to detect and record one eye movement track of described experimenter; C) videotape is finished, produce eye movement track data and a stimulus process image file; D) utilize described The stimulation process image file is played by the superimposed playback module between the stimulus and the region of interest; e) the display device presents one of the stimulation process image files; f) the region of interest is used The setting module performs the setting of a time interval and a space area on the stimulation history image; g) generates a set file of an area of interest and an area of interest; h) uses the image space-time synchronization module to The stimulation process image file and the set file of the region of interest and the region of interest are synchronized in time and space; i) a space-time synchronization file is generated; j) the superimposed playback module of the stimulus and the region of interest is used to play the k) the display device presents a process image with a ROI frame process image file; m) using the time synchronization module to display the process image with the ROI frame process The image file is synchronized with the second time stamp of the eye movement track data; n) a time synchronization file is generated; p) the eye movement track data is corrected; q) using the ROI eye movement index analysis module to analyze the time synchronization file; and r) generating an eye movement index file.

An eye movement index analysis method for a region of interest of a dynamic stimulus of the present invention is to capture and record the user's visual field range through a stimulus recording device, and the eye movement analysis module synchronously detects the user's visual field. The eye movement track is used to generate an eye movement track data file, analyze and process it, and then synchronize the processed data in time, and obtain the eye movement index file, so as to provide the recording and integration of the eye movement track, so as to analyze and obtain the subject data. The eye movement index data of the subject's eyes falling into the region of interest, and the above purpose is achieved.

1: Stimulus recording device

2: Eye movement detection device

2': Head-mounted eye movement detection device

3: Processing device

4: Display device

31: Stimulus and area of interest overlay playback module

32: Region of interest setting module

33: Image space-time synchronization module

34: Time synchronization module

35: Area of interest eye movement index analysis module

301~319, 401~419, 3191~3193: Steps

1 is a block diagram of an eye movement analysis system for an eye movement index analysis method of a region of interest of a dynamic stimulus according to an embodiment of the present case; FIG. 2 is a block diagram of a processing module of the eye movement index analysis method of FIG. 1 Block diagram; Figure 3 is a flowchart of the eye movement index analysis method of Figure 1; Figure 4 is a block diagram of an eye movement analysis system for an eye movement index analysis method for a region of interest of a dynamic stimulus according to an embodiment of the present case Fig. 5 is the flow chart of the eye movement index analysis method of Fig. 4; Fig. 6 is the schematic diagram that the region of interest of the eye movement index analysis method of Fig. 3 and Fig. 5 is superimposed on the stimulus history image file; 7 is a flowchart of an eye movement index analysis method for a region of interest of a dynamic stimulus according to another embodiment of the present application; and FIG. 8 is a state transition of the region of interest generated by the eye movement index analysis method of FIG. 7 Schematic diagram of the table.

An eye movement index analysis method for a region of interest of a dynamic stimulus according to an embodiment of the present invention is an eye movement index analysis system for a region of interest of a dynamic stimulus for analyzing a subject's eye movement index. The attention distribution status of the visual field for at least one stimulus, as shown in Figure 1 and Figure 2, the eye movement index analysis system includes a stimulus recording device 1, an eye movement detection device 2, a processing device 3 and a display The device 4 and the processing device 3 are signal-connected to the stimulus recording device 1, the eye movement detection device 2 and the display device 4, respectively. The processing device 3 has a stimuli and a region of interest overlay playback module 31, and a region of interest setting The module 32 , an image space-time synchronization module 33 , a time synchronization module 34 and a region of interest eye movement index analysis module 35 .

In this embodiment, the stimulus recording device 1 can be a screen skimmer, which can perform content skimming on a screen (not shown) of a stimulus display module (not shown) that the user is observing , in order to capture the displayed content with the stimuli. Referring to FIG. 3, at the beginning, as in step 301, the eye movement detection device 2 is used to display the stimuli on the screen (not shown) of the stimuli display module (not shown). Perform eye movement correction in a display range (not shown), and in step 302, use the stimuli recording device 1 to start recording all the behaviors of the screen content browsed by the subject, in step 303, use the eye movement detection device 3 Detect and record an eye movement trajectory of the subject, and in step 304, after the video recording is completed, generate eye movement trajectory data and a stimulus process image file, in step 305, use the stimulus The stimulation process image file is played by the superimposed playing module 31 with the region of interest.

In step 306 , the display device 4 presents one of the stimulation history image files. In step 307 , the region of interest setting module 32 is used to set a time interval and a -spatial area for the stimulation history image, in step 308 . , generate a region of interest and region of interest profile, as in step 309, use the image spatiotemporal synchronization module 33 to synchronize the stimulation process image file, region of interest and region of interest profile in time and space, as in step 310, generate A time-space synchronization file, as in step 311, utilizes the stimulus and the ROI overlay playback module 31 to play the synchronization file, as in step 312, the display device 4 presents a process image with a ROI frame process image file, In step 313, it is determined whether another ROI and ROI profiles need to be set up. If necessary, go back to step 307 to generate another ROI and ROI profile in the same way.

If it is not needed, in step 314, use the time synchronization module 34 to synchronize the frame history image file with the region of interest with the two time stamps of the eye movement track data, in step 315, generate a time synchronization file, in step 316, use The superimposed playback module 31 of the stimuli and the region of interest determines whether the time synchronization file is successful. If it is unsuccessful, go to step 317 to perform time correction on the eye movement track data. If the synchronization is successful, go to step 318 to use the eye movement of the region of interest. The index analysis module 35 analyzes the time synchronization file, and finally, in step 319 , generates an eye movement index file.

According to another embodiment of the present invention, an eye movement index analysis method for a region of interest of a dynamic stimulus is shown in FIG. 2 , FIG. 4 and FIG. 5 . The eye movement detection device is replaced by the head-mounted eye movement detection device 2', and is used to capture the real-world image within the user's field of view. The beginning is shown in step 401. For the head-mounted eye movement The detection device 2' performs eye movement correction, and in step 402, the stimuli recording device 1 is used to start video recording of the real world where the subject browses the field, and then in step 403, the head-mounted eye movement detection device is used 2' Detect and record an eye movement track of the subject, and as in step 404, after video recording, generate eye movement track data and a stimulus process image file, as in step 405, use the stimulus and the region of interest to overlay the playback module 31 Play the stimulus process image file.

In step 406 , the display device 4 presents one of the stimulation history image files. In step 407 , the region of interest setting module 32 is used to set a time interval and a -spatial area for the stimulation history image, in step 408 . , generate a region of interest and region of interest profile, as in step 409, use the image spatiotemporal synchronization module 33 to synchronize the stimulation process image file, region of interest and region of interest profile in time and space, as in step 410, generate A time-space synchronization file, as in step 411, utilizes the stimulus and the ROI overlay playback module 31 to play the synchronization file, as in step 412, the display device 4 presents a process image with a ROI frame process image file, In step 413, it is determined whether another ROI and ROI profiles need to be reconfigured. If necessary, go back to step 407 to generate another ROI and ROI profile in the same way.

If it is not required, in step 414, use the time synchronization module 34 to synchronize the frame history image file with the region of interest with the two time stamps of the eye tracking data, in step 415, generate a time synchronization file, in step 416, use The superimposed playback module 31 of the stimuli and the region of interest determines whether the time synchronization file is successful. If it is unsuccessful, go to step 417 to perform time correction on the eye movement track data. If the synchronization is successful, go to step 418 and use the eye movement of the region of interest. The index analysis module 35 analyzes the time synchronization file, and finally, in step 419 , generates an eye movement index file.

The ROI and ROI profiles in the aforementioned two embodiments include a ROI name information, a start time information, an end time information, a region start upper left X coordinate axis information, a region start upper left y Coordinate axis information, a region start lower right X axis information, a region start lower right y axis information, a region end upper left X axis information, a region end upper left y axis information, a region end lower right X axis information, a region end right The lower y-axis information and an annotation information, as shown in Figure 6, are described according to the user's area of interest (AOI) in the format of "AOI name, start time, end time, upper left x, upper left y , lower right x, lower right y, [upper left x, upper left y, lower right x, lower right y]//annotation", such as "heart,14.3,15.5,649,172,709,248,377,18,978,655//heart displacement becomes larger", representing The AOI with the name "heart" will appear in the film between 14.3 and 15.5 seconds, and the rectangle when it first appears (14.3 seconds) is the upper left coordinate (649,172) and the lower right coordinate (709,248), and it will continue to pan in the middle. And change the size, until the end (15.5 seconds), the rectangular range of the AOI becomes the upper left (377, 18) and the lower right (978, 655). If it is a static AOI that does not move and changes size, it can be simplified as "heart, 11.4, 14.3, 649,172,709,248//heart", which means that the AOI named "heart" will appear in the video between 11.4 and 14.3 seconds, during which the size and position of the AOI are fixed, and the rectangle in the video is the upper left ( 649,172) lower right (709,248).

Since an AOI will have complex changing behaviors in the movie (such as repeated appearance and disappearance, changing size and position, etc.), it usually requires multiple lines of description to define. For example, the complete definition example of the AOI of "heart" is as follows: heart, 11.4, 14.3, 649, 172, 709, 248 //heart

heart,14.3,15.5,649,172,709,248,377,18,978,655//Heart displacement becomes larger

heart,15.5,40.4,377,18,978,655//heart

heart,43.5,57,423,31,913,636//heart

According to another embodiment of the present invention, an eye movement analysis method for an eye movement analysis system is shown in FIG. 3 and FIG. 7 together, wherein step 319 further includes the following steps. First, in step 3191, the eye movement index file is Each of the information is simplified and converted into an original data file. Next, in step 3192, referring to FIG. 8, the original data file is converted into an ROI state conversion table, wherein the state conversion table includes a start time. information, an end time information, a status information, a region of interest name information, and a duration information.

In the state transition table of the region of interest, the duration of many states is very short, and many states are less than 100 milliseconds. Therefore, the states with too short time can be deleted. For example, the gaze time of row 08 is only 6 milliseconds and will be deleted. .

Finally, as in step 3193, the eye movement index statistical file is generated, wherein the eye movement index statistical file has a set of fields, and the fields include the total gaze time of each student, the first gaze time of each area of interest, each The duration of the first gaze in the ROI, the total number of gazes per ROI, and the total gaze duration per ROI combined.

An eye movement index analysis method for a region of interest of dynamic stimuli of the present invention is to capture and record the user's visual field range by a stimulus recording device, and synchronously detect the user's eye movement track by an eye movement analysis module In order to generate an eye movement trajectory data file, each unit in the processing module will analyze, process and time synchronization, and record and integrate the user's eye movement trajectory to analyze and obtain the subject's eye falling into the region of interest. dynamic indicator data, and achieve the above goals of.

Although the present invention has been disclosed above with preferred embodiments, it is not intended to limit the present invention. Anyone skilled in the art can make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, the present invention The scope of protection shall be determined by the scope of the appended patent application.

301~319: Steps

Claims (7)

An eye movement analysis method, which is an eye movement index analysis system for a region of interest of a dynamic stimulus, for analyzing the attention distribution of a subject's visual field for at least one stimulus, the eye movement index The analysis system includes a stimulus recording device, an eye movement detection device, a processing device and a display device, the processing device is signal-connected to the stimulus recording device, the eye movement detection device and the display device, respectively. The device has a stimuli and a region of interest overlay playback module, a region of interest setting module, an image space-time synchronization module, a time synchronization module and a region of interest eye movement index analysis module, wherein the eye movement The analysis method includes: a) using the stimulus recording module to record the visual field content browsed by the subject; b) using the eye movement detection device to detect and record an eye movement track of the subject; c) video recording After finishing, generating eye movement trajectory data and a stimulation process image file; d) playing the stimulation process image file using the stimuli and the ROI overlay playback module; e) the display device presents one of the stimulation process image files on the display device history image; f) use the region of interest setting module to set a time interval and a space region for the stimulus history image; g) generate a region of interest and region of interest setting file; h) use the image to synchronize time and space The module performs time and space synchronization on the stimulation process image file, the region of interest and the region of interest configuration file, wherein the region of interest and the region of interest configuration file includes a region of interest name information, a start time information, a End time information, a region start upper left X-axis information, a region start upper left y-axis information, a region start lower right X-axis information, a region start lower right y-axis information, a region End the upper left X-coordinate axis information, a region end the upper left y-coordinate axis information, a region end the lower right X-coordinate axis information, a region end the lower right y-coordinate axis information and an annotation information; i) generate a space-time synchronization file; j) use the stimulus Playing the synchronization file with the ROI overlay playback module; k) the display device presents a process image with a ROI frame process image file; l) judging whether it is necessary to set another ROI and an ROI setting file, if it is judged necessary, then perform step f) again; m) use the time synchronization module to synchronize the frame history image file with the region of interest with the two time stamps of the eye movement track data; n) generate a time synchronization file; p) modifying the eye movement track data; q) analyzing the time synchronization file by using the eye movement index analysis module of the region of interest; and r) generating an eye movement index file. The eye movement analysis method described in claim 1, further comprising a step a') before the step a) using the eye movement detection device to display a display area of a screen of a stimulus display module Perform eye movement correction. The eye movement analysis method as described in claim 2, wherein the step a) includes a step a1) the stimulus recording device starts recording all the behaviors of the subject browsing the screen content. The eye movement analysis method as described in claim 1, wherein the eye movement detection device is a head-mounted eye movement detection device, and the eye movement analysis method further comprises a "a" before the step a). ) to perform eye movement correction on the head-mounted eye movement detection device. The eye movement analysis method as described in claim 2, wherein the step a) comprises a step a2) the stimulus recording device starts recording the real world in a field where the subject is browsing. The eye movement analysis method described in item 1 of the scope of the patent application further comprises a step o) between the step n) and the step p), using the stimulus and the region of interest overlay playback module to determine the time synchronization file Whether it is successful or not, if the judgment is unsuccessful, time correction is performed on the eye movement track data. The eye movement analysis method described in item 1 of the scope of the application, wherein step r) comprises the following steps: r1) simplify and convert each information of the eye movement index file into an original data file; r2) convert the original data The data file is converted into an ROI state transition table, and the information whose occurrence time is lower than a first predetermined time is deleted. The state transition table includes a start time information, an end time information, a state information, and a ROI name information and a duration information; and r3) produce an eye movement index statistic file, wherein the eye movement index statistic file has a set of fields including the total gaze time of each student, the when you stare for the first time time, duration of the first gaze per ROI, total number of gazes per ROI, total gaze duration per ROI combined.

Images