CN114397964B - Method and device for detecting effective fixation point, electronic equipment and storage medium - Google Patents
Method and device for detecting effective fixation point, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN114397964B CN114397964B CN202210297818.9A CN202210297818A CN114397964B CN 114397964 B CN114397964 B CN 114397964B CN 202210297818 A CN202210297818 A CN 202210297818A CN 114397964 B CN114397964 B CN 114397964B
- Authority
- CN
- China
- Prior art keywords
- effective fixation
- fixation point
- sequence
- eye movement
- speed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Eye Examination Apparatus (AREA)
Abstract
The invention discloses an effective fixation point detection method, an effective fixation point detection device, electronic equipment and a storage medium. The method comprises the following steps: performing original model training based on the collected eye movement information to obtain an effective fixation point detection model, wherein a speed threshold value used for determining an effective fixation point in the effective fixation point detection model is adaptively adjusted according to information input to the effective fixation point detection model, and the collected eye movement information comprises a position sequence and a time sequence; the effective fixation point detection model obtains an effective fixation point corresponding to the eye movement information based on the collected eye movement information; and polymerizing the effective fixation point to obtain the polymerized effective fixation point. By using the method, the detection of the effective fixation point corresponding to the eye movement information can be realized according to the established effective fixation point detection model, and meanwhile, the effective fixation point is subjected to polymerization treatment to obtain the polymerized effective fixation point, so that the accuracy of the detection of the effective fixation point is improved.
Description
Technical Field
The embodiment of the invention relates to the technical field of eye control, in particular to an effective fixation point detection method and device, electronic equipment and a storage medium.
Background
The eye tracking technique is a technique for recording the state of eye movement by receiving infrared rays reflected by the cornea and pupil by an optical receiver.
Typically, the eye movement position of each frame acquired by an optical receiver (e.g., an eye tracker) does not identify an effective gaze point, which affects the interpretation of the gaze trajectory map, the heat map, and the availability of raw data in the visualization. Accurate identification of effective gaze points is therefore extremely important for studying psychological and cognitive problems.
In the existing algorithm, the detection of the effective fixation point is mainly based on a fixed threshold. In many scenes, such as an infant eye movement test scene, the problems of low accuracy, low stability, high noise, larger individual difference and the like of an original fixation point exist, and at the moment, if the traditional algorithm with a fixed threshold value is adopted for processing, the error division of an effective fixation point is easily caused, so that the research result is influenced; in addition, many infant eye movement studies or examinations usually adopt dynamic stimulation means such as video, animation, etc., and at this time, if the conventional algorithm with a fixed threshold is adopted for processing, there may be a case that the detection performance is greatly reduced due to too large change of the watching mode.
Disclosure of Invention
The embodiment of the invention provides an effective fixation point detection method, an effective fixation point detection device, electronic equipment and a storage medium, and aims to improve the accuracy of effective fixation point detection.
In a first aspect, an embodiment of the present invention provides an effective gaze point detection method, including:
performing original model training based on the collected eye movement information to obtain an effective fixation point detection model, wherein a speed threshold value used for determining an effective fixation point in the effective fixation point detection model is adaptively adjusted according to information input to the effective fixation point detection model, and the collected eye movement information comprises a position sequence and a time sequence;
the effective fixation point detection model obtains an effective fixation point corresponding to the eye movement information based on the collected eye movement information, wherein the eye movement information comprises a position sequence and a time sequence;
and polymerizing the effective fixation point to obtain the polymerized effective fixation point.
In a second aspect, an embodiment of the present invention further provides an effective gazing point detection apparatus, including:
the system comprises a training module, a detection module and a control module, wherein the training module is used for carrying out original model training based on collected eye movement information to obtain an effective fixation point detection model, a speed threshold value used for determining an effective fixation point in the effective fixation point detection model is adaptively adjusted according to information input to the effective fixation point detection model, and the collected eye movement information comprises a position sequence and a time sequence;
the detection module is used for acquiring an effective fixation point corresponding to the eye movement information based on the acquired eye movement information by the effective fixation point detection model, and the eye movement information comprises a position sequence and a time sequence;
and the aggregation processing module is used for carrying out aggregation processing on the effective fixation point to obtain an aggregated effective fixation point.
In a third aspect, an embodiment of the present invention further provides an electronic device, including:
one or more processors;
storage means for storing one or more programs;
the one or more programs are executed by the one or more processors, so that the one or more processors implement the effective gazing point detection method provided by the embodiment of the present invention.
In a fourth aspect, the embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the effective gazing point detection method provided by the embodiment of the present invention.
The embodiment of the invention provides an effective fixation point detection method, an effective fixation point detection device, electronic equipment and a storage medium. The method comprises the following steps: performing original model training based on the collected eye movement information to obtain an effective fixation point detection model, wherein a speed threshold value used for determining an effective fixation point in the effective fixation point detection model is adaptively adjusted according to information input to the effective fixation point detection model, and the collected eye movement information comprises a position sequence and a time sequence; the effective fixation point detection model obtains an effective fixation point corresponding to the eye movement information based on the collected eye movement information, wherein the eye movement information comprises a position sequence and a time sequence; and polymerizing the effective fixation point to obtain the polymerized effective fixation point. By using the technical scheme, the detection of the effective fixation point corresponding to the eye movement information can be realized according to the established effective fixation point detection model, and meanwhile, the effective fixation point is subjected to polymerization treatment to obtain the polymerized effective fixation point, so that the accuracy of the detection of the effective fixation point is improved.
Drawings
Fig. 1 is a schematic flowchart of an effective gazing point detection method according to an embodiment of the present invention;
fig. 2 is a schematic flowchart of an effective gazing point detection method according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of an effective gaze point detection method according to a second embodiment of the present invention;
fig. 4 is a schematic structural diagram of an original model training according to a second embodiment of the present invention;
fig. 5 is a schematic structural diagram of an effective gazing point detection apparatus according to a third embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the operations (or steps) as a sequential process, many of the operations can be performed in parallel, concurrently or simultaneously. In addition, the order of the operations may be re-arranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like. In addition, the embodiments and features of the embodiments in the present invention may be combined with each other without conflict.
The term "include" and variations thereof as used herein are intended to be open-ended, i.e., "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment".
It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used for distinguishing corresponding contents, and are not used for limiting the order or interdependence relationship.
It is noted that references to "a", "an", and "the" modifications in the present invention are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that reference to "one or more" unless the context clearly dictates otherwise.
Example one
Fig. 1 is a flowchart illustrating an effective gazing point detection method according to an embodiment of the present invention, which can be applied to the case of eye tracking based on an optical receiver, and the method can be executed by an effective gazing point detection apparatus, where the apparatus can be implemented by software and/or hardware and is generally integrated on an electronic device, where the electronic device includes but is not limited to: computers, notebook computers, tablet computers, mobile phones and other devices.
It should be noted that, compared with other technologies, the eye tracking technology has the characteristics of no contact, low cost, strong portability, easy operation, and the like, and is widely applied in the fields of scientific research, clinical diagnosis, economic activities, and the like.
In the eye movement data, the fixation point can be considered to be generated by real recognition or observation of an object by human eyes, and the distance between the fixation point and the fixation point can be called eye jump, and in general, the process of the eye jump is not processed by cognition, and even a swept target is not really seen. The fixation point and eye jump (and sometimes also eye blinking) are the true content of interest in the eye movement data analysis process.
The original gaze point data corresponds to the unanalyzed data, i.e. all frames of eye movement positions collected by the eye tracker have no valid gaze point identified. The effective fixation point influences the interpretability of the fixation track graph, the heat map and the usability of the original data in the visualization, wherein circles in the fixation track graph represent the fixation point, and lines represent eye jumps. I.e., a large number of meaningless "hotness" may be stacked without identifying a valid point of regard; meanwhile, in the derived text data, it is generally necessary to identify the gaze point ID and the gaze duration, and all data points that are not recognized as valid gaze points are discarded as eye jumps. Therefore, accurate identification of an effective fixation point is extremely important for studying psychological and cognitive problems.
Based on the above, the embodiment of the invention provides an effective fixation point detection method, which is characterized in that an effective fixation point prediction model is established by acquiring the original coordinates and time sequence of the eyes of an individual and based on a machine learning method. Compared with the existing detection algorithm, the embodiment of the invention can automatically identify the threshold value based on the speed, and allows the threshold value to be different in the x and y directions, the left eye and the right eye, tasks, individuals and instruments; at the same time, a more robust analysis can be provided, particularly in the presence of significant noise; in addition, the embodiment of the invention can be operated on line in real time.
As shown in fig. 1, a method for detecting an effective gaze point according to a first embodiment of the present invention includes the following steps:
s110, training an original model based on collected eye movement information to obtain an effective fixation point detection model, wherein a speed threshold value used for determining an effective fixation point in the effective fixation point detection model is adaptively adjusted according to information input to the effective fixation point detection model, and the collected eye movement information comprises a position sequence and a time sequence.
In this embodiment, the collected eye movement information may refer to data collected based on an optical receiver, such as an eye tracker; the collected eye movement information may include a position sequence and a time sequence, and the position sequence may be understood as a coordinate sequence of the eyes when the trainer performs a certain action, for example, the coordinate sequence of the eyes when the trainer opens a certain application program; the time series may be characterized as a time series corresponding to the position series, wherein the trainer may refer to a person randomly selected when constructing the collected eye movement information.
The original model may include one or more units for training by using the collected eye movement information to obtain an effective gaze point detection model, for example, the original model may include one or more of a signal preprocessing unit, a hidden markov model unit, a minimum covariance determinant estimation unit, a multivariate control chart unit, and the like. The signal preprocessing unit may be a unit that preprocesses a position sequence and a time sequence in the acquired eye movement information; a hidden markov model unit may be a unit for classifying the velocity sequence; the minimum covariance determinant estimation unit can be used for a unit for primarily screening the classified speed sequences and the corresponding position sequences; and the multivariate control chart unit can be used for re-screening the speed sequence after primary screening to obtain a unit of screened eye movement data.
The effective gaze point detection model may refer to a model that performs effective gaze point detection based on input eye movement information. The effective fixation point detection model may include a speed threshold for determining the effective fixation point, and the speed threshold may be adaptively adjusted according to information input to the effective fixation point detection model, instead of a fixed value.
Specifically, the original model training can be performed based on the collected eye movement information to obtain an effective fixation point detection model. The embodiment does not limit the specific steps of obtaining the effective gaze point detection model, for example, a plurality of position sequences and time sequences in the collected eye movement information may be grouped, and the effective gaze point detection model is obtained by combining after training based on each group; or preprocessing the position sequence and the time sequence to obtain a speed sequence; then classifying the speed sequences to obtain classified speed sequences; then screening the classified speed sequences and the corresponding position sequences to obtain preliminarily screened position sequences, corresponding speed sequences and corresponding speed classification information; secondly, obtaining screened eye movement data according to the preliminarily screened speed classification information and the corresponding speed sequence; finally, the speed classification is re-predicted according to the eye movement data and the corresponding label data, and model parameters of the original model are adjusted until the training end condition is met so as to reduce errors in the classification; the position sequence and the time sequence can be preprocessed to obtain a speed sequence; then, the velocity sequence is classified and predicted again, and the model parameters of the original model are adjusted to obtain the model parameters, which is not limited in this embodiment.
And S120, the effective fixation point detection model obtains an effective fixation point corresponding to the eye movement information based on the collected eye movement information, wherein the eye movement information comprises a position sequence and a time sequence.
After the effective fixation point detection model is determined, the collected eye movement information can be input into the effective fixation point detection model for detection, so that an effective fixation point corresponding to the eye movement information is obtained. The method for obtaining the effective fixation point corresponding to the eye movement information is not limited in the step, and for example, the speed sequence and the model parameters of the effective fixation point detection model can be classified according to the speed to obtain speed classification information; then according to the speed classification information and the corresponding position sequence, obtaining the position sequence after preliminary screening, the corresponding speed sequence and the corresponding speed classification information; and finally, re-screening the preliminarily screened speed classification information and the corresponding speed sequence to obtain the effective fixation point corresponding to the eye movement information.
In addition, the collected eye movement information is input into the effective fixation point detection model, and eye jump data corresponding to the eye movement information can be obtained, wherein the eye jump data can be regarded as data between the effective fixation point and the effective fixation point, such as data of a blinking process.
And S130, carrying out polymerization treatment on the effective fixation points to obtain the polymerized effective fixation points.
After the effective fixation point is obtained, the effective fixation point can be subjected to polymerization treatment to obtain a polymerized effective fixation point, so that the effective fixation point is subjected to smooth treatment, and the accuracy of effective fixation point detection is further improved. The means of the aggregation process is not limited as long as the effective fixation point after aggregation can be screened out, and for example, adjacent fixation points may be smoothly aggregated by using a specific condition and a statistical method, and the specific condition and the statistical method are not expanded in this step.
The method for detecting the effective fixation point provided by the embodiment of the invention comprises the following steps: performing original model training based on the collected eye movement information to obtain an effective fixation point detection model, wherein a speed threshold value used for determining an effective fixation point in the effective fixation point detection model is adaptively adjusted according to information input to the effective fixation point detection model, and the collected eye movement information comprises a position sequence and a time sequence; the effective fixation point detection model obtains an effective fixation point corresponding to the eye movement information based on the collected eye movement information, wherein the eye movement information comprises a position sequence and a time sequence; and polymerizing the effective fixation point to obtain the polymerized effective fixation point. By using the method, the detection of the effective fixation point corresponding to the eye movement information can be realized according to the established effective fixation point detection model, and meanwhile, the effective fixation point is subjected to polymerization treatment to obtain the polymerized effective fixation point, so that the accuracy of the detection of the effective fixation point is improved.
On the basis of the above-described embodiment, a modified embodiment of the above-described embodiment is proposed, and it is to be noted herein that, in order to make the description brief, only the differences from the above-described embodiment are described in the modified embodiment.
In one embodiment, the aggregating the effective gaze point to obtain an aggregated effective gaze point includes:
and performing aggregation processing on the effective fixation points, determining the fixation points meeting the set conditions as the aggregated effective fixation points, wherein different aggregation methods correspond to different set conditions.
The setting condition may be understood as a condition that the effective fixation points are filtered again to obtain the aggregated effective fixation points, and the fixation points that satisfy the setting condition may be determined as the aggregated effective fixation points. The setting condition may be preset by a system or a related person, and this embodiment does not limit this. It is understood that different polymerization methods may correspond to different set conditions.
In one embodiment, the method further comprises:
and determining the fixation point which does not meet the set condition as the eye jump data.
In this embodiment, when the gaze point does not satisfy the setting condition, it is described that the gaze point is not the aggregated effective gaze point, and the gaze point may be determined as the eye jump data.
In one embodiment, the method further comprises:
and aggregating the eye jump data detected by the effective fixation point detection model and the eye jump data determined by the set conditions to obtain aggregated eye jump data.
In this step, the eye jump data detected by the effective gaze point detection model and the eye jump data determined by the set conditions may be aggregated to obtain aggregated eye jump data. The means of polymerization is not limited herein, and may be, for example, adjacent.
Example two
Fig. 2 is a schematic flow chart of an effective gaze point detection method according to a second embodiment of the present invention, where the second embodiment is optimized based on the foregoing embodiments.
In this embodiment, the original model training is performed on the basis of the collected eye movement information, and the effective gaze point detection model is further embodied as: inputting the position sequence and the time sequence into a signal preprocessing unit to obtain a speed sequence; and inputting the speed sequence into an algorithm reconstruction subunit of the hidden Markov model unit to obtain model parameters.
Meanwhile, in the second embodiment, the obtaining of the effective gaze point corresponding to the eye movement information by the effective gaze point detection model based on the collected eye movement information may further be embodied as: inputting the speed sequence and the model parameters into a classification subunit of the hidden Markov model unit to obtain speed classification information; inputting the speed classification information and the corresponding position sequence into a minimum covariance determinant estimation unit to obtain a position sequence after primary screening, a corresponding speed sequence and corresponding speed classification information; inputting the preliminarily screened speed classification information and the corresponding speed sequence into the multivariate control chart unit to obtain screened eye movement data, wherein different preliminarily screened speed sequences correspond to different speed thresholds, and the screened eye movement data are effective fixation points corresponding to the eye movement information.
Please refer to the first embodiment for a detailed description of the present embodiment.
As shown in fig. 2, a method for detecting an effective gaze point according to a second embodiment of the present invention includes the following steps:
and S210, inputting the position sequence and the time sequence into a signal preprocessing unit to obtain a speed sequence.
A velocity sequence is understood to be a time sequence with respect to velocity, which may be determined, for example, on the basis of displacement and time.
Specifically, the position sequence and the time sequence may be input to the signal preprocessing unit to obtain the velocity sequence. For example, by processing the position sequence, the speed of the eye movement can be calculated according to the distance between two adjacent points and the time sequence, and a speed sequence is obtained.
And S220, inputting the speed sequence into an algorithm reconstruction subunit of the hidden Markov model unit to obtain model parameters.
Hidden Markov Models (HMMs) can be considered as a statistical Model to characterize the transition of states over time that are not directly observable, e.g. the states that the Hidden Markov Model unit uses for detection include the velocity profile of the fixation point and the velocity profile of the eye jump point. The hidden markov model may include an algorithm reconstruction subunit, which may be understood as a unit for re-estimating parameters of the original model, so as to accurately train the effective gaze point detection model. The model parameters may be considered as parameters of the effective gaze point detection model, and it can be understood that different effective gaze point detection models correspond to different model parameters, and the model parameters are not limited here.
In this embodiment, the speed sequence may be input to the algorithm reconstruction subunit of the hidden markov model unit to adjust the model parameters until the training end condition is satisfied, and finally the model parameters are obtained. The algorithm reconstruction subunit may re-estimate the model parameters based on the Baum-Welch algorithm.
And S230, inputting the speed sequence and the model parameters into a classification subunit of the hidden Markov model unit to obtain speed classification information.
In this step, the hidden markov model may further include a classification subunit, which may be understood as a unit for classifying the velocity sequence. The velocity classification information may refer to information obtained by classifying velocities in the obtained velocity sequence. In this embodiment, the velocity sequence and the model parameters may be input into the classification subunit to obtain velocity classification information, for example, the velocity state in the velocity sequence may be detected and the velocity may be preliminarily classified into a low velocity and a high velocity, which correspond to the gazing point and the eye jumping point, respectively.
Illustratively, the model parameters may be estimated based on the Baum-Welch algorithm first, and the velocity classification of the eye, i.e., velocity classification information, predicted according to the Viterbi algorithm second, thereby allowing the velocity threshold to vary between eye movement directions, tasks, and individuals (rather than pre-setting a uniform velocity threshold).
S240, inputting the speed classification information and the corresponding position sequence into a minimum covariance determinant estimation unit to obtain the position sequence after primary screening, the corresponding speed sequence and the corresponding speed classification information.
A Minimum Covariance Determinant (MCD) may be understood as a statistical method for obtaining a more robust estimate to more effectively detect outliers. In this embodiment, the minimum covariance determinant estimation unit may be used to quickly screen the fixation point and the eye jump point and re-estimate the velocity threshold.
Specifically, the velocity classification information and the corresponding position sequence may be input into the minimum covariance determinant estimation unit, and the position sequence after the preliminary screening, the corresponding velocity sequence, and the corresponding velocity classification information may be obtained by screening outliers.
And S250, inputting the preliminarily screened speed classification information and the corresponding speed sequences into a multivariate control chart unit to obtain screened eye movement data, wherein different preliminarily screened speed sequences correspond to different speed thresholds, and the screened eye movement data are effective fixation points corresponding to the eye movement information.
The multivariate control chart unit can be regarded as a unit for obtaining screened eye movement data by screening again according to the speed sequence after primary screening. Specifically, the preliminarily screened speed classification information and the corresponding speed sequence may be input to the multivariate control chart unit, so as to obtain screened eye movement data, and different preliminarily screened speed sequences correspond to different speed thresholds. For example, the mean and standard deviation of the velocities in the velocity sequence can be determined to screen each of the initially screened velocity sequences.
For example, each velocity threshold may be input to a multivariate Shewhart control chart program (multivariate Shewhart control procedure) and then statistically dependent
Principle, eye location points are further classified as gaze or eye jump.
And S260, carrying out aggregation processing on the effective fixation points to obtain aggregated effective fixation points.
The effective fixation point detection method provided by the embodiment of the invention realizes the establishment of the effective fixation point detection model through training based on the signal preprocessing unit in the original model and the algorithm reconstruction subunit in the hidden markov model unit, and then improves the performance of effective fixation point detection through the classification subunit, the minimum covariance determinant estimation unit and the multivariate control chart unit, thereby improving the accuracy of effective fixation point detection.
In one embodiment, the inputting the position sequence and the time sequence into a signal preprocessing unit to obtain a velocity sequence includes:
filtering the position sequence with the attribute information of the missing signal in the position sequence through the signal preprocessing unit to obtain the filtered position sequence and the corresponding time sequence;
and converting the filtered position sequence and the corresponding time sequence into a speed sequence through the signal preprocessing unit.
It can be understood that there may be a signal missing situation in the position sequence, and each position point in the position sequence has corresponding attribute information, and the attribute information may indicate that the corresponding sequence is a non-missing signal and a missing signal, so if the attribute information in the position sequence is a missing signal, the validity of the current position point is represented, and it needs to be filtered out; if the attribute information in the position sequence is a non-missing signal, the validity of the current position point is represented to be strong, and the current position point can be reserved for subsequent operation.
In this embodiment, first, a position sequence with attribute information as a missing signal in the position sequence may be filtered by the signal preprocessing unit to obtain a filtered position sequence and a corresponding time sequence; the filtered position sequence and the corresponding time sequence can then be converted into a velocity sequence by the signal preprocessing unit, but the means for converting into a velocity sequence is not limited in this embodiment as long as the filtered position sequence and the corresponding time sequence can be converted into a velocity sequence.
For example, if the number of missing position values in the position sequence reaches a certain value, the quality of the eye movement data for training may be considered to be too poor and should be discarded.
Fig. 3 is a schematic structural diagram of an effective gaze point detection method according to a second embodiment of the present invention, and as shown in fig. 3, the information acquisition module may acquire original eye data (i.e., eye movement information) through a desktop type eye tracker. Specifically, the information acquisition module can acquire a position sequence and time sequence data (i.e., a time sequence) of the eyes, and then convert the physiological signals into digital signals through the eye tracker and transmit and store the digital signals to the acquired eye movement information.
The model training module (i.e., the original model) can preprocess the original coordinates (i.e., the position sequence and the time sequence) of the gaze point to obtain a speed signal (i.e., a speed sequence) based on the established collected eye movement information, train a classifier by using a hidden markov model to predict the speed threshold of the individual, screen outliers by using minimum covariance determinant analysis and multivariate control charts, establish a speed prediction model, and classify the eye state (i.e., the effective gaze point and the eye jump) of the individual in real time.
The effective fixation point classification module can input the information acquired by the information acquisition module into the effective fixation point classification model based on the acquired eye movement information and the effective fixation point classification model (namely, the effective fixation point detection model), and classify the eye state of the individual to the current task in real time.
The aggregation module may smoothly aggregate neighboring gazing points using a specific condition and a statistical method.
Fig. 4 is a schematic structural diagram of original model training according to a second embodiment of the present invention, and as shown in fig. 4, the model training module (i.e., the original model) includes a signal preprocessing unit, a hidden markov model unit, a minimum covariance determinant estimation unit, a multivariate control graph unit, and an algorithm reconstruction unit.
The signal preprocessing unit can process the eye position sequence (i.e. the position sequence), and calculate the speed of the eye movement by the distance and time between two adjacent points, so as to obtain a time sequence (i.e. the speed sequence) of the speed.
Hidden Markov Model (HMM), a Hidden Markov Model unit, is a statistical Model that characterizes the transition of states over time that are not directly observable. A two-state HMM (velocity profile representing point of gaze and eye jump, respectively) is defined, where the hidden state is the velocity classification (low, high) and the observed state is the velocity signature, first estimating the model parameters based on the Baum-Welch algorithm, and second predicting the velocity classification of the eye according to the Viterbi algorithm, allowing the velocity thresholds to be different between eye movement direction, task and individual (instead of presetting a uniform velocity threshold).
The Minimum Covariance Determinant estimation unit, MCD, is a statistical method to obtain a more robust estimator to achieve more efficient detection of outliers. MCD can be used to quickly screen fixation points and eye jump points and estimate velocity thresholds separately.
The multivariate control graph element may input these velocity thresholds to a multivariate Shewhart control graph program (multivariate Shewhart control procedure), based on statistics
Principle, the eye position point is preliminarily classified as gaze or eye jump.
The algorithm reconstruction unit is capable of re-estimating the model probability parameters based on the Baum-Welch algorithm and re-predicting the velocity classification of the eye based on the Viterbi algorithm to minimize errors in state assignment. The parameter re-estimation may be performed as many times as necessary.
It should be noted that the algorithm reconstruction unit may be integrated in the hidden markov model unit, or may be disposed behind the multivariate control chart unit. The different means for setting the position adjustment model parameters are different, and are not limited herein, for example, if the algorithm reconstruction unit is disposed behind the multivariate control chart unit, the model parameters may be adjusted based on the classification result. When the algorithm reconstruction unit is arranged in the hidden markov model unit, the algorithm reconstruction unit can be called as an algorithm reconstruction subunit.
The following describes an exemplary effective gazing point detection method.
Firstly, using the desktop eye tracker, the platform can select the platform corresponding to the desktop eye tracker, measure the eye movement information of the subject, and derive the eye coordinate sequence and the time sequence (i.e. the position sequence and the time sequence) through the platform interface.
Then, based on the collected eye movement information, the position signal (i.e., the position sequence) and the time signal (i.e., the time sequence) are preprocessed, and a velocity sequence is calculated from the distance and time between the two points. Establishing a hidden Markov model, and defining a two-state HMM (respectively representing the speed distribution of a fixation point and an eye jump point), wherein the hidden state is a speed classification (low speed and high speed), and the observation state is a speed signal; model parameters were estimated based on the Baum-Welch algorithm.
Secondly, the velocity sequence and the model parameters are input into a classification subunit of a hidden Markov model unit, and the velocity classification of the eyes is predicted according to a Viterbi algorithm (namely velocity classification information is obtained). And (4) rapidly screening outliers by using a minimum covariance determinant statistical method, and re-estimating a speed threshold (namely obtaining a position sequence after primary screening, a corresponding speed sequence and corresponding speed classification information).
Then, inputting the primarily screened speed classification information and the corresponding speed sequence into a multi-element Shewhart control chart (i.e. a multi-element control chart unit), and carrying out statistics according to the speed classification information and the corresponding speed sequence
Principle, screening data to obtainAnd obtaining the screened eye movement data as the effective fixation point corresponding to the eye movement information. Finally, the classified gaze points may be smoothly aggregated.
Through the description, the effective fixation point detection method provided by the embodiment of the invention allows the speed threshold to be different according to people and tasks, provides a more strict standard for predicting the effective fixation point, and has an obvious effective fixation point detection effect aiming at scenes with large fixation point change, such as the characteristics of strong individual difference and much noise in the eye movement task of infants.
Meanwhile, the effective fixation point detection method based on the embodiment of the invention can obtain more stable and accurate fixation point division, thereby ensuring that psychological and cognitive experiments are more reliable and effective, and providing guarantee for psychological and cognitive analysis and clinical diagnosis application based on the eye movement technology.
EXAMPLE III
Fig. 5 is a schematic structural diagram of an effective gazing point detection apparatus according to a third embodiment of the present invention, which is suitable for use in a case of eye tracking based on an optical receiver, wherein the apparatus can be implemented by software and/or hardware and is generally integrated on an electronic device.
As shown in fig. 5, the apparatus includes:
a training module 310, configured to perform original model training based on collected eye movement information to obtain an effective gaze point detection model, where a speed threshold for determining an effective gaze point in the effective gaze point detection model is adaptively adjusted according to information input to the effective gaze point detection model, and the collected eye movement information includes a position sequence and a time sequence;
a detection module 320, configured to obtain, by the effective gaze point detection model, an effective gaze point corresponding to the eye movement information based on the collected eye movement information, where the eye movement information includes a position sequence and a time sequence;
and the aggregation processing module 330 is configured to perform aggregation processing on the effective gaze point to obtain an aggregated effective gaze point.
In this embodiment, the device performs original model training based on the collected eye movement information through a training module 310 to obtain an effective gaze point detection model, where a speed threshold for determining an effective gaze point in the effective gaze point detection model is adaptively adjusted according to information input to the effective gaze point detection model, and the collected eye movement information includes a position sequence and a time sequence; obtaining, by the detection module 320, an effective gaze point corresponding to the eye movement information based on the collected eye movement information by the effective gaze point detection model, where the eye movement information includes a position sequence and a time sequence; and performing aggregation processing on the effective fixation point through an aggregation processing module 330 to obtain an aggregated effective fixation point. By the device, the detection of the effective fixation point corresponding to the eye movement information can be realized according to the established effective fixation point detection model, and meanwhile, the effective fixation point is subjected to polymerization treatment to obtain the polymerized effective fixation point, so that the accuracy of the detection of the effective fixation point is improved.
Further, the original model comprises a signal preprocessing unit, a hidden Markov model unit, a minimum covariance determinant estimation unit and a multivariate control chart unit; accordingly, training module 310 includes:
the speed sequence unit is used for inputting the position sequence and the time sequence into the signal preprocessing unit to obtain a speed sequence;
the model parameter unit is used for inputting the speed sequence into an algorithm reconstruction subunit of the hidden Markov model unit to obtain model parameters;
accordingly, the detection module 320 includes:
the speed classification information unit is used for inputting the speed sequence and the model parameters into a classification subunit of the hidden Markov model unit to obtain speed classification information;
the preliminary screening unit is used for inputting the speed classification information and the corresponding position sequence into the minimum covariance determinant estimation unit to obtain the preliminarily screened position sequence, the corresponding speed sequence and the corresponding speed classification information;
the eye movement data unit is used for inputting the primarily screened speed classification information and the corresponding speed sequences into the multivariate control chart unit to obtain screened eye movement data, different primarily screened speed sequences correspond to different speed thresholds, and the screened eye movement data are effective fixation points corresponding to the eye movement information;
further, the states detected by the hidden markov model unit include the velocity distribution of the fixation point and the velocity distribution of the eye jump point.
Further, the speed sequence unit is specifically configured to:
filtering the position sequence with the attribute information of the missing signal in the position sequence through the signal preprocessing unit to obtain a filtered position sequence and a corresponding time sequence;
and converting the filtered position sequence and the corresponding time sequence into a speed sequence through the signal preprocessing unit.
Further, the aggregation processing module 330 includes:
and performing aggregation processing on the effective fixation points, determining the fixation points meeting the set conditions as the aggregated effective fixation points, wherein different aggregation methods correspond to different set conditions.
Further, the apparatus further comprises: a determination module to:
and determining the fixation point which does not meet the set condition as the eye jump data.
Further, the apparatus further comprises: an aggregation module to:
and aggregating the eye jump data detected by the effective fixation point detection model and the eye jump data determined by the set conditions to obtain aggregated eye jump data.
The effective fixation point detection device can execute the effective fixation point detection method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
Example four
Fig. 6 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present invention. As shown in fig. 6, an electronic device provided in the fourth embodiment of the present invention includes: one or more processors 41 and storage 42; the processor 41 in the electronic device may be one or more, and one processor 41 is taken as an example in fig. 6; storage 42 is used to store one or more programs; the one or more programs are executed by the one or more processors 41 such that the one or more processors 41 implement an effective point of regard detection method as described in any of the embodiments of the present invention.
The electronic device may further include: an input device 43 and an output device 44.
The processor 41, the storage device 42, the input device 43 and the output device 44 in the electronic apparatus may be connected by a bus or other means, and the connection by the bus is exemplified in fig. 6.
The storage device 42 in the electronic device is used as a computer-readable storage medium for storing one or more programs, which may be software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the effective gazing point detecting method provided in one or two embodiments of the present invention (for example, the modules in the effective gazing point detecting apparatus shown in fig. 5 include a training module 310, a detecting module 320, and an aggregation processing module 330). The processor 41 executes various functional applications and data processing of the electronic device by running software programs, instructions and modules stored in the storage device 42, that is, implements the effective gaze point detection method in the above-described method embodiments.
The storage device 42 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to use of the electronic device, and the like. Further, the storage 42 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, the storage 42 may further include memory located remotely from the processor 41, which may be connected to the device over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 43 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function controls of the electronic apparatus. The output device 44 may include a display device such as a display screen.
And, when the one or more programs included in the above-mentioned electronic device are executed by the one or more processors 41, the programs perform the following operations:
performing original model training based on the collected eye movement information to obtain an effective fixation point detection model, wherein a speed threshold value used for determining an effective fixation point in the effective fixation point detection model is adaptively adjusted according to information input to the effective fixation point detection model, and the collected eye movement information comprises a position sequence and a time sequence;
the effective fixation point detection model obtains an effective fixation point corresponding to the eye movement information based on the collected eye movement information, wherein the eye movement information comprises a position sequence and a time sequence;
and carrying out polymerization treatment on the effective fixation points to obtain the polymerized effective fixation points.
EXAMPLE five
An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to perform an effective gaze point detection method, where the method includes:
performing original model training based on the collected eye movement information to obtain an effective fixation point detection model, wherein a speed threshold value used for determining an effective fixation point in the effective fixation point detection model is adaptively adjusted according to information input to the effective fixation point detection model, and the collected eye movement information comprises a position sequence and a time sequence;
the effective fixation point detection model obtains an effective fixation point corresponding to the eye movement information based on the collected eye movement information, wherein the eye movement information comprises a position sequence and a time sequence;
and polymerizing the effective fixation point to obtain the polymerized effective fixation point.
Optionally, the program, when executed by the processor, may be further configured to perform an effective gaze point detection method provided by any of the embodiments of the present invention.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM), a flash Memory, an optical fiber, a portable CD-ROM, an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. A computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take a variety of forms, including, but not limited to: an electromagnetic signal, an optical signal, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, Radio Frequency (RF), etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing description is only exemplary of the invention and that the principles of the technology may be employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (9)
1. A method for detecting an effective gaze point, comprising:
performing original model training based on the collected eye movement information to obtain an effective fixation point detection model, wherein a speed threshold value used for determining an effective fixation point in the effective fixation point detection model is adaptively adjusted according to information input to the effective fixation point detection model, and the collected eye movement information comprises a position sequence and a time sequence;
the effective fixation point detection model obtains an effective fixation point corresponding to the eye movement information based on the collected eye movement information, wherein the eye movement information comprises a position sequence and a time sequence;
polymerizing the effective fixation point to obtain a polymerized effective fixation point;
the original model comprises a signal preprocessing unit, a hidden Markov model unit, a minimum covariance determinant estimation unit and a multivariate control chart unit; correspondingly, the training of the original model based on the collected eye movement information to obtain the effective fixation point detection model includes:
inputting the position sequence and the time sequence into a signal preprocessing unit to obtain a speed sequence;
inputting the speed sequence into an algorithm reconstruction subunit of the hidden Markov model unit to obtain model parameters;
correspondingly, the obtaining, by the effective gaze point detection model, an effective gaze point corresponding to the eye movement information based on the collected eye movement information includes:
inputting the speed sequence and the model parameters into a classification subunit of the hidden Markov model unit to obtain speed classification information;
inputting the speed classification information and the corresponding position sequence into a minimum covariance determinant estimation unit to obtain a position sequence after primary screening, a corresponding speed sequence and corresponding speed classification information;
inputting the preliminarily screened speed classification information and the corresponding speed sequence into the multivariate control chart unit to obtain screened eye movement data, wherein different preliminarily screened speed sequences correspond to different speed thresholds, and the screened eye movement data are effective fixation points corresponding to the eye movement information.
2. The method of claim 1, wherein the states used by the hidden Markov model element to detect include a velocity profile of a point of regard and a velocity profile of a jump of eye point.
3. The method of claim 1, wherein inputting the position sequence and the time sequence to a signal preprocessing unit to obtain a velocity sequence comprises:
filtering the position sequence with the attribute information of the missing signal in the position sequence through the signal preprocessing unit to obtain a filtered position sequence and a corresponding time sequence;
and converting the filtered position sequence and the corresponding time sequence into a speed sequence through the signal preprocessing unit.
4. The method of claim 1, wherein aggregating the effective gaze point to obtain an aggregated effective gaze point comprises:
and performing aggregation processing on the effective fixation points, determining the fixation points meeting the set conditions as the aggregated effective fixation points, wherein different aggregation methods correspond to different set conditions.
5. The method of claim 4, further comprising:
and determining the fixation point which does not meet the set condition as the eye jump data.
6. The method of claim 5, further comprising:
and aggregating the eye jump data detected by the effective fixation point detection model and the eye jump data determined by the set conditions to obtain aggregated eye jump data.
7. An effective fixation point detection device, comprising:
the system comprises a training module, a detection module and a control module, wherein the training module is used for carrying out original model training based on collected eye movement information to obtain an effective fixation point detection model, a speed threshold value used for determining an effective fixation point in the effective fixation point detection model is adaptively adjusted according to information input to the effective fixation point detection model, and the collected eye movement information comprises a position sequence and a time sequence;
the detection module is used for acquiring an effective fixation point corresponding to the eye movement information based on the acquired eye movement information by the effective fixation point detection model, and the eye movement information comprises a position sequence and a time sequence;
the aggregation processing module is used for carrying out aggregation processing on the effective fixation points to obtain aggregated effective fixation points;
the original model comprises a signal preprocessing unit, a hidden Markov model unit, a minimum covariance determinant estimation unit and a multivariate control chart unit;
correspondingly, the training module comprises:
the speed sequence unit is used for inputting the position sequence and the time sequence into the signal preprocessing unit to obtain a speed sequence;
the model parameter unit is used for inputting the speed sequence into an algorithm reconstruction subunit of the hidden Markov model unit to obtain model parameters;
correspondingly, the detection module comprises:
the speed classification information unit is used for inputting the speed sequence and the model parameters into a classification subunit of the hidden Markov model unit to obtain speed classification information;
the primary screening unit is used for inputting the speed classification information and the corresponding position sequence into the minimum covariance determinant estimation unit to obtain the position sequence after primary screening, the corresponding speed sequence and the corresponding speed classification information;
and the eye movement data unit is used for inputting the primarily screened speed classification information and the corresponding speed sequences into the multi-element control chart unit to obtain screened eye movement data, different primarily screened speed sequences correspond to different speed thresholds, and the screened eye movement data are effective fixation points corresponding to the eye movement information.
8. An electronic device, comprising:
one or more processors;
storage means for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-6.
9. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210297818.9A CN114397964B (en) | 2022-03-25 | 2022-03-25 | Method and device for detecting effective fixation point, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210297818.9A CN114397964B (en) | 2022-03-25 | 2022-03-25 | Method and device for detecting effective fixation point, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114397964A CN114397964A (en) | 2022-04-26 |
| CN114397964B true CN114397964B (en) | 2022-07-01 |
Family
ID=81234469
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210297818.9A Active CN114397964B (en) | 2022-03-25 | 2022-03-25 | Method and device for detecting effective fixation point, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114397964B (en) |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8224652B2 (en) * | 2008-09-26 | 2012-07-17 | Microsoft Corporation | Speech and text driven HMM-based body animation synthesis |
| CN104463916B (en) * | 2014-12-31 | 2017-04-12 | 武汉大学 | Eye movement fixation point measurement method based on random walk |
| CN111399658B (en) * | 2020-04-24 | 2022-03-15 | Oppo广东移动通信有限公司 | Calibration method and device for eyeball fixation point, electronic equipment and storage medium |
| CN112507799B (en) * | 2020-11-13 | 2023-11-24 | 幻蝎科技(武汉)有限公司 | Image recognition method based on eye movement fixation point guidance, MR glasses and medium |
-
2022
- 2022-03-25 CN CN202210297818.9A patent/CN114397964B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN114397964A (en) | 2022-04-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108875833B (en) | Neural network training method, face recognition method and device | |
| US10219129B2 (en) | Autonomous semantic labeling of physical locations | |
| CN101216884B (en) | A method and system for face authentication | |
| BR112020001729A2 (en) | gesture recognition device, apparatus and device | |
| EP3035235B1 (en) | Method for setting a tridimensional shape detection classifier and method for tridimensional shape detection using said shape detection classifier | |
| US20180125405A1 (en) | Mental state estimation using feature of eye movement | |
| CN110796199B (en) | Image processing method and device and electronic medical equipment | |
| CN108596087B (en) | Driving fatigue degree detection regression model based on double-network result | |
| CN106648078B (en) | Multi-mode interaction method and system applied to intelligent robot | |
| Haseeb et al. | Wisture: Rnn-based learning of wireless signals for gesture recognition in unmodified smartphones | |
| CN108986137B (en) | Human body tracking method, device and equipment | |
| CN103902978A (en) | Face detection and identification method | |
| CN108133197B (en) | Method and apparatus for generating information | |
| CN107862295A (en) | A kind of method based on WiFi channel condition informations identification facial expression | |
| CN111199238A (en) | Behavior identification method and equipment based on double-current convolutional neural network | |
| TW202101384A (en) | Image processing method, proposal evaluation method, and related devices | |
| CN112149615A (en) | Face living body detection method, device, medium and electronic equipment | |
| CN113763348A (en) | Image quality determination method and device, electronic equipment and storage medium | |
| CN110472673B (en) | Parameter adjustment method, fundus image processing device, fundus image processing medium and fundus image processing apparatus | |
| CN116881853B (en) | Attention assessment method, system, equipment and medium based on multi-mode fusion | |
| CN114397964B (en) | Method and device for detecting effective fixation point, electronic equipment and storage medium | |
| CN111985432B (en) | Multi-modal data fusion method based on Bayesian theorem and adaptive weight adjustment | |
| CN116152576B (en) | Image processing method, device, equipment and storage medium | |
| CN116704609A (en) | Online hand hygiene assessment method and system based on time sequence attention | |
| Iyengar | Decision-making with heterogeneous sensors-a copula based approach |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |