CN111738234B

CN111738234B - Automatic co-situation ability identification method based on individual eye movement characteristics

Info

Publication number: CN111738234B
Application number: CN202010816246.1A
Authority: CN
Inventors: 吴奇; 周萍; 李思琦
Original assignee: Hunan Normal University
Current assignee: Hunan Normal University
Priority date: 2020-08-14
Filing date: 2020-08-14
Publication date: 2020-11-24
Anticipated expiration: 2040-08-14
Also published as: CN111738234A

Abstract

The invention discloses an automatic common-situation ability identification method based on individual eye movement characteristics, which comprises the following steps: acquiring test eye movement data of a testee based on a preset visual material; acquiring shared situation response data of the testee; extracting eye movement characteristics of the testee according to the test eye movement data, wherein the eye movement characteristics comprise global eye movement characteristics and local eye movement characteristics; inputting the eye movement characteristics and the sympathy response data into a machine learning model for evaluating sympathy capacity so as to train a prediction model; the method comprises the steps of collecting test eye movement data based on preset visual materials, inputting the test eye movement data into a prediction model obtained after training, and determining the shared situation ability level of a testee corresponding to the test eye movement data according to output data of the prediction model. The technical scheme of the invention aims to provide a non-invasive, more efficient and convenient common situation capacity identification method.

Description

Automatic co-situation ability identification method based on individual eye movement characteristics

Technical Field

The invention relates to the technical field of machine learning, in particular to an automatic co-emotion capacity identification method based on individual eye movement characteristics.

Background

In psychology, sympathy often refers to the behavior of individuals to perceive and comprehend the emotions of other individuals and to react correctly, which has always been a hot problem in current psychological studies. The core meaning of the sympathy is "understanding and feeling of someone".

Due to the important role of the co-estrus in the human social cooperation process, the individual co-estrus capacity is accurately measured, and the method has very important value in various fields. For example, to understand the structure and function of sympathy, a reliable measurement method of the sympathy ability is needed, which is the basis of scientific research; the method has the advantages that the measurement of the shared situation ability of the individuals can help people to predict numerous psychology and behavior processes of cooperation, social behavior, moral behavior, mental health state and the like of the individuals, and the method has wide and important application value in various fields relating to interpersonal cooperation and interaction, such as medical treatment, education, judicial science, human resources, commerce, advertisement, military affairs, politics and the like. Because of the importance of sympathy in human life, researchers have been discussing the manner in which they are evaluated and measured. At present, the individual co-morbidities are mainly measured in a manual measuring mode.

In the process of manually measuring the co-emotional ability, the testee and the tester adopt face-to-face cooperation test, and the test result is a behavior reaction result presented by the testee under the condition that the testee realizes that the testee is tested, observed and controlled (namely, an invasive measurement method). Such a common situation measurement method cannot avoid the possibility that the subject realizes that the subject is observed and evaluated by the subject and the behavior of the subject changes. For example, an individual may change his or her response due to social approval, make a false response, or the like. Besides, errors caused by memory, fatigue, individual reaction deviation, repeated measurement and the like are difficult problems which cannot be solved by the method. Furthermore, in the manual measurement process, one-time measurement can only be performed on one testee, and multiple testees cannot be measured simultaneously, so that the measurement efficiency is low.

Therefore, it is desirable to provide a non-invasive and more efficient method for identifying common situation ability.

Disclosure of Invention

The invention mainly aims to provide an automatic shared emotion capacity identification method based on individual eye movement characteristics, and aims to provide a non-invasive and more efficient shared emotion capacity identification method.

In order to achieve the above object, the present invention provides an automatic sympathy capability identification method based on individual eye movement characteristics, comprising the following steps:

acquiring test eye movement data of a testee based on a preset visual material;

acquiring shared situation response data of the testee;

extracting eye movement characteristics of the testee according to the test eye movement data, wherein the eye movement characteristics comprise global eye movement characteristics and local eye movement characteristics;

inputting the eye movement characteristics and the sympathy response data into a machine learning model for evaluating sympathy capacity so as to train a prediction model;

the method comprises the steps of collecting test eye movement data based on preset visual materials, inputting the test eye movement data into a prediction model obtained after training, and determining the shared situation ability level of a testee corresponding to the test eye movement data according to output data of the prediction model.

Preferably, the step of acquiring the test eye movement data of the human subject based on the preset visual material comprises:

acquiring eye movement coordinate data of a testee based on a preset visual material;

collecting pupil diameter data of a testee based on a preset visual material;

acquiring blink event data of a testee based on preset visual materials;

and acquiring time sequence data of the testee based on preset visual materials.

Preferably, the step of extracting the eye movement characteristics of the subject from the test eye movement data includes:

detecting eye movement events other than blink events based on the eye movement coordinate data;

performing feature extraction on a first type of eye movement features and a second type of eye movement features based on the eye movement coordinate data and all eye movement events, wherein the first type of eye movement features comprises: based on features of the gaze, saccade and blink event data, the second type of eye movement features includes: based on the eye movement abscissa, eye movement ordinate and characteristics of the pupil diameter raw data.

Preferably, the step of extracting the features of the first type and the second type based on the eye movement coordinate data and all eye movement events includes:

detecting gaze events in a loop;

when the distance between the current eye movement coordinate data and the center coordinate is larger than a preset distance threshold value, carrying out settlement on the current gazing event for the effective gazing event, and detecting the next gazing event until the traversal of all the gazing events is completed;

the method comprises the following steps of determining the gazing event meeting a first preset condition as an effective gazing event, determining the gazing event not meeting the first preset condition as an ineffective gazing event, wherein the first preset condition comprises the following steps: the duration of the fixation is above the first time and the data error rate is lower than the first error rate.

Preferably, the step of extracting features of the first type and the second type based on the eye movement coordinate data and all eye movement events further includes:

detecting a glance event based on a gaze event;

performing the settlement of the current glance event on each effective glance event, and detecting the next glance event until the traversal of all glance events is completed;

wherein the glance event is an eye movement between two segments of the gaze event, the glance event satisfying a second predetermined condition is determined to be a valid glance event, the glance event not satisfying the second predetermined condition is determined to be an invalid gaze event, and the second predetermined condition comprises: the duration of the glance is more than the second time, the correct amount of samples is more than the preset number, the maximum speed in the glance process is more than the preset speed, and the total displacement in the glance process is less than the preset displacement.

Preferably, between the step of detecting eye movement events other than blinking events based on the eye movement coordinate data and the step of extracting features of the first kind of eye movement features and the second kind of eye movement features based on the eye movement coordinate data and all eye movement events, further comprising:

sliding a preset time length as a time window on the time sequence without overlapping the eye movement coordinate data and all eye movement events of each measured person to obtain a series of time windows;

the step of extracting the features of the first type of eye movement features and the second type of eye movement features based on the eye movement coordinate data and all eye movement events comprises the following steps:

extracting the first type of eye movement features and the second type of eye movement features based on the eye movement coordinate data and all eye movement events in each of the time windows.

Preferably, after the step of extracting the eye movement characteristics of the subject according to the test eye movement data, the method further includes:

screening the eye movement characteristics according to preset screening conditions;

and performing dimension reduction treatment on the eye movement characteristics after screening treatment.

Preferably, the step of inputting the eye movement characteristics and the sympathy response data into a machine learning model for evaluating sympathy abilities to train a prediction model comprises:

determining the co-estrus response data as dependent variables;

determining the eye movement characteristics as independent variables, wherein the eye movement characteristics are eye movement characteristics of a first eye movement record of the human subject;

establishing a statistical model according to a leave-one-out method and a grid search method;

based on the leave-one-out method, the eye movement features are used as a data set, the shared situation ability score of the testee is used as an effect target, and the machine learning model is used for extracting and learning the eye movement features so as to train a prediction model.

Preferably, before the step of acquiring the test eye movement data of the human subject based on the preset visual material, the method further comprises:

acquiring multiple times of binocular eye movement data of a fixation point of a testee on a fixation display interface;

positioning the positions of the eyes of the testee according to the binocular eye movement data for a plurality of times;

when the error value of the positions of the two eyes is in a preset range, executing the step of acquiring the test eye movement data of the testee based on a preset visual material;

when the error value of the positions of the two eyes exceeds a preset range, the step of obtaining the data of the multiple times of the double-eye movement of the fixation point on the fixation display interface of the testee is executed again;

and/or

Before the step of acquiring the test eye movement data based on the preset visual material, the method further comprises the following steps:

and when the error value of the positions of the two eyes exceeds a preset range, executing the step of obtaining the data of the multiple times of the double-eye movement of the fixation point on the fixation display interface of the testee again.

Preferably, before the step of acquiring the data of the plurality of binocular eye movements of the fixation point on the display interface, the method further includes:

sequentially playing preset visual materials according to a preset playing frequency and a preset playing sequence;

before each piece of the preset visual material is played, the fixation point is displayed on a display interface, so that the fixation position of the testee is calibrated through the fixation point.

According to the technical scheme, a prediction model is trained through test eye movement data and sympathy response data, then the collected test eye movement data are used as input data and input into the prediction model, and the sympathy capability level of a testee corresponding to the test eye movement data is determined through output data of the prediction model. In the model training process, firstly, the test eye movement data of a testee based on a preset visual material is collected, wherein the preset visual material can be a picture, the collected test eye movement data can be collected through equipment (for example, infrared sensing equipment and a camera), the collection equipment can finish data collection by aiming at eyeballs of the testee, and the display of the visual material can be realized through projection equipment or display equipment. According to the technical scheme, the equipment is adopted to collect the test eye movement data of the testee when the testee views the visual materials, so that the observability of the testee caused by the fact that the testee and the testee test face to face can be avoided, and for the same preset visual materials, a plurality of testees can be arranged at one time for testing, and only the corresponding collection equipment is required to collect the eyeball data of each testee; and secondly, acquiring the co-emotional response data of the testee, extracting the eye movement characteristics of the testee according to the test eye movement data, and inputting the eye movement characteristics and the co emotional response data into a machine learning model to train the prediction model, so that the trained model is favorable for obtaining the co emotional ability recognition data with higher authenticity. After the training of the prediction model is completed, in the subsequent testing process, the common situation ability level data of the testee can be obtained by only acquiring the testing eye movement data of the testee based on the preset visual materials through the testing eye movement data and the prediction model, so that the inaccuracy of the testing data brought by face-to-face testing can be avoided, the eye movement data of a plurality of testees can be acquired in batch for common situation ability recognition, and the testing efficiency of the common situation ability can be improved. Furthermore, different from other behavior data generated by the individual, the eye movement process of the individual has higher spontaneity and is more difficult to be forged intentionally, and the eye movement data can be collected smoothly through the collecting equipment, so that an invasive common situation ability identification mode is avoided, and therefore, the accuracy and the convenience of the common situation ability test can be further improved by the technical scheme provided by the invention.

Drawings

FIG. 1 is a schematic flow chart illustrating a first embodiment of an automated sympathy capability identification method based on individual eye movement characteristics according to the present invention;

FIG. 2 is a correlation result of the co-estrus response data of the co-estrus capacity prediction score and the co-estrus measurement table obtained by using a linear regression model;

FIG. 3 is a correlation result of the shared situation response data of the shared situation ability prediction score and the shared situation measurement table obtained by using a random forest model;

FIG. 4 is a correlation result of the shared situation ability prediction score obtained by the KNN model and shared situation response data of the shared situation measurement table;

FIG. 5 shows the correlation results of the co-estrus response data of the co-estrus capacity prediction score and the co-estrus measurement table obtained by the SVR model.

The objects, features and advantages of the present invention will be further explained with reference to the accompanying drawings.

Detailed Description

It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

In the following description, suffixes such as "module", "component", or "unit" used to denote elements are used only for facilitating the explanation of the present invention, and have no specific meaning in itself. Thus, "module", "component" or "unit" may be used mixedly.

Referring to fig. 1, to achieve the above object, a first embodiment of the present invention provides an automatic comorbidity recognition method based on eye movement characteristics of an individual, including the following steps:

step S10, acquiring the test eye movement data of the testee based on the preset visual material;

step S20, obtaining shared situation response data of the testee;

step S30, extracting the eye movement characteristics of the testee according to the test eye movement data, wherein the eye movement characteristics comprise global eye movement characteristics and local eye movement characteristics;

step S40, inputting the eye movement characteristics and the sympathy response data into a machine learning model for evaluating the sympathy ability so as to train a prediction model;

and step S50, acquiring test eye movement data, inputting the test eye movement data into the prediction model, and determining the comorbidity level of the testee corresponding to the test eye movement data according to the output data of the prediction model.

Specifically, in the step of acquiring the eye movement data of the human subject based on the preset visual material, the eye movement information of the human subject based on the preset visual material can be acquired through a remote sensing type infrared sensing eye movement instrument; and determining the eye movement data according to the eye movement information. The model of the remote infrared sensing eye tracker can adopt an SMI RED500 remote infrared sensing eye tracker (the sampling rate is 500Hz, the screen resolution is 1920 multiplied by 1080, and information of two eyes is collected), the visual material can adopt a standardized ESP-MTTI shared emotion response picture set, shared emotion response data can be obtained through an IRI-C interpersonal response pointer scale, and tested shared emotion score data can be obtained according to the scoring standard of the scale. In one identification process, one test for a subject may be arranged, or a plurality of tests for a subject may be arranged at the same time.

In order to solve the problems of low efficiency, complexity, interference from social promises and the like of the conventional shared situation measuring mode, on the basis of research results in the field of machine learning, a standardized visual material is adopted, test eye movement data of a tested perceptual visual material are collected, global and local eye movement characteristics of the tested eye movement are extracted, and a universal machine learning model is used for automatic classification learning, so that a system for automatically judging the shared situation capacity level by a machine is formed.

In one embodiment, the training process of the prediction model may be: the method comprises the steps of collecting eye movement information of 166 testees watching a semi-projected common situation picture (namely an ESP-MTTI picture set) through an SMI RED500 remote sensing type infrared sensing eye movement instrument, collecting an IRI-C interpersonal reaction pointer scale of the testees, obtaining the common situation score of the testees as common situation reaction data according to the scale scoring standard, extracting eye movement characteristics of the testees as independent variables, and obtaining original data of the eye movement which respectively comprise coordinate data, pupil diameters, blink events and corresponding time sequences. Through a series of processing such as feature extraction, screening, dimensionality reduction and the like, the IRI-C score to be tested is taken as a dependent variable, a leave-one-out method is adopted, and a machine learning method for regression problem is applied, wherein the method comprises the following steps: and any one of Linear Regression, K-nearest neighbor KNN, Random Forest, and support vector machine Regression SVR extracts and learns the eye movement characteristics to realize the automatic prediction of individual shared feeling capability.

The invention uses the original eye movement characteristics obtained by watching the semi-projected picture as the basis and carries out the co-emotion capability prediction by using the SVR model to obtain the prediction system with the highest performance. On 166 samples, the consensus score obtained by the leave-one-out method was significantly correlated with the consensus score in the real IRI-C questionnaire, and the pearson correlation coefficient was 0.95. The prediction error in the four IRI-C dimensions and the total score is between 6.5% and 10.5%. While the random forest and KNN are mainly used for classification tasks, but can also be used for regression tasks and are widely applied to emotion recognition, the Pearson coefficient r = 0.63-0.70 between the predicted value and the observed value of the co-emotion score in the invention. That is, the results show that different models have large differences in learning of the experimental eye movement data, but it is found that it is feasible to predict sympathy by machine learning based on the experimental eye movement data. Meanwhile, the possible relation between the eye movement characteristics such as time series dynamic indexes, heat point diagrams and the like and the shared situation capability is also shown.

The co-emotional ability prediction method based on the eye movement data has a good effect and high accuracy. The measurement of the shared conditions does not depend on manual test, and a more objective, accurate, efficient and real measurement mode can be constructed by machine learning. And the conclusion further verifies that the method for collecting the eye movement data of the testee has important contribution to the shared situation ability identification. And extracting eye movement characteristics and the sympathy response data, inputting the extracted eye movement characteristics and the sympathy response data into a machine learning model, and finishing the training of a prediction model so as to determine the sympathy capability level of the testee corresponding to the test eye movement data through the prediction model in the subsequent recognition process, wherein the accuracy is higher.

Referring to fig. 2, according to the first embodiment of the method for automatically recognizing co-emotional ability based on the eye movement characteristics of the individual according to the present invention, and the second embodiment of the method for automatically recognizing co-emotional ability based on the eye movement characteristics of the individual according to the present invention, the step S10 includes:

step S11, collecting the eye movement coordinate data of the testee based on the preset visual material;

step S12, collecting pupil diameter data of the testee based on the preset visual material;

step S13, collecting blink event data of the testee based on preset visual materials;

in step S14, time-series data of the subject based on the preset visual material is acquired.

Vision is the most prominent channel of sensory information. Most of the sensory information about the outside world obtained by humans is obtained visually. In the visual perception process, the eye movement process plays a very important role, and there is a myriad of connections with attention. Studies have shown that humans focus most attention on emotionally meaningful stimuli and that eye movements are found to play a crucial role in facial expression recognition and facial memory. Because the eye movement process is closely related to the individual psychological state and behavior, people often understand and predict the psychological state of others through the eye movement process of others in daily communication. The same sense of mind also affects the adult face detection performance and attention process.

Unlike other behavioral data generated by an individual, the individual's eye movement process is more spontaneous and more difficult to forge intentionally. The generation of eye movement data is also more mandatory for the individual (eye movements are generated as long as the process of visual attention is available, but the individual may not necessarily generate facial expressions, body movements, verbal information, etc.). Compared with electroencephalogram or brain imaging and the like, the method for acquiring the eye movement data has the advantages of cost and convenience. Therefore, the present invention selects a series of eye movement data to extract the eye movement characteristics of the subject.

In order to enable the eye movement characteristics to be expressed through parameters, an eye movement coordinate system is established, wherein eye movement coordinate data are coordinate points of pupils of a tested person on the eye movement coordinate system, and the position movement of the pupils is described through the coordinate points; various eye movement characteristics can be described through coordinate points of the pupil on the eye movement coordinate system. In the eye movement coordinate system, the values of all coordinate points are normalized to be between-1 and 1, and the influence caused by the equipment model and the resolution ratio is eliminated.

When there are multiple subjects, the original data in idf format is obtained after the experiment, the original data is exported to be a file in txt format (Bagaze can be used), the data of each subject is divided in csv format, and the original data respectively has coordinate data, pupil diameter, eye movement event (only using blinking time) and corresponding time sequence. Wherein the coordinate data are scaled to the range of [ -1,1 ].

The time series data are time series associated with eye movement coordinate data, pupil diameter data, and blink event data, respectively.

Based on the second embodiment of the method for automatically recognizing co-estrus based on eye movement characteristics of individual of the present invention, and the third embodiment of the method for automatically recognizing co-estrus based on eye movement characteristics of individual of the present invention, the step S30 includes:

step S31 of detecting an eye movement event other than a blink event based on the eye movement coordinate data;

step S32, performing feature extraction on a first type of eye movement features and a second type of eye movement features based on the eye movement coordinate data and all eye movement events, where the first type of eye movement features include: based on features of the gaze, saccade and blink event data, the second type of eye movement features includes: based on the characteristics of the raw data of the eye movement abscissa (X-coordinate), the eye movement ordinate (Y-coordinate) and the pupil diameter (D).

All eye movement events include gaze events, saccade events and blink events based on eye movement coordinate data and are correlated with the pupil diameter data and the time series data.

In this embodiment, first, eye movement events except for a blink event are detected based on eye movement coordinate data, and then, based on the eye movement coordinate and all eye movement events, the following two types of feature extraction are performed on the data using a time window:

1. features based on gaze, glance and blink event data;

2. based on the characteristics of the XYD raw data (X-coordinate, Y-coordinate, D-pupil diameter).

Based on the third embodiment of the method for automatically recognizing co-estrus ability based on individual eye movement features of the present invention, in the fourth embodiment of the method for automatically recognizing co-estrus ability based on individual eye movement features of the present invention, the step S32 includes:

step S321, circularly detecting a fixation event;

step S322, when the distance between the current eye movement coordinate data and the center coordinate is larger than a preset distance threshold, the current gaze event is settled for the effective gaze event, and the next gaze event is detected until the traversal of all the gaze events is completed;

Gaze events are detected in a loop, and when the duration of the gaze is less than a first time (e.g., 0.1 s), or the data error rate reaches a first error rate (e.g., fifty percent), the current gaze event is invalid. When the distance between the current coordinate data and the center coordinate is larger than a preset distance threshold (for example, 0.01), the fixation event is settled (the settlement comprises the steps of determining the average value and the variance of the coordinates x and y, the fixation starting and ending time and the corresponding index, the average value and the variance of the pupil diameter during fixation, and the average value and the variance of the deviation angle during fixation), and a new fixation event is entered, and the loop is repeated until all data are traversed.

Based on the third embodiment or the fourth embodiment of the method for automatically recognizing co-estrus based on individual eye movement features of the present invention, in the fifth embodiment of the method for automatically recognizing co-estrus based on individual eye movement features of the present invention, the step S32 further includes:

step S323, detecting a glance event based on a gaze event;

step S324, the current glance event is settled for each effective glance event, and the next glance event is detected until the traversal of all glance events is completed;

After a fixation event is detected, a saccade event is detected based on the fixation event, and eye movement between two fixations may be considered a saccade. In this embodiment, when the duration is less than 0.1s, the number of correct samples is less than a preset number (e.g., 2), or the maximum velocity during panning is less than 0.2 and the total displacement during panning is greater than 0.02, the saccade is determined to be invalid. Each valid glance event is settled, and the settled data comprises coordinates of the start and the end of the glance, a glance deviation angle, a glance start and end time and corresponding indexes respectively, a mean variance of pupil diameters during the glance, a maximum speed of the glance, a total displacement during the glance and a type of the glance.

Based on the third to fifth embodiments of the automatic passivity recognition method based on individual eye movement characteristics of the present invention, in the sixth embodiment of the automatic passivity recognition method based on individual eye movement characteristics of the present invention, between the step S31 and the step S32, the method further includes:

step S33, sliding the preset time length as a time window without overlapping on the time sequence for the eye movement coordinate data and all the eye movement events of each tested person to obtain a series of time windows;

the step S32 includes:

step S325, in each time window, extracting the first type of eye movement features and the second type of eye movement features based on the eye movement coordinate data and all eye movement events.

Extracting eye movement features using a time window, specifically comprising: a sliding window approach is used to process the time series of pupil diameter data. That is, only data from a certain length of time is considered in each data of each subject. Each sliding window includes a start event and an end time, and their corresponding indices. A series of time windows are obtained by sliding time windows of length 3 seconds, without overlapping in time series. In each valid time window, two types of data are acquired, based on the characteristics of the raw data XYD, based on the characteristics of the glance-gaze blink events, respectively. The data are detailed in tables 1 and 2.

TABLE 1 statistics based on raw data (X-coordinate, Y-coordinate, D pupil diameter)

In table 1, the average angle of the two original gazing points, i.e. the angle between the x-axis and the vector connecting the two gazing points.

The 8 × 8-unit gaze point thermal map means that 2.5% of gaze points on the upper, lower, left and right sides from the boundary of the whole interface are not considered, that is, only gaze points between 2.5% and 97.5% in the vertical and transverse directions are considered, the considered interval is divided into 64 cells with the size of 8 × 8 (8 rows and 8 columns), and the occurrence frequency of each cell gaze point in the time window is calculated.

The quartile range of the distribution of XYD is 75% -25% quantile of the distribution of XYD.

The subsequent difference refers to the difference between the latter point of gaze and the current point of gaze XYD.

TABLE 2 statistics based on gaze, glance and blink event data

In table 2, the fixation/saccade rate is the number of times eye movement time occurs per second (e.g., 5 fixations per second).

The glance-fixation ratio is the ratio between the number of occurrences in a time window.

The saccade pupil diameter mean/variance is the mean/variance of the pupil diameter for a single sustained saccade event.

The mean/variance of the saccade pupil diameter means/variance is the mean/variance of the pupil diameter means/variance of all saccade events within the time window.

The subsequent gaze angle is the arctangent of the slope of the tangents made by adjacent gaze points within a single sustained gaze event.

The subsequent gaze angle mean is the mean of the subsequent gaze angles within a single sustained gaze event.

The average of the subsequent gaze angle means is the average of the subsequent gaze angle means of all gaze events within the time window.

In this embodiment, each time window obtains 143 features, each person slides the time window 78 times without overlapping, the features of each time window correspond to the features of each picture, the features are spliced, and finally, a single piece of data of each person has 11154 features; finally, after feature extraction, 166 testees correspond to 166 pieces of data, and each piece of data contains 11154 features.

Based on the first to sixth embodiments of the method for automatically recognizing co-estrus based on individual eye movement features of the present invention, in a seventh embodiment of the method for automatically recognizing co-estrus based on individual eye movement features of the present invention, after the step S30, the method further includes:

step S60, screening the eye movement characteristics according to preset screening conditions;

and step S70, performing dimension reduction processing on the eye movement characteristics after the screening processing.

Because the original data has the characteristics of high dimensionality, incapability of being identified by a computer and the like, the original data cannot be directly used for constructing a model for learning, and a series of processing needs to be carried out on the model, including feature extraction, data standardization and feature selection. The obtained experimental eye movement data has low dimensionality and can be well recognized and learned by a computer.

The data normalization includes: there is a difference in magnitude level between the parameters of different features, such as a difference of about one thousand times between the coordinates and the pupil diameter. When different eigenvalues of the features are very different, there is an effect of improving the features with higher values and an effect of weakening the indexes with lower values, so that the data needs to be normalized, and the distribution of the data set is converted into a standard normal distribution with a mean value of 0 and a standard deviation of 1 by performing an operation z = (x-u)/s (where z is a processed sample, x is a corresponding sample, u is a mean value of a training sample, and s is a standard deviation of the training sample).

Feature selection packageComprises the following steps: the feature selection can eliminate irrelevant or redundant features, so that the aims of reducing the number of features, improving the accuracy of a model and reducing the running time are fulfilled. And (3) performing score optimization of correlation coefficients on the feature selection method of each algorithm by a grid search method, and selecting an Embedded method based on a linear least square L2 regularization algorithm and a Filter method based on correlation sorting. For the former, the algorithm adjusts the penalty term coefficient corresponding to each feature in minimizing the loss function. At the end of the algorithm, there is a case where the penalty term coefficient of the feature becomes very close to 0 ((<

) I.e. unsuitable features are screened out. For the latter, the features are sorted by calculating the relevance between different features of the data set to select the top k features with relevance ranking. For the determination of k, continuously searching an optimal value through a continuously reduced range of spans of 1000, 500, 100, 50, 10 and 1, taking 1000 spans as an example, respectively comparing the effects of retaining 1000, 2000 to 11000 dimensional data, finding a highest-scoring interval under the same algorithm, and gradually reducing the spans to finally obtain the optimal value of k.

Based on the first to seventh embodiments of the automatic passivity identifying method based on individual eye movement characteristics of the present invention, in an eighth embodiment of the automatic passivity identifying method based on individual eye movement characteristics of the present invention, the step S40 includes:

step S41, determining the sympathy response data as a dependent variable;

step S42, determining the eye movement characteristics as independent variables, wherein the eye movement characteristics are eye movement characteristics recorded by the first eye movement of the human subject;

step S43, establishing a statistical model according to a leave-one-out method and a grid search method;

and step S44, based on the leave-one-out method, taking the eye movement characteristics as a data set, taking the shared situation ability score of the testee as an effect target, and extracting and learning the eye movement characteristics by using the machine learning model so as to train a prediction model.

Specifically, the eye movement characteristics and the sympathy response data are input into a machine learning model; mutually exclusive division is carried out on a data set consisting of all the samples of the testee, one data in the data set is used as a test set and the other data are used as a training set repeatedly each time, and corresponding prediction scores are obtained according to the test set and the training set; combining the prediction scores of the test set, and calculating a correlation coefficient with the actual score of the test set to obtain a group of optimal coefficients; and determining the shared situation score of the testee according to the optimal coefficient.

In this embodiment, for the above models, the parameters are adjusted by using a grid search method, that is, the possible combinations of all the parameters in the given range are traversed to obtain the parameter combination with the highest evaluation. Taking the KNN model as an example, whether the distance reciprocal is adopted by the neighbor nodes in the model as the weight or not and the number of the neighbor nodes is a parameter regulated in the model, if the range of the number of the neighbor nodes is set to be 2 to 20, and whether the weight is adopted or not is 2 possible, the total number is 19 x 2 parameter combinations, the correlation coefficient of each combination prediction data and the original data is calculated, and finally the parameter combination with the highest value is selected. Machine learning models suffer from overfitting, and machine learning-based models are often very complex, with linearity and parametric diversity. In order to alleviate the overfitting problem, in this embodiment, a leave-one method is adopted in each parameter search, that is, a data set composed of 166 samples is divided in a mutually exclusive manner, one data is used as a test set repeatedly each time, the rest data is used as a training set to obtain corresponding scores, and finally, the prediction results of the test set are combined, and further, the calculation of the correlation coefficient is performed with the actual scores of the test set.

In the invention, the Pearson correlation coefficient r is taken as an evaluation index, and the evaluation of the model by adopting Mean Absolute Percentage Error (MAPE) is also considered, so that the purpose of more accurate and effective evaluation mode is achieved. Therefore, the invention adopts the two common evaluation indexes to verify the four machine learning models so as to analyze and detect which machine learning model can better learn the eye movement data of the test to obtain more accurate common emotion scores.

The average absolute percentage error can better reflect the actual situation of the error of the predicted value, and the smaller the MAPE value is, the better the accuracy of the prediction model is. The calculation may employ the following formula (1):

（1）；

wherein, in formula (1): the observed is a true value, the predicted value is a predicted value, the subscript t is the t-th sample, and n is the total number of samples.

The machine learning model adopted by the invention comprises one of Linear Regression (Linear Regression), K-nearest neighbor KNN, Random Forest and support vector machine (SVR).

(1) Linear regression model: the linear regression model has the advantages of simple calculation, simple realization and understanding and high running speed under the condition of large data volume, and the relationship among all components of the eye movement characteristic vector is fitted by constructing the linear regression model on the existing test eye movement data set, and the fitted linear model is utilized to predict the score of the common situation. To get a more accurate prediction score, the linear model is fitted by minimizing the sum of the squared residuals between the observed values in the dataset and the predicted values of the linear model, resulting in an optimal set of coefficients w = (w)₁，…，w_p) And then predicting a final co-occurrence score using an optimal coefficient, wherein w₁，…，w_pRespectively, the prediction weights of the features corresponding to the subscript numbers.

(2) K nearest neighbor KNN model: the training time of the k-nearest neighbor algorithm is low in complexity. Meanwhile, the method mainly carries out prediction through limited samples around, does not need to have hypothesis on data distribution, and is insensitive to abnormal points. Based on the above algorithm advantages and the wide application in emotion recognition, the present study attempted to predict sympathy using this model.

After a grid screening method, the number K considered by the neighboring samples of each sample is set to be 16, and the Minkowski distance with the p value set to be 2 is used for judging sample points close to the current point in the process; the prediction value of the current point is then obtained by the average of 16 neighboring points, where the sample points closer to the prediction target have higher weights. Wherein the minkowski distance is calculated using the following formula (2):

（2）

in formula (2), (x, y) is the form in which the sample point is expressed as a feature vector, i is the ith component of the feature vector, n is the dimension of the feature vector, and P is a constant.

(3) Random forest model: the Random Forest is an integrated model fused with a plurality of decision trees and can be used for regression and classification tasks, and the regression tasks are Random Forest regressors. Random sampling is carried out on a random forest, generalization capability is strong, and meanwhile, node division is carried out on numerous decision trees randomly, so that the random forest still has high-efficiency performance when facing high-dimensional data. Many different attribute feature types, such as binary, categorical and numeric, can be processed simultaneously, and the relative importance of each feature to the prediction can be measured very quickly. In the research, the eye movement data are of various types and comprise time sequences, coordinate values, hot spot maps and the like, so that a random forest algorithm is adopted for the experiment selection of another algorithm model.

The random forest is an algorithm model based on a bagging random sampling basis and an idea of adding feature random selection. Selecting a CART decision tree as a weak learner under a bagging framework, taking the research as an example, 166 data can be divided into 165 training set samples and 1 test set sample by one-out-of-one method each time, for the 165 training set samples, bagging can obtain a sampling set by repeatedly sampling the training set for 165 times, and the probability that each sample is selected in the sampling process is always 1/165. Through the parameter search of the grid search method, 190 weak learners are finally set in the algorithm, so that 190 sampling sets can be obtained by repeating the process for 190 times. For the sampling sets, 190 weak learners can be trained independently, so that the calculation amount is reduced while under-fitting is not easy to cause. And adopting an averaging method with weights for the results of the weak learners to obtain a final result.

(4) Support vector machine regression SVR: the SVR model is an application of an SVM (support vector machine) to solving the regression problem, and has very good generalization performance in solving the problems of small samples, nonlinearity and high-dimensional classification. SVR maps input linear irreparable data into a high-dimensional feature space such that the input data is separable in its feature space, which can be achieved using a kernel function in a model, in which a gaussian Radial Basis (RBF) function has the following advantages: the RBF is used as the kernel function of the SVR because of its strong locality, good performance for large and small samples, and fewer parameters than other polynomial kernels. The gaussian Radial Basis (RBF) function is determined using the following equation (3):

（3）

wherein, in the formula (3), x and x_iTwo different training samples are represented, respectively, as free parameters.

Based on the first to eighth embodiments of the method for automatically recognizing co-estrus based on eye movement characteristics of individual according to the present invention, in the ninth embodiment of the method for automatically recognizing co-estrus based on eye movement characteristics of individual according to the present invention, before the step S10, the method further includes:

step S80, acquiring multiple times of binocular test eye movement data of a fixation point on a fixation display interface of a testee;

step S90, positioning the positions of the two eyes of the testee according to the data of the two-eye test eye movements for a plurality of times;

when the error value of the both-eye position is in a preset range, the step S10 is executed;

when the error value of the both-eye position is out of the preset range, the step S90 is performed again.

And/or

Before the step S50, the method further includes:

when the error value of the both-eye position is in a preset range, the step S50 is executed;

In this embodiment, the subject first adjusts the seat to the optimal collection position according to the guidance information displayed on the display interface. In order to avoid the positional deviation of both eyes, the eye movement is corrected a plurality of times, for example, the subject needs to watch the red dot at the center of the white globule on the screen a plurality of times, for example, 9 times, thereby locating the distance between both eyes of the subject. If X, Y coordinate values are less than 1.0, the space key is pressed to start the experiment, and if X or Y coordinate value is greater than 1.0, recalibration is required. And (5) entering an eye movement experiment after the calibration is successful.

Based on the first to ninth embodiments of the method for automatically recognizing co-estrus based on eye movement characteristics of individual of the present invention, in the tenth embodiment of the method for automatically recognizing co-estrus based on eye movement characteristics of individual of the present invention, before the step S80, the method further includes:

step S100, sequentially playing preset visual materials according to a preset playing frequency and a preset playing sequence;

step S110, before each preset visual material is played, displaying the fixation point on a display interface, so as to calibrate the fixation position of the human subject through the fixation point.

A preset amount of visual material is presented in a fixed sequence, each picture presentation time is 3000ms, and the "+" gaze point is presented 500ms on the screen before each picture presentation. In this embodiment, an SMI RED500 eye movement instrument is used to record the whole eye movement process of the human subject within the presentation time of the shared situation picture (sampling frequency 500 Hz). Preset questions related to the pictures appear randomly on the display interface between the pictures and prompt the testee to answer orally so that the testee focuses attention on the pictures.

The automatic sympathy ability identification method based on the individual eye movement features adopts the following corresponding prediction results of different machine learning models.

(1) The linear regression model predicts the result: after parameter adjustment, the model finally determines that a Fliter method based on a correlation coefficient as an evaluation index is adopted, 353 dimensions of features are selected, the linear regression model learns the eye movement features, the final obtained result is that the total score of the predicted common conditions and the scores of the four sub-dimensions are obviously related to the observed value of the common conditions, and the Pearson correlation coefficient r = 0.65-0.82, so that strong correlation is achieved. And positive correlation can be found according to the scatter diagram and the fitted curve, and the detail is shown in figure 2. In fig. 2, the abscissa represents the prediction score and the ordinate represents the observed value. The average absolute percentage error is 7.3% MAPE in the prediction of the total score, and is relatively high for the average absolute percentage error of four sub-dimensions, and reaches 10.5% -29.5%.

(2) And (3) predicting a result by using a random forest model: after parameter adjustment, the model finally determines that a Fliter method based on the correlation coefficient as an evaluation index is adopted, and 200 dimensions of the features are selected. The random forest model trains and learns the following characteristics, and the result shows that the predicted co-emotion scores are obviously related to corresponding observed values in the total score and four dimensions, and the Pearson coefficient r = 0.50-0.68, so that strong correlation is achieved. And the scatter diagram and the fitted curve show that the two are in positive correlation. See figure 3 for details. In fig. 3, the abscissa represents the predicted score and the ordinate represents the observed value. The average absolute percentage error is MAPE of 10.4% in the prediction of the total score, and the average absolute percentage error in four sub-dimensions is relatively high and reaches 9.15% -35.4%.

(3) Prediction results of the KNN model: after parameter tuning, the model finally determines that a Fliter method based on a correlation coefficient as an evaluation index is adopted, the 350-dimensional characteristic is selected, the KNN model learns the corresponding eye movement characteristic, the final obtained result is that the total score of the predicted common conditions and the scores of the four sub-dimensions are obviously related to the observed value of the common conditions, and the Pearson correlation coefficient r = 0.56-0.74 achieves strong correlation. Both the scatter plot and the fitted curve were found to be positively correlated. See figure 4 for details. In fig. 4, the abscissa represents the predicted score and the ordinate represents the observed value. The average absolute percentage error is MAPE of 10.6% in the prediction of the total score, and the average absolute percentage error in four sub-dimensions is relatively high and reaches 15.3% -37.2%.

(4) SVR model prediction results: after parameter tuning, the model finally determines that an Embedded method based on a linear least square L2 regularization algorithm is adopted, the 4662-dimensional characteristics are kept, the SVR model learns the eye movement characteristics, the final obtained result is that the total score of the predicted common conditions and the scores of the four sub-dimensions are obviously related to the observed value of the common conditions, and the Pearson correlation coefficient r = 0.93-0.95. And positive correlation can be found from the scatter plot and the fitted curve. See figure 5 for details. In fig. 5, the abscissa represents the predicted score and the ordinate represents the observed value. The MAPE on the prediction of the total score is 6.6%, and the average absolute percentage error on the four sub-dimensions is relatively high and reaches 9.2% -24.0%.

The invention adopts the most suitable characteristic selection mode aiming at different machine learning models, screens out unsuitable characteristics, and sorts the characteristics by calculating the correlation among different characteristics of the data set. After processing, the prediction accuracy of the model for the co-situation capability is greatly improved, which shows that the research is very effective for the mode of selecting and extracting the eye movement data features.

The machine learning method is combined with the behavior science method, the adopted eye movement characteristics show the potential of developing non-invasive and low-cost methods to monitor the individual mental health in real time, in order to utilize the eye movement data generated by the individual to evaluate the shared emotion ability of the individual, and developing an automatic sympathy capability assessment system based on individual eye movement characteristics with important scientific research and practical application values, and subsequent researchers continue to develop machine learning methods suitable for mining sympathic eye movement characteristics that provide a standard testing platform, and the prediction of various psychology and behaviors such as individual societal behavior, moral behavior, mental health state and the like provides an efficient, convenient and interference-free automatic tool and provides a research basis.

In the description herein, references to the description of the term "an embodiment," "another embodiment," "other embodiments," or "first through Xth embodiments," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, method steps, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.

The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.

Claims

1. An automatic sympathy ability identification method based on individual eye movement characteristics is characterized by comprising the following steps:

acquiring test eye movement data of a testee based on a preset visual material, wherein the visual material adopts a standardized ESP-MTTI common situation response picture set;

acquiring shared situation response data of the testee;

screening the eye movement characteristics according to preset screening conditions; performing dimension reduction processing on the eye movement characteristics after screening processing;

acquiring test eye movement data based on a preset visual material, inputting the test eye movement data into a prediction model obtained after training, and determining the shared situation capability level of a testee corresponding to the test eye movement data according to output data of the prediction model;

the step of collecting the test eye movement data of the testee based on the preset visual materials comprises the following steps:

collecting pupil diameter data of a testee based on a preset visual material;

acquiring blink event data of a testee based on preset visual materials;

2. The method for automatically recognizing co-emotional ability based on the individual eye movement features of the human subject according to claim 1, wherein the step of extracting the eye movement features of the human subject according to the test eye movement data comprises:

3. The method for automatically recognizing shared emotion capacity based on individual eye movement characteristics according to claim 2, wherein the step of performing feature extraction on the first type of eye movement characteristics and the second type of eye movement characteristics based on the eye movement coordinate data and all eye movement events comprises:

detecting gaze events in a loop;

4. The method for automatically recognizing shared emotion capability based on eye movement characteristics of individuals according to claim 2, wherein the step of performing feature extraction on the first type of eye movement characteristics and the second type of eye movement characteristics based on the eye movement coordinate data and all eye movement events further comprises:

detecting a glance event based on a gaze event;

5. The method of claim 2, wherein between the step of detecting eye movement events other than blinking events based on the eye movement coordinate data and the step of extracting the features of the first type and the second type based on the eye movement coordinate data and all eye movement events, the method further comprises:

6. The method for automatically recognizing co-emotional capacity based on the individual eye movement characteristics according to any one of claims 1 to 5, wherein the step of inputting the eye movement characteristics and the co emotional response data into a machine learning model for evaluating co emotional capacity so as to train a prediction model comprises the following steps:

determining the co-estrus response data as dependent variables;

7. The method for automatically recognizing co-emotional ability based on the individual eye movement characteristics according to any one of claims 1 to 5, wherein the step of acquiring the test eye movement data of the human subject based on the preset visual materials is preceded by the following steps:

and/or

8. The method for automatically recognizing shared emotion capability based on individual eye movement characteristics according to claim 7, wherein the step of obtaining the data of a plurality of binocular eye movements of the fixation point on the fixation display interface of the human subject is preceded by the steps of: