CN113591550B - Method, device, equipment and medium for constructing personal preference automatic detection model - Google Patents
Method, device, equipment and medium for constructing personal preference automatic detection model Download PDFInfo
- Publication number
- CN113591550B CN113591550B CN202110672709.6A CN202110672709A CN113591550B CN 113591550 B CN113591550 B CN 113591550B CN 202110672709 A CN202110672709 A CN 202110672709A CN 113591550 B CN113591550 B CN 113591550B
- Authority
- CN
- China
- Prior art keywords
- pir
- sequence
- user
- window
- personal preference
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 33
- 238000000034 method Methods 0.000 title claims abstract description 26
- 210000001747 pupil Anatomy 0.000 claims abstract description 61
- 238000007781 pre-processing Methods 0.000 claims abstract description 23
- 238000012549 training Methods 0.000 claims abstract description 22
- 230000008859 change Effects 0.000 claims abstract description 13
- 239000013598 vector Substances 0.000 claims abstract description 10
- 210000000554 iris Anatomy 0.000 claims description 30
- 238000010276 construction Methods 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 10
- 239000006185 dispersion Substances 0.000 claims description 9
- 238000005286 illumination Methods 0.000 claims description 7
- 238000010606 normalization Methods 0.000 claims description 7
- 238000004422 calculation algorithm Methods 0.000 claims description 6
- 230000011218 segmentation Effects 0.000 claims description 6
- 238000013135 deep learning Methods 0.000 claims description 5
- 230000002123 temporal effect Effects 0.000 claims description 4
- 238000012545 processing Methods 0.000 claims description 2
- 230000006461 physiological response Effects 0.000 abstract description 3
- 230000004044 response Effects 0.000 description 6
- 230000000694 effects Effects 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 3
- 230000004439 pupillary reactions Effects 0.000 description 3
- 230000003542 behavioural effect Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000005065 mining Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000035790 physiological processes and functions Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 230000001149 cognitive effect Effects 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 210000000653 nervous system Anatomy 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000007637 random forest analysis Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9535—Search customisation based on user profiles and personalisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Eye Examination Apparatus (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
Abstract
The invention discloses a pupil change-based method, a device, equipment and a medium for constructing an automatic personal preference detection model, wherein the method comprises the following steps: collecting video of eyes of a user when reading given content, and extracting the ratio PIR of the pupil and the iris of the eyes from a video frame; preprocessing the PIR sequence and assigning corresponding tags, wherein the tags identify the interest degree of a user in the read given content; extracting given several characteristics from the PIR sequence with the label to obtain a characteristic vector with the label, and forming a sample of the user; acquiring a plurality of samples aiming at each label, wherein all samples form a training sample set; and training parameters of the multi-layer perceptron by using the training sample set to obtain the automatic detection module of the personal preference of the user. The invention can use the instant physiological response information of pupil change directly related to the user's mind to infer the user's preference for the specific content on the intelligent device, thereby capturing the relationship between the watching content and the user's preference.
Description
Technical Field
The invention belongs to the technical field of Internet big data application, and particularly relates to a pupil change-based automatic personal preference detection model construction method, device, equipment and medium.
Background
With the rapid development of internet economies and the widespread use of information services, particularly popular e-commerce platforms, both personal and content service platforms need to use advanced tools to search for appropriate information and make selections that meet user needs and desires, thereby improving user experience and overall satisfaction with online services. If the content provided is not attractive to the user, the user will soon discard the internet platform, resulting in a significant revenue loss for the service provider. To address this issue, it is critical to build an efficient user preference model that can capture useful information to personalize the user's experience and accurately infer the user's interests or preferences.
But building accurate user preference models based on complex behavioral activities such as web browsing, merchandise purchase, content clicking, scoring and commenting is very challenging. This is because we can never know exactly what the user really is. Existing models mainly use these explicit and implicit activities that are not directly related to the user's intrinsic perception to model user preferences, which can increase model uncertainty and introduce prediction errors. The effectiveness of content that has been widely used by electronic commerce platforms and content push systems remains far from satisfactory due to the lack of user preference models. Thus, it still needs further research to break the indirect inference limitations of explicit and implicit behavioral activity by more insight into more accurate and reliable information, thereby better inferring the user's preferences.
The pupil is the window of human heart, controlled by the nervous system. The change in pupil size is strongly related to the user's mind when viewing a specific content. In other words, the physiological process of pupillary response can potentially be used to express the extent to which the user is interested in viewing content. Furthermore, with the rapid development of mobile internet technology and the wide application of smart devices with built-in front-facing cameras and enhanced computing power, people increasingly turn to online shopping, education and entertainment, and in actual operation, by following privacy protection policies, pupil responses of users are captured on the smart devices.
Disclosure of Invention
The invention provides a method, a device, equipment and a medium for constructing an automatic personal preference detection model based on pupil variation, which utilize instant physiological response information of pupil variation directly related to the mind of a user to infer the preference of the user, thereby capturing the relationship between watching content and the preference degree of the user.
In order to achieve the technical purpose, the invention adopts the following technical scheme:
a method for constructing an automatic personal preference detection model based on pupil change comprises the following steps:
step 1, collecting video of eyes of a user when reading given content, and extracting the ratio PIR of the pupils and the irises of the eyes from video frames to obtain a PIR sequence;
step 2, carrying out data preprocessing on the PIR sequence obtained in the step 1, and endowing the PIR sequence with a corresponding label, wherein the label identifies the interest degree of a user on the read given content;
step 3, extracting a given plurality of characteristics from the PIR sequence with the label obtained in the step 2 to obtain a characteristic vector with the label, and forming a sample of the user;
step 4, obtaining a plurality of samples according to the steps 1-3 aiming at each label, wherein all the samples form a training sample set;
and step 5, training parameters of the multi-layer perceptron by using the training sample set to obtain the automatic detection model of the personal preference of the user.
In a more preferable technical solution, the extracting the ratio PIR of the pupil to the iris of the eye from the video frame specifically includes:
step 1.1, analyzing videos by using a Haar cascade classifier of OpenCV, and detecting eyes of a user from the videos;
step 1.2, adopting a deep learning network U-ne to divide an eye area image into pupils and irises;
and 1.3, obtaining the pupil diameter and the iris diameter according to the minimum circle corresponding to the split pupil and iris fitting, and further calculating the ratio of the pupil diameter to the iris diameter, namely the ratio PIR of the pupil to the iris.
In a more preferable technical scheme, the given content is provided on the network platform of the electronic equipment according to the interest degree of the user, and the video of the eyes of the user when the given content is read is acquired through a camera on the electronic equipment.
In a more preferable technical scheme, the data preprocessing in the step 2 includes data denoising, specifically: and (3) denoising the PIR sequence acquired in the step (1) by using a box graph, and correcting the noise point by taking the average value of two data points around the noise point.
In a more preferred technical solution, the data preprocessing in step 2 includes dividing PIR sequences of an effective gazing phase by using an I-DT algorithm, specifically:
1) Extracting geometric center coordinates (x, y) of eyes from the video frame to obtain an eye center coordinate sequence corresponding to the video;
2) The initial size of the preset window is the number of center coordinates contained in a given duration time threshold;
3) Traversing the eye center coordinate sequence using a window: calculating the coordinate dispersion of the current window according to the maximum and minimum values of the coordinates of the current window: d= [ max (x) -min (x) ]+[ max (y) -min (y) ]; max (x), min (x), max (y), min (y) are respectively the maximum value and the minimum value of the abscissa and the maximum value and the minimum value of the ordinate of the current window;
4) If the current window dispersion D is higher than the discrete threshold value given by the I-DT tool, the current window is not represented as gazing, and the current window is moved to the right by one point; otherwise, representing that the current window represents fixation, and expanding the current window rightward by one point;
5) Returning to the step 3) until all the eye center coordinates are traversed, wherein the window obtained finally is a fixation window;
6) And taking the PIR sequence corresponding to the central coordinate in the gazing window as the PIR sequence of the effective gazing stage obtained by segmentation, namely the PIR sequence obtained by current pretreatment.
In a more preferable technical scheme, the data preprocessing in the step 2 includes normalization processing, specifically: subtracting PIR data average value under the same illumination condition from the PIR sequence of the effective gazing stage to obtain a PIR sequence obtained by normalization, and taking the PIR sequence as a PIR sequence obtained by pretreatment.
In a preferred embodiment, the given several features include: the average value of the PIR sequence, the slope of the first 1/3 segment PIR sequence, the slope of the last 1/3 segment PIR sequence, the variance of the middle 1/3 segment PIR sequence, the temporal complexity of the middle 1/3 segment PIR sequence, and the sample entropy of the middle 1/3 segment PIR sequence.
An automatic personal preference detection model construction device based on pupil variation, comprising:
the original sequence acquisition module is used for: collecting video of eyes of a user when reading given content, and extracting the ratio PIR of the pupil and the iris of the eyes from a video frame to obtain a PIR sequence;
the data preprocessing module is used for: performing data preprocessing on the PIR sequence obtained by the original sequence obtaining module, and endowing the obtained PIR sequence with a corresponding label, wherein the label identifies the interest degree of a user on the read given content;
a sample construction module for: extracting given characteristics from the PIR sequence with the tag obtained by data preprocessing to obtain a characteristic vector with the tag, and forming a sample of the user;
model training module for: and training parameters of the multi-layer perceptron by using the training sample set to obtain the automatic detection model of the personal preference of the user.
An electronic device includes a memory and a processor, where the memory stores a computer program, and when the computer program is executed by the processor, the processor implements the method for building an automatic personal preference detection model according to any one of the above technical solutions.
A computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the personal preference automatic detection model construction method according to any one of the above-described aspects.
Advantageous effects
The invention provides a method, a device, equipment and a medium for constructing an automatic personal preference detection model based on pupil change, in particular to a general physiological process for mining pupil response when specific content on intelligent equipment is checked, and the ratio PIR of pupil to iris is extracted to reflect the interest degree of a user on the specific content on the intelligent equipment. In addition, in order to solve the diversity of pupil response variation caused by inherent individual differences, the present invention uses a multi-layered perceptron to automatically train and adjust the importance of key features of each person, and then generates an automatic personal preference detection model related to the pupil response of the user. Therefore, the invention can use the instant physiological response information of pupil change directly related to the user's mind to infer the user's preference for the specific content on the intelligent device, thereby capturing the relationship between the watching content and the user's preference.
Drawings
FIG. 1 is a technical layout of a method according to an embodiment of the present application;
FIG. 2 is a schematic flow chart of PIR data collection according to an embodiment of the present application;
fig. 3 is a block diagram of a multi-layer perceptron in accordance with an embodiment of the present application.
Detailed Description
The following describes in detail the embodiments of the present invention, which are developed based on the technical solution of the present invention, and provide detailed embodiments and specific operation procedures, and further explain the technical solution of the present invention.
Example 1
The embodiment provides a method for constructing an automatic personal preference detection model based on pupil change, as shown in fig. 1, comprising the following steps:
and step 1, acquiring videos of eyes of a user when reading given content, and extracting the ratio PIR of the pupils and the irises of the eyes from video frames to obtain a PIR sequence.
The given content is provided on the network platform of the electronic equipment according to the interest degree of the user, and the video of the eyes of the user when the given content is read is acquired through a camera on the electronic equipment.
Fig. 2 shows a PIR data acquisition flow. When a user views content displayed on the smart device, a Haar cascades classifier with built-in cameras and opencvs is used to capture corresponding PIR information. The captured pupil-related video is divided into individual frames, and each frame may be considered to contain an image of the pupil outline. For each frame, a deep learning network U-net is employed to segment the pupil and iris. The U-net may train the deep learning network and the generated model may effectively convert the eye region into a segmented picture. Finally, PIR values are calculated by fitting circles around the iris and pupil by an algorithm that finds the circle containing the smallest area of the iris and pupil of the 2D point set.
The specific process of sorting the ratio PIR of pupil to iris of the eye extracted from the video frame in this embodiment is:
step 1.1, analyzing videos by using a Haar cascade classifier of OpenCV, and detecting eyes of a user from the videos;
step 1.2, adopting a deep learning network U-ne to divide an eye area image into pupils and irises;
and 1.3, obtaining the pupil diameter and the iris diameter according to the minimum circle corresponding to the split pupil and iris fitting, and further calculating the ratio of the pupil diameter to the iris diameter, namely the ratio PIR of the pupil to the iris.
And 2, carrying out data preprocessing on the PIR sequence obtained in the step 1, and endowing the PIR sequence with a corresponding label, wherein the label identifies the interest degree of a user in the read given content.
The preprocessing of PIR sequences in this embodiment includes: denoising, segmentation of PIR sequences at the effective fixation stage, normalization, and explanation of each are performed below.
(1) Denoising method
PIR sequences vary linearly with time. However, due to the effects of illumination variations and human movement, the PIR data obtained still typically contain some outliers. For example, PIR values of some outliers are close to 0 or 1, which violates the physiological phenomenon of pupillary response. The present embodiment detects these outliers using a box graph. The principle of determining the outlier through the box graph is that the quartile and the quartile spacing are adopted, the quartile has a certain degree of resistance, up to 25% of data can be arbitrarily far without greatly interfering with the quartile, therefore, the outlier cannot influence the data shape of the box graph, and the result of identifying the outlier through the box graph is more objective. It can be seen that the box plot has certain advantages in identifying outliers. And (3) denoising the PIR sequence by adopting a box graph, and correcting by using an average value of two points around the outlier.
(2) PIR sequence for segmenting active gaze phase
When people watch content displayed on the smart device screen, they have to look at the content. Thus, gaze time when viewing different content may be used to represent an effective pupillary response. Since the gaze point during gaze is certainly fixed at a specific location, the gaze points in time of gaze often tend to be closely clustered together. The present invention identifies gaze as a set of consecutive points within a particular dispersion or maximum interval.
The embodiment utilizes an I-DT algorithm to segment PIR sequences of an effective gazing phase, and specifically comprises the following steps:
1) Extracting geometric center coordinates (x, y) of eyes from the video frame to obtain an eye center coordinate sequence corresponding to the video;
2) The initial size of the preset window is the number of center coordinates contained in a given duration time threshold;
3) Traversing the eye center coordinate sequence using a window: calculating the coordinate dispersion of the current window according to the maximum and minimum values of the coordinates of the current window: d= [ max (x) -min (x) ]+[ max (y) -min (y) ]; max (x), min (x), max (y), min (y) are respectively the maximum value and the minimum value of the abscissa and the maximum value and the minimum value of the ordinate of the current window;
4) If the current window dispersion D is higher than the discrete threshold value given by the I-DT tool, the current window is not represented as gazing, and the current window is moved to the right by one point; otherwise, representing that the current window represents fixation, and expanding the current window rightward by one point;
5) Returning to the step 3) until all the eye center coordinates are traversed, wherein the window obtained finally is a fixation window;
6) And taking the PIR sequence corresponding to the central coordinate in the gazing window as the PIR sequence of the effective gazing stage obtained by segmentation, namely the PIR sequence obtained by current pretreatment.
(3) Normalization: subtracting PIR data average value under the same illumination condition from the PIR sequence of the effective gazing stage to obtain a PIR sequence obtained by normalization, and taking the PIR sequence as a PIR sequence obtained by pretreatment.
Variations in ambient light intensity can affect the diameter of the pupil and cause errors in the PIR in the time domain. Using a model of the relationship between pupil diameter, cognitive load and lighting conditions, as follows:
PD＝PD light +PD task ，
where PD is the pupil diameter measured directly. PD (potential difference) device light Is the average pupil diameter, PD, under given illumination conditions task Is the normalized pupil diameter resulting from a particular task. In the present embodiment, PD task Corresponds to the pupil diameter caused by viewing content on the smart device screen, thus PD task ＝PD-PD light 。
In the case where the light intensity of the viewing content during gaze (recorded by the built-in light sensor) remains unchanged, PD can be obtained by simply subtracting the average pupil diameter under the current illumination conditions from the measured pupil diameter PD task . However, when the light intensity is dynamically changing during gaze, the present invention can effectively find and record the point of abrupt light intensity changes to divide the gaze phase into several smaller parts. Each section will repeat the above procedure to pre-process the original PIR values during fixation.
The iris diameter does not change after 7 years of age, so the effect of ambient light intensity on pupil diameter is directly reflected on PIR changes. In this embodiment, in order to reduce the influence of the ambient light intensity on the PIR sequence, the PIR sequence in the effective gazing stage obtained by segmentation is subtracted from the PIR data average value under the same illumination condition, so that the PIR sequence free from the influence of the ambient light intensity can be obtained, and the PIR sequence can show the interest degree on the reading content.
And 3, extracting a given plurality of characteristics from the PIR sequence with the label obtained in the step 2 to obtain a characteristic vector with the label, and forming one sample of the user.
First, in order to mine out key features that describe user preferences, the PIR change process is regarded as a time-varying sequence response value, and a large number of time-series features that may be related to user preferences are extracted. By segmenting and preprocessing the pupil responses while viewing different content, the present embodiment uses the Python module Tsfresh to perform feature mining extraction on the extracted PIR sequences, so that the extracted features can be used to describe or cluster time sequences. Furthermore, they can also be used to build models that perform classification or regression tasks on time series. In general, these features can provide new insights into the time series and its dynamic characteristics.
In order to find key features that effectively distinguish PIR sequences, the present embodiment uses the feature extractor tool of the Tsfresh module to perform feature extraction and preliminary feature filtering to screen potentially useful features. To determine key features from among a multitude of features, random forest algorithms in ensemble learning are employed to weigh the importance of each function in relation to user preferences. Obtaining 6 key features with larger weight for constructing an automatic personal preference detection model: average PIR sequence, slope of the first 1/3 segment PIR sequence, slope of the last 1/3 segment PIR sequence, variance of the middle 1/3 segment PIR sequence, temporal complexity of the middle 1/3 segment PIR sequence, and sample entropy of the middle 1/3 segment PIR sequence.
(1) PIR sequence mean
By giving a PIR segment S PIR The mean of PIR sequences can be calculated by the following formula:
wherein x is i Represent S PIR And n is the length of the PIR sequence.
(2) Slope k of the first 1/3 segment PIR curve 1 Slope k of the PIR curve of the 1/3 segment after the sum 2
Slope k of the first 1/3 segment PIR curve 1 And the post 1/3 segment PIR curveSlope k of (2) 2 Is the slope of the regression line obtained by least squares regression of the first 1/3 segment and the last 1/3 segment PIR data. The calculation of the least squares regression function is shown below.
Where (x, y) is a pair of observations, and x= [ x ] 1 ,x 2 ,…,x n ] T ∈R n Is a time series, y i Represent S PIR PIR data in (a), m represents observation data (x i ,y i ) The number of groups of (i=1, …, m) and y=f (x, w) is a theoretical function, where w= [ w ] 1 ,w 2 ,…,w n ] T Is a parameter to be determined. L (L) i (x) (i=1, 2, …, m) is a residual function. Thus, the slope k is calculated as follows:
(3) Variance var of intermediate 1/3 segment PIR sequences
PIR sequence S PIR The variance var of the middle 1/3 segment is calculated as:
(4) Temporal complexity of intermediate 1/3 segment PIR sequences
Considering that the actual PIR sequences collected are not necessarily of equal length and that the time complexity CE is also affected by the length of the time series, we divide the value of CE by the length of the time series to get a normalized CE. Normalized time complexity CE is calculated as
Wherein x is i Represent S PIR I-th PIR data in (a).
(5) Sample entropy of intermediate 1/3 segment PIR sequence
For a PIR sequence { x (N) } =x (1), x (2), …, x (N) consisting of N PIR data, the sample entropy samplen is calculated as follows:
a. forming a group of vector sequences with m dimension according to sequence numbers, X m (1),…,X m (N-m+1) wherein X m (i) = { x (i), x (i+1), …, x (i+m-1) }, 1.ltoreq.i.ltoreq.N-m+1. These vectors represent m consecutive x values starting from the i-th point.
b, define vector X m (i) And X is m (j) Distance d [ X ] between m (i),X m (j)]The absolute value of the maximum difference in the corresponding elements is:
d[X m (i),X m (j)]＝max k＝0,…,m-1 (|x(i+k)-x(j+k)|)
c, for a given X m (i) Statistics of X m (i) And X is m (j) The number of j (1.ltoreq.j.ltoreq.N-m, j.noteq.i) with a distance between them of r or less is denoted as B i . For 1.ltoreq.i.ltoreq.N-m, define:
d, definition B (m) (r) is:
e, increasing the dimension number to m+1, and calculating X m+1 (i) And X is m+1 (j) The number that the distance (1.ltoreq.j.ltoreq.N-m, j.noteq.i) is less than or equal to r is denoted as A i 。The definition is as follows:
f, definition A (m) (r) is:
thus B (m) (r) is the probability that two sequences match m points with a similar tolerance r, while A (m) (r) is the probability that two sequences match m+1 points. Sample entropy is defined as:
when N is a finite value, the sample entropy can be estimated by:
step 4, obtaining a plurality of samples according to the steps 1-3 aiming at each label, wherein all the samples form a training sample set;
in this embodiment, the interest level of the user is divided into: five levels of uninteresting, likely uninteresting, general, likely interesting and interesting, each level corresponding to a label, multiple samples of the user need to be taken for each level of interest to construct a training sample set.
And step 5, training parameters of the multi-layer perceptron by using the training sample set to obtain the automatic detection model of the personal preference of the user.
Fig. 3 is an illustration of a multi-layer perceptron. The invention classifies the interestingness for representing user preference into 5 levels: not of interest, possibly not of interest, in general, possibly of interest and very of interest. Accordingly, personalized categorization of user preferences may be considered a typical multi-class problem. The invention adopts a multi-layer perceptron (MLP), can be used for multiple classification tasks, is excellent in nonlinear data, and can classify PIR sequences of all segments and marks. Wherein the input of the multi-classifier is the value of the 6 key features described above and the output is one of 5 levels of interest.
The final obtained automatic detection model of the personal preference of the user can be used for: (1) When the user reads any content on the intelligent device, collecting video of eyes during reading, and extracting the ratio PIR of the pupils and the irises of the eyes from video frames to obtain a PIR sequence; (2) Performing data preprocessing on the PIR sequence obtained currently according to the same method in the step 2; (3) Extracting a plurality of given characteristics from the PIR sequence obtained by pretreatment according to the same method in the step 3; (4) And inputting the extracted given plurality of characteristics into an automatic personal preference detection model of the user, and outputting the interest degree of the user in the current reading content. Further, it may be determined whether to recommend certain related content to the user based on the level of interest.
Example 2
The embodiment provides a personal preference automatic detection model construction device based on pupil change, which comprises:
the original sequence acquisition module is used for: collecting video of eyes of a user when reading given content, and extracting the ratio PIR of the pupil and the iris of the eyes from a video frame to obtain a PIR sequence;
the data preprocessing module is used for: performing data preprocessing on the PIR sequence obtained by the original sequence obtaining module, and endowing the obtained PIR sequence with a corresponding label, wherein the label identifies the interest degree of a user on the read given content;
a sample construction module for: extracting given characteristics from the PIR sequence with the tag obtained by data preprocessing to obtain a characteristic vector with the tag, and forming a sample of the user;
model training module for: and training parameters of the multi-layer perceptron by using the training sample set to obtain the automatic detection model of the personal preference of the user.
Example 3
The present embodiment provides an electronic device, including a memory and a processor, where the memory stores a computer program, and when the computer program is executed by the processor, the processor implements the method for building an automatic personal preference detection model according to embodiment 1.
Example 4
The present embodiment provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the personal preference automatic detection model construction method as described in embodiment 1.
The above embodiments are preferred embodiments of the present application, and various changes or modifications may be made on the basis thereof by those skilled in the art, and such changes or modifications should be included within the scope of the present application without departing from the general inventive concept.
Claims (9)
1. The method for constructing the automatic personal preference detection model based on pupil change is characterized by comprising the following steps of:
step 1, collecting video of eyes of a user when reading given content, and extracting the ratio PIR of the pupils and the irises of the eyes from video frames to obtain a PIR sequence;
step 2, carrying out data preprocessing on the PIR sequence obtained in the step 1, and endowing the PIR sequence with a corresponding label, wherein the label identifies the interest degree of a user on the read given content;
the data preprocessing in the step 2 comprises dividing the PIR sequence of the effective gazing phase by using an I-DT algorithm, and specifically comprises the following steps:
1) Extracting geometric center coordinates (x, y) of eyes from the video frame to obtain an eye center coordinate sequence corresponding to the video;
2) The initial size of the preset window is the number of center coordinates contained in a given duration time threshold;
3) Traversing the eye center coordinate sequence using a window: calculating the coordinate dispersion of the current window according to the maximum and minimum values of the coordinates of the current window: d= [ max (x) -min (x) ]+[ max (y) -min (y) ]; max (x), min (x), max (y), min (y) are respectively the maximum value and the minimum value of the abscissa and the maximum value and the minimum value of the ordinate of the current window;
4) If the current window dispersion D is higher than the discrete threshold value given by the I-DT tool, the current window is not represented as gazing, and the current window is moved to the right by one point; otherwise, representing that the current window represents fixation, and expanding the current window rightward by one point;
5) Returning to the step 3) until all the eye center coordinates are traversed, wherein the window obtained finally is a fixation window;
6) The PIR sequence corresponding to the central coordinate in the fixation window is used as the PIR sequence of the effective fixation phase obtained by segmentation, namely the PIR sequence obtained by current pretreatment;
step 3, extracting a given plurality of characteristics from the PIR sequence with the label obtained in the step 2 to obtain a characteristic vector with the label, and forming a sample of the user;
step 4, obtaining a plurality of samples according to the steps 1-3 aiming at each label, wherein all the samples form a training sample set;
and step 5, training parameters of the multi-layer perceptron by using the training sample set to obtain the automatic detection model of the personal preference of the user.
2. The method for constructing the automatic personal preference detection model according to claim 1, wherein the extracting the ratio PIR of the pupil to the iris of the eye from the video frame is specifically as follows:
step 1.1, analyzing videos by using a Haar cascade classifier of OpenCV, and detecting eyes of a user from the videos;
step 1.2, adopting a deep learning network U-ne to divide an eye area image into pupils and irises;
and 1.3, obtaining the pupil diameter and the iris diameter according to the minimum circle corresponding to the split pupil and iris fitting, and further calculating the ratio of the pupil diameter to the iris diameter, namely the ratio PIR of the pupil to the iris.
3. The method for constructing an automatic personal preference detection model according to claim 1, wherein the given content is provided on a network platform of the electronic device according to the interest level of the user, and the video of the eyes of the user when the given content is read is acquired through a camera on the electronic device.
4. The method for constructing an automatic personal preference detection model according to claim 1, wherein the data preprocessing in step 2 includes data denoising, specifically: and (3) denoising the PIR sequence acquired in the step (1) by using a box graph, and correcting the noise point by taking the average value of two data points around the noise point.
5. The method for constructing an automatic personal preference detection model according to claim 1, wherein the data preprocessing in step 2 includes normalization processing, specifically: subtracting PIR data average value under the same illumination condition from the PIR sequence of the effective gazing stage to obtain a PIR sequence obtained by normalization, and taking the PIR sequence as a PIR sequence obtained by pretreatment.
6. The method for building an automatic personal preference detection model according to claim 1, wherein the given plurality of features include: the average value of the PIR sequence, the slope of the first 1/3 segment PIR sequence, the slope of the last 1/3 segment PIR sequence, the variance of the middle 1/3 segment PIR sequence, the temporal complexity of the middle 1/3 segment PIR sequence, and the sample entropy of the middle 1/3 segment PIR sequence.
7. A pupil change-based personal preference automatic detection model construction apparatus, comprising:
the original sequence acquisition module is used for: collecting video of eyes of a user when reading given content, and extracting the ratio PIR of the pupil and the iris of the eyes from a video frame to obtain a PIR sequence;
the data preprocessing module is used for: performing data preprocessing on the PIR sequence obtained by the original sequence obtaining module, and endowing the obtained PIR sequence with a corresponding label, wherein the label identifies the interest degree of a user on the read given content;
the data preprocessing comprises dividing PIR sequences of effective gazing phases by using an I-DT algorithm, and specifically comprises the following steps:
1) Extracting geometric center coordinates (x, y) of eyes from the video frame to obtain an eye center coordinate sequence corresponding to the video;
2) The initial size of the preset window is the number of center coordinates contained in a given duration time threshold;
3) Traversing the eye center coordinate sequence using a window: calculating the coordinate dispersion of the current window according to the maximum and minimum values of the coordinates of the current window: d= [ max (x) -min (x) ]+[ max (y) -min (y) ]; max (x), min (x), max (y), min (y) are respectively the maximum value and the minimum value of the abscissa and the maximum value and the minimum value of the ordinate of the current window;
4) If the current window dispersion D is higher than the discrete threshold value given by the I-DT tool, the current window is not represented as gazing, and the current window is moved to the right by one point; otherwise, representing that the current window represents fixation, and expanding the current window rightward by one point;
5) Returning to the step 3) until all the eye center coordinates are traversed, wherein the window obtained finally is a fixation window;
6) The PIR sequence corresponding to the central coordinate in the fixation window is used as the PIR sequence of the effective fixation phase obtained by segmentation, namely the PIR sequence obtained by current pretreatment;
a sample construction module for: extracting given characteristics from the PIR sequence with the tag obtained by data preprocessing to obtain a characteristic vector with the tag, and forming a sample of the user;
model training module for: and training parameters of the multi-layer perceptron by using the training sample set to obtain the automatic detection model of the personal preference of the user.
8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein the computer program, when executed by the processor, causes the processor to implement the method for automatically detecting personal preference model construction according to any one of claims 1 to 6.
9. A computer-readable storage medium, on which a computer program is stored, characterized in that the computer program, when executed by a processor, implements the personal preference automatic detection model construction method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110672709.6A CN113591550B (en) | 2021-06-17 | 2021-06-17 | Method, device, equipment and medium for constructing personal preference automatic detection model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110672709.6A CN113591550B (en) | 2021-06-17 | 2021-06-17 | Method, device, equipment and medium for constructing personal preference automatic detection model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113591550A CN113591550A (en) | 2021-11-02 |
| CN113591550B true CN113591550B (en) | 2024-03-26 |
Family
ID=78243896
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110672709.6A Active CN113591550B (en) | 2021-06-17 | 2021-06-17 | Method, device, equipment and medium for constructing personal preference automatic detection model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113591550B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115862810B (en) * | 2023-02-24 | 2023-10-17 | 深圳市铱硙医疗科技有限公司 | VR rehabilitation training method and system with quantitative evaluation function |
| CN117314559B (en) * | 2023-09-21 | 2024-08-02 | 淮阴工学院 | Personalized commodity optimization pushing method and system based on user pupil response |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102253714A (en) * | 2011-07-05 | 2011-11-23 | 北京工业大学 | Selective triggering method based on vision decision |
| CN103169485A (en) * | 2013-02-01 | 2013-06-26 | 广东平成广告有限公司 | Cognition curve generation system and cognition curve generation method based on video |
| CN105516446A (en) * | 2014-09-26 | 2016-04-20 | 中兴通讯股份有限公司 | Terminal screen backlight adjusting method and terminal |
| CN105744881A (en) * | 2013-09-19 | 2016-07-06 | 儿童国家医疗中心 | Controlling a luminous means having at least two semiconductor light sources |
| CN106169063A (en) * | 2016-06-22 | 2016-11-30 | 江苏大学 | A kind of method in automatic identification user's reading interest district |
| CN109376304A (en) * | 2018-11-30 | 2019-02-22 | 维沃移动通信有限公司 | A kind of information recommendation method and device |
| CN109740466A (en) * | 2018-12-24 | 2019-05-10 | 中国科学院苏州纳米技术与纳米仿生研究所 | Acquisition methods, the computer readable storage medium of advertisement serving policy |
| CN111291701A (en) * | 2020-02-20 | 2020-06-16 | 哈尔滨理工大学 | Sight tracking method based on image gradient and ellipse fitting algorithm |
| CN111344703A (en) * | 2017-11-24 | 2020-06-26 | 三星电子株式会社 | User authentication device and method based on iris recognition |
| CN111708939A (en) * | 2020-05-29 | 2020-09-25 | 平安科技（深圳）有限公司 | Push method and device based on emotion recognition, computer equipment and storage medium |
| CN112230774A (en) * | 2020-10-15 | 2021-01-15 | 北京津发科技股份有限公司 | Method and system for analyzing reaction behavior based on human-computer interaction |
| WO2021068387A1 (en) * | 2019-10-08 | 2021-04-15 | 梅州市青塘实业有限公司 | Non-contact vital sign detection device and system |
-
2021
- 2021-06-17 CN CN202110672709.6A patent/CN113591550B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102253714A (en) * | 2011-07-05 | 2011-11-23 | 北京工业大学 | Selective triggering method based on vision decision |
| CN103169485A (en) * | 2013-02-01 | 2013-06-26 | 广东平成广告有限公司 | Cognition curve generation system and cognition curve generation method based on video |
| CN105744881A (en) * | 2013-09-19 | 2016-07-06 | 儿童国家医疗中心 | Controlling a luminous means having at least two semiconductor light sources |
| CN105516446A (en) * | 2014-09-26 | 2016-04-20 | 中兴通讯股份有限公司 | Terminal screen backlight adjusting method and terminal |
| CN106169063A (en) * | 2016-06-22 | 2016-11-30 | 江苏大学 | A kind of method in automatic identification user's reading interest district |
| CN111344703A (en) * | 2017-11-24 | 2020-06-26 | 三星电子株式会社 | User authentication device and method based on iris recognition |
| CN109376304A (en) * | 2018-11-30 | 2019-02-22 | 维沃移动通信有限公司 | A kind of information recommendation method and device |
| CN109740466A (en) * | 2018-12-24 | 2019-05-10 | 中国科学院苏州纳米技术与纳米仿生研究所 | Acquisition methods, the computer readable storage medium of advertisement serving policy |
| WO2021068387A1 (en) * | 2019-10-08 | 2021-04-15 | 梅州市青塘实业有限公司 | Non-contact vital sign detection device and system |
| CN111291701A (en) * | 2020-02-20 | 2020-06-16 | 哈尔滨理工大学 | Sight tracking method based on image gradient and ellipse fitting algorithm |
| CN111708939A (en) * | 2020-05-29 | 2020-09-25 | 平安科技（深圳）有限公司 | Push method and device based on emotion recognition, computer equipment and storage medium |
| CN112230774A (en) * | 2020-10-15 | 2021-01-15 | 北京津发科技股份有限公司 | Method and system for analyzing reaction behavior based on human-computer interaction |
Non-Patent Citations (1)
| Title |
|---|
| 基于视频图像的眼动追踪系统算法;王际航;刘富;袁雨桐;刘星;;吉林大学学报(信息科学版)(02);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113591550A (en) | 2021-11-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Linardos et al. | Simple vs complex temporal recurrences for video saliency prediction | |
| US11386284B2 (en) | System and method for improving speed of similarity based searches | |
| Kumar et al. | The p-destre: A fully annotated dataset for pedestrian detection, tracking, and short/long-term re-identification from aerial devices | |
| Borji et al. | Analysis of scores, datasets, and models in visual saliency prediction | |
| US8724910B1 (en) | Selection of representative images | |
| US10108852B2 (en) | Facial analysis to detect asymmetric expressions | |
| Daza et al. | mEBAL: A multimodal database for eye blink detection and attention level estimation | |
| US20170095192A1 (en) | Mental state analysis using web servers | |
| Do et al. | Deep neural network-based fusion model for emotion recognition using visual data | |
| CN113591550B (en) | Method, device, equipment and medium for constructing personal preference automatic detection model | |
| US11430561B2 (en) | Remote computing analysis for cognitive state data metrics | |
| EP2915101A1 (en) | Method and system for predicting personality traits, capabilities and suggested interactions from images of a person | |
| US9013591B2 (en) | Method and system of determing user engagement and sentiment with learned models and user-facing camera images | |
| Zhang et al. | Eye center localization and gaze gesture recognition for human–computer interaction | |
| CN113766330A (en) | Method and device for generating recommendation information based on video | |
| CN105979366A (en) | Smart television and content recommending method and content recommending device thereof | |
| Beyan et al. | Investigation of small group social interactions using deep visual activity-based nonverbal features | |
| KR102191044B1 (en) | Advertising systems that are provided through contents analytics and recommendation based on artificial intelligence facial recognition technology | |
| Jiang et al. | DeepVS2. 0: A saliency-structured deep learning method for predicting dynamic visual attention | |
| Xiang et al. | Salad: A multimodal approach for contextual video advertising | |
| Van Huynh et al. | Emotion recognition by integrating eye movement analysis and facial expression model | |
| KR20180082950A (en) | Display apparatus and service providing method of thereof | |
| Gupta | PERSIST: Improving micro-expression spotting using better feature encodings and multi-scale Gaussian TCN | |
| CN112685596A (en) | Video recommendation method and device, terminal and storage medium | |
| US9501710B2 (en) | Systems, methods, and media for identifying object characteristics based on fixation points |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |