CN108415955B - Interest point database establishing method based on eye movement fixation point moving track - Google Patents
Interest point database establishing method based on eye movement fixation point moving track Download PDFInfo
- Publication number
- CN108415955B CN108415955B CN201810116386.0A CN201810116386A CN108415955B CN 108415955 B CN108415955 B CN 108415955B CN 201810116386 A CN201810116386 A CN 201810116386A CN 108415955 B CN108415955 B CN 108415955B
- Authority
- CN
- China
- Prior art keywords
- point
- dimensional
- interest
- model
- eye movement
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/51—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
Abstract
The invention discloses a method for establishing an interest point database based on eye movement fixation point movement tracks. The method comprises the steps of projecting a three-dimensional model into a two-dimensional picture by using a projection technology, making the picture into a video, placing the video on an eye tracker to be watched by a volunteer, further collecting data of an eye movement fixation point, then dividing the video according to frames, extracting the eye movement fixation point in the picture, and obtaining a coordinate point of the eye movement fixation point on the three-dimensional model by using a two-dimensional to three-dimensional mapping technology. And finally, establishing the obtained eye movement fixation point data into a point-of-interest database by using an algorithm for establishing a group Truth. The invention is more suitable for reconstructing the model, because the eye tracker can be used for dividing the part of the most concerned region and the part of the no concerned region of the human eyes on the model. The reconstruction accuracy can be enhanced for the interested places of human beings through the interest point standard library, and the reconstruction accuracy is relatively reduced for the non-interested regions, so that the workload and the storage capacity of model reconstruction are reduced.
Description
Technical Field
The invention relates to a method for establishing a point-of-interest database of a three-dimensional model by collecting eye movement fixation point movement tracks when a human eye observes the three-dimensional model by using an eye movement instrument, analyzing and processing the tracks.
Background
The point-of-interest database, which is currently used as a three-dimensional model with a large reference amount, is a text "Evaluation of 3D interest point detection technologies via human-generated ground route" published in Vis Comut by Helin Dutagaci et al in 2012. In the experiment, 24 three-dimensional models are displayed on a window of a website, 23 experimenters can obtain views from different angles of the models through keys arranged on the window, and the experimenters can mark the interested areas of the models through a mouse after observing the views. And collecting data marked by the experimenter, eliminating pseudo interest points through an algorithm, and integrating an interest point data set of the three-dimensional model.
The points of interest obtained by this way of the database are not objective. Because the experimenter observes the data for a long time, the selected corner points are strongly subjective through brain thinking. This is in contrast to the point of interest where the model comes into sight of the human eye, drawing the attention of the human vision the first time. The invention establishes the interest point database through the eye movement fixation point moving track, and can truly reflect the interest points when the human observes the model.
Disclosure of Invention
In view of the fact that the current three-dimensional model interest point data set cannot reflect the reality of a human being when observing the model. The invention provides a method for establishing an interest point database based on eye movement fixation point movement tracks. The interest point database established by the method can reflect the real condition of human eyes when observing the model, and the data is more real and reliable.
In order to achieve the purpose, the invention is realized by the following technical scheme, which specifically comprises the following 4 steps:
and (1) collecting the three-dimensional model and making the three-dimensional model into a video material required by the experiment.
And (2) placing the video on an eye movement instrument for an experimenter to watch, obtaining data of the eye movement fixation point, and synthesizing the video with the eye movement fixation point through corresponding software.
And (3) generating a three-dimensional model with the eye movement fixation point by using the video with the eye movement fixation point through a motion point extraction and three-dimensional mapping algorithm.
And (4) analyzing the eye movement fixation points of the experimenters, sorting the eye movement fixation points of all the experimenters to obtain a three-dimensional model interest point set, discarding some inappropriate and abnormal data, merging the interest points and establishing an interest point database.
The method for making the video material in the step (1) is as follows:
the 24 stanford three-dimensional model libraries and models in the SHREC2007 model database were selected, and these three-dimensional models are widely used in the standard libraries for three-dimensional model research. And saving two groups of data for each three-dimensional model by using MATLAB, wherein one group is formed by sequentially rotating the three-dimensional models around an X axis at intervals of 60 degrees: three-dimensional models which rotate 0 degree, 60 degree, 120 degree, 180 degree, 240 degree, 300 degree and 360 degree respectively; one group is that the three-dimensional model rotates around the Y axis at intervals of 60 degrees in sequence: three-dimensional models which rotate 0 degree, 60 degree, 120 degree, 180 degree, 240 degree, 300 degree and 360 degree respectively; the Z-axis direction is selected as a viewpoint, and the model at 12 angles is projected on the XOY plane, so that 12 two-dimensional projection pictures are obtained by one model.
A formula for three-dimensional model rotation and a formula for projection onto the XOY plane.
Rotation around the X axis:
z′＝zcosθ-xsinθ
x′＝xcosθ+zsinθ
y′＝y
rotation around the Y axis:
z′＝zcosθ-ysinθ
x′＝x
y′＝ycosθ+zsinθ
formula for parallel projection:
x′＝x
y′＝y
z′＝0
and transforming a picture every 1.5 seconds by using a video editor Movie Maker, synthesizing 12 two-dimensional pictures of each model into a short video, inserting a blank picture of ten seconds between every two models as rest time, and synthesizing 6 models into a long video. A total of 4 long videos were made from 24 models.
The precondition for the experimenter to carry out the experiment in the step (2) is as follows:
a) the display device of the image is placed on the left side, and the experimenter sits right in front of the display device of the image and keeps the distance at 70cm, while the experimenter keeps the eyes in accordance with the height of the screen, and can look up the center of the screen.
b) The operating personnel is on the right side, and with computer control video broadcast on display device, the operating personnel is kept apart with the light shield baffle with the experimenter.
c) The experiment interference to the experimenter caused by other light sources is prevented by isolating baffles around the experimenter and isolating curtains around the laboratory.
d) The sound of the environment of the laboratory is kept not more than 30dB, an ideal quiet environment is created, and the experiment interference of other sound sources to an experimenter is prevented.
The specific experiment in the step (2) is as follows:
the eye position of the test person was first adjusted using the iViewX software. After the pupil Image of the experimenter appears in the Eye Image frame, an operator needs to adjust the relative position of the screen and the experimenter, so that the pupil Image on the screen can be stably displayed in a centered mode. Wherein, the slight movement of the head of the experimenter can not influence the projection, and the loss of the image caused by blinking can be quickly recovered.
The movement locus of the eye movement fixation point when the experimenter watches the video is collected by using the expert Center software. The sight of the eyes of an experimenter needs to be calibrated, after calibration is finished, calibration feedback, namely deviation in the X, Y direction, can occur, when the deviation X and the deviation Y are smaller than 1.0, an experiment can be carried out, and then model video playing can occur;
and finally, synthesizing the material video and the eye movement fixation point tracking track by using BeGaze analysis software to obtain the model video with the experimenter eye movement fixation point track.
And (3) cutting the model video with the experimenter eye movement fixation point track acquired by the eye tracker according to frames, acquiring eye movement fixation point coordinates on each picture by a moving point extraction algorithm, and converting two-dimensional coordinates of a moving point on the picture into three-dimensional coordinates on a space by a three-dimensional mapping algorithm.
Firstly, extracting coordinates of a two-dimensional eye movement fixation point, and utilizing FFmpeg built software to cut a synthesized video into pictures according to frames to obtain two-dimensional pictures; and acquiring the color of the eye movement fixation point in the picture, wherein the movement track of the eye movement fixation point is embodied in a mode that the eye movement fixation point moves on the model, and the eye movement fixation point is orange through software setting in the last step. We call it the moving point color here and set the color tolerance value; then setting a two-dimensional array, initializing the two-dimensional array to set the value to be 1, comparing the two-dimensional array with a two-dimensional picture, and if the two-dimensional array is compared with the moving point color, obtaining three RGB values which are all in a tolerance range and meet the condition; otherwise, the point is not the eye movement point we are looking for. If no point satisfies the condition, the point of interest flag is set to 0, otherwise to 1. And (3) putting the points meeting the conditions into a point array, respectively recording the maximum and minimum row and column of the points meeting the conditions, taking the median of the maximum and minimum row and column coordinates, namely the coordinate (row, rank) of the interest point, and outputting the coordinate point of the interest point.
And then obtaining a three-dimensional model with an eye movement fixation point through a three-dimensional mapping algorithm. Because the two-dimensional picture of the model is obtained by projection on the three-dimensional XOY plane, we can see from the nature of the projection that the two-dimensional and three-dimensional coordinate points are linked to each other. The three-dimensional coordinates (x, y, z) are determined by first projecting the model onto the XOY plane so that x, y coordinate values are determined and the z coordinate is taken of the vertex of the model surface closest to the viewpoint and the z coordinate value of the vertex on the three-dimensional model is the z coordinate value of the eye gaze point on the three-dimensional model, and if the vertex is not present on the model surface, selecting the coordinate value of the model vertex at a distance within a threshold range as the coordinate value of the eye gaze point. Further, since the model is rotated around the coordinate axis when preparing the picture material, it is now required to be rotated in the reverse direction by a corresponding angle. And finally, obtaining the data of the interest points collected by the eye tracker.
And (4) correspondingly analyzing the experimental data result of the experimenter, sorting all the extracted eye movement fixation points to obtain a database of an interest point set, discarding some inappropriate and abnormal data, and appropriately merging the interest points, wherein the integration method comprises the following steps:
when constructing an evaluation library that evaluates the point of interest operator, two criteria are selected, one being the radius of the region of interest and the other being the number of action points within the region. The radius of the region of interest is set to σ dM, where dM represents the model diameter, i.e., the maximum euclidean distance between all vertices of the model M, and σ represents a constant coefficient. All interest points with a measurement distance of less than 2 σ dM are divided into the same region, and if the data volume of different experimenters in the region is less than n, the interest points in the region are discarded. Selecting a point from each region as a representative, and using the point as a standard interest point, wherein the standard is as follows: these points of interest that are selected as criteria need to satisfy the minimum sum of their geometric distances from all other points of interest in the area in which they are located. Note that this is also reasonable if the interest points of the two regions overlap. If the distance between two regions is less than 2 σ dM, the representation with the smaller number of points in the two regions is discarded from the interest point set of the evaluation criterion, and the representation with the larger number of points in the regions is taken as the interest point of the criterion. We denote the point of interest criteria library by the parameters n and σ, i.e., GM(n, σ) represents the point of interest data set for a particular model M. The corresponding values of these two parameters determine the point of interest criteria library. When the value of n is taken to be correspondingly higher, more moving points will fall within the interest area as σ increases, which are considered reasonable because not all volunteers select the details of a certain model as interest points, and σ increases accept more local variations of the labeled points. However, as σ is further increased, it doesOften, the defined regions will contain different regions of interest, so that the closely interesting points marked on different structures start to merge. The average number of point of interest evaluation criteria libraries given will vary with n and σ.
The invention has the following beneficial effects:
the method for establishing the interest point database is more suitable for reconstructing the model, because the eye tracker can be used for dividing the most concerned region part and the non-concerned region part of human eyes on the model. The reconstruction accuracy can be enhanced for the interested places of human beings through the interest point standard library, and the reconstruction accuracy is relatively reduced for the uninterested regions, so that the workload and the storage capacity of model reconstruction can be reduced.
Drawings
FIG. 1 is a database design flow diagram.
Fig. 2 is a video timing diagram.
Fig. 3 is an experimental environment diagram.
FIG. 4 is a flow chart of two-dimensional eye movement extraction.
Fig. 5 is a flow chart of data point integration.
Detailed Description
The invention is further described below with reference to the accompanying drawings.
A design flow of a method for establishing a point of interest database based on an eye movement fixation point moving track is shown in figure 1, and specifically comprises the following 4 steps:
step (1) firstly, a three-dimensional database model needs to be collected, and materials needed by an experiment are manufactured. The specific operation is as follows:
the 24 stanford three-dimensional model libraries and models in the SHREC2007 model database were selected, and these three-dimensional models are widely used in the standard libraries for three-dimensional model research. Storing two groups of data for each three-dimensional model by using MATLAB, wherein one group of data is the three-dimensional model which is sequentially rotated by 60 degrees around an X axis and is respectively rotated by 0 degree, 60 degrees, 120 degrees, 180 degrees, 240 degrees, 300 degrees and 360 degrees; one group is three-dimensional models of which the three-dimensional models rotate by 60 degrees around the Y axis in sequence, and the three-dimensional models rotate by 0 degree, 60 degrees, 120 degrees, 180 degrees, 240 degrees, 300 degrees and 360 degrees respectively; the Z-axis direction is selected as a viewpoint, and the model at 12 angles is projected on the XOY plane, so that 12 two-dimensional projection pictures are obtained by one model. The following are the equations for the rotation of the three-dimensional model and the equations projected onto the XOY plane.
Rotation around the X axis:
z′＝zcosθ-xsinθ
x′＝xcosθ+zsinθ
y′＝y
rotation around the Y axis:
z′＝zcosθ-ysinθ
x′＝x
y′＝ycosθ+zsinθ
formula for parallel projection:
x′＝x
y′＝y
z′＝0
and transforming a picture every 1.5 seconds by using a video editor Movie Maker, synthesizing 12 two-dimensional pictures of each model into a short video, inserting a blank picture of ten seconds between every two models as rest time, and synthesizing 6 models into a long video. The 24 models were made into 4 long videos in total, and the timing diagram is shown in fig. 2.
And (2) putting the video on an eye tracker for an experimenter to watch, and acquiring experimental data by using the eye tracker and corresponding software, as shown in fig. 3.
The specific experimental conditions were as follows:
a) the display device of the image is placed on the left side, and the experimenter sits right in front of the display device of the image and keeps the distance at 70cm, while the experimenter keeps the eyes in accordance with the height of the screen, and can look up the center of the screen.
b) The operating personnel is on the right side, and with computer control video broadcast on display device, the operating personnel is kept apart with the light shield baffle with the experimenter.
c) The experiment interference to the experimenter caused by other light sources is prevented by isolating baffles around the experimenter and isolating curtains around the laboratory.
d) The sound of the environment of the laboratory is kept not more than 30dB, an ideal quiet environment is created, and the experiment interference of other sound sources to an experimenter is prevented.
The eye position of the test person was adjusted using the iViewX software. After the pupil Image of the experimenter appears in the Eye Image frame, an operator needs to adjust the relative position of the screen and the experimenter, so that the pupil Image on the screen can be stably displayed in a centered mode. Wherein, the slight movement of the head of the experimenter can not influence the projection, and the loss of the image caused by blinking can be quickly recovered. The movement locus of the eye movement fixation point when the experimenter watches the video is collected by using the expert Center software. The eye sight of an experimenter needs to be calibrated, after calibration is finished, calibration feedback, namely deviation in the X, Y direction, can occur, when the deviation X and Y are smaller than 1.0, an experiment can be carried out, and a model video is played on a screen; and finally, synthesizing the material video and the eye movement fixation point tracking track by using BeGaze analysis software to obtain the video with the eye movement fixation point track of the experimenter.
And (3) cutting the video with the experimenter eye movement fixation point track acquired by the eye movement instrument according to frames, extracting eye movement fixation point coordinates on each picture, and converting two-dimensional coordinates of the moving point on the picture into three-dimensional coordinates on a space through mapping. The specific operation comprises the following two steps:
1. coordinate extraction of two-dimensional eye movement fixation point
The flow of extracting the eye movement fixation point is shown in fig. 4. Firstly, utilizing FFmpeg built software to cut a synthesized video into pictures according to frames to obtain two-dimensional pictures; the color of the moving point in the picture is obtained, the moving track of the eye movement fixation point is embodied in the mode that the moving point moves on the model, and the moving point is orange through software setting in the last step. We call it the moving point color here and set the color tolerance value; then setting a two-dimensional array, initializing the two-dimensional array to set the value to be 1, comparing the two-dimensional array with a two-dimensional picture, and if the two-dimensional array is compared with the moving point color, obtaining three RGB values which are all in a tolerance range and meet the condition; otherwise, the point is not the eye movement point we are looking for. If no point satisfies the condition, the point of interest flag is set to 0, otherwise to 1. And (3) putting the points meeting the conditions into a point array, respectively recording the maximum and minimum row and column of the points meeting the conditions, taking the median of the maximum and minimum row and column coordinates, namely the coordinate (row, rank) of the interest point, and outputting the coordinate point of the interest point.
2. Three-dimensional mapping
The two-dimensional picture of the model is obtained by projection on a three-dimensional XOY plane, and by the nature of the projection, we can see that two-dimensional and three-dimensional coordinate points are related to each other. The three-dimensional coordinates (x, y, z) are determined by first projecting the model onto the XOY plane so that x, y coordinate values are determined and the z coordinate is taken of the vertex of the model surface closest to the viewpoint and the z coordinate value of the vertex on the three-dimensional model is the z coordinate value of the eye gaze point on the three-dimensional model, and if the vertex is not present on the model surface, selecting the coordinate value of the model vertex at a distance within a threshold range as the coordinate value of the eye gaze point. Further, since the model is rotated around the coordinate axis when preparing the picture material, it is now required to be rotated in the reverse direction by a corresponding angle. And finally, obtaining a data model of the interest point collected by the eye tracker.
And (4) correspondingly analyzing the experimental data result of the experimenter, sorting all the extracted eye movement fixation points to obtain a database of an interest point set, discarding some inappropriate and abnormal data, and appropriately combining the interest points, wherein the integration flow of the data is shown in fig. 5, so that the interest point database based on the eye movement fixation point movement track can be obtained. The specific operation is as follows:
when constructing an evaluation library that evaluates the point of interest operator, two criteria are selected, one being the radius of the region of interest and the other being the number of action points within the region. The radius of the region of interest is set to σ dM, where dM represents the model diameter, i.e., the maximum euclidean distance between all vertices of the model M, and σ represents a constant coefficient. All interest points with a measurement distance of less than 2 σ dM are divided into the same region, and if the data volume of different experimenters in the region is less than n, the interest points in the region are discarded. Selecting a point from each region as a representative by using a criterion, the criterion being a point of interest of the criterionComprises the following steps: these points of interest that are selected as criteria need to satisfy the minimum sum of their geometric distances from all other points of interest in the area in which they are located. Note that this is also reasonable if the interest points of the two regions overlap. If the distance between two regions is less than 2 σ dM, the representation with the smaller number of points in the two regions is discarded from the interest point set of the evaluation criterion, and the representation with the larger number of points in the regions is taken as the interest point of the criterion. We denote the point of interest criteria library by the parameters n and σ, i.e., GM(n, σ) represents the point of interest data set for a particular model M. The corresponding values of these two parameters determine the point of interest criteria library. When the value of n is taken to be correspondingly higher, more moving points will fall within the interest area as σ increases, which are considered reasonable because not all volunteers select the details of a certain model as interest points, and σ increases accept more local variations of the labeled points. However, as σ is further increased, the regions it defines tend to contain different regions of interest, so the closely interesting points marked on different structures begin to merge. The average number of point of interest evaluation criteria libraries given will vary with n and σ.
Claims (6)
1. A method for establishing a point of interest database based on eye movement fixation point movement tracks is characterized by comprising the following steps:
collecting a three-dimensional model and making the three-dimensional model into a video material required by an experiment;
synthesizing two-dimensional projection pictures of the three-dimensional model on a plurality of angles into the video material;
placing the video on an eye movement instrument for an experimenter to watch, obtaining data of an eye movement fixation point, and synthesizing the video with the eye movement fixation point through corresponding software;
step (3) generating a three-dimensional model with an eye movement fixation point by a video with the eye movement fixation point through a motion point extraction and three-dimensional mapping algorithm;
cutting the video with the eye movement fixation point, obtaining the eye movement fixation point coordinate on each picture through a moving point extraction algorithm, and converting the two-dimensional coordinate of the moving point on the picture into a three-dimensional coordinate on a space through a three-dimensional mapping algorithm;
and (4) analyzing the eye movement fixation points of the experimenters, sorting the eye movement fixation points of all the experimenters to obtain a three-dimensional model interest point set, discarding abnormal data in the set, merging the interest points and establishing an interest point database.
2. The method for building a point-of-interest database based on the eye movement fixation point movement track according to claim 1, wherein the method for creating the video material in step (1) is as follows:
selecting 24 Stanford three-dimensional model libraries and models in a SHREC2007 model database, wherein the three-dimensional models are widely used in a standard library for three-dimensional model research; and saving two groups of data for each three-dimensional model by using MATLAB, wherein one group is formed by sequentially rotating the three-dimensional models around an X axis at intervals of 60 degrees: three-dimensional models which rotate 0 degree, 60 degree, 120 degree, 180 degree, 240 degree, 300 degree and 360 degree respectively; one group is that the three-dimensional model rotates around the Y axis at intervals of 60 degrees in sequence: three-dimensional models which rotate 0 degree, 60 degree, 120 degree, 180 degree, 240 degree, 300 degree and 360 degree respectively; selecting a Z-axis direction as a viewpoint, and projecting the model at 12 angles on an XOY plane, so that 12 two-dimensional projection pictures can be obtained by one model;
a formula of three-dimensional model rotation and a formula of projection on an XOY plane;
rotation around the X axis:
z′＝zcosθ-xsinθ
x′＝xcosθ+zsinθ
y′＝y
rotation around the Y axis:
z′＝zcosθ-ysinθ
x′＝x
y′＝ycosθ+zsinθ
formula for parallel projection:
x′＝x
y′＝y
z′＝0
converting a picture every 1.5 seconds by using a video editor Movie Maker, synthesizing 12 two-dimensional pictures of each model into a short video, inserting a blank picture of ten seconds between each two models as rest time, and synthesizing 6 models into a long video; a total of 4 long videos were made from 24 models.
3. The method for building a point-of-interest database based on the eye movement fixation point movement track as claimed in claim 2, wherein the precondition for the experimenter to perform the experiment in step (2) is as follows:
a) the display device of the image is placed on the left side, the experimenter sits right in front of the display device of the image and keeps a distance of 70cm, and meanwhile, the experimenter keeps the height of eyes consistent with that of the screen and can level up the center of the screen;
b) the operator is on the right side, the computer is used for controlling the video to be played on the display equipment, and the operator and the experimenter are isolated by the light-isolating baffle;
c) the periphery of the experimenter needs to be isolated by a light-isolating baffle, and a light-isolating curtain is used around the laboratory to prevent other light sources from causing experimental interference to the experimenter;
d) the sound of the environment of the laboratory is kept not more than 30dB, an ideal quiet environment is created, and the experiment interference of other sound sources to an experimenter is prevented.
4. The method for establishing the point-of-interest database based on the eye movement fixation point movement track as claimed in claim 3, wherein the specific experiment in the step (2) is as follows:
firstly, adjusting the eye position of a tester by using iViewX software; after the pupil Image of the experimenter appears in the Eye Image frame, an operator enables the pupil Image on the screen to be stably displayed in a centered mode by adjusting the relative position of the screen and the experimenter;
then, collecting the movement track of the eye movement fixation point when the experimenter watches the video by using the expert Center software; the sight of the eyes of an experimenter needs to be calibrated before the collection process, after calibration is finished, calibration feedback, namely deviation in the direction of X, Y, can occur, when the deviation X and the deviation Y are smaller than 1.0, an experiment can be carried out, and then model material video playing can occur;
and finally, synthesizing the model material video and the eye movement fixation point tracking track by using BeGaze analysis software to obtain the model video with the eye movement fixation point track of the experimenter.
5. The method for establishing the interest point database based on the eye movement fixation point movement track according to the claim 4, characterized in that the step (3) comprises the steps of cutting the model video with the experimenter eye movement fixation point track acquired by the eye tracker according to frames, obtaining the eye movement fixation point coordinate on each picture by the motion point extraction algorithm, and converting the two-dimensional coordinate of the motion point on the picture into the three-dimensional coordinate on the space by the three-dimensional mapping algorithm, which is specifically realized as follows:
firstly, extracting coordinates of a two-dimensional eye movement fixation point, and utilizing FFmpeg built software to cut a synthesized video into pictures according to frames to obtain two-dimensional pictures; acquiring the color of an eye movement fixation point in a picture, wherein the movement track of the eye movement fixation point is embodied in a mode that the eye movement fixation point moves on a model, and in the step (2), the eye movement fixation point is set to be orange through the setting of BeGaze analysis software, namely the color of the eye movement fixation point is orange, and a color tolerance value is set;
then setting a two-dimensional array, and initializing the two-dimensional array to set the value to 1; comparing the initialized two-dimensional array with the two-dimensional picture, and if the initialized two-dimensional array is compared with the moving point color, obtaining three RGB values which are all in the range of color tolerance values and meet the conditions; otherwise, the point is not the eye movement point to be found; if no point meets the condition, setting the interest point mark to be 0, and otherwise, setting the interest point mark to be 1; putting the points meeting the conditions into a point array, respectively recording the maximum and minimum row and column of the points meeting the conditions, taking the median of the maximum and minimum row and column coordinates, namely the coordinate (row, rank) of the interest point, and outputting the coordinate point of the interest point;
then obtaining a three-dimensional model with an eye movement fixation point through a three-dimensional mapping algorithm; since the two-dimensional picture of the model is obtained by projection on the three-dimensional XOY plane, it can be seen from the nature of the projection that two-dimensional and three-dimensional coordinate points are linked to each other: firstly, projecting the model on an XOY plane, so that x and y coordinate values are determined, wherein a z coordinate is a vertex of the model surface, which is closest to the viewpoint, the z coordinate value of the vertex on the three-dimensional model is the z coordinate value of the eye movement fixation point on the three-dimensional model, and if the vertex does not exist on the model surface, selecting the coordinate value of the model vertex, which is within a threshold value range, as the coordinate value of the eye movement fixation point, so as to determine the three-dimensional coordinate (x, y and z); in addition, because the model is rotated around the coordinate axis when the picture material is prepared, the model needs to be rotated reversely by a corresponding angle at present; and finally, obtaining the data of the interest points collected by the eye tracker.
6. The method for establishing an interest point database based on the eye movement fixation point movement track of claim 5, wherein the step (4) is implemented by analyzing the experimental data result of the experimenter correspondingly, sorting all the extracted eye movement fixation points to obtain a database of an interest point set, discarding abnormal data, and merging the interest points, and the integration method comprises the following steps:
when an evaluation library for evaluating an interest point operator is constructed, two criteria are selected, wherein one criterion is the radius of an interest area, and the other criterion is the number of action points in the area; setting the radius of the region of interest as sigma dM, wherein dM represents the diameter of the model, namely the maximum Euclidean distance between all vertexes of the model M, and sigma represents a constant coefficient; dividing all interest points with the measuring distance smaller than 2 sigma dM into the same region, and if the data volume of different experimenters in the region is smaller than n, discarding the interest points in the region;
selecting a point from each region as a representative, and using the point as a standard interest point, wherein the standard is as follows: the interest point selected as the criterion needs to satisfy the condition that the sum of the geometric distances between the interest point and all other interest points in the area where the interest point is located is minimum; if the distance between the two regions is less than 2 sigma dM, the representative with a smaller number of points in the two regions is discarded from the interest point set of the evaluation criterion, and the representative with a larger number of points in the regions is taken as the interest point of the criterion; expressed by the parameters n and sigmaPoint of interest criteria library, i.e. GM(n, σ) a point of interest data set represented as a particular model M; the corresponding values of the two parameters determine a point of interest standard library; when the value of n is higher, more moving points will fall into the interest region along with the increase of sigma, and the change of sigma is considered reasonable because not all experimenters select the details of a certain model as the interest points, but the increase of sigma accepts more local changes of the marked points; but as σ is further increased, the region it defines tends to contain different regions of interest, so the closely interesting points marked on different structures start to merge; the average number of point of interest evaluation criteria libraries given will vary with n and σ.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810116386.0A CN108415955B (en) | 2018-02-06 | 2018-02-06 | Interest point database establishing method based on eye movement fixation point moving track |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810116386.0A CN108415955B (en) | 2018-02-06 | 2018-02-06 | Interest point database establishing method based on eye movement fixation point moving track |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108415955A CN108415955A (en) | 2018-08-17 |
| CN108415955B true CN108415955B (en) | 2021-12-28 |
Family
ID=63126834
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810116386.0A Active CN108415955B (en) | 2018-02-06 | 2018-02-06 | Interest point database establishing method based on eye movement fixation point moving track |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108415955B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109255342B (en) * | 2018-11-20 | 2020-07-10 | 武汉大学 | Image region-of-interest extraction method and system based on two-step clustering of eye movement trajectory data |
| CN111382591B (en) * | 2018-12-27 | 2023-09-29 | 海信集团有限公司 | Binocular camera ranging correction method and vehicle-mounted equipment |
| CN109887583B (en) * | 2019-03-11 | 2020-12-22 | 数坤（北京）网络科技有限公司 | Data acquisition method/system based on doctor behaviors and medical image processing system |
| CN110032278B (en) * | 2019-03-29 | 2020-07-14 | 华中科技大学 | Pose identification method, device and system for human eye interested object |
| CN110786868A (en) * | 2019-10-18 | 2020-02-14 | 天津大学 | Non-invasive detection and eye movement analysis method for ASD screening |
| CN111079740A (en) * | 2019-12-02 | 2020-04-28 | 咪咕文化科技有限公司 | Image quality evaluation method, electronic device, and computer-readable storage medium |
| CN111510630B (en) * | 2020-04-24 | 2021-09-28 | Oppo广东移动通信有限公司 | Image processing method, device and storage medium |
| CN111966225A (en) * | 2020-08-31 | 2020-11-20 | 东莞市东全智能科技有限公司 | Visual data-based multimedia data interaction platform and analysis method |
| CN113158879B (en) * | 2021-04-19 | 2022-06-10 | 天津大学 | Three-dimensional fixation point estimation and three-dimensional eye movement model establishment method based on matching characteristics |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102521595A (en) * | 2011-12-07 | 2012-06-27 | 中南大学 | Method for extracting image region of interest based on eye movement data and bottom-layer features |
| CN205563453U (en) * | 2015-12-16 | 2016-09-07 | 中国科学院西安光学精密机械研究所 | Sight is tracked and people's eye zone location system interested |
| CN107179829A (en) * | 2017-05-22 | 2017-09-19 | 贵州大学 | The method that user individual preference is predicted using eye movement data |
| CN107506751A (en) * | 2017-09-13 | 2017-12-22 | 重庆爱威视科技有限公司 | Advertisement placement method based on the dynamic control of eye |
| CN107656613A (en) * | 2017-09-08 | 2018-02-02 | 国网山东省电力公司电力科学研究院 | A kind of man-machine interactive system and its method of work based on the dynamic tracking of eye |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9852543B2 (en) * | 2015-03-27 | 2017-12-26 | Snap Inc. | Automated three dimensional model generation |
| US10674139B2 (en) * | 2015-06-03 | 2020-06-02 | University Of Connecticut | Methods and systems for human action recognition using 3D integral imaging |
| US9849364B2 (en) * | 2016-02-02 | 2017-12-26 | Bao Tran | Smart device |
-
2018
- 2018-02-06 CN CN201810116386.0A patent/CN108415955B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102521595A (en) * | 2011-12-07 | 2012-06-27 | 中南大学 | Method for extracting image region of interest based on eye movement data and bottom-layer features |
| CN205563453U (en) * | 2015-12-16 | 2016-09-07 | 中国科学院西安光学精密机械研究所 | Sight is tracked and people's eye zone location system interested |
| CN107179829A (en) * | 2017-05-22 | 2017-09-19 | 贵州大学 | The method that user individual preference is predicted using eye movement data |
| CN107656613A (en) * | 2017-09-08 | 2018-02-02 | 国网山东省电力公司电力科学研究院 | A kind of man-machine interactive system and its method of work based on the dynamic tracking of eye |
| CN107506751A (en) * | 2017-09-13 | 2017-12-22 | 重庆爱威视科技有限公司 | Advertisement placement method based on the dynamic control of eye |
Non-Patent Citations (4)
| Title |
|---|
| "Evaluation of 3D interest point detection techniques via human-generated ground truth";Helin Dutagaci 等;《The Visual Computer》;20120629;第28卷;第901-917页 * |
| "基于感兴趣区域图像眼动数据库的建立";闫苗苗 等;《西安理工大学学报》;20140331;第30卷(第1期);第85-90页 * |
| "基于眼动感知信息的彩色图像感兴趣区域提取算法";李双 等;《光学工程》;20110430;第38卷(第4期);第115-123页 * |
| Wolf Kienzle 等."How to Find Interesting Locations in Video: A Spatiotemporal Interest Point Detector Learned from Human Eye Movements".《Pattern Recognition》.2007, * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108415955A (en) | 2018-08-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108415955B (en) | Interest point database establishing method based on eye movement fixation point moving track | |
| CN107945282B (en) | Rapid multi-view three-dimensional synthesis and display method and device based on countermeasure network | |
| US10560687B2 (en) | LED-based integral imaging display system as well as its control method and device | |
| US8551015B2 (en) | System and method for evaluating and diagnosing patients based on ocular responses | |
| US8636361B2 (en) | Learning-based visual attention prediction system and method thereof | |
| CN103442773B (en) | The sensing apparatus of virtual golf analogue means and use thereof and method for sensing | |
| Fuhl et al. | Eye movement velocity and gaze data generator for evaluation, robustness testing and assess of eye tracking software and visualization tools | |
| CN111427150B (en) | Eye movement signal processing method used under virtual reality head-mounted display and wearable device | |
| US20170366758A1 (en) | System and method for acquiring virtual and augmented reality scenes by a user | |
| Coutrot et al. | Influence of soundtrack on eye movements during video exploration | |
| Martin et al. | Scangan360: A generative model of realistic scanpaths for 360 images | |
| CN113436136A (en) | Object-centric scanning | |
| EP3509037B1 (en) | Image generating device and method of generating an image | |
| CA3003966A1 (en) | Uav detection | |
| Choi et al. | “Starkfest” Vision and Clinic Science Special Issue: String Editing Analysis of Human Visual Search | |
| JP5306940B2 (en) | Moving image content evaluation apparatus and computer program | |
| CN112597967A (en) | Emotion recognition method and device for immersive virtual environment and multi-modal physiological signals | |
| EP3503036A1 (en) | Determining pixel values using reference images | |
| CN115933930A (en) | Method, terminal and device for analyzing attention of learning object in education meta universe | |
| Muthumanickam et al. | Identification of temporally varying areas of interest in long-duration eye-tracking data sets | |
| CN116682140A (en) | Three-dimensional human body posture estimation algorithm based on attention mechanism multi-mode fusion | |
| CN116719176B (en) | Intelligent display system of intelligent exhibition hall | |
| Sundstedt et al. | A systematic review of visualization techniques and analysis tools for eye-tracking in 3d environments | |
| CN112954313A (en) | Method for calculating perception quality of panoramic image | |
| EP2112619A1 (en) | Video data processing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |