CN107067438A - Two-way direction of visual lines method of estimation and device based on linear regression - Google Patents
Two-way direction of visual lines method of estimation and device based on linear regression Download PDFInfo
- Publication number
- CN107067438A CN107067438A CN201710183955.9A CN201710183955A CN107067438A CN 107067438 A CN107067438 A CN 107067438A CN 201710183955 A CN201710183955 A CN 201710183955A CN 107067438 A CN107067438 A CN 107067438A
- Authority
- CN
- China
- Prior art keywords
- visual lines
- linear regression
- eyes
- eye
- lrm
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
Landscapes
- Image Analysis (AREA)
- Eye Examination Apparatus (AREA)
Abstract
The present invention proposes a kind of two-way direction of visual lines method of estimation and device based on linear regression, wherein, method includes：The eyes direction of visual lines of continuous predetermined quantity frame of video is tracked as initial data set；Calculate the linear regression model (LRM) of two direction of visual lines between eyes respectively according to initial data set；Eye tracking is carried out to current video frame, if eye tracking results abnormity, direction of visual lines estimation is carried out by linear regression model (LRM).Thus; by the way that the potential relation that eyes direction of visual lines is present is fitted into linear regression model (LRM); if so as to which the direction of visual lines of this eye is estimated using the direction of visual lines of linear regression model (LRM) and another eyes when the tracking result exception or direction of visual lines of eye during carrying out eye tracking can not be obtained; a rational binocular direction of visual lines result is obtained, increases the robustness of eye tracking.
Description
Technical field
The present invention relates to field of computer technology, more particularly to a kind of two-way direction of visual lines estimation side based on linear regression
Method and device.
Background technology
At present, eye tracking be in the fields such as computer vision, computer graphics one it is important and basic the problem of, it
Also there is application widely in fields such as man-machine interaction, virtual reality and augmented realities.For example, in computer vision, eye
Blinkpunkt of the eyeball on screen can be used for completing various human-computer interaction functions, for another example in computer graphics and virtual reality
In, direction of visual lines can be used for driving virtual eyeball phantom, to generate more lively real faceform's animation.Further,
In augmented reality, direction of visual lines can be used for the content of adjustment display to produce the more preferably sense of reality.Can just because of eyes
The emotion that the expression mankind enrich, the research of eye tracking has high scientific research and application value.
However, existing Visual Trace Technology still has defect, generally it is vulnerable to the influence of many external factor and produces
The problem of life tracking result is inaccurate, such as lighting condition, image motion are fuzzy, head rotation.Meanwhile, in face tracking compared with
One of intractable problem is exactly occlusion issue, and when there is object to block before face, the extraction of facial feature points would generally be very
Unstable, this just causes influence to the accuracy of pupil and iris tracking；Ocular even is appeared in when shelter, at all
Real eyes image can not be obtained, so that the rotation of three-dimensional eyeball phantom can not be driven to generate rational people in the case
Face model animation.
The content of the invention
It is contemplated that at least solving one of technical problem in correlation technique to a certain extent.
Therefore, first purpose of the present invention is to propose a kind of two-way direction of visual lines estimation side based on linear regression
Method, this method by the way that the potential relation that eyes direction of visual lines is present is fitted into linear regression model (LRM) so that carry out sight with
If utilizing linear regression model (LRM) and another when the tracking result exception or direction of visual lines of eye can not be obtained during track
The direction of visual lines of eyes estimates the direction of visual lines of this eye, obtains a rational binocular direction of visual lines result.
Second object of the present invention is to propose a kind of two-way direction of visual lines estimation unit based on linear regression.
For up to above-mentioned purpose, first aspect present invention embodiment proposes a kind of two-way direction of visual lines based on linear regression
Method of estimation, including：The eyes direction of visual lines of continuous predetermined quantity frame of video is tracked as initial data set；According to described initial
Data set calculates the linear regression model (LRM) of two direction of visual lines between eyes respectively；Eye tracking is carried out to current video frame, such as
Really described eye tracking results abnormity, then carry out direction of visual lines by the direction of visual lines of the linear regression model (LRM) and an eyes
Estimation.
The two-way direction of visual lines method of estimation based on linear regression of the embodiment of the present invention, by tracking continuous predetermined quantity
The eyes direction of visual lines of frame of video calculates according to initial data set two sight sides between eyes respectively as initial data set
To linear regression model (LRM), finally to current video frame carry out eye tracking, in eye tracking results abnormity by linearly returning
Model is returned to carry out direction of visual lines estimation.Thus, by the way that the potential relation that eyes direction of visual lines is present is fitted into linear regression mould
Type, if so as to be utilized when the tracking result exception or direction of visual lines of eye during carrying out eye tracking can not be obtained
The direction of visual lines of linear regression model (LRM) and another eyes estimates the direction of visual lines of this eye, obtains a rational binocular vision
Line direction result, increases the robustness of eye tracking.
In addition, the two-way direction of visual lines method of estimation according to the above embodiment of the present invention based on linear regression can also have
There is technical characteristic additional as follows：
Alternatively, described method, in addition to:By PCA extract the feature of the initial data set to
Amount；If the eye tracking result is normal, the direction of visual lines of current video frame is expressed as to the shape of characteristic vector weighted sum
Formula.
Alternatively, described method, in addition to：The eyes direction of visual lines of the current video frame is updated to described initial
In data set, and extract new characteristic vector.
Alternatively, the continuous predetermined quantity frame of video is effective video frame.
Alternatively, the linear regression mould of two direction of visual lines calculated respectively according to the initial data set between eyes
Type, including：Obtain the first model matrix that right eye is estimated from left eye；Obtain the second model matrix that left eye is estimated from right eye；It is logical
Cross least square method first model matrix and second model matrix are fitted and obtain described two direction of visual lines
Linear regression model (LRM).
For up to above-mentioned purpose, second aspect of the present invention embodiment proposes a kind of two-way direction of visual lines based on linear regression
Estimation unit, including：Tracking module, for tracking the eyes direction of visual lines of continuous predetermined quantity frame of video as primary data
Collection；Computing module, the linear regression model (LRM) for calculating two direction of visual lines between eyes respectively according to the initial data set；
Processing module, for carrying out eye tracking to current video frame, passes through described linear time in the eye tracking results abnormity
The direction of visual lines of model and an eyes is returned to carry out direction of visual lines estimation.
The two-way direction of visual lines estimation unit based on linear regression of the embodiment of the present invention, by tracking continuous predetermined quantity
The eyes direction of visual lines of frame of video calculates according to initial data set two sight sides between eyes respectively as initial data set
To linear regression model (LRM), finally to current video frame carry out eye tracking, in eye tracking results abnormity by linearly returning
Model is returned to carry out direction of visual lines estimation.Thus, by the way that the potential relation that eyes direction of visual lines is present is fitted into linear regression mould
Type, if so as to be utilized when the tracking result exception or direction of visual lines of eye during carrying out eye tracking can not be obtained
The direction of visual lines of linear regression model (LRM) and another eyes estimates the direction of visual lines of this eye, obtains a rational binocular vision
Line direction result, increases the robustness of eye tracking.
In addition, the two-way direction of visual lines estimation unit according to the above embodiment of the present invention based on linear regression can also have
There is technical characteristic additional as follows：
Alternatively, described device, in addition to:Extraction module, for extracting the initial number by PCA
According to the characteristic vector of collection；The processing module, is additionally operable to the sight of current video frame when the eye tracking result is normal
Direction is expressed as the form of characteristic vector weighted sum.
Alternatively, described device, in addition to：Update module, for by the eyes direction of visual lines of the current video frame
Update the primary data to concentrate, and extract new characteristic vector.
Alternatively, the continuous predetermined quantity frame of video is effective video frame.
Alternatively, the computing module is used for：Obtain the first model matrix that right eye is estimated from left eye；Acquisition is estimated from right eye
Calculate the second model matrix of left eye；First model matrix and second model matrix are intended by least square method
Close the linear regression model (LRM) for obtaining described two direction of visual lines.
The additional aspect of the present invention and advantage will be set forth in part in the description, and will partly become from the following description
Obtain substantially, or recognized by the practice of the present invention.
Brief description of the drawings
Of the invention above-mentioned and/or additional aspect and advantage will become from the following description of the accompanying drawings of embodiments
Substantially and be readily appreciated that, wherein：
Fig. 1 is that a kind of flow for two-way direction of visual lines method of estimation based on linear regression that the embodiment of the present invention is proposed is shown
It is intended to；
Fig. 2 is the flow for another two-way direction of visual lines method of estimation based on linear regression that the embodiment of the present invention is proposed
Schematic diagram；
Fig. 3 is a kind of signal of the frame of video for faceform's animation comprising direction of visual lines that the embodiment of the present invention is proposed
Figure；
Fig. 4 is the signal of the frame of video for another faceform's animation comprising direction of visual lines that the embodiment of the present invention is proposed
Figure；
Fig. 5 is that drawing axis has a frame of shelter, the direction of visual lines knot of mistake in the video that the embodiment of the present invention is proposed
Fruit and the schematic diagram for the direction of visual lines result estimated using this method；
Fig. 6 is a frame of direction of visual lines tracking result apparent error in the video that the embodiment of the present invention is proposed, mistake is regarded
Line direction result and the direction of visual lines result estimated using this method.
Fig. 7 is that a kind of structure for two-way direction of visual lines estimation unit based on linear regression that the embodiment of the present invention is proposed is shown
It is intended to；
Fig. 8 is the structure for another two-way direction of visual lines estimation unit based on linear regression that the embodiment of the present invention is proposed
Schematic diagram.
Embodiment
Embodiments of the invention are described below in detail, the example of the embodiment is shown in the drawings, wherein from beginning to end
Same or similar label represents same or similar element or the element with same or like function.Below with reference to attached
The embodiment of figure description is exemplary, it is intended to for explaining the present invention, and be not considered as limiting the invention.
Below with reference to the accompanying drawings the two-way direction of visual lines method of estimation based on linear regression and dress of the embodiment of the present invention described
Put.
Generally, face or when being that eye has shelter, causes eye tracking result inaccurate, thus influence sight with
Application effect of the track in fields such as man-machine interaction, virtual reality and augmented realities.
In order to solve the above problems, the present invention proposes a kind of two-way direction of visual lines method of estimation based on linear regression, leads to
Cross linear regression model (LRM) and carry out direction of visual lines estimation.Thus, by the way that the potential relation that eyes direction of visual lines is present is fitted into line
Property regression model, if so that the tracking result exception or direction of visual lines of eye can not be obtained during eye tracking is carried out
The direction of visual lines of this eye is estimated when obtaining using the direction of visual lines of linear regression model (LRM) and another eyes, one is obtained rationally
Binocular direction of visual lines result, increase eye tracking robustness.It is specific as follows：
Fig. 1 is that a kind of flow for two-way direction of visual lines method of estimation based on linear regression that the embodiment of the present invention is proposed is shown
It is intended to.
Comprise the following steps as shown in figure 1, being somebody's turn to do the two-way direction of visual lines method of estimation based on linear regression：
Step 101, the eyes direction of visual lines for tracking continuous predetermined quantity frame of video is used as initial data set.
Specifically, the feature of eyes is different between different people, such as eyeball position.Thus, the direction of visual lines of different people
It is different.
In order to further improve the accuracy of direction of visual lines estimation, it is necessary to obtain the difference in continuous predetermined quantity frame of video
The direction of visual lines data of people are trained.Avoid training in advance static models, it is impossible to which the eyes for better conforming to more people are special
The shortcoming levied.
It should be noted that the numerical value of continuous predetermined quantity can need to carry out selection setting according to practical application, such as
Continuous 7 frame of video etc..
More specifically, the eyes direction of visual lines obtained can be the spherical coordinate in Binocular vison line direction, by eyes sight side
To spherical coordinate constitute initial data set.Wherein, continuous predetermined quantity frame of video is effective video frame, you can correctly to track
Direction of visual lines.
In order to which those skilled in the art are more clear how the spherical coordinate of eyes direction of visual lines constituting initial data set, below
It is described as follows by taking preceding P frame of video in video as an example：
Direction of visual lines number before being obtained by sight tracing of the prior art in video in P frame of video
It is that abnormal conditions are not present in valid frame i.e. tracking result according to, frame of video here.The direction of visual lines of two eyes can use sphere
Coordinate is expressed asWithWherein, L and R represent left eye and right eye respectively.
Thus, P groups eyes sight bearing data constitutes S ° of initial data set, is expressed as：
Step 102, the linear regression model (LRM) of two direction of visual lines between eyes is calculated respectively according to initial data set.
Specifically, the mode of the linear regression model (LRM) of two direction of visual lines between eyes is calculated respectively according to initial data set
There are many kinds, can select to set as needed.In the present embodiment, the first model of right eye can be estimated from left eye by obtaining
The second model matrix of left eye is estimated in matrix and acquisition from right eye, finally by least square method to the first model matrix and second
Model matrix is fitted the linear regression model (LRM) for obtaining two direction of visual lines.
In order to become apparent from how description sets up linear regression model (LRM).Continue with the initial data set in above-mentioned steps 101Exemplified by explanation.
Specifically, the first model matrix isFirst model matrix is
Wherein MLRepresent the model matrix from left eye estimation right eye, MRThe model matrix from right eye estimation left eye is represented, both at 3 × 3
Matrix.Thus, above-mentioned two formula can be containing MLOr MRIn 9 parameters linear system, further utilize least square method
And initial data set, it just can must fit two linear regression model (LRM)s.
Step 103, eye tracking is carried out to current video frame, if eye tracking results abnormity, passes through linear regression
Model carries out direction of visual lines estimation.
Specifically, current video frame refers to tracking other any frame of video behind continuous predetermined quantity frame of video.This
Invention can individually handle the eye tracking result of each frame of video, and the progress direction of visual lines estimation in results abnormity.
Specifically, eye tracking results abnormity during eye tracking is carried out to current video frame, and (such as one eyes can not be obtained
Obtain its direction of visual lines, eye tracking result mistake etc.), it is possible to use the eyes that can not obtain its direction of visual lines are corresponding linear time
The direction of visual lines of model and another eyes is returned to estimate the direction of visual lines for the eyes that can not obtain its direction of visual lines, so as to obtain one
Individual rational eyes sight tracking result.
Thus, the direction of visual lines of certain eyes can be used to estimate the sight side of another eyes when continuing frame after treatment
To so as to reach the purpose of the abnormal eye tracking result of processing.In order to how more clearly describe by the sights of certain eyes
The direction of visual lines of direction estimation another eyes, is continued with and is illustrated with the example in step 101 and step 102.
Specifically, for certain eyes e (e ∈ { L, R }), if its direction of visual lines can not be obtained or its eye tracking result is bright
Aobvious mistake, then according to another eyes f (f ∈ { L, R }, f ≠ e) direction of visual linesWith estimation e linear regression mould
Type MfEstimation e direction of visual lines be：So as to obtain a rational eyes eye tracking
As a result.
In summary, the two-way direction of visual lines method of estimation based on linear regression of the embodiment of the present invention, passes through the company of tracking
The eyes direction of visual lines of continuous predetermined quantity frame of video is calculated according to initial data set between eyes respectively as initial data set
The linear regression model (LRM) of two direction of visual lines, finally carries out eye tracking, in eye tracking results abnormity to current video frame
Direction of visual lines estimation is carried out by linear regression model (LRM).Thus, by the way that the potential relation that eyes direction of visual lines is present is fitted to
Linear regression model (LRM), if so that the tracking result exception or direction of visual lines of eye can not during eye tracking is carried out
The direction of visual lines of this eye is estimated during acquisition using the direction of visual lines of linear regression model (LRM) and another eyes, a conjunction is obtained
The binocular direction of visual lines result of reason, increases the robustness of eye tracking.
Fig. 2 is the flow for another two-way direction of visual lines method of estimation based on linear regression that the embodiment of the present invention is proposed
Schematic diagram.
Comprise the following steps as shown in Fig. 2 being somebody's turn to do the two-way direction of visual lines method of estimation based on linear regression：
Step 201, the eyes direction of visual lines for tracking continuous predetermined quantity frame of video is used as initial data set.
Step 202, the linear regression model (LRM) of two direction of visual lines between eyes is calculated respectively according to initial data set.
It should be noted that step S201-S202 description is corresponding with above-mentioned steps S101-S102, thus to step
Rapid S201-S202 description will not be repeated here with reference to above-mentioned steps S101-S102 description.
Step 203, the characteristic vector of initial data set is extracted by PCA.
Specifically, the characteristic vector for extracting initial data set using PCA (PCA) constitutes PCA space.
Specifically, it is using the characteristic vector of S ° of initial data set in PCA methods extraction step 101：
Feature for representing initial data set, can be avoided updated
Data explosion problem in journey, alsos for constantly updating optimization linear regression mould using the eye tracking result of subsequent video frame
Type.
Step 204, eye tracking is carried out to current video frame, if eye tracking results abnormity, passes through linear regression
The direction of visual lines of model and an eyes carries out direction of visual lines estimation.
Step 205, if eye tracking result is normal, the direction of visual lines of current video frame is expressed as characteristic vector and added
Weigh the form of sum.
Step 206, the eyes direction of visual lines of current video frame is updated to primary data and concentrated, and extract new feature to
Amount.
It is understood that when handling other subsequent video frames using linear regression model (LRM), on the one hand will be according to sight
The result of tracking is decided whether to carry out direction of visual lines estimation using linear regression model (LRM), on the other hand updated using more data
Linear regression model (LRM) and data set, make estimation result more accurate.
Specifically, before being processed, it is S by the direction of visual lines data set definition subsequently used, its characteristic vector is
Z, and be its to assign initial value respectively be S=S °, Z=Z °.It is whether to be normally carried out according to present frame eye tracking result below
Different step：
Example one：For certain eyes e (e ∈ { L, R }), if its direction of visual lines can not be obtained or its eye tracking result is bright
Aobvious mistake, then according to another eyes f (f ∈ { L, R }, f ≠ e) direction of visual linesWith estimation e linear regression mould
Type MfEstimate e direction of visual lines：So as to obtain a rational eyes eye tracking knot
Really.
Example two, if the eye tracking result of two eyes is normal, the sight of present frame is represented using characteristic vector Z
Direction G is Gz, i.e.,：
If error is more than tolerance ε, i.e.,：||G-Gz 2>ε2>ε.Then represent that the direction of visual lines of present frame is included to be beyond expression in S
Binocular visual line characteristics, therefore G is added into S and least square fitting M is reusedLAnd MRAnd reuse PCA methods
Extract characteristic vector Z.
In summary, the two-way direction of visual lines method of estimation based on linear regression of the embodiment of the present invention, passes through the company of tracking
The eyes direction of visual lines of continuous predetermined quantity frame of video is calculated according to initial data set between eyes respectively as initial data set
The linear regression model (LRM) of two direction of visual lines, then by the characteristic vector of PCA extraction initial data set, it is finally right
Current video frame carries out eye tracking, and direction of visual lines estimation is carried out by linear regression model (LRM) in eye tracking results abnormity,
The direction of visual lines of current video frame is expressed as to the form of characteristic vector weighted sum when eye tracking result is normal, and will be current
The eyes direction of visual lines of frame of video, which is updated to primary data, to be concentrated, and extracts new characteristic vector.Thus, by by eyes sight
The potential relation that direction is present is fitted to linear regression model (LRM), if so that during eye tracking is carried out eye with
Track results abnormity or direction of visual lines estimate this when can not obtain using the direction of visual lines of linear regression model (LRM) and another eyes
The direction of visual lines of eyes, obtains a rational binocular direction of visual lines result, increases after the robustness of eye tracking, and utilization
The eye tracking result of continuous frame of video constantly updates optimization linear regression model (LRM), further improves the accurate of direction of visual lines estimation
Property.
Understand the specific implementation process of above-described embodiment in order to which those skilled in the art can add, illustrated with reference to specific example
It is bright as follows：
Specifically, the video of two sections of human face actions is included in the present embodiment, wherein one section of video bag is containing one section of frame of video
In a drawing axis have a shelter, tracking result of another section of video bag containing an eyes in one section of frame sequence occurs obvious wrong
By mistake.It should be noted that using the three-dimensional face rebuild in the result in order to show overall faceform's animation, embodiment
Model is used as input.
The direction of visual lines of continuous some frame of video in the first step, tracking video, take wherein preceding P=150 valid frame with
The spherical coordinate of track result, i.e. binocular direction of visual linesWithConstitute S ° of initial data set.Two sections of video sequences
The image of each frame and the model animation of eye tracking result be as shown in Figure 3 and Figure 4 in row.
Second step, two linear regression model (LRM) M between eyes are calculated according to S ° of initial data set respectivelyLAnd MR, by following formula table
Show：WithUtilize least square method and initial number
Two initial linear regression models are fitted according to the P group binocular sight bearing datas collected in S °.Then, extracted using PCA methods
S ° of characteristic vectorFeature for representing initial data set.
3rd step, after treatment before continuous frame of video, first by S ° of initial data set and Z ° of assignment of characteristic vector in S and Z.So
Eye tracking is carried out to present frame afterwards.
Specifically, the first situation, eye tracking results abnormity, such as drawing axis have shelter can not obtain direction of visual lines
(as shown in B in Fig. 5), or eye tracking result apparent error (in Fig. 6 shown in B).In the case, in order to estimate that right eye is correct
Direction of visual linesUse the linear regression model (LRM) M from left eye to right eyeLWith the direction of visual lines of left eyeEstimate
The direction of visual lines of right eye is counted, i.e.,Binocular vision knot fruit after estimation is as in Fig. 5 and Fig. 6
C shown in.
Specifically, second of situation, eye tracking result is normal, then the direction of visual lines G of present frame is expressed as into feature in Z
The form of vectorial weighted sum, i.e.,：Then the error e after so representing is calculated, i.e.,：||G-
Gz 2>ε||2>ε.If e>ε=0.05 | | G | |, then G is added into S, and reuse least square fitting MLAnd MRAnd make again
Characteristic vector Z is extracted with PCA methods.
It should be noted that the hardware PC configurations of the method for the present embodiment；CPU：Intel(R)Core(TM)i7-
47903.6GHz；Internal memory：16G；Operating system：Windows 8.
In summary, the two-way direction of visual lines method of estimation based on linear regression of the embodiment of the present invention, passes through the company of tracking
The eyes direction of visual lines of continuous predetermined quantity frame of video is calculated according to initial data set between eyes respectively as initial data set
The linear regression model (LRM) of two direction of visual lines, then by the characteristic vector of PCA extraction initial data set, it is finally right
Current video frame carries out eye tracking, and direction of visual lines estimation is carried out by linear regression model (LRM) in eye tracking results abnormity,
The direction of visual lines of current video frame is expressed as to the form of characteristic vector weighted sum when eye tracking result is normal, and will be current
The eyes direction of visual lines of frame of video, which is updated to primary data, to be concentrated, and extracts new characteristic vector.Thus, by by eyes sight
The potential relation that direction is present is fitted to linear regression model (LRM), if so that during eye tracking is carried out eye with
Track results abnormity or direction of visual lines estimate this when can not obtain using the direction of visual lines of linear regression model (LRM) and another eyes
The direction of visual lines of eyes, obtains a rational binocular direction of visual lines result, increases after the robustness of eye tracking, and utilization
The eye tracking result of continuous frame of video constantly updates optimization linear regression model (LRM), further improves the accurate of direction of visual lines estimation
Property.
In order to realize above-described embodiment, the present invention also proposes a kind of two-way direction of visual lines estimation dress based on linear regression
Put.
Fig. 7 is that a kind of structure for two-way direction of visual lines estimation unit based on linear regression that the embodiment of the present invention is proposed is shown
It is intended to.
As shown in fig. 7, being somebody's turn to do the two-way direction of visual lines estimation unit based on linear regression includes：Tracking module 11, calculating mould
Block 12 and processing module 13.
Wherein, tracking module 11, for tracking the eyes direction of visual lines of continuous predetermined quantity frame of video as primary data
Collection.
Computing module 12, the linear regression mould for calculating two direction of visual lines between eyes respectively according to initial data set
Type.
Processing module 13, for carrying out eye tracking to current video frame, in eye tracking results abnormity by linear
The direction of visual lines of regression model and an eyes carries out direction of visual lines estimation.
In one embodiment of the invention, continuous predetermined quantity frame of video is effective video frame.
Further, in a kind of possible implementation of the embodiment of the present invention, computing module 12 is used for：Obtain from a left side
First model matrix of eye estimation right eye；Obtain the second model matrix that left eye is estimated from right eye；By least square method to
One model matrix and the second model matrix are fitted the linear regression model (LRM) for obtaining two direction of visual lines.
It should be noted that the foregoing explanation to the two-way direction of visual lines method of estimation embodiment based on linear regression
The two-way direction of visual lines estimation unit based on linear regression of the present embodiment is also applied for, here is omitted.
In summary, the two-way direction of visual lines method of estimation based on linear regression of the embodiment of the present invention, passes through the company of tracking
The eyes direction of visual lines of continuous predetermined quantity frame of video is calculated according to initial data set between eyes respectively as initial data set
The linear regression model (LRM) of two direction of visual lines, finally carries out eye tracking, in eye tracking results abnormity to current video frame
Direction of visual lines estimation is carried out by linear regression model (LRM).Thus, by the way that the potential relation that eyes direction of visual lines is present is fitted to
Linear regression model (LRM), if so that the tracking result exception or direction of visual lines of eye can not during eye tracking is carried out
The direction of visual lines of this eye is estimated during acquisition using the direction of visual lines of linear regression model (LRM) and another eyes, a conjunction is obtained
The binocular direction of visual lines result of reason, increases the robustness of eye tracking.
For an embodiment in clear explanation, another two-way direction of visual lines based on linear regression is present embodiments provided
Estimation unit.
Fig. 8 is the structure for another two-way direction of visual lines estimation unit based on linear regression that the embodiment of the present invention is proposed
Schematic diagram.
As shown in figure 8, on the basis of a upper embodiment, the device also includes：Extraction module 14 and update module 15.
Wherein, extraction module 14, the characteristic vector for extracting initial data set by PCA.
Processing module 13, is additionally operable to represent to be characterized by the direction of visual lines of current video frame when eye tracking result is normal
The form of vectorial weighted sum.
Update module 15, is concentrated for the eyes direction of visual lines of current video frame to be updated to primary data, and extracts new
Characteristic vector.
It should be noted that the foregoing explanation to the two-way direction of visual lines method of estimation embodiment based on linear regression
The two-way direction of visual lines estimation unit based on linear regression of the present embodiment is also applied for, here is omitted.
In summary, the two-way direction of visual lines method of estimation based on linear regression of the embodiment of the present invention, passes through the company of tracking
The eyes direction of visual lines of continuous predetermined quantity frame of video is calculated according to initial data set between eyes respectively as initial data set
The linear regression model (LRM) of two direction of visual lines, then by the characteristic vector of PCA extraction initial data set, it is finally right
Current video frame carries out eye tracking, and direction of visual lines estimation is carried out by linear regression model (LRM) in eye tracking results abnormity,
The direction of visual lines of current video frame is expressed as to the form of characteristic vector weighted sum when eye tracking result is normal, and will be current
The eyes direction of visual lines of frame of video, which is updated to primary data, to be concentrated, and extracts new characteristic vector.Thus, by by eyes sight
The potential relation that direction is present is fitted to linear regression model (LRM), if so that during eye tracking is carried out eye with
Track results abnormity or direction of visual lines estimate this when can not obtain using the direction of visual lines of linear regression model (LRM) and another eyes
The direction of visual lines of eyes, obtains a rational binocular direction of visual lines result, increases after the robustness of eye tracking, and utilization
The eye tracking result of continuous frame of video constantly updates optimization linear regression model (LRM), further improves the accurate of direction of visual lines estimation
Property.
In the description of this specification, reference term " one embodiment ", " some embodiments ", " example ", " specifically show
The description of example " or " some examples " etc. means to combine specific features, structure, material or the spy that the embodiment or example are described
Point is contained at least one embodiment of the present invention or example.In this manual, to the schematic representation of above-mentioned term not
Identical embodiment or example must be directed to.Moreover, specific features, structure, material or the feature of description can be with office
Combined in an appropriate manner in one or more embodiments or example.In addition, in the case of not conflicting, the skill of this area
Art personnel can be tied the not be the same as Example or the feature of example and non-be the same as Example or example described in this specification
Close and combine.
In addition, term " first ", " second " are only used for describing purpose, and it is not intended that indicating or implying relative importance
Or the implicit quantity for indicating indicated technical characteristic.Thus, define " first ", the feature of " second " can express or
Implicitly include at least one this feature.In the description of the invention, " multiple " are meant that at least two, such as two, three
It is individual etc., unless otherwise specifically defined.
Any process described otherwise above or method description are construed as in flow chart or herein, represent to include
Module, fragment or the portion of the code of one or more executable instructions for the step of realizing custom logic function or process
Point, and the scope of the preferred embodiment of the present invention includes other realization, wherein can not be by shown or discussion suitable
Sequence, including according to involved function by it is basic simultaneously in the way of or in the opposite order, carry out perform function, this should be of the invention
Embodiment person of ordinary skill in the field understood.
It should be appreciated that each several part of the present invention can be realized with hardware, software, firmware or combinations thereof.Above-mentioned
In embodiment, the software that multiple steps or method can in memory and by suitable instruction execution system be performed with storage
Or firmware is realized.Such as, if realized with hardware with another embodiment, following skill well known in the art can be used
Any one of art or their combination are realized：With the logic gates for realizing logic function to data-signal from
Scattered logic circuit, the application specific integrated circuit with suitable combinational logic gate circuit, programmable gate array (PGA), scene can be compiled
Journey gate array (FPGA) etc..
Those skilled in the art are appreciated that to realize all or part of step that above-described embodiment method is carried
Rapid to can be by program to instruct the hardware of correlation to complete, described program can be stored in a kind of computer-readable storage medium
In matter, the program upon execution, including one or a combination set of the step of embodiment of the method.
In addition, each functional unit in each embodiment of the invention can be integrated in a processing module, can also
That unit is individually physically present, can also two or more units be integrated in a module.Above-mentioned integrated mould
Block can both be realized in the form of hardware, it would however also be possible to employ the form of software function module is realized.The integrated module is such as
Fruit is realized using in the form of software function module and as independent production marketing or in use, can also be stored in a computer
In read/write memory medium.
Storage medium mentioned above can be read-only storage, disk or CD etc..Although having been shown and retouching above
Embodiments of the invention are stated, it is to be understood that above-described embodiment is exemplary, it is impossible to be interpreted as the limit to the present invention
System, one of ordinary skill in the art can be changed to above-described embodiment, change, replace and become within the scope of the invention
Type.
Claims (10)
1. a kind of two-way direction of visual lines method of estimation based on linear regression, it is characterised in that comprise the following steps：
The eyes direction of visual lines of continuous predetermined quantity frame of video is tracked as initial data set；
Calculate the linear regression model (LRM) of two direction of visual lines between eyes respectively according to the initial data set；
Eye tracking is carried out to current video frame, if the eye tracking results abnormity, passes through the linear regression model (LRM)
Direction of visual lines estimation is carried out with the direction of visual lines of an eyes.
2. the method as described in claim 1, it is characterised in that also include:
The characteristic vector of the initial data set is extracted by PCA；
If the eye tracking result is normal, the direction of visual lines of current video frame is expressed as to the shape of characteristic vector weighted sum
Formula.
3. method as claimed in claim 2, it is characterised in that also include：
The eyes direction of visual lines of the current video frame is updated to the primary data and concentrated, and extracts new characteristic vector.
4. the method as described in claim 1, it is characterised in that the continuous predetermined quantity frame of video is effective video frame.
5. the method as described in claim 1, it is characterised in that described to be calculated respectively between eyes according to the initial data set
The linear regression model (LRM) of two direction of visual lines, including：
Obtain the first model matrix that right eye is estimated from left eye；
Obtain the second model matrix that left eye is estimated from right eye；
First model matrix and second model matrix are fitted and obtain described two regard by least square method
The linear regression model (LRM) in line direction.
6. a kind of two-way direction of visual lines estimation unit based on linear regression, it is characterised in that including following：
Tracking module, for tracking the eyes direction of visual lines of continuous predetermined quantity frame of video as initial data set；
Computing module, the linear regression mould for calculating two direction of visual lines between eyes respectively according to the initial data set
Type；
Processing module, for carrying out eye tracking to current video frame, the line is passed through in the eye tracking results abnormity
Property regression model and the direction of visual lines of eye carry out direction of visual lines estimation.
7. device as claimed in claim 6, it is characterised in that also include:
Extraction module, the characteristic vector for extracting the initial data set by PCA；
The processing module, is additionally operable to that the direction of visual lines of current video frame is expressed as into spy when the eye tracking result is normal
Levy the form of vectorial weighted sum.
8. device as claimed in claim 7, it is characterised in that also include：
Update module, is concentrated, and extract for the eyes direction of visual lines of the current video frame to be updated to the primary data
New characteristic vector.
9. device as claimed in claim 6, it is characterised in that the continuous predetermined quantity frame of video is effective video frame.
10. device as claimed in claim 6, it is characterised in that the computing module is used for：
Obtain the first model matrix that right eye is estimated from left eye；
Obtain the second model matrix that left eye is estimated from right eye；
First model matrix and second model matrix are fitted and obtain described two regard by least square method
The linear regression model (LRM) in line direction.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710183955.9A CN107067438B (en) | 2017-03-24 | 2017-03-24 | Bidirectional sight line direction estimation method and device based on linear regression |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710183955.9A CN107067438B (en) | 2017-03-24 | 2017-03-24 | Bidirectional sight line direction estimation method and device based on linear regression |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107067438A true CN107067438A (en) | 2017-08-18 |
| CN107067438B CN107067438B (en) | 2019-12-24 |
Family
ID=59620496
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710183955.9A Active CN107067438B (en) | 2017-03-24 | 2017-03-24 | Bidirectional sight line direction estimation method and device based on linear regression |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107067438B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019128675A1 (en) * | 2017-12-25 | 2019-07-04 | 北京七鑫易维信息技术有限公司 | Method and apparatus for determining parameter in gaze tracking device |
| CN111368589A (en) * | 2018-12-25 | 2020-07-03 | 北京三星通信技术研究有限公司 | Method and device for sight line estimation and electronic equipment |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104089606A (en) * | 2014-06-30 | 2014-10-08 | 天津大学 | Free space eye tracking measurement method |
| CN104966070A (en) * | 2015-06-30 | 2015-10-07 | 北京汉王智远科技有限公司 | Face recognition based living body detection method and apparatus |
| CN105303170A (en) * | 2015-10-16 | 2016-02-03 | 浙江工业大学 | Human eye feature based sight line estimation method |
| WO2016034021A1 (en) * | 2014-09-02 | 2016-03-10 | Hong Kong Baptist University | Method and apparatus for eye gaze tracking |
| CN105955465A (en) * | 2016-04-25 | 2016-09-21 | 华南师范大学 | Desktop portable sight line tracking method and apparatus |
| CN106030614A (en) * | 2014-04-22 | 2016-10-12 | 史內普艾德有限公司 | System and method for controlling a camera based on processing an image captured by other camera |
-
2017
- 2017-03-24 CN CN201710183955.9A patent/CN107067438B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106030614A (en) * | 2014-04-22 | 2016-10-12 | 史內普艾德有限公司 | System and method for controlling a camera based on processing an image captured by other camera |
| CN104089606A (en) * | 2014-06-30 | 2014-10-08 | 天津大学 | Free space eye tracking measurement method |
| WO2016034021A1 (en) * | 2014-09-02 | 2016-03-10 | Hong Kong Baptist University | Method and apparatus for eye gaze tracking |
| CN104966070A (en) * | 2015-06-30 | 2015-10-07 | 北京汉王智远科技有限公司 | Face recognition based living body detection method and apparatus |
| CN105303170A (en) * | 2015-10-16 | 2016-02-03 | 浙江工业大学 | Human eye feature based sight line estimation method |
| CN105955465A (en) * | 2016-04-25 | 2016-09-21 | 华南师范大学 | Desktop portable sight line tracking method and apparatus |
Non-Patent Citations (5)
| Title |
|---|
| FENG LU 等: "Adaptive Linear Regression for Appearance-Based Gaze Estimation", 《IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE》 * |
| SANKHA S. MUKHERJEE 等: "Deep Head Pose:Gaze-Direction Estimation in Multimodal Video", 《IEEE TRANSACTIONS ON MULTIMEDIA》 * |
| 蔡海斌: "稳定的非接触式眼球跟踪与视线估计", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
| 郑雨: "基于双目视觉的驾驶员视线估计关键技术研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
| 黄莹 等: "基于双光源的实时视线追踪系统", 《中国工程科学》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019128675A1 (en) * | 2017-12-25 | 2019-07-04 | 北京七鑫易维信息技术有限公司 | Method and apparatus for determining parameter in gaze tracking device |
| US11380134B2 (en) | 2017-12-25 | 2022-07-05 | Beijing 7Invensun Technology Co., Ltd. | Method and device for determining parameter for gaze tracking device |
| CN111368589A (en) * | 2018-12-25 | 2020-07-03 | 北京三星通信技术研究有限公司 | Method and device for sight line estimation and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107067438B (en) | 2019-12-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Sun et al. | Colorectal polyp segmentation by U-Net with dilation convolution | |
| CN105339868B (en) | Vision enhancement based on eyes tracking | |
| CN108122234A (en) | Convolutional neural networks training and method for processing video frequency, device and electronic equipment | |
| CN106095089A (en) | A kind of method obtaining interesting target information | |
| CN106856009A (en) | Retina female is rendered | |
| CN109816624A (en) | Appearance inspection device | |
| US20210166450A1 (en) | Motion trajectory drawing method and apparatus, and device and storage medium | |
| EP3511803A1 (en) | Method and apparatus to determine trigger intent of user | |
| CN109753850A (en) | The training method and training equipment of face recognition model | |
| CN110136522A (en) | Skull base surgery simulation teching training system | |
| US11238651B2 (en) | Fast hand meshing for dynamic occlusion | |
| CN107067438A (en) | Two-way direction of visual lines method of estimation and device based on linear regression | |
| Cunningham et al. | Fully automated image-based estimation of postural point-features in children with cerebral palsy using deep learning | |
| JP2023545189A (en) | Image processing methods, devices, and electronic equipment | |
| CN107422844A (en) | A kind of information processing method and electronic equipment | |
| US20240161652A1 (en) | Systems and methods for assessing surgical skill | |
| US11321865B1 (en) | Synthetic infrared image generation for machine learning of gaze estimation | |
| Courty et al. | A new application for saliency maps: Synthetic vision of autonomous actors | |
| CN108475114A (en) | Feedback for subject poses tracker | |
| CN109886198A (en) | A kind of information processing method, device and storage medium | |
| CN113723187A (en) | Semi-automatic labeling method and system for gesture key points | |
| CN108197274A (en) | Abnormal individual character detection method and device based on dialogue | |
| CN110503072A (en) | Face age estimation method based on multiple branch circuit CNN framework | |
| CN115661142A (en) | Tongue diagnosis image processing method, device and medium based on key point detection | |
| CN104680134B (en) | Quick human body detecting method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |