CN102749991B - A kind of contactless free space sight tracing being applicable to man-machine interaction - Google Patents

A kind of contactless free space sight tracing being applicable to man-machine interaction Download PDF

Info

Publication number
CN102749991B
CN102749991B CN201210107182.3A CN201210107182A CN102749991B CN 102749991 B CN102749991 B CN 102749991B CN 201210107182 A CN201210107182 A CN 201210107182A CN 102749991 B CN102749991 B CN 102749991B
Authority
CN
China
Prior art keywords
eye
image
human eye
model
pupil
Prior art date
Application number
CN201210107182.3A
Other languages
Chinese (zh)
Other versions
CN102749991A (en
Inventor
黄若浩
Original Assignee
广东百泰科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 广东百泰科技有限公司 filed Critical 广东百泰科技有限公司
Priority to CN201210107182.3A priority Critical patent/CN102749991B/en
Publication of CN102749991A publication Critical patent/CN102749991A/en
Application granted granted Critical
Publication of CN102749991B publication Critical patent/CN102749991B/en

Links

Abstract

The invention provides a kind of contactless free space sight tracing being applicable to man-machine interaction, comprise the following steps: real-time face and eyes locating and tracking, eye move biological information extract, set up based on ocular bioavailability characteristic information eye movement model and build the mapping relations model etc. of eye movement model and eye gaze object.The present invention relates to multiple crossing domains such as image procossing, computer vision, pattern-recognition, the fields such as, Aero-Space association area auxiliary man-machine interaction of new generation, disabled person, sports, carplane are driven, virtual reality and game have a wide range of applications, in addition to raising disabled person life and self-care level, build a harmonious society, improve China man-machine interaction, the capability of independent innovation of the high-technology field such as unmanned has great realistic meaning.

Description

A kind of contactless free space sight tracing being applicable to man-machine interaction

Technical field

The present invention relates to the eye tracking and technical field that are applied to man-machine interaction, relate to a kind of contactless free space sight tracing being applicable to man-machine interaction based on Computer Image Processing and area of pattern recognition basis are developed especially.

Background technology

Traditional man-machine interaction mode is " centered by computing machine ", namely requires that the regulation that user will obey computing machine could use, and the training therefore sometimes needing corresponding specialty just can be carried out.And along with technical development and universal, information society more requires to carry out " man-machine interaction focusing on people ", computing machine is made to be everyone service in society, the center of man-machine interaction should be transformed into that focus be put on man, reach people and during computer interactive just as if the mutual the same effect of mankind itself.Wherein, in three dimensions, computing machine has also been embedded in various domestic electric appliance, living space and apparatus for human lives and activity, and the apparatus that before not being, that large volume position is fixing, the form of expression is diversification all the more.So man-machine interaction also demand fulfillment can make user can be convenient to use computing machine in three dimensions, instead of must sit in face of computing machine, is undertaken by the mode such as keyboard, mouse.

In fact, from the development history of man-machine interaction, from the most ancient man-machine interaction punched card, most main mode is become to keyboard and mouse, utilize now the sensation of people and action (as voice, hand-written, posture, sight line, expression etc.) as the rise of the research and development application of input mode, man-machine interaction experienced by and adapts to computing machine from people and constantly adapt to man-based development process to computing machine.Allow and calculate that function is listened, can be seen, talkative, the main development direction that can feel to be considered to following man-machine interaction.

And computing machine will be made to realize above-mentioned functions, simple hand motion operation keyboard and mouse obviously can not meet the demands, so other sense organ organ of people also management and participating in computing machine gradually, wherein Visual Trace Technology is the most important thing wherein, the object of this technology is the content that the information inference people that watch attentively from user are interested or arouse attention, and obtain its referents by the object that people watches attentively, the relation between hint object.The field such as early stage Visual Trace Technology is mainly used in psychological study, help the disabled, was just applied to the usability engineering such as compression of images and man-machine interaction afterwards.

Visual Trace Technology has broad application prospects, such as, can help paralytic or quadriplegia, and speechless people realizes normal interactive process.In addition, can also be controlled external unit by eye gazing, and realize multi-job operation, such as militarily, if pilot has found target, when manual operation is dealt with and do not come, while can being aimed at by eyes, control the transmitting of fire control system with eyes, just enhance operational efficiency greatly like this.The research of Visual Trace Technology relates to multiple crossing domain, its achievement in research Aero-Space association area, sports, etc. every field have a wide range of applications.

But at present, volume, weight for realizing the system equipment of eye tracking are all larger, also limit the degree of freedom of people simultaneously, larger to the interference of people, use very inconvenient, and the price of commercial product is generally also costly, so, make the universal of view line tracking device become more difficult.Therefore, reduce the hardware cost of gaze tracking system or equipment, development non-intrusion type Visual Trace Technology is a kind of development trend.

Summary of the invention

The object of the invention is to overcome the deficiencies in the prior art, a kind of contactless free space sight tracing being applicable to man-machine interaction is provided.

The present invention is achieved through the following technical solutions:

Be applicable to a contactless free space sight tracing for man-machine interaction, comprise the following steps.

1) real-time face and eyes locating and tracking

Captured the image of face and human eye by convention video tracking camera in real time, facial image is positioned to the analyzing and processing of pupil, described analyzing and processing mode is specially: adopt Viola algorithm to set up face classification device and detect face; On human face region, use Viola algorithm to set up human eye sorter to locate human eye area again simultaneously; Then, adopt based on the legal position pupil center of image gray projection, realize human face region to human eye area, and human eye area is to the image procossing simplification process of pupil region.

The method that described Viola algorithm sets up face classification device is:

Set up the living things feature recognition algorithm model based on cascade cascade searching algorithm Face datection, class rectangle (Haar-like) feature based on integrogram is extracted to the facial image database set up, adopt adaboost to advance training algorithm carry out sorter training to facial image database and classify and obtain face classification device, do pre-service in conjunction with colour of skin coupling.

The method that described VViola algorithm sets up human eye sorter is:

Set up the living things feature recognition algorithm model based on cascade cascade searching algorithm human eye detection, class rectangle (Haar-like) feature based on integrogram is extracted to the facial image database set up, adaboost is adopted to advance training algorithm carry out sorter training to facial image database and classify and obtain human eye sorter, can detection & localization human eye accurately by the sorter that obtains.

The described method based on the legal position pupil center of image gray projection is:

Change human eye area image into gray level image, size is m*n, by formula:

Ph y ( y ) = Σ x = 0 n - 1 I ( x , y ) ,

Ph x ( x ) = Σ y = 0 m - 1 I ( x , y )

Do the Gray Projection of horizontal and vertical, the Gray Projection of horizontal and vertical, wherein respectively have a minimal value in the vertical direction of pupil center and the Gray Projection of horizontal direction, can try to achieve pupil center Q is:

(x 0,y 0)wherePh y(y 0)=Max{Ph y(y)}andPh x(x 0)=Max{Ph x(x)}。

2) eye moves biological information extraction

Detect and gather the position of human eye in image, extracting eyes subimage, wherein adopting Viola algorithm to set up face classification device and face is detected, on human face region, use Viola algorithm to set up human eye sorter to locate human eye area more simultaneously; Further, by extracting the mobile message of pupil based on the method for corneal reflection principle and image procossing, pass through based on the legal position pupil center of image gray projection, and adopt the EHMM based on 2D-DCT feature, analyze by the eye image collected and set up Hidden Markov model, realizing the differentiation of human eye state.

The knowledge method for distinguishing that the described EHMM based on 2D-DCT feature carries out human eye state is specially:

Eye image sampled and 2D-DCT conversion is carried out to each sample window, forming observation sequence vector by the low frequency coefficient after 2D-DCT converts, according to the vector initialising EHMM parameter of observation obtained after status number and image uniform segmentation; Further, carry out the extracting method moving information based on the eye of pupil: by dual nested Viterbi algorithm, eye image is split again, by Baum-welch algorithm revaluation model parameter, to EHMM model training, obtain the human eye state recognition classifier based on EHMM, when wherein human eye state being identified, first constructed by eye image to be identified and observe sequence vector, then calculate each training pattern and produce the likelihood value observing sequence vector, the training pattern with maximum likelihood value is object belonging to eye image to be identified.

3) eye movement model based on ocular bioavailability characteristic information is set up

Set up the model of the rotation center measuring eyeball and the three-dimensional eye movement model based on two-dimentional pupil mobile message and eyeball irregular spheroid rotation information, the bivector of definition from Purkinje spot to pupil center is pupil-corneal reflection vector, be denoted as P-CR, and by carrying out the real-time acquisition eyeball information of shooting to human eye and in conjunction with P-CR, generating human eye to the 3 dimension space direction vectors watching object attentively.

4) the mapping relations model of eye movement model and eye gaze object is built

The pan of people when observing various outdoor scene and screen message is selected and watches process attentively to adopt eye tracking to learn, thus obtain visually-perceptible and the association mechanism of people, set up the transformation relation between visual field coordinate system and pupil coordinate system, obtain the coordinate of the true blinkpunkt of human eye in the coordinate system of visual field, and calculate eye gaze point, what finally blinkpunkt is mapped to user's reality watches attentively on object, what complete visual field (actual eye gaze point) and eye image mates work, realizes the corresponding of video tracking camera field of view and eyes visual field.

Wherein, the method building eye movement model is:

The three-dimensional vector (representing the actual direction of gaze of sight line) formed by the combination of pupil two-dimensional signal and eyeball shape information, and then set up eye movement model, concrete is: calculate eyeball radius according to eye image information, eyeglass center, location, then calculate human eye to the three-dimensional space direction vector watching object attentively; Then, by the Purkinje image point legal parallactic angle film curved surface centre of sphere (O of image procossing, improvement cornea), in conjunction with the two dimensional surface information of pupil center, generate the three-dimensional model that an eye is dynamic.

In step 4, to be admired (Purkinje) spotting method by pul, at screen four angles, infrared LED is set respectively as light source, corneal reflection on pupil, by obtaining each two field picture with the camera of optical filter, wherein, at camera collection in eye image, there will be four obvious bright spots around pupil center, by the method for the geometrical constraint of image procossing, the outline map of original image is first obtained with Canny boundary operator, adopt Hough transform that eyeball image is projected to parameter space from plane space again, find out the hot spot center of circle, accurately can locate the relative position of pupil center and four bright spots, then using reflection spot as reference point, the coordinate figure of pupil center is performed mathematical calculations with it, and then judge the two dimensional motion in-plane of eyeball.

Meanwhile, also comprise the scaling method of blinkpunkt in step 4, be specially:

1) mapping relations equation is built

If vectorial y is the eye gaze point of visual field reference system, vector x is pupil center's subpoint thereon in (human eye) reference system, by the transformation relation of function F (*) representative from x to y, P representative is determined statistically comprehensive parameters vector in calibration process, namely the parameter vector in original unknown F (*), then have:

y=F(x,P);

Determine the concrete form of function F (x, P), and try to achieve the estimated value p ' of statistically comprehensive parameters vector P, thus obtain the estimated value y ' of eye gaze point position:

y′=F(x,P′)。

2) statistically comprehensive parameters vector P is determined

Determine the valuation P ' of statistically comprehensive parameters vector P, concrete, adopt the calibration algorithm based on least square curve fitting, design a merit function, for the degree of consistency between metric measurement data and the parameter model of selection according to one group of measurement data; Regulate model parameter simultaneously, make merit function value minimum, obtain best fit parameters P.

Assuming that P is M dimension, total N number of test point, by this M adjustable parameter Pi (i=1,2 ..., M) model to N number of test data (xi, yi) i=1 ..., N carries out matching.Definition has the vectorial b of N number of component, then have: bi=y/Ri, i=1 ..., N.

Wherein R ibe the measuring error of i-th data point, default value is 1; To vectorial P and N number of data, have: yi (xi)=∑ pkXk (xi), i=1 ..., N.

X k(x i) be called one group of basis function.Then matrix A=(a is defined ij) n × M, its element by M basis function at N number of coordinate x ion value and N number of measuring error calculate, that is:

Definition merit function X 2=| A*P-b| 2, try to achieve parameter vector P, make X2 be minimum value.In over-determined systems situation, the optimal approximation solution under least square meaning can be drawn by the method that SVD decomposes.When carrying out matching by SVD least square method to eye-movement measurement data, the form of fitting function can be specified as required.After trying to achieve parameter vector P, the eye that can obtain y=F (x, P) function moves the mapping relations model of information model to eye gaze object, in man-machine interaction actual for eye tracking application.

Compared with prior art, the present invention has following beneficial effect:

The invention provides more creationary method, judge the two dimensional motion in-plane of eyeball, the biological information that eye is dynamic can accurately be extracted, ensure the data accuracy of the process moving the eye tracking of biological information based on eye, simultaneously, adopt the vectorial actual direction of gaze etc. representing eyes of the three-dimensional formed by the combination of pupil two-dimensional signal and eyeball shape information, the error that the good eye movement that compensate for irregular spheroid brings, obtains great breakthrough to the accuracy improving Visual Trace Technology, in addition, also apply dynamic calibration algorithm, when overcoming static demarcating, subject's head keeps the deficiency of absolute rest in eye-movement measurement process, and the head rotation etc. of user affects the problem of eye tracking free walker, consider eye tracking working mechanism, what complete visual field (actual eye gaze point) and eye movement model mates work, effectively solve the nonlinear problem of the mapping relations between visual field coordinate system and pupil coordinate system, construct dynamic eye and move the mapping model of information model to eye gaze object, the perfect low defect not strong with practicality of Visual Trace Technology degree of freedom now.

The present invention relates to multiple crossing domains such as image procossing, computer vision, pattern-recognition, its achievement in research is auxiliary man-machine interaction of new generation, disabled person, Aero-Space association area, sports, carplane are driven, the field such as virtual reality and game has a wide range of applications.To raising disabled person life and self-care level, build a harmonious society, improve China man-machine interaction, the capability of independent innovation of the high-technology field such as unmanned has great realistic meaning.

Accompanying drawing explanation

Below in conjunction with embodiment and accompanying drawing, the present invention is described in further detail:

Figure 1 shows that the overall flow schematic diagram of the present invention one specific embodiment;

Figure 2 shows that the method modular diagram of the present invention one specific embodiment;

Figure 3 shows that the eyeball phantom schematic diagram involved by the present invention one specific embodiment.

Embodiment

Below in conjunction with embodiment and accompanying drawing, the present invention is described in further detail, but embodiments of the present invention are not limited thereto.

As shown in FIG. 1 to 3 for can be used as a kind of contactless free space sight tracing being applicable to man-machine interaction of present pre-ferred embodiments, comprise the following steps.

1) real-time face and eyes locating and tracking

The image of face and human eye is captured in real time by convention video tracking camera, facial image is positioned to the analyzing and processing of pupil, convention video tracking camera can Real-time Collection to facial image, although head is in continuous activity, the localization and tracking of face and human eye in image can be realized by the image analyzing and processing technology of existing software.

Described analyzing and processing mode is specially: utilize the method for image procossing and pattern-recognition to locate pupil, because human eye area is little, if directly adopt the algorithm at entire image location pupil, the problems such as false drop rate is high, calculated amount is large can be produced, detect so adopt Viola algorithm to set up face classification device to face; On human face region, use Viola algorithm to set up human eye sorter to locate human eye area again simultaneously; Then, due to the accurate location based on human eye area, adopt based on the legal position pupil center of image gray projection, realize human face region to human eye area, and human eye area is to the image procossing simplification process of pupil region.

The method that described Viola algorithm sets up face classification device is:

Set up the living things feature recognition algorithm model based on cascade cascade searching algorithm Face datection, class rectangle (Haar-like) feature based on integrogram is extracted to the facial image database set up, adopt adaboost to advance training algorithm carry out sorter training to facial image database and classify and obtain face classification device, do pre-service in conjunction with colour of skin coupling.

Ongoing research confirms, after Viola algorithm does pre-service in conjunction with colour of skin coupling, and more accurate Face datection model under complex background and light conditions can be obtained.

The method that described Viola algorithm sets up human eye sorter is:

Set up the living things feature recognition algorithm model based on cascade cascade searching algorithm human eye detection, class rectangle (Haar-like) feature based on integrogram is extracted to the facial image database set up, adaboost is adopted to advance training algorithm carry out sorter training to facial image database and classify and obtain human eye sorter, can detection & localization human eye accurately by the sorter that obtains.

The described method based on the legal position pupil center of image gray projection is:

Change human eye area image into gray level image, size is m*n, by formula:

Ph y ( y ) = Σ x = 0 n - 1 I ( x , y ) ,

Ph x ( x ) = Σ y = 0 m - 1 I ( x , y )

Do the Gray Projection of horizontal and vertical, the Gray Projection of horizontal and vertical, wherein respectively have a minimal value in the vertical direction of pupil center and the Gray Projection of horizontal direction, can try to achieve pupil center Q is:

(x 0,y 0)wherePh y(y 0)=Max{Ph y(y)}andPh x(x 0)=Max{Ph x(x)}。

2) eye moves biological information extraction

Detect and gather the position of human eye in image, extracting eyes subimage, wherein adopting Viola algorithm to set up face classification device and face is detected, on human face region, use Viola algorithm to set up human eye sorter to locate human eye area more simultaneously.

Analysis chart comprises as eye movement characteristics: watch (fixation) attentively, beat (saccades) and smoothly trail tracking (smoothpursuit) etc., these eye movement characteristics main manifestations are that pupil center moves, in conjunction with Purkinje image point method, by extracting the mobile message of pupil based on the method for corneal reflection principle and image procossing, pass through based on the legal position pupil center of image gray projection, and adopt the EHMM based on 2D-DCT feature, analyze by the eye image collected and set up Hidden Markov model, realizing the differentiation of human eye state.

The knowledge method for distinguishing that the described EHMM based on 2D-DCT feature carries out human eye state is specially:

Eye image sampled and 2D-DCT conversion is carried out to each sample window, forming observation sequence vector by the low frequency coefficient after 2D-DCT converts, according to the vector initialising EHMM parameter of observation obtained after status number and image uniform segmentation; Further, carry out the extracting method moving information based on the eye of pupil: by dual nested Viterbi algorithm, eye image is split again, by Baum-welch algorithm revaluation model parameter, to EHMM model training, obtain the human eye state recognition classifier based on EHMM, when wherein human eye state being identified, first constructed by eye image to be identified and observe sequence vector, then calculate each training pattern and produce the likelihood value observing sequence vector, the training pattern with maximum likelihood value is object belonging to eye image to be identified.

3) eye movement model based on ocular bioavailability characteristic information is set up

The direction of visual lines of eyes is the rectilinear directions connecting eyeball center and pupil center, and the change of direction of visual lines is that centre of sphere axle rotates a certain angle with eyeball.

Set up the model of the rotation center measuring eyeball and the three-dimensional eye movement model based on two-dimentional pupil mobile message and eyeball irregular spheroid rotation information, the bivector of definition from Purkinje spot to pupil center is pupil-corneal reflection vector, be denoted as P-CR, and by carrying out the real-time acquisition eyeball information of shooting to human eye and in conjunction with P-CR, generating human eye to the 3 dimension space direction vectors watching object attentively.

4) the mapping relations model of eye movement model and eye gaze object is built

The pan of people when observing various outdoor scene and screen message is selected and watches process attentively to adopt eye tracking to learn, thus obtain visually-perceptible and the association mechanism of people, set up the transformation relation between visual field coordinate system and pupil coordinate system, obtain the coordinate of the true blinkpunkt of human eye in the coordinate system of visual field, and calculate eye gaze point, what finally blinkpunkt is mapped to user's reality watches attentively on object, what complete visual field (actual eye gaze point) and eye image mates work, realizes the corresponding of video tracking camera field of view and eyes visual field.

Wherein, the method building eye movement model is:

The eyeball of people is a spheroid substantially, the natural functions that human eye has is realized by forms of motion different in eye movement, as the means of man-machine interaction, be worth it is of concern that target after eye movement, instead of its motion process, so need the three-dimensional vector (representing the actual direction of gaze of sight line) that the eye movement model set up namely is formed by the combination of pupil two-dimensional signal and eyeball shape information, and then set up eye movement model, concrete is:

Calculate eyeball radius according to eye image information, eyeglass center, location, human eye can be calculated to the three-dimensional space direction vector watching object attentively.But eyeball is not exclusively spheroid, if therefore can certain defect be there is with conventional algorithm.When Physiologic Studies proves eye movement, its center is not a point of fixity, but along a movement in a curve, is called the shifting movement of eyeball, but there are some researches show eyeball, the extreme sport of rotating the centre of form when rotating in the scope of ± 38 ° from primary position of eye is less than 2mm simultaneously; When eyeball rotates within the scope of ± 3 °, the motion of the centre of form is less than 0.2mm, and for eye tracking, the center of rotation of eyeball is fixing.Eyeball physical arrangement is in fact inlayed by former and later two spheroids to form, and the spheroid radius-of-curvature accounting for volume 1/6 is above about 8mm, and radius of sphericity is below about 12mm, and the spherula center of circle is the cornea curved surface centre of sphere, and the direction of sight line is the upper O of figure corneato (optical axis VisualAxis) on the line of actual blinkpunkt, so the optical axis determines the direction of gaze of sight line.

Described on total, then analyze eyeball physical arrangement further, in conjunction with its characteristic, by the Purkinje image point legal parallactic angle film curved surface centre of sphere (O of image procossing, improvement cornea), and in conjunction with the two dimensional surface information of pupil center, finally generate the dynamic three-dimensional model of an eye.

Based on the Visual Trace Technology of pupil one corneal reflection vector method and image procossing, there is non-invasive advantage, achieve very fast progress in recent years, near-infrared light source light determines the moving direction of pupil at the hot spot (glint) of cornea eye reflection generation and the position relationship of pupil center.So in step 4, to be admired (Purkinje) spotting method by pul, at screen four angles, infrared LED is set respectively as light source, corneal reflection on pupil, by obtaining each two field picture with the camera of optical filter, wherein, at camera collection in eye image, there will be four obvious bright spots around pupil center, by the method for the geometrical constraint of image procossing, the outline map of original image is first obtained with Canny boundary operator, adopt Hough transform that eyeball image is projected to parameter space from plane space again, find out the hot spot center of circle, accurately can locate the relative position of pupil center and four bright spots, then using reflection spot as reference point, the coordinate figure of pupil center is performed mathematical calculations with it, and then judge the two dimensional motion in-plane of eyeball.

Simultaneously, the scaling method of blinkpunkt is also comprised in step 4, as the mapping relations between the direction of gaze that eye tracking will be used for just must completing in man-machine interaction sight line to computer screen point vector, a key point of practical application can be carried out so the demarcation of blinkpunkt is eye tracking, it is the condition precedent of system worked well, also be the key that system can move towards practical, be specially:

1) mapping relations equation is built

The problems such as distortion are extracted owing to there are data when actual eye moves information, there is unintentional nonlinearity factor in the mapping relations F (*) that eye camera image coordinate is tied between screen coordinate system, so F (*) can not describe by simple linear relationship.In order to determine mapping relations F (*), if vectorial y is the eye gaze point of visual field reference system, vector x is pupil center's subpoint thereon in (human eye) reference system, by the transformation relation of function F (*) representative from x to y, P representative is determined statistically comprehensive parameters vector in calibration process, namely the parameter vector in original unknown F (*), then have:

y=F(x,P);

Determine the concrete form of function F (x, P), and try to achieve the estimated value p ' of statistically comprehensive parameters vector P, thus obtain the estimated value y ' of eye gaze point position:

y′=F(x,P′)。

Further, make | y-y ' | the form of the function of → 0 determines primarily of the conversion of two described in step 1 is common.Due to the characteristic present point after eye rotation angles and eye imaging---there is unintentional nonlinearity between pupil center location; Simultaneously cause the distortion of data owing to being difficult to ensure the absolute rest of head in test process; In addition the impact of the sphere of eyes and the factor such as the position of light source and intensity also causes the abundant and even of very difficult underwriter's eye light, and in test process, easily cause the distortion of data because of absent minded.These also all result in F (*) and can not describe by simple linear relationship.So the task that mapping relations equation of the present invention will complete is the working mechanism considering system, what complete visual field (actual eye gaze point) and eye image mates work, solve in measuring process the nonlinear problem occurred as far as possible, by the measured value of system in certain accuracy rating " reduction " in the reference system of visual field.

2) statistically comprehensive parameters vector P is determined

The key link of demarcating how to determine the valuation P ' of statistically comprehensive parameters vector P, concrete, adopt the calibration algorithm based on least square curve fitting, design a merit function, for the degree of consistency between metric measurement data and the parameter model of selection according to one group of measurement data; Regulate model parameter simultaneously, make merit function value minimum, obtain best fit parameters P.

Assuming that P is M dimension, total N number of test point, by this M adjustable parameter Pi (i=1,2 ..., M) model to N number of test data (xi, yi) i=1 ..., N carries out matching.Definition has the vectorial b of N number of component, then have: bi=y/Ri, i=1 ..., N.

Wherein R ibe the measuring error of i-th data point, default value is 1; To vectorial P and N number of data, have: yi (xi)=∑ pkXk (xi), i=1 ..., N.

X k(x i) be called one group of basis function.Then matrix A=(a is defined ij) n × M, its element by M basis function at N number of coordinate x ion value and N number of measuring error calculate, that is:

Definition merit function X 2=| A*P-b| 2, try to achieve parameter vector P, make X2 be minimum value.In over-determined systems situation, the optimal approximation solution under least square meaning can be drawn by the method that SVD decomposes.When carrying out matching by SVD least square method to eye-movement measurement data, the form of fitting function can be specified as required.After trying to achieve parameter vector P, the eye that can obtain y=F (x, P) function moves the mapping relations model of information model to eye gaze object, in man-machine interaction actual for eye tracking application.

The application of the present invention in actual man-machine interactive operation:

By adopting the method for the invention, come operating computer or other equipment by eyes, current application can be embodied in: the controlling functions 1. realizing eye-controlled mouse, as: control text reading and webpage rolling, play music and other multimedia; 2. according to the needs of embody rule, various eye movement characteristics is corresponded in the concrete operating function of software, as in electric athletic game, just can transfer a certain stunt function etc., for game enthusiasts provides a kind of interactive mode of fashion by eyes; 3. from the physiological medical science feature of people, people is once occur that fatigue is easy to move rule from eye reflect, as frequency of wink, blink, doze off and divert attention, by the physiological medical science feature of above combine with technique fatigue, realize carrying out detection early warning to driver, important or dangerous post, voice reminder staff take care, tired alarm can be sent to the Surveillance center of enterprise by network simultaneously, or by 3G wireless network, alarm is sent in the mobile phone of managerial personnel, facilitate Enterprises Leader grasp important information in time and make a policy.

Claims (8)

1. be applicable to a contactless free space sight tracing for man-machine interaction, it is characterized in that comprising the following steps:
1) real-time face and eyes locating and tracking:
Captured the image of face and human eye by video tracking video camera in real time, facial image is positioned to the analyzing and processing of pupil, described analyzing and processing mode is specially: adopt Viola algorithm to set up face classification device and detect face; On human face region, use Viola algorithm to set up human eye sorter to locate human eye area again simultaneously; Then, adopt based on the legal position pupil center of image gray projection, realize human face region to human eye area, and human eye area is to the image procossing simplification process of pupil region;
2) eye moves biological information extraction:
Detect and gather the position of human eye in image, extracting eyes subimage, wherein adopting Viola algorithm to set up face classification device and face is detected, on human face region, use Viola algorithm to set up human eye sorter to locate human eye area more simultaneously; Further, by extracting the mobile message of pupil based on the method for corneal reflection principle and image procossing, pass through based on the legal position pupil center of image gray projection, and adopt the EHMM based on 2D-DCT feature, analyze by the eye image collected and set up Hidden Markov model, realizing the differentiation of human eye state;
3) eye movement model based on ocular bioavailability characteristic information is set up:
Set up the model of the rotation center measuring eyeball and the three-dimensional eye movement model based on two-dimentional pupil mobile message and eyeball irregular spheroid rotation information, the bivector of definition from Purkinje spot to pupil center is pupil-corneal reflection vector, be denoted as P-CR, and by carrying out the real-time acquisition eyeball information of shooting to human eye and in conjunction with P-CR, generating human eye to the 3 dimension space direction vectors watching object attentively;
4) the mapping relations model of eye movement model and eye gaze object is built:
The pan of people when observing various outdoor scene and screen message is selected and watches process attentively to adopt eye tracking to learn, thus obtain visually-perceptible and the association mechanism of people, set up the transformation relation between visual field coordinate system and pupil coordinate system, obtain the coordinate of the true blinkpunkt of human eye in the coordinate system of visual field, and calculate eye gaze point, what finally blinkpunkt is mapped to user's reality watches attentively on object, what complete actual eye gaze point and eye image mates work, realizes the corresponding of video tracking camera field of view and eyes visual field.
2. the contactless free space sight tracing being applicable to man-machine interaction according to claim 1, is characterized in that the process that described Viola algorithm sets up face classification device is:
Set up the living things feature recognition algorithm model based on cascade cascade searching algorithm Face datection, class rectangle (Haar-like) feature based on integrogram is extracted to the facial image database set up, adopt adaboost to advance training algorithm carry out sorter training to facial image database and classify and obtain face classification device, do pre-service in conjunction with colour of skin coupling.
3. the contactless free space sight tracing being applicable to man-machine interaction according to claim 1, is characterized in that the process that described Viola algorithm sets up human eye sorter is:
Set up the living things feature recognition algorithm model based on cascade cascade searching algorithm human eye detection, class rectangle (Haar-like) feature based on integrogram is extracted to the facial image database set up, adaboost is adopted to advance training algorithm carry out sorter training to facial image database and classify and obtain human eye sorter, by the detection of classifier and the location human eye that obtain.
4. the contactless free space sight tracing being applicable to man-machine interaction according to claim 1, is characterized in that the described process based on the legal position pupil center of image gray projection is:
Change human eye area image into gray level image, its size is m*n, by formula:
Ph y ( y ) = Σ x = 0 n - 1 I ( x , y ) ,
Ph x ( x ) = Σ y = 0 m - 1 I ( x , y )
Do the Gray Projection of horizontal and vertical, wherein respectively have a minimal value in the vertical direction of pupil center and the Gray Projection of horizontal direction, can try to achieve pupil center Q is:
(x 0,y 0)wherePh y(y 0)=Max{Ph y(y)}andPh x(x 0)=Max{Ph x(x)}。
5. the contactless free space sight tracing being applicable to man-machine interaction according to claim 1, is characterized in that being specially based on the method for the EHMM of 2D-DCT feature in described step 2:
Eye image sampled and 2D-DCT conversion is carried out to each sample window, forming observation sequence vector by the low frequency coefficient after 2D-DCT converts, according to the vector initialising EHMM parameter of observation obtained after status number and image uniform segmentation; Further, carry out the extracting method moving information based on the eye of pupil: by dual nested Viterbi algorithm, eye image is split again, by Baum-welch algorithm revaluation model parameter, to EHMM model training, obtain the human eye state recognition classifier based on EHMM, when wherein human eye state being identified, first constructed by eye image to be identified and observe sequence vector, then calculate each training pattern and produce the likelihood value observing sequence vector, the training pattern with maximum likelihood value is object belonging to eye image to be identified.
6. the contactless free space sight tracing being applicable to man-machine interaction according to claim 1, is characterized in that the mode setting up eye movement model in described step 4 is specially:
The three-dimensional vector formed by the combination of pupil two-dimensional signal and eyeball shape information, and then set up eye movement model, concrete is: calculate eyeball radius according to eye image information, eyeglass center, location, then calculates human eye to the three-dimensional space direction vector watching object attentively; Then, by the Purkinje image point legal parallactic angle film curved surface centre of sphere (O of image procossing, improvement cornea), in conjunction with the two dimensional surface information of pupil center, generate the three-dimensional model that an eye is dynamic.
7. the contactless free space sight tracing being applicable to man-machine interaction according to claim 1, it is characterized in that in described step 4, to be admired (Purkinje) spotting method by pul, at screen four angles, infrared LED is set respectively as light source, corneal reflection on pupil, by obtaining each two field picture with the camera of optical filter, wherein, at camera collection in eye image, there will be four obvious bright spots around pupil center, by the method for the geometrical constraint of image procossing, the outline map of original image is first obtained with Canny boundary operator, adopt Hough transform that eyeball image is projected to parameter space from plane space again, find out the hot spot center of circle, accurately can locate the relative position of pupil center and four bright spots, then using reflection spot as reference point, the coordinate figure of pupil center is performed mathematical calculations with it, and then judge the two dimensional motion in-plane of eyeball.
8. the contactless free space sight tracing being applicable to man-machine interaction according to claim 1, is characterized in that described step 4 comprises the computing method of actual eye gaze point, specific as follows:
1) mapping relations equation is built:
If vectorial y is the eye gaze point of visual field reference system, vector x is pupil center's subpoint thereon in (human eye) reference system, by the transformation relation of function F (*) representative from x to y, P representative is determined statistically comprehensive parameters vector in calibration process, namely the parameter vector in original unknown F (*), then have:
y=F(x,P);
Determine the concrete form of function F (x, P), and try to achieve the estimated value p ' of statistically comprehensive parameters vector P, thus obtain the estimated value y ' of eye gaze point position:
y′=F(x,P′);
2) statistically comprehensive parameters vector P is determined:
Determine the valuation P ' of statistically comprehensive parameters vector P, concrete, adopt the calibration algorithm based on least square curve fitting, design a merit function, for the degree of consistency between metric measurement data and the parameter model of selection according to one group of measurement data; Regulate model parameter simultaneously, make merit function value minimum, obtain best fit parameters P;
Assuming that P is M dimension, total N number of test point, by this M adjustable parameter Pi (i=1,2 ... M) model to N number of test data (xi, yi) i=1 ..., N carries out matching, and definition has the vectorial b of N number of component, then have: bi=y/Ri, i=1 ..., N;
Wherein R ibe the measuring error of i-th data point, default value is 1; To vectorial P and N number of data, have: yi (xi)=Σ pkXk (xi), i=1 ..., N;
X k(x i) be called one group of basis function, then define matrix A=(a ij) n × M, its element by M basis function at N number of coordinate x ion value and N number of measuring error calculate, that is:
Definition merit function X 2=| A*P – b| 2try to achieve parameter vector P, make X2 be minimum value, in over-determined systems situation, the optimal approximation solution under least square meaning can be drawn by the method that SVD decomposes, when matching being carried out to eye-movement measurement data by SVD least square method, the form of fitting function can be specified as required, after trying to achieve parameter vector P, can obtain y=F (x, P) eye of function moves the mapping relations model of information model to eye gaze object, in man-machine interaction actual for eye tracking application.
CN201210107182.3A 2012-04-12 2012-04-12 A kind of contactless free space sight tracing being applicable to man-machine interaction CN102749991B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201210107182.3A CN102749991B (en) 2012-04-12 2012-04-12 A kind of contactless free space sight tracing being applicable to man-machine interaction

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201210107182.3A CN102749991B (en) 2012-04-12 2012-04-12 A kind of contactless free space sight tracing being applicable to man-machine interaction

Publications (2)

Publication Number Publication Date
CN102749991A CN102749991A (en) 2012-10-24
CN102749991B true CN102749991B (en) 2016-04-27

Family

ID=47030252

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201210107182.3A CN102749991B (en) 2012-04-12 2012-04-12 A kind of contactless free space sight tracing being applicable to man-machine interaction

Country Status (1)

Country Link
CN (1) CN102749991B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10528849B2 (en) 2015-08-28 2020-01-07 Beijing Kuangshi Technology Co., Ltd. Liveness detection method, liveness detection system, and liveness detection device

Families Citing this family (28)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102930252B (en) * 2012-10-26 2016-05-11 广东百泰科技有限公司 A kind of sight tracing based on the compensation of neutral net head movement
CN102957931A (en) * 2012-11-02 2013-03-06 京东方科技集团股份有限公司 Control method and control device of 3D (three dimensional) display and video glasses
CN102981616B (en) * 2012-11-06 2017-09-22 中兴通讯股份有限公司 The recognition methods of object and system and computer in augmented reality
CN103870796B (en) * 2012-12-13 2017-05-24 汉王科技股份有限公司 Eye sight evaluation method and device
CN103927670B (en) * 2013-01-10 2017-11-28 上海通用汽车有限公司 Quantify the method for the region attention rate of object
CN104076915A (en) * 2013-03-29 2014-10-01 英业达科技有限公司 Exhibition system capable of adjusting three-dimensional models according to sight lines of visitors and method implemented by exhibition system
CN104133548A (en) * 2013-05-03 2014-11-05 中国移动通信集团公司 Method and device for determining viewpoint area and controlling screen luminance
WO2014192103A1 (en) * 2013-05-29 2014-12-04 三菱電機株式会社 Information display device
CN103440038B (en) * 2013-08-28 2016-06-15 中国人民大学 A kind of information acquisition system based on eye recognition and application thereof
TWI562007B (en) * 2013-11-15 2016-12-11 Utechzone Co Ltd
CN103761519B (en) * 2013-12-20 2017-05-17 哈尔滨工业大学深圳研究生院 Non-contact sight-line tracking method based on self-adaptive calibration
CN104978548B (en) * 2014-04-02 2018-09-25 汉王科技股份有限公司 A kind of gaze estimation method and device based on three-dimensional active shape model
EP3137968B1 (en) * 2014-04-29 2019-10-16 Hewlett-Packard Development Company, L.P. Gaze detector using reference frames in media
TWI577327B (en) * 2014-08-14 2017-04-11 由田新技股份有限公司 Method, apparatus and computer program product for positioning pupil
CN104253944B (en) * 2014-09-11 2018-05-01 陈飞 Voice command based on sight connection assigns apparatus and method
JP2017528889A (en) * 2014-09-25 2017-09-28 フィリップス ライティング ホールディング ビー ヴィ Lighting control
CN105138961A (en) * 2015-07-27 2015-12-09 华南师范大学 Eyeball tracking big data based method and system for automatically identifying attractive person of opposite sex
CN105184277A (en) * 2015-09-29 2015-12-23 杨晴虹 Living body human face recognition method and device
CN105867611A (en) * 2015-12-29 2016-08-17 乐视致新电子科技(天津)有限公司 Space positioning method, device and system in virtual reality system
CN105700677A (en) * 2015-12-29 2016-06-22 努比亚技术有限公司 Mobile terminal and control method thereof
CN107180441B (en) * 2016-03-10 2019-04-09 腾讯科技(深圳)有限公司 The method and apparatus for generating eye image
CN105892691A (en) * 2016-06-07 2016-08-24 京东方科技集团股份有限公司 Method and device for controlling travel tool and travel tool system
CN106445115A (en) * 2016-08-31 2017-02-22 中国人民解放军海军医学研究所 Eye movement data-based user help information automatic triggering apparatus and method
CN107103293B (en) * 2017-04-13 2019-01-29 西安交通大学 It is a kind of that the point estimation method is watched attentively based on joint entropy
CN108721070A (en) * 2017-04-24 2018-11-02 河北工业大学 A kind of intelligent vision functional training system and its training method based on eyeball tracking
CN110096130A (en) * 2018-01-29 2019-08-06 美的集团股份有限公司 Control method and device, water heater and computer readable storage medium
CN108572733B (en) * 2018-04-04 2019-03-12 西安交通大学 A kind of eye movement behavior visual search target prediction method based on condition random field
CN108960106B (en) * 2018-06-25 2019-09-20 西安交通大学 A kind of human eye fixation point estimation method based on quantization Minimum error entropy criterion

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101344919A (en) * 2008-08-05 2009-01-14 华南理工大学 Sight tracing method and disabled assisting system using the same
CN101540090A (en) * 2009-04-14 2009-09-23 华南理工大学 Driver fatigue monitoring device based on multivariate information fusion and monitoring method thereof
CN101576771A (en) * 2009-03-24 2009-11-11 山东大学 Scaling method for eye tracker based on nonuniform sample interpolation

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101344919A (en) * 2008-08-05 2009-01-14 华南理工大学 Sight tracing method and disabled assisting system using the same
CN101576771A (en) * 2009-03-24 2009-11-11 山东大学 Scaling method for eye tracker based on nonuniform sample interpolation
CN101540090A (en) * 2009-04-14 2009-09-23 华南理工大学 Driver fatigue monitoring device based on multivariate information fusion and monitoring method thereof

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10528849B2 (en) 2015-08-28 2020-01-07 Beijing Kuangshi Technology Co., Ltd. Liveness detection method, liveness detection system, and liveness detection device

Also Published As

Publication number Publication date
CN102749991A (en) 2012-10-24

Similar Documents

Publication Publication Date Title
Matarić et al. Fixation behavior in observation and imitation of human movement
Kapoor et al. Towards a learning companion that recognizes affect
Zhang et al. RGB-D-based action recognition datasets: A survey
Zhang et al. Multimodal spontaneous emotion corpus for human behavior analysis
Grauman et al. Communication via eye blinks and eyebrow raises: Video-based human-computer interfaces
Tran et al. 3-D posture and gesture recognition for interactivity in smart spaces
CN103443742B (en) For staring the system and method with gesture interface
Li et al. Learning to predict gaze in egocentric video
Valenti et al. Combining head pose and eye location information for gaze estimation
US7043056B2 (en) Facial image processing system
Suarez et al. Hand gesture recognition with depth images: A review
US20160128450A1 (en) Information processing apparatus, information processing method, and computer-readable storage medium
US20160210503A1 (en) Real time eye tracking for human computer interaction
CN100487568C (en) Enhanced real natural interactive helmet with sight line follow-up function
Le et al. Human posture recognition using human skeleton provided by Kinect
CN103597515A (en) System for recognizing an open or closed hand
CN109799900A (en) The wireless wrist connected for three-dimensional imaging, mapping, networking and interface calculates and controls device and method
JPH11175246A (en) Sight line detector and method therefor
Plantard et al. Validation of an ergonomic assessment method using Kinect data in real workplace conditions
Chen et al. Probabilistic gaze estimation without active personal calibration
US20140049465A1 (en) Gesture operated control for medical information systems
Al-Rahayfeh et al. Eye tracking and head movement detection: A state-of-art survey
Fitzpatrick From First Contact to Close Encounters: A developmentally deep perceptual system for a humanoid robot
CN101561710A (en) Man-machine interaction method based on estimation of human face posture
Pamplona et al. The image-based data glove

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
C14 Grant of patent or utility model
GR01 Patent grant