CN103761519A - Non-contact sight-line tracking method based on self-adaptive calibration - Google Patents

Abstract

The invention provides a non-contact sight-line tracking method based on self-adaptive calibration. A light spot characteristic extracting method combining the BFS algorithm, image geometric features and gray features is used for accurately matching light spots with corresponding light sources; by means of a fitting method comprising the steps that circular fitting is conducted through a one-dimension edge detection operator and least square ellipse fitting, and noisy points are removed until the center of an ellipse is fixed, the accurate centers of pupils are obtained finally. Besides, a dynamically self-adaptive calibration is provided, and existing space mapping model precision is improved effectively.

Description

A kind of contactless sight line method for tracing based on adaptive calibration

Technical field

The invention belongs to the method for tracing of computer vision field, relate generally to a kind of contactless Eye Tracking Technique based on adaptive calibration.

Background technology

The mankind are species that a height relies on vision, calculate people's blinkpunkt, and the application that can produce every aspect, comprises psychology, Industrial Engineering, commercial advertisement etc.For example, can utilize visual pursuit technology to carry out:

1) notice analysis.The web page contents that analysis user is most interested, finds interest in infant growth process etc.

2) man-machine interaction.Can replace hand to carry out various operations with human eye, such as carrying out auxiliary, the auxiliary driving of disabled person, controlling intelligent appliance, webpage and PPT page turning etc.Visual pursuit technological penetration, in people's life, will produce huge effect to improving people's quality of life.

Summary of the invention

For existing based on CCD(Charge Coupled Device, charge-coupled image sensor) the Visual Trace Technology pupil of video camera extracts inaccurate, the coordinate setting of people's eye fixation is inaccurate, and the deficiency to aspects such as posture restriction are too much, the present invention utilizes the intensity profile of image to carry out human eye fast and just locates.The present invention proposes a kind of combination BFS(BFS (Breadth First Search), Breadth First Search) the beam pattern extracting method of algorithm, image geometry feature and gray feature, can carry out exact matching with corresponding light source by hot spot; Utilize one dimension edge detection operator and the least square ellipse matching matching that circulates, remove noise, until the fixing approximating method of elliptical center finally obtains accurate pupil center; And a kind of calibration steps of dynamic self-adapting is proposed, effectively raise existing spatial mappings model accuracy.

The invention provides a kind of contactless sight line method for tracing based on adaptive calibration, comprise the following steps:

Step 1: convert the coloured image getting to gray-scale map, image is carried out to denoising;

Step 2: analyze the minimum subregion of gray-scale value in image, usining this gray-scale value is divided into binary image as thresholding by image, utilize all point (xi that are less than threshold value, yi) (i=1 ... N), N is the number that in histogram, gray scale is less than the point of threshold value, at binary image coarse positioning pupil center location (x _pupil, y _pupil), suc as formula (1) and formula (2):

$x_{\mathrm{pupil}}=\frac{1}{N}\underset{n=1}{\overset{N}{\mathrm{\Σ}{x}_n}}---\left(1\right)$

$y_{\mathrm{pupil}}=\frac{1}{N}\underset{n=1}{\overset{N}{\mathrm{\Σ}{y}_n}}---\left(2\right);$

Step 3: according to thick fixed pupil center, the rectangular area centered by it is set to region of interest ROI;

Step 4: utilize maximum variance between clusters to set an initial threshold, then the ROI image after cutting apart is assessed, if search is less than the hot spot satisfying condition from utilize the image current threshold is cut apart, automatically adjust threshold value, again cut apart image, again search for, directly search out the whole hot spots that satisfy condition;

Step 5: utilize centroid method, determine the centre coordinate of five hot spots;

Step 6: based on geometric relationship coupling hot spot and light source;

Step 7: utilize the corneal reflection spot area obtaining in hot spot leaching process, the gray scale of the point set of spot area is replaced to the optimal self-adaptive threshold value in hot spot leaching process, then adopt classical Threshold sementation to carry out binaryzation to image, image is cut apart according to the optimal self-adaptive threshold value in hot spot leaching process, obtained the general area of pupil;

Step 8: the edge point set that extracts pupil;

Step 9: utilize ellipse fitting, obtain the centre coordinate of pupil: first use least square ellipse fitting process, candidate point is carried out to ellipse fitting, remove in candidate marginal from the point of the hypertelorism of elliptical center; Loop matching, until obtain a stable elliptical center position, the elliptical center position that this is stable, is exactly the centre coordinate of pupil;

Step 10: calculate blinkpunkt;

Step 11: calibration blinkpunkt.

As a further improvement on the present invention, in step 1, with median filtering method, remove noise.

As a further improvement on the present invention, in step 2, utilize histogram analysis publish picture picture in the minimum subregion of gray-scale value.

As a further improvement on the present invention, in step 3, the scope of rectangular area is: the rectangular area of 70*70 pixel is to the rectangular area of 90*90 pixel.

As a further improvement on the present invention, in step 4: adopt breadth first method search hot spot.

As a further improvement on the present invention, in step 8, the edge point set that extracts pupil is specific as follows: spot area gray scale is used after the gray-scale value replacement lower than thresholding, calculate the barycenter of pupil region, centered by this barycenter, along all directions, carrying out the scanning of one dimension marginal point, search the point set that meets gray scale sequence [111000] in binary image, is exactly the edge point set of pupil.

As a further improvement on the present invention, in step 9, remove in candidate marginal from the requirement of the point of the hypertelorism of elliptical center and be: surpass other all marginal point 5-10 pixels; Variation is less than 3 pixels and just thinks stable center.

As a further improvement on the present invention, in step 10, calculate blinkpunkt specific as follows:

According to four hot spots on the screen calculating and center coordinate of eye pupil, utilize the cross ratio invariability principle in projective geometry to carry out the mapping of volume coordinate, p represents the center of pupil, and g represents the blinkpunkt of human eye on screen, V1, V2, V3, V4 is respectively the near-infrared light source IR1 on four angles of screen, IR2, IR3, imaginary point corresponding to hot spot that IR4 produces;

U _v, U _v2, U _v3, U _v4respectively imaginary point V1, V2, V3, the mapping point of V4 on the plane of delineation, so these 4 points are also imaginary points, and actual hot spot Gt1 on human eye, Gt2, Gt3, Gt4 corresponding point in image is U _r1, U _r2, U _r3, U _r4; Between imaginary point in the hot spot of catching in image and spatial mappings relation on the plane of delineation, have a fixing proportionate relationship, shown in (7), wherein α is that a value is similar to 2.0 theoretical constant, U _rpand U _rican from the image of catching, calculate U _rpit is the hot spot producing near the near-infrared light source of position of camera optic axis in system; So, when α value is constant 2.0, U _v1, U _v2, U _v3, U _v4just can calculate by through type (7), the relation between imaginary point and hot spot is just established like this,

U _Vi=U _RP+α(U _Ri-U _RP)(i=1,2,3,4) (7)

According to cross ratio invariability principle, IR ₁iR ₂double ratio value equal the double ratio value of V1V2, and the double ratio value of V1V2 equals U _v1u _v2double ratio value, so IR ₁iR ₂double ratio value equal U _v1u _v2double ratio value;

4 imaginary point coordinates supposing the plane of delineation are U _vi=(x _i ^v, y _i ^v) (i=1,2,3,4), take these 4 points to expand as the intersection point in Fig. 4 (a) as basis, the coordinate of supposing these intersection points is U _m=(x _i ^m, y _i ^m) (i=1,2,3,4), U _v1u _v2double ratio value suc as formula shown in (8), U _v2u _v3double ratio value suc as formula shown in (9),

${\mathrm{CR}}_{\mathrm{image}}^x=\frac{(x_1^V{y}_1^M-x_1^M{y}_1^V)(x_2^M{y}_2^V-x_2^V{y}_2^M)}{(x_1^V{y}_2^M-x_2^M{y}_1^V)(x_1^M{y}_2^V-x_2^V{y}_1^M)}---\left(8\right)$

${\mathrm{CR}}_{\mathrm{image}}^y=\frac{(x_2^V{y}_3^M-x_3^M{y}_2^V)(x_4^M{y}_3^V-x_3^V{y}_4^M)}{(x_2^V{y}_4^M-x_4^M{y}_2^V)(x_3^M{y}_3^V-x_3^V{y}_3^M)}---\left(9\right)$

Suppose that the sight line blinkpunkt coordinate of human eye on screen is g (x _g, y _g), the width of screen and be highly respectively w and h has M1 (x in the plane at screen place _g, 0) and M2 (w/2,0), IR ₁iR ₂double ratio value suc as formula shown in (10), the double ratio value of IR1 and IR4, suc as formula shown in (11), is utilized double ratio equation ${\mathrm{CR}}_{\mathrm{image}}^x={\mathrm{CR}}_{\mathrm{screen}}^x$ With ${\mathrm{CR}}_{\mathrm{image}}^y={\mathrm{CR}}_{\mathrm{screen}}^y,$ Solve the blinkpunkt g of human eye,

${\mathrm{CR}}_{\mathrm{screen}}^x=\frac{(w-\frac{w}{2})x_g}{(w-x_g)\frac{w}{2}}=\frac{w_g}{w-x_g}---\left(10\right)$

${\mathrm{CR}}_{\mathrm{screen}}^y=\frac{(h-\frac{h}{2})y_g}{(h-y_g)\frac{h}{2}}=\frac{y_g}{h-y_g}---\left(11\right).$

As a further improvement on the present invention, the condition in step 4 is: the pixel number that gray scale is greater than in the single connected region of threshold value surpasses N, and the value of N depends on eye image size and resolution.For example: in the eye image of 768*576 pixel size, N value is 100.

As a further improvement on the present invention, in step 11, calibration blinkpunkt is specific as follows:

Screen area has been divided into the rectangular area of n formed objects equably, the center of all rectangular areas is exactly calibration point set, the coordinate of these calibration points can determine according to the size of screen and rectangular area, and the value of n depends on size and the desired accuracy rating reaching of screen

The coordinate of calibration point is that set is C, shown in (12), when tested, watches calibration point C attentively _itime, the blinkpunkt coordinate that actual computation goes out is E _i, shown in (13), the point in formula (14) is on the plane of delineation, the hot spot being produced by the near-infrared light source on 4 angles of screen, point in formula (15) is on the plane of delineation, the hot spot being produced by the near-infrared light source near position of camera optic axis, and formula (16) is U _rPcorresponding imaginary point,

$C_i=(x_i^C,y_i^C)(i=\mathrm{1,2,3,4}...n)---\left(12\right)$

$E_i=(x_i^E,y_i^E)(i=\mathrm{1,2,3,4}...n)---\left(13\right)$

$U_{\mathrm{Ri}}=(x_i^R,y_i^R)(i=\mathrm{1,2,3,4}...n)---\left(14\right)$

$U_{\mathrm{RP}}=(x_R^P,y_R^P)---\left(15\right)$

$U_{\mathrm{VP}}=(x_V^P,y_V^P)---\left(16\right)$

In extracting alpha parameter matrix process, each alpha parameter that extracts a screen subregion, between parameter, do not interfere with each other mutually, in initial, alpha parameter is assigned 2.0, at this moment can calculate tested rough blinkpunkt, according to this rough blinkpunkt and the difference between calibration point is counter releases an optimum alpha parameter, according to formula (7), can push type (17), when wushu (17) also enters to (8), (9), unique unknown number is exactly α, so can be by solving an equation

with calculate the alpha parameter that current region is corresponding,

$U_{\mathrm{Vi}}=(x_R^P+\mathrm{\α}(x_i^R-x_R^P),y_R^P+\mathrm{\α}(y_i^R-y_R^P))---\left(17\right)$

Each calibration point C _icorresponding alpha parameter is stored in a matrix, remembers that this matrix is α Matrix, C _ialso be stored in a matrix, remember that this matrix is CMatrix (C ₁₁=C ₁, C ₁₂=C ₂..., C _st=C _n), n=s*t, U _r={ U _r1, U _r2, U _r3, U _r4, U _rP, s and t are the row and column numbers of the subregion that is divided into of screen, U _rbe the real-time hot spot set of obtaining from the plane of delineation, formula (4-19) represents C _ijaccording to current U _ralpha parameter is calculated in set, make s=4, t=4, screen has been divided into the subregion of 16 formed objects, α Matrix (shown in (20)) just can utilize formula (18) to calculate by a known CMatrix (shown in (19))

α _ij=F(C _ij,U _R)(i=1...s,j=1...t) (18)

$\mathrm{cMatrix}=\left[\begin{array}{cccc}{C}_{11}& C_{12}& C_{13}& C_{14}\\ C_{21}& C_{22}& C_{23}& C_{24}\\ C_{31}& C_{32}& C_{33}& C_{34}\\ C_{41}& C_{42}& C_{43}& C_{44}\end{array}\right]---\left(19\right)$

$\mathrm{\αMatrix}=\left[\begin{array}{cccc}{\mathrm{\α}}_{11}& {\mathrm{\α}}_{12}& {\mathrm{\α}}_{13}& {\mathrm{\α}}_{14}\\ {\mathrm{\α}}_{21}& {\mathrm{\α}}_{22}& {\mathrm{\α}}_{23}& {\mathrm{\α}}_{24}\\ {\mathrm{\α}}_{31}& {\mathrm{\α}}_{32}& {\mathrm{\α}}_{33}& {\mathrm{\α}}_{34}\\ {\mathrm{\α}}_{41}& {\mathrm{\α}}_{42}& {\mathrm{\α}}_{43}& {\mathrm{\α}}_{44}\end{array}\right]---\left(20\right)$

All alpha parameters in a tested α Matrix all extract complete after, just can watch experiment attentively, for each local subregion of screen, (center is C _ij), in α Matrix, there is an alpha parameter corresponding with it, when initial, the alpha parameter of α Matrix center is set to initial current alpha parameter, when tested current sight line is watched drop point attentively in certain sub regions, current alpha parameter is just dynamically converted to alpha parameter corresponding to this sub regions in α Matrix, guarantees that current alpha parameter is always adapted to the optimum alpha parameter of current local subregion.

The invention has the beneficial effects as follows:

The present invention proposes the beam pattern extracting method of a kind of combination BFS algorithm, image geometry feature and gray feature, can accurately search hot spot, and hot spot accurately can be mated with corresponding light source; Utilize one dimension boundary operator and least square circulation ellipse fitting method to carry out matching pupil, circulate each time and from marginal point set, filter some away from the point of the current pupil center simulating, until pupil center is stable; Propose a kind of dynamic self-adapting calibration steps simultaneously, effectively raise the precision of existing space mapping model.

Accompanying drawing explanation

Fig. 1 is the corresponding relation structural representation of system source of the present invention and hot spot;

Fig. 2 is pupil ellipse fitting result of the present invention;

Fig. 3 is spatial mappings graph of a relation of the present invention;

Fig. 4 a is the imaginary point on the plane of delineation, and Fig. 4 b is the point on screen;

Fig. 5 is screen calibration point distribution of the present invention (n=4*4) figure.

Embodiment

Below in conjunction with accompanying drawing, the present invention will be further described.

Complicated for existing visual pursuit technical equipment, to posture restriction too much, want head-mount specialized equipment, the deficiency of the aspect such as precision is low and efficiency is slow, on the basis of forefathers' invention, propose a kind of new method that realizes visual pursuit technology, adopt five near infrared light modulations as the light source of corneal reflection.The method does not need to be with any helmet, and can adapt to the proper motion of head, and the sight line positional precision that the method for design is calculated is higher.Utilizing the intensity profile of image to carry out human eye fast just locates.Propose the beam pattern extracting method of a kind of combination BFS algorithm, image geometry feature and gray feature, hot spot can be carried out to exact matching with corresponding light source.Utilization, with edge detection operator and the least square ellipse matching matching that circulates, is removed noise, until the fixing approximating method of elliptical center finally obtains accurate pupil center.Utilize cross ratio invariability principle to carry out coordinate mapping, calculate blinkpunkt coordinate, and by a kind of calibration steps of dynamic self-adapting, effectively raise existing spatial mappings model accuracy.Utilize the Realizing Achievement of above-mentioned technology, realize eye controlled mouse system, show the effect of visual pursuit.

1 feature extraction

1.1 human eye area location

In sight line tracking system, human eye feature extraction accurately in early stage is the later stage to promote the assurance that sight line is estimated accuracy rate, so feature extraction is particularly important.And in a real-time sight line tracking system, in order to ensure the real-time of sight line tracking system, can not the too high method of complexity service time, thereby need design can locate fast the method for human eye area.The present invention, according to the morphological feature of human eye and gray feature, has adopted a kind of location algorithm fast.

(1) medium filtering denoising

First convert the coloured image getting to gray-scale map, due in using eye image, the marking on eyebrow, eyelashes and skin all may disturb processing to produce, noise penalty quality, to feature extraction, bring obstruction, so after catching eye image, first need image to carry out denoising.Through test, the present invention has selected medium filtering to remove noise.

(2) human eye histogram analysis

Then utilize the histogram analysis minimum subregion (being also first significant crest gray-scale value) of gray-scale value in picture of publishing picture, using this gray-scale value, as thresholding, image is divided into binary image.Utilize all point (xi, yi) (i=1 that are less than threshold value ... N), N is the number that in histogram, gray scale is less than the point of threshold value, at binary image coarse positioning pupil center location (x _pupil, y _pupil).Suc as formula (1) and formula (2).

$x_{\mathrm{pupil}}=\frac{1}{N}\underset{n=1}{\overset{N}{\mathrm{\Σ}{x}_n}}---\left(1\right)$

$y_{\mathrm{pupil}}=\frac{1}{N}\underset{n=1}{\overset{N}{\mathrm{\Σ}{y}_n}}---\left(2\right)$

(3) human eye area location

According to thick fixed pupil center, the rectangular area of the 80*80 pixel centered by it is set to region of interest ROI (region-of-interest), the scope of processing with regard to downscaled images like this, only operates area-of-interest, has improved image treatment effeciency.

1.2 feature hot spots extract

Next be to calculate spot center coordinate, the present invention proposes the beam pattern extracting method of a kind of combination BFS algorithm, image geometry feature and gray feature, hot spot can be carried out to exact matching with corresponding light source.

(1) adaptive threshold split plot design

First utilize maximum variance between clusters (Otsu algorithm) to set an initial threshold, then the ROI image after cutting apart is assessed, if search is less than the hot spot satisfying condition from utilize the image current threshold is cut apart, automatically adjust threshold value, again cut apart image, again search for, directly search out the whole hot spots that satisfy condition.

(2) breadth First algorithm search hot spot

In search hot spot, the present invention has adopted breadth First algorithm to search for, because being one, the search procedure of hot spot constantly expands the process that traversal scope is found optimum solution, the feature that is just meeting breadth First ergodic algorithm, so the present invention adopts breadth First algorithm to search for hot spot from image, can search out respectively 5 continuous maximum spot area.

(3) utilize centroid method, determine the centre coordinate of five hot spots

In (2), the hot spot producing in corneal reflection by breadth First algorithm search to 5 light source, we,, by centroid method, obtain respectively the centre coordinate of these five spot area here, are also spot center

(4) based on geometric relationship coupling hot spot and light source

The present invention is directed to the particular problem of invention, observe following feature: in the ROI region of human eye, 4 hot spots that the near-infrared light source on 4 angles of screen produces are that 4 angular distances in distance R OI region are respectively nearest.And from the distance at 4 angles of ROI, be not nearest by the hot spot producing near the near-infrared light source of ccd video camera optical axis.According to this feature, propose a kind of method of determining 5 hot spots according to distance R OI region herein, utilize this feature, can be more fast and extract exactly the corresponding relation (as shown in Figure 1) of correct 5 hot spots and light source.

1.3 pupil center location

After having determined spot center coordinate, next need accurately to locate pupil center.As another key character of the visual pursuit technology based on corneal reflection principle, whether the quality of pupil detection quality accurately plays vital effect for the result of spatial mappings.The present invention is in conjunction with the feature in corneal reflection visual pursuit technology, on the basis of forefathers' invention, adopt a kind of by carry out the detection of pupil edge point in human eye ROI image, and then utilize the marginal point of discounting to carry out pupil matching, finally by the pupil edge simulating, determine the method for pupil center.

In the visual pursuit technology based on corneal reflection, due to the existence of pupil corneal reflection hot spot, directly adopt gray threshold that image is divided into after binary image, in pupil region, have some cavities, and difficulty will be brought to rim detection in these cavities.In fact, in the leaching process of corneal reflection hot spot, known the point set of spot area, so can utilize this point, before extracting pupil, first the gray-scale value of the point of spot area is all replaced to the optimal self-adaptive threshold value in hot spot leaching process, like this, in binaryzation, just can not produce the cavity of corneal reflection spot area.On this basis, the present invention proposes and a kind ofly utilize one dimension edge detection operator and least square circulation pupil approximating method to realize pupil accurately to locate.

(1) image pre-service

Utilize the corneal reflection spot area obtaining in hot spot leaching process, the gray scale of the point set of spot area is replaced to the optimal self-adaptive threshold value in hot spot leaching process.Then adopt classical Threshold sementation to carry out binaryzation to image, image is cut apart according to the optimal self-adaptive threshold value in hot spot leaching process, just can obtain the general area of pupil.Wherein also comprise some possible noises, but can eliminate by medium filtering.

(2) marginal point extracts

In order to adopt the method for ellipse fitting to simulate pupil, just need to from image, obtain the marginal point of pupil.And the edge detection method adopting in visual pursuit technology based on corneal reflection is mainly to carry out based on one dimension edge detection operator.This one dimension edge detection algorithm is mainly for ellipse fitting, and in order to meet the real-time of system, aspect minimizing calculated amount, performance well, also can guarantee the accuracy of rim detection simultaneously.

Spot area gray scale is used after the gray-scale value replacement lower than thresholding, can be calculated the barycenter of pupil region, centered by this barycenter, along all directions, carry out the scanning of one dimension marginal point.Owing to having gray scale sudden change in pupil edge region, so utilize pupil gray threshold that image is divided into after binary image, adopt BFS (Breadth First Search) (the Breadth First Search using in hot spot leaching process, BFS) algorithm, in binary image, searching the point set that meets gray scale sequence [111000], is exactly the edge point set of pupil.

(3) pupil ellipse fitting

First the present invention uses least square ellipse fitting process, and candidate point is carried out to ellipse fitting.Remove in candidate marginal from the point (surpassing other all marginal point 5-10 pixels) of the hypertelorism of elliptical center.Loop matching, until obtain a stable elliptical center position (situation that absolute stability is constant is unlikely, and in most cases variation is less than 3 pixels and just thinks stable center).The elliptical center position that this is stable is exactly the centre coordinate of pupil.Fig. 2 is the result that algorithm that the present invention adopts is obtained.

2 spatial mappings

2.1 blinkpunkts calculate

The present invention, according to four hot spots on the screen calculating and center coordinate of eye pupil, utilizes the cross ratio invariability principle in projective geometry to carry out the mapping of volume coordinate.Computation model of the present invention as shown in Figure 3.P represents the center of pupil, and g represents the blinkpunkt of human eye on screen.Supposing has the cornea tangent plane shown in figure on eye cornea surface.V1, V2, V3, V4 is respectively the near-infrared light source IR1 on four angles of screen, IR2, IR3, imaginary point corresponding to hot spot that IR4 produces.Introducing the object of these imaginary points, is for spatial mappings relation and double ratio principle being combined.The hot spot that these imaginary points are not, the just mapping point that meets in the ideal case certain condition of hot spot.

In Fig. 3, U _v, U _v2, U _v3, U _v4respectively imaginary point V1, V2, V3, the mapping point of V4 on the plane of delineation, so these 4 points are also imaginary points, and actual hot spot Gt1 on human eye, Gt2, Gt3, Gt4 corresponding point in image is U _r1, U _r2, U _r3, U _r4.Between imaginary point in the hot spot of catching in image (real point) and spatial mappings relation on the plane of delineation, have a fixing proportionate relationship, shown in (7), wherein α is that a value is similar to 2.0 theoretical constant, U _rpand U _rican from the image of catching, calculate (U _rpthe hot spot producing near the near-infrared light source of position of camera optic axis in system).So, when α value is constant 2.0, U _v1, U _v2, U _v3, U _v4just can calculate by through type (7).Relation between imaginary point and hot spot has just been established like this.

U _Vi=U _RP+α(U _Ri-U _RP)(i=1,2,3,4) (7)

After relation between known imaginary point and hot spot, just can directly by imaginary point (indirectly passing through hot spot), utilize double ratio principle to carry out the spatial mappings in Fig. 3.Fig. 4 a and Fig. 4 b show while utilizing double ratio principle to carry out spatial mappings, the relation between point used.In Fig. 3, according to cross ratio invariability principle, IR ₁iR ₂double ratio value equal the double ratio value of V1V2, and the double ratio value of V1V2 equals U _v1u _v2double ratio value, so IR ₁iR ₂double ratio value equal U _v1u _v2double ratio value.Utilize double ratio principle, so just the plane of delineation and screen plane have been set up and contacted directly by cornea tangent plane, subsequent calculations has not just needed to consider cornea tangent plane, only need in the plane of delineation and screen plane, carry out analytical calculation.

Point coordinate involved in the image of catching by video camera all can calculate, so IR ₁iR ₂on double ratio value just can calculate.4 imaginary point coordinates supposing the plane of delineation are U _vi=(x _i ^v, y _i ^v) (i=1,2,3,4), take these 4 points to expand as the intersection point in Fig. 4 a as basis, the coordinate of supposing these intersection points is U _m=(x _i ^m, y _i ^m) (i=1,2,3,4).U _v1u _v2double ratio value suc as formula shown in (8).U _v2u _v3double ratio value suc as formula shown in (9).

${\mathrm{CR}}_{\mathrm{image}}^x=\frac{(x_1^V{y}_1^M-x_1^M{y}_1^V)(x_2^M{y}_2^V-x_2^V{y}_2^M)}{(x_1^V{y}_2^M-x_2^M{y}_1^V)(x_1^M{y}_2^V-x_2^V{y}_1^M)}---\left(8\right)$

${\mathrm{CR}}_{\mathrm{image}}^y=\frac{(x_2^V{y}_3^M-x_3^M{y}_2^V)(x_4^M{y}_3^V-x_3^V{y}_4^M)}{(x_2^V{y}_4^M-x_4^M{y}_2^V)(x_3^M{y}_3^V-x_3^V{y}_3^M)}---\left(9\right)$

Suppose that the sight line blinkpunkt coordinate of human eye on screen is g (x _g, y _g), the width of screen and be highly respectively w and h.Similarly, as shown in Figure 4 b, in the plane at screen place, there is M1 (x _g, 0) and M2 (w/2,0), IR ₁iR ₂double ratio value suc as formula shown in (10), the double ratio value of IR1 and IR4 is suc as formula shown in (11).Utilize double ratio equation

with just can solve the blinkpunkt g of human eye.

${\mathrm{CR}}_{\mathrm{screen}}^x=\frac{(w-\frac{w}{2})x_g}{(w-x_g)\frac{w}{2}}=\frac{w_g}{w-x_g}---\left(10\right)$

${\mathrm{CR}}_{\mathrm{screen}}^y=\frac{(h-\frac{h}{2})y_g}{(h-y_g)\frac{h}{2}}=\frac{y_g}{h-y_g}---\left(11\right)$

2.2 blinkpunkt calibrations

The dynamic self-adapting calibration steps that the present invention proposes adapts to different screen areas to each user with an alpha parameter matrix, the alpha parameter of current use be from matrix Real-time Obtaining to optimal parameter.For the target blinkpunkt (calibration point) that verification is used is relatively evenly distributed on screen, screen area has been divided into the rectangular area of n formed objects equably.The center of all rectangular areas is exactly calibration point set, and the coordinate of these calibration points can be determined according to the size of screen and rectangular area.The value of n depends on size and the desired accuracy rating reaching of screen.In certain rectangle size, the value of n is larger, and the precision finally obtaining is higher.But more calibration point means more numerous and diverse calibration process, and calibration process is more numerous and diverse, calibrate the needed time just longer.If n=4*4, the distribution situation of calibration point on screen as shown in Figure 5.

The coordinate of calibration point is that set is C, shown in (12).When tested, watch calibration point C attentively _itime, the blinkpunkt coordinate that actual computation goes out is E _i, shown in (13).Point in formula (14) is on the plane of delineation, the hot spot being produced by the near-infrared light source on 4 angles of screen.Point in formula (15) is on the plane of delineation, the hot spot being produced by the near-infrared light source near position of camera optic axis.Formula (16) is U _rPcorresponding imaginary point.

$C_i=(x_i^C,y_i^C)(i=\mathrm{1,2,3,4}...n)---\left(12\right)$

$E_i=(x_i^E,y_i^E)(i=\mathrm{1,2,3,4}...n)---\left(13\right)$

$U_{\mathrm{Ri}}=(x_i^R,y_i^R)(i=\mathrm{1,2,3,4}...n)---\left(14\right)$

$U_{\mathrm{RP}}=(x_R^P,y_R^P)---\left(15\right)$

$U_{\mathrm{VP}}=(x_V^P,y_V^P)---\left(16\right)$

In extracting alpha parameter matrix process, extract the alpha parameter of a screen subregion at every turn, between parameter, do not interfere with each other mutually.In initial, alpha parameter is assigned 2.0, at this moment can calculate tested rough blinkpunkt, according to the difference between this rough blinkpunkt and calibration point (central point of subregion) is counter, releases an optimum alpha parameter.According to formula (7), can push type (17), when wushu (17) also enters to (8), (9), unique unknown number is exactly α, so can be by solving an equation

with

calculate the alpha parameter that current region is corresponding.

$U_{\mathrm{Vi}}=(x_R^P+\mathrm{\α}(x_i^R-x_R^P),y_R^P+\mathrm{\α}(y_i^R-y_R^P))---\left(17\right)$

Each calibration point C _icorresponding alpha parameter is stored in a matrix, remembers that this matrix is α Matrix, C _ialso be stored in a matrix, remember that this matrix is CMatrix (C ₁₁=C ₁, C ₁₂=C ₂..., C _st=C _n), n=s*t, U _r={ U _r1, U _r2, U _r3, U _r4, U _rP.S and t are the row and column numbers of the subregion that is divided into of screen, U _rbe the real-time hot spot set of obtaining from the plane of delineation, formula (4-19) represents C _ijaccording to current U _ralpha parameter is calculated in set.Make s=4, t=4, screen has been divided into the subregion of 16 formed objects.α Matrix (shown in (20)) just can utilize formula (18) to calculate by a known CMatrix (shown in (19)).

α _ij=F(C _ij,U _R)(i=1...s,j=1...t) (18)

$\mathrm{cMatrix}=\left[\begin{array}{cccc}{C}_{11}& C_{12}& C_{13}& C_{14}\\ C_{21}& C_{22}& C_{23}& C_{24}\\ C_{31}& C_{32}& C_{33}& C_{34}\\ C_{41}& C_{42}& C_{43}& C_{44}\end{array}\right]---\left(19\right)$

$\mathrm{\αMatrix}=\left[\begin{array}{cccc}{\mathrm{\α}}_{11}& {\mathrm{\α}}_{12}& {\mathrm{\α}}_{13}& {\mathrm{\α}}_{14}\\ {\mathrm{\α}}_{21}& {\mathrm{\α}}_{22}& {\mathrm{\α}}_{23}& {\mathrm{\α}}_{24}\\ {\mathrm{\α}}_{31}& {\mathrm{\α}}_{32}& {\mathrm{\α}}_{33}& {\mathrm{\α}}_{34}\\ {\mathrm{\α}}_{41}& {\mathrm{\α}}_{42}& {\mathrm{\α}}_{43}& {\mathrm{\α}}_{44}\end{array}\right]---\left(20\right)$

All alpha parameters in a tested α Matrix all extract complete after, just can watch attentively and test.For each local subregion of screen, (center is C _ij), in α Matrix, there is an alpha parameter corresponding with it.When initial, the alpha parameter of α Matrix center is set to initial current alpha parameter.When tested current sight line is watched drop point attentively in certain sub regions, current alpha parameter is just dynamically converted to alpha parameter corresponding to this sub regions in α Matrix, guarantees that current alpha parameter is always adapted to the optimum alpha parameter of current local subregion.By such strategy, can solve the problem of overall offset, the method for precision Ye Bi global optimum is high simultaneously.

Above content is in conjunction with concrete preferred implementation further description made for the present invention, can not assert that specific embodiment of the invention is confined to these explanations.For general technical staff of the technical field of the invention, without departing from the inventive concept of the premise, can also make some simple deduction or replace, all should be considered as belonging to protection scope of the present invention.

Claims (10)

1. the contactless sight line method for tracing based on adaptive calibration, is characterized in that, comprises the following steps:

Step 1: convert the coloured image getting to gray-scale map, image is carried out to denoising;

Step 2: analyze the minimum subregion of gray-scale value in image, usining this gray-scale value is divided into binary image as thresholding by image, utilize all point (xi that are less than threshold value, yi) (i=1 ... N), N is the number that in histogram, gray scale is less than the point of threshold value, at binary image coarse positioning pupil center location (x _pupil, y _pupil), suc as formula (1) and formula (2):

$x_{\mathrm{pupil}}=\frac{1}{N}\underset{n=1}{\overset{N}{\mathrm{\Σ}{x}_n}}---\left(1\right)$

$y_{\mathrm{pupil}}=\frac{1}{N}\underset{n=1}{\overset{N}{\mathrm{\Σ}{y}_n}}---\left(2\right);$

Step 3: according to thick fixed pupil center, the rectangular area centered by it is set to region of interest ROI;

Step 4: utilize maximum variance between clusters to set an initial threshold, then the ROI image after cutting apart is assessed, if search is less than the hot spot satisfying condition from utilize the image current threshold is cut apart, automatically adjust threshold value, again cut apart image, again search for, directly search out the whole hot spots that satisfy condition;

Step 5: utilize centroid method, determine the centre coordinate of five hot spots;

Step 6: based on geometric relationship coupling hot spot and light source;

Step 7: utilize the corneal reflection spot area obtaining in hot spot leaching process, the gray scale of the point set of spot area is replaced to the optimal self-adaptive threshold value in hot spot leaching process, then adopt classical Threshold sementation to carry out binaryzation to image, image is cut apart according to the optimal self-adaptive threshold value in hot spot leaching process, obtained the general area of pupil;

Step 8: the edge point set that extracts pupil;

Step 9: utilize ellipse fitting, obtain the centre coordinate of pupil: first use least square ellipse fitting process, candidate point is carried out to ellipse fitting, remove in candidate marginal from the point of the hypertelorism of elliptical center; Loop matching, until obtain a stable elliptical center position, the elliptical center position that this is stable, is exactly the centre coordinate of pupil;

Step 10: calculate blinkpunkt;

Step 11: calibration blinkpunkt.

2. a kind of contactless sight line method for tracing based on adaptive calibration according to claim 1, is characterized in that: in step 1, with median filtering method, remove noise.

3. a kind of contactless sight line method for tracing based on adaptive calibration according to claim 1, is characterized in that: in step 2, utilize the histogram analysis minimum subregion of gray-scale value in picture of publishing picture.

4. a kind of contactless sight line method for tracing based on adaptive calibration according to claim 1, is characterized in that: in step 3, the scope of rectangular area is: the rectangular area of 70*70 pixel is to the rectangular area of 90*90 pixel.

5. a kind of contactless sight line method for tracing based on adaptive calibration according to claim 1, is characterized in that: in step 4: adopt breadth first method search hot spot.

6. a kind of contactless sight line method for tracing based on adaptive calibration according to claim 1, it is characterized in that: in step 8, the edge point set that extracts pupil is specific as follows: spot area gray scale is used after the gray-scale value replacement lower than thresholding, calculate the barycenter of pupil region, centered by this barycenter, along all directions, carrying out the scanning of one dimension marginal point, search the point set that meets gray scale sequence [111000] in binary image, is exactly the edge point set of pupil.

7. a kind of contactless sight line method for tracing based on adaptive calibration according to claim 1, it is characterized in that: in step 9, remove in candidate marginal from the requirement of the point of the hypertelorism of elliptical center and be: surpass other all marginal point 5-10 pixels; Variation is less than 3 pixels and just thinks stable center.

8. a kind of contactless sight line method for tracing based on adaptive calibration according to claim 1, is characterized in that: in step 10, calculate blinkpunkt specific as follows:

According to four hot spots on the screen calculating and center coordinate of eye pupil, utilize the cross ratio invariability principle in projective geometry to carry out the mapping of volume coordinate, p represents the center of pupil, and g represents the blinkpunkt of human eye on screen, V1, V2, V3, V4 is respectively the near-infrared light source IR1 on four angles of screen, IR2, IR3, imaginary point corresponding to hot spot that IR4 produces;

U _v, U _v2, U _v3, U _v4respectively imaginary point V1, V2, V3, the mapping point of V4 on the plane of delineation, so these 4 points are also imaginary points, and actual hot spot Gt1 on human eye, Gt2, Gt3, Gt4 corresponding point in image is U _r1, U _r2, U _r3, U _r4; Between imaginary point in the hot spot of catching in image and spatial mappings relation on the plane of delineation, have a fixing proportionate relationship, shown in (7), wherein α is that a value is similar to 2.0 theoretical constant, U _rpand U _rican from the image of catching, calculate U _rpit is the hot spot producing near the near-infrared light source of position of camera optic axis in system; So, when α value is constant 2.0, U _v1, U _v2, U _v3, U _v4just can calculate by through type (7), the relation between imaginary point and hot spot is just established like this,

U _Vi=U _RP+α(U _Ri-U _RP)(i=1,2,3,4) (7)

According to cross ratio invariability principle, IR ₁iR ₂double ratio value equal the double ratio value of V1V2, and the double ratio value of V1V2 equals U _v1u _v2double ratio value, so IR ₁iR ₂double ratio value equal U _v1u _v2double ratio value;

4 imaginary point coordinates supposing the plane of delineation are U _vi=(x _i ^v, y _i ^v) (i=1,2,3,4), take these 4 points to expand as the intersection point in Fig. 4 (a) as basis, the coordinate of supposing these intersection points is U _m=(x _i ^m, y _i ^m) (i=1,2,3,4), U _v1u _v2double ratio value suc as formula shown in (8), U _v2u _v3double ratio value suc as formula shown in (9),

${\mathrm{CR}}_{\mathrm{image}}^x=\frac{(x_1^V{y}_1^M-x_1^M{y}_1^V)(x_2^M{y}_2^V-x_2^V{y}_2^M)}{(x_1^V{y}_2^M-x_2^M{y}_1^V)(x_1^M{y}_2^V-x_2^V{y}_1^M)}---\left(8\right)$

${\mathrm{CR}}_{\mathrm{image}}^y=\frac{(x_2^V{y}_3^M-x_3^M{y}_2^V)(x_4^M{y}_3^V-x_3^V{y}_4^M)}{(x_2^V{y}_4^M-x_4^M{y}_2^V)(x_3^M{y}_3^V-x_3^V{y}_3^M)}---\left(9\right)$

Suppose that the sight line blinkpunkt coordinate of human eye on screen is g (x _g, y _g), the width of screen and be highly respectively w and h has M1 (x in the plane at screen place _g, 0) and M2 (w/2,0), IR ₁iR ₂double ratio value suc as formula shown in (10), the double ratio value of IR1 and IR4, suc as formula shown in (11), is utilized double ratio equation ${\mathrm{CR}}_{\mathrm{image}}^x={\mathrm{CR}}_{\mathrm{screen}}^x$ With ${\mathrm{CR}}_{\mathrm{image}}^y={\mathrm{CR}}_{\mathrm{screen}}^y,$ Solve the blinkpunkt g of human eye,

${\mathrm{CR}}_{\mathrm{screen}}^x=\frac{(w-\frac{w}{2})x_g}{(w-x_g)\frac{w}{2}}=\frac{w_g}{w-x_g}---\left(10\right)$

${\mathrm{CR}}_{\mathrm{screen}}^y=\frac{(h-\frac{h}{2})y_g}{(h-y_g)\frac{h}{2}}=\frac{y_g}{h-y_g}---\left(11\right).$

9. a kind of contactless sight line method for tracing based on adaptive calibration according to claim 1, it is characterized in that: the condition in step 4 is: the pixel number that gray scale is greater than in the single connected region of threshold value surpasses N, and the value of N depends on eye image size and resolution.

10. a kind of contactless sight line method for tracing based on adaptive calibration according to claim 1, is characterized in that: in step 11, calibration blinkpunkt is specific as follows:

Screen area has been divided into the rectangular area of n formed objects equably, the center of all rectangular areas is exactly calibration point set, the coordinate of these calibration points can determine according to the size of screen and rectangular area, and the value of n depends on size and the desired accuracy rating reaching of screen

The coordinate of calibration point is that set is C, shown in (12), when tested, watches calibration point C attentively _itime, the blinkpunkt coordinate that actual computation goes out is E _i, shown in (13), the point in formula (14) is on the plane of delineation, the hot spot being produced by the near-infrared light source on 4 angles of screen, point in formula (15) is on the plane of delineation, the hot spot being produced by the near-infrared light source near position of camera optic axis, and formula (16) is U _rPcorresponding imaginary point,

$C_i=(x_i^C,y_i^C)(i=\mathrm{1,2,3,4}...n)---\left(12\right)$

$E_i=(x_i^E,y_i^E)(i=\mathrm{1,2,3,4}...n)---\left(13\right)$

$U_{\mathrm{Ri}}=(x_i^R,y_i^R)(i=\mathrm{1,2,3,4}...n)---\left(14\right)$

$U_{\mathrm{RP}}=(x_R^P,y_R^P)---\left(15\right)$

$U_{\mathrm{VP}}=(x_V^P,y_V^P)---\left(16\right)$

In extracting alpha parameter matrix process, each alpha parameter that extracts a screen subregion, between parameter, do not interfere with each other mutually, in initial, alpha parameter is assigned 2.0, at this moment can calculate tested rough blinkpunkt, according to this rough blinkpunkt and the difference between calibration point is counter releases an optimum alpha parameter, according to formula (7), can push type (17), when wushu (17) also enters to (8), (9), unique unknown number is exactly α, so can be by solving an equation

with calculate the alpha parameter that current region is corresponding,

$U_{\mathrm{Vi}}=(x_R^P+\mathrm{\α}(x_i^R-x_R^P),y_R^P+\mathrm{\α}(y_i^R-y_R^P))---\left(17\right)$

Each calibration point C _icorresponding alpha parameter is stored in a matrix, remembers that this matrix is α Matrix, C _ialso be stored in a matrix, remember that this matrix is CMatrix (C ₁₁=C ₁, C ₁₂=C ₂..., C _st=C _n), n=s*t, U _r={ U _r1, U _r2, U _r3, U _r4, U _rP, s and t are the row and column numbers of the subregion that is divided into of screen, U _rbe the real-time hot spot set of obtaining from the plane of delineation, formula (4-19) represents C _ijaccording to current U _ralpha parameter is calculated in set, make s=4, t=4, screen has been divided into the subregion of 16 formed objects, α Matrix (shown in (20)) just can utilize formula (18) to calculate by a known CMatrix (shown in (19))

α _ij=F(C _ij,U _R)(i=1...s,j=1...t) (18)

$\mathrm{cMatrix}=\left[\begin{array}{cccc}{C}_{11}& C_{12}& C_{13}& C_{14}\\ C_{21}& C_{22}& C_{23}& C_{24}\\ C_{31}& C_{32}& C_{33}& C_{34}\\ C_{41}& C_{42}& C_{43}& C_{44}\end{array}\right]---\left(19\right)$

$\mathrm{\αMatrix}=\left[\begin{array}{cccc}{\mathrm{\α}}_{11}& {\mathrm{\α}}_{12}& {\mathrm{\α}}_{13}& {\mathrm{\α}}_{14}\\ {\mathrm{\α}}_{21}& {\mathrm{\α}}_{22}& {\mathrm{\α}}_{23}& {\mathrm{\α}}_{24}\\ {\mathrm{\α}}_{31}& {\mathrm{\α}}_{32}& {\mathrm{\α}}_{33}& {\mathrm{\α}}_{34}\\ {\mathrm{\α}}_{41}& {\mathrm{\α}}_{42}& {\mathrm{\α}}_{43}& {\mathrm{\α}}_{44}\end{array}\right]---\left(20\right)$

All alpha parameters in a tested α Matrix all extract complete after, just can watch experiment attentively, for each local subregion of screen, (center is C _ij), in α Matrix, there is an alpha parameter corresponding with it, when initial, the alpha parameter of α Matrix center is set to initial current alpha parameter, when tested current sight line is watched drop point attentively in certain sub regions, current alpha parameter is just dynamically converted to alpha parameter corresponding to this sub regions in α Matrix, guarantees that current alpha parameter is always adapted to the optimum alpha parameter of current local subregion.

Images