RU2815470C1 - Method for determining direction of view - Google Patents

Abstract

FIELD: computer technology.

SUBSTANCE: invention relates to devices and methods for determining the direction of view and can be used in various fields of technology, including robotics. The method for determining the direction of the user's view includes obtaining an image of the left eye and an image of the right eye containing glare created respectively by light sources of the left eye and light sources of the right eye, determining the position of the pupil of the eye, determining the position and numbering of the glare on the cornea of the eye, determining the optical axis of the eye and determining view directions based on the specific optical axis of the eye and information about the surrounding scene.

EFFECT: improved accuracy in determining the direction of view, high data processing speed, low hardware requirements, and reduced weight of the wearable device.

11 cl, 5 dwg

Description

The present invention relates to devices and methods for determining the direction of gaze and can be used in various fields of technology, including robotics.

RF patent RU 2696042 describes an invention related to technologies used to determine areas of gaze fixations during eye movements, and can be used for an objective assessment of the processes of visual attention, control of computer interfaces by means of gaze direction, in camera work, marketing, etc. According to the claimed method determining areas of gaze fixation during eye movement, the respondent with a fixed head position is presented with at least one stimulator image during the presentation of the stimulus image. In this case, the eye area is covered with infrared radiation, the reflected signal is recorded, and the resulting video sequence is processed to determine the coordinates of the center of the pupil in each video frame. Next, the coordinates are converted into the coordinate system of the stimulus image, forming a two-dimensional array of coordinate values of the center of the pupil in a chronological sequence corresponding to the frames of the video sequence, and then segment analysis is used to select at least three points considered by the area of gaze fixation.

The disadvantage of this technical solution is that gaze fixation during eye movement is carried out with a fixed position of the respondent’s head relative to the presented image. In addition, the accuracy of determining the direction of gaze is not high enough, which is especially important when it is necessary to determine the direction to a specific object located among many other objects.

US Patent US 10234940 describes an eye tracking method comprising the following steps: recording video images of a person's eye such that the pupil of the eye and the glare on the eyeball caused by a light source are recorded; processing the video images to calculate an offset between a position of a predetermined spatial element and a predetermined position relative to the highlight; and, through a light source such as a display, emitting light from the light pattern at a location selected from a plurality of preconfigured light pattern locations to the human eye. The location is controlled by a feedback signal; controlling the location of the light pattern among the predetermined locations of the light patterns in response to the displacement, such that the predetermined position relative to the glare caused by the light source tracks the spatial feature of human eyes. In this case, the above steps are repeated to establish a control loop with the location of the light pattern controlled by the feedback signal. The gaze tracker is configured to filter video images to identify one or more highlights that may arise from the light pattern, wherein a predetermined position relative to the highlight is calculated relative to the identified one or more points.

This device allows you to increase the accuracy of recognizing the gaze position and forming the resulting image, but it places increased demands on the hardware.

US Pat. No. 9,830,512 describes an eye tracking method that includes the following steps: determining the position of a central point of the cornea using at least two light reflection points detected in the eyeball region of a first image of the user's face; calculating a first vector relative to at least two fixed feature points detected from the first face image and the position of the central point of the cornea; calculating a position of a central point of the cornea relative to the eyeball region of the second face image using the position of a feature point detected from the second face image and a first vector when at least two light reflection points are not detected from the eyeball region of the second face image of the user; calculating a second vector using the calculated position of the center point of the cornea and the position of the center point of the pupil detected from the eyeball region of the second face image; and tracking the user's gaze using the second vector.

This known method and system for eye tracking can improve the accuracy of recognizing the position of the gaze and forming the resulting image, but are quite complex and demanding on computing resources.

As the closest analogue of the present invention, a gaze tracking method based on adaptive tomographic mapping according to US patent US 9684827 can be selected. The tracking method includes the steps of estimating gaze, in which an adaptive transformation of the tomographic mapping is used to correct the displacement, while the adaptive transformation of the tomographic the mapping is trained by minimizing an objective function based on data corresponding to multiple head positions and gaze directions to compensate for spatially varying gaze errors or head pose-dependent errors relative to the calibration position; capturing current glare data and pupil-related data in an image using a plurality of light sources and a camera; provide the current glare data and pupil-related data processed from the image as features to obtain head pose-dependent data based on the recognized adaptive tomographic mapping transformation, which are used to determine the current gaze information.

The well-known method of eye tracking based on adaptive tomographic matching also makes it possible to increase the accuracy of recognition of the gaze position and the formation of the resulting image, but is quite complex and demanding of computing resources, and in addition, requires additional actions from the user.

Thus, there is a task to create a method for determining the direction of gaze, in which a one-time calibration for one user is sufficient, which simplifies the repeated use of the system, which increases the accuracy of determining the direction of gaze even with small changes in the position of the system on the user.

The technical result of the claimed invention is an increase in the accuracy of determining the direction of gaze, high data processing speed, which means low hardware requirements, which leads to a reduction in the weight of the wearable device, and the ability to use high-performance high-resolution cameras to further improve the accuracy of determining the direction of gaze.

The problem is solved, and the technical result is achieved in a method for determining the direction of the user's gaze, which includes the stages of obtaining an image of the left eye and an image of the right eye containing glare from light sources of the left eye and right eye, determining the position of the pupil of the eye, determining the position and numbering of glare on the cornea eyes, determining the optical axis of the direction of gaze of each eye and determining the direction of gaze.

When determining the position of the pupil of each eye in the image of each eye, a preliminary search for the pupil is performed, a preliminary ellipse of the pupil is constructed, and a pupil ellipse is constructed using its nodal points.

When determining the position and numbering of highlights on the cornea of each eye, a search for highlights is performed in the eye image, the size of the iris is calculated, highlights located outside the iris are excluded, and highlights are numbered to determine the gaze direction vector.

When determining the optical axis of the gaze direction of each eye, the nodal point of the eye, the refractive point for the center of the pupil, and the position of the center of the pupil in the coordinate system of the scene camera are determined.

Determination of the gaze direction is performed based on the determined optical axis of the gaze direction and the calibration of the gaze direction.

In the preferred embodiment of the claimed method, for preliminary search of the pupil, the preliminary position of the center of the pupil is determined, as well as the number of pixels in the area of the pupil, which preliminarily characterizes its size. When constructing a preliminary pupil ellipse, preferably, a binarization threshold is found in the preliminary pupil region and binarization is performed to determine the pupil boundary for constructing a preliminary pupil ellipse. To construct a pupil ellipse from the nodal points of the boundary of the preliminary pupil ellipse, you can use the least squares method. In this case, it is preferable to filter out the nodal points so that they form a convex figure.

The search for glare on the cornea of the eye can be performed, for example, by threshold processing of the eye image with cluster selection and filtering of clusters by brightness, size and deviation from roundness parameter.

When calculating the iris size, information about the average size of the human iris and information about the distance from the corresponding left eye camera or right eye camera to the pupil are preferably used, which improves the accuracy of determining the direction of gaze.

Numbering of highlights can be done from one highlight from the top pair, closest to the bridge of the nose, in a circle, away from the bridge of the nose, i.e. clockwise for the right eye and counterclockwise for the left eye.

When determining the optical axis of the gaze direction of each eye, the nodal point of the eye, the refractive point for the center of the pupil, and the position of the center of the pupil in the coordinate system of the scene camera are determined.

The claimed method for determining gaze direction may also include a gaze direction calibration step, which is performed either at one of these steps or in advance. In this case, it is enough to perform calibration once for a specific user, and repeated calibration will no longer be required. In particular, when calibrating the gaze direction, the individual characteristics of the user and the relative positions of the left and right eye cameras and the scene camera are taken into account.

Next, the invention, as well as some possible embodiments of its implementation, are explained in detail with reference to the figures, which show:

Figure 1 shows a general view of a device variant for implementing the claimed method;

Figure 2 schematically shows a front view of a variant of the device for implementing the claimed method;

Fig. 3 schematically shows a rear view of a variant of the device for implementing the claimed method;

Fig. 4 schematically shows the relative position of the left or right eye, the camera of the left or right eye and one light source of the left or right eye, as well as a diagram of the path of the rays;

Fig. 5a shows the stage of finding the center of the pupil;

Fig. 5b shows the step of approximating the position of the pupil with an ellipse;

Fig. 5c shows the step of determining the position of highlights;

Fig. 5d shows an example of highlight numbering;

Fig. 5e shows the step of determining the direction of the optical axis of the eye.

In the figures, reference numerals are indicated by:

1 - device for determining the direction of view;

2 - body;

3.1 - left rim;

3.2 - right rim; 4 - nose;

5.1, 5.2 - side clamping strips;

6.1 - camera of the left eye;

6.2 - camera of the right eye;

7 - stage camera;

8 - light source;

8i - i-th light source 8;

9 - computing module;

10 - control module;

11 - eye;

12 - iris of the eye;

C - nodal point of the eye;

P - center of the pupil;

R - refraction point;

Qi is the reflection point of the i-th light source 8;

O - nodal point of the camera;

V - image of the center of the pupil;

Ui is the image of the highlight of the i-th light source 8.

The claimed method for determining the direction of gaze can be implemented using a device, the preferred embodiment of which is described in detail below.

The device 1 for determining the direction of the user's gaze contains a body 2, which is essentially made in the form of glasses having a left rim 3.1 and a right rim 3.2, a nosepiece 4 and two side clamping bars 5.1, 5.2.

The body 2 is preferably made of durable and lightweight material suitable for permanent or long-term wearing of the device 1, taking into account the anthropometric data of the user's head. For greater user convenience and to ensure the possibility of adjusting the position of the device 1 for the purpose of proper operation of all systems, replaceable nosepieces 4 can be used, which adjust the position of the body 2 in height and are selected under the bridge of the nose to prevent it from being pinched. The correct selection of the distance from the eye to the rims 3.1, 3.2 contributes to the correct operation of the eye image recognition system.

In the lower parts of the rims 3.1, 3.2, preferably in the areas of the light openings in planes passing through the vertical axes of the light openings, a left eye camera 6.1 and a right eye camera 6.2 are installed, respectively, hereinafter for brevity also called eye cameras 6.1, 6.2.

Eye cameras 6.1, 6.2 are designed to register and record a video sequence depicting the process of eye movement and obtain images of the corresponding eye. At the same time, the movement of both the user’s left and right eyes is recorded, which is necessary to accurately determine the direction of gaze and eliminate false positives. The eye cameras 6.1,6.2 contain photosensitive matrices and corresponding optical systems (lenses) for forming an enlarged, focused image of the eyes on the photosensitive matrices.

To correctly determine the direction of gaze, images of the pupil and glare are registered (described below) and the subsequent determination of the coordinates of the center of the pupil and each of the glare in the coordinate system of cameras 6.1, 6.2 of the eye (local coordinates).

To improve the ergonomic characteristics of the product, the photosensitive matrix of the eye cameras 6.1, 6.2 is located in a plane at an angle to the main planes of the optical system of the eye 11, as shown schematically in Fig.4. This arrangement will not create obstacles and block the user’s view.

To determine the direction of gaze at any position of the pupil, one eye camera 6.1, 6.2 is used to record the image of each eye 11 from one angle - from below (Fig. 4). The chosen angle is determined by the fact that the lower eyelashes are significantly shorter than the upper ones and do not create obstacles for image registration. The angle between the optical axis of the lens and the viewing direction axis is selected based on a compromise between the best image recording angle (corresponding to an angle of 0°) and the angle at which the camera is guaranteed not to fall into the user's field of view (corresponding to an angle of 90°). Most preferably, said angle is between 45° and 60°.

It is preferable to use a color photosensitive matrix as the photosensitive matrix. This is due to the fact that displaying the eye in a color palette allows the use of a procedure for binarizing a color image according to the levels of each color component, which, in turn, makes it possible to reliably identify detected objects due to their achromatic colors (pupil - sharply black, highlights - bright white) on the stage of eye image processing. This approach simplifies the algorithm for recognizing detected objects by eliminating additional particle filtering procedures, which lead to the loss of important image details.

The optical system of cameras 6.1, 6.2 eyes (not shown in the figures) is designed to form an image in the plane of the photosensitive matrix.

The optical system contains several lenses and a light filter. The distance from the exit pupil of the lens to the eye changes slightly and can be considered fixed. To form an undistorted image in the plane of the matrix, the optical design of a compact four-component lens with an output aspherical lens to compensate for distortion can be used.

On the nosepiece 4 there is a scene camera 7 (Fig. 2), which is used to obtain an image of the surrounding scene and ensure fixation of the environment, to which the viewing direction vector is subsequently linked. The scene camera 7 is preferably located on the upper part of the glasses body 2.

On each of the rims 3.1, 3.2 around the light openings, light sources 8 are installed, forming highlights on the corresponding eye (Fig. 3). Light sources 8 are designed to create the minimum required level of illumination and form point highlights due to reflections from the cornea of the eye.

An IR radiation source, in particular an IR LED, can be used as a light source 8.

Off-axis IR illumination of the eye creates the effect of a dark pupil and forms images of light sources 8 due to the reflection of radiation from the cornea of the eye. The images of light sources 8 formed by reflection from the cornea are called first Purkinje images, or flares. The dark pupil and glare are then visualized by the optical system of cameras 6.1, 6.2 of the eye and captured by light-sensitive matrices that have sufficient sensitivity in the near-IR region of the spectrum. The pupil and glare images move in proportion to the rotation of the eyeball, but along different trajectories. The difference vector between these two features is used to determine the gaze direction vector.

The central wavelength of the light sources 8 is selected preferably from the wavelength range from 860 to 990 nm, which in the best embodiment of the claimed device 1 corresponds to the operating range of the eye cameras 6.1, 6.2 (near-infrared range). The choice of near-infrared radiation for light sources 8 is due to several reasons, in particular:

- near-IR radiation is invisible to the human eye, does not distract the user’s attention and does not cause pupil dilation;

- near-infrared radiation does not cause degradation and destruction of the receptor apparatus of the human eye;

- near-IR radiation is recorded by the same means as visible radiation;

- when using IR radiation, the image of the pupil is characterized by high contrast due to reflection from the retina (lighting scheme with a dark pupil);

- the use of IR radiation makes it possible to separate useful information from external illumination falling in the visible part of the spectrum.

The number of light sources 8 determines the number of glares on the cornea, relative to which the distance to the center of the pupil will be measured. The gaze direction detection device 1 uses a six-glare measurement method for each eye. This approach improves not only the accuracy, but also the reliability of the device. In addition, the use of six glares makes it possible to increase the working range of angles of device 1 due to the reliable restoration of information about the direction of the gaze vector at large angles of rotation of the eyeball. At such angles, part of the glare falls on the sclera of the eye and is not detected by cameras 6.1, 6.2 of the eye, but due to the use of six glares, at least four of them fall on the pupil or iris 12 (Fig. 4), and their coordinates are determined with fairly high reliability.

Light sources 8 are installed in rims 3.1,3.2 in the area of the left and right light openings and are located relative to the light opening in such a way that six highlights on the cornea of each eye form a control pattern in the form of a hexagon (five are clearly visible in Figs. 5a-5e highlights, and the sixth highlight, located next to the fifth highlight, is dim).

Light sources 8 can be installed on platforms located in special grooves on the rims 3.1, 3.2. The angles of the areas are calculated to ensure uniform illumination of the eye area.

Preferably, irradiation is performed with non-collimated divergent beams to uniformly illuminate the entire analyzed area. Since an off-axis (relative to the optical axis of the lenses of cameras 6.1, 6.2 eyes) lighting scheme is selected, the angle between the optical axis of the lens of cameras 6.1, 6.2 eyes and the normal to the light-emitting area of the light sources 8 lies in the range from 0 to 90° and is much greater than zero (see Fig.4). This scheme allows the use of six separate light sources 8 for each light opening and simplifies the process of arranging the device 1. In this case, it is preferable to place the light sources 8 symmetrically relative to the horizontal and vertical axes of the light opening, at equal distances from the optical axes of the eye cameras 6.1, 6.2.

In addition, it is also preferable if the light sources 8 operate in modulation mode according to a periodic law to increase the efficiency of the process of isolating useful information against the background of external illumination, extending the service life, reducing energy consumption and reducing irradiation of the cornea and retina of the human eye. The intensity of the light sources 8 is controlled by modulating the supply current.

Device 1 also contains a computing module 9, a control module 10 and a power supply (not shown in the figures). The placement of the computing module 9 and the control module 10 can be any, for example, on the clamping bars 5.1, 5.2, as shown in Fig. 1,3.

Computing module 9 is designed to implement an algorithm for determining the direction of gaze and performs the following functions:

- determination of the coordinates of the gaze direction vector in the coordinate system of cameras 6.1, 6.2 eyes;

- transformation of the coordinates of the gaze direction vector from the coordinate system of cameras 6.1, 6.2 of the eye into the coordinate system of camera 7 of the scene,

- calibrating the direction of view;

- if necessary, transfer information about the global coordinates of the viewing direction vector to an external device.

To perform its functions, the computing module 9 contains a detector of the position and size of the pupil (not shown in the figures; hereinafter also DPRZ) or is connected to it as an external device. DPRZ is intended for processing the image of the eye in real time with a frame rate of at least 50 Hz in order to construct an ellipse that coincides with the contour of the pupil, and subsequently determine the coordinates of its center (Figs. 5a-5e). DPRZ also calculates the positions of glare.

The control module 10 provides two functions:

- coordination of the interface of cameras 6.1, 6.2 eyes with the interface of the computing module 9;

- control of the brightness of light sources 8, as mentioned above, for example, using pulse width control.

Data from cameras 6.1, 6.2 of the eye is supplied through control module 10 to the DPRZ. The parameters of the ellipse and highlights, determined by the DPRZ from the image of the pupil, are transferred to the computing module 9, where they are used to calculate the direction of gaze.

The power supply can be any suitable power source, rechargeable (battery) or non-rechargeable.

In addition, the device 1 may further comprise an information storage unit (not shown in the figures), in particular for storing the results of the gaze direction calibration, as discussed below.

Device 1 continuously monitors the direction of the user's gaze and transmits the coordinates of the intersection of the gaze direction vector with the image plane received from the scene camera 7, provided that the gaze direction vector is in the field of view of the scene camera 7. Fixing a point (area) of attention can be done by volitional blinking, maintaining attention on this point (area) for a fixed time, or by another method.

According to the present invention, the method for determining the direction of gaze, or the gaze direction vector, uses the method of digital infrared video oculography with subsequent linking of the found gaze direction vector to the image of the surrounding environment. In this method, the eye is illuminated with light, in particular infrared light, which is reflected from the cornea and lens of the eye 11, and then recorded using cameras 6.1, 6.2 of the eye. The position of the pupil is calculated as the center of the area of sharp contrast within the iris, which is observed when illuminated by light sources 8. The glare on the cornea of the eye caused by corneal reflection is used as a reference point to measure the direction of gaze. The vector of the difference between the coordinates of the center of the pupil and the glare changes when the direction of gaze changes and is connected by geometric relationships with the vector of the direction of gaze. The binding of the gaze direction vector to the surrounding environment is carried out by superimposing the found vector coordinates on the image of the surrounding scene, which can be obtained using the scene camera 7. Increasing the accuracy of determining the gaze direction vector is achieved, among other things, by calibrating the device, which can be performed both during the implementation of the method and in advance, once for a given user.

Image processing is performed in real time.

Device 1 can operate in two modes: calibration mode and operating mode.

An example calibration option includes the following steps:

- form a calibration mark, for example, by displaying it on a computer monitor screen;

- receive images from the scene camera 7, thereby also determining the coordinate system of the scene camera 7 (hereinafter also referred to as the global coordinate system);

- determine the coordinates of the center of the calibration mark in the coordinate system of the camera 7 of the scene;

- determine the vector of the direction of view in the coordinate system of cameras 6.1, 6.2 of the eye (hereinafter also referred to as the local coordinate system);

- calculate conversion factors from the local coordinate system to the global coordinate system.

It is enough to carry out the calibration procedure once for a specific user.

After the calibration procedure, you can use device 1 in operating mode, in which, using cameras 6.1, 6.2, the eyes determine the gaze direction vector in the local coordinate system (i.e., in the coordinate system of cameras 6.1, 6.2 eyes), recalculate the local coordinates of the gaze direction vector in global coordinates of the gaze direction vector (i.e., into the coordinate system of the scene camera 7) and obtain the corresponding gaze fixation point on the scene image.

The following describes the main steps in determining the position of the pupil of the eye.

First stage. Obtaining images of the left and right eyes from cameras 6.1, 6.2 eyes. The eye image comes from eye cameras 6.1, 6.2 with a frequency of at least 50 Hz. The field of view of the eye cameras 6.1, 6.2 is selected so that the eye always remains in the frame regardless of variations in the position of the device 1 on the user's head. The position of the eye in the eye image frame can change; it is not always located in the center of the frame.

Second phase. Preliminary search for the pupil. The pupil is the largest coherent dark area in an image of the eye. The preliminary position of the center of the pupil is determined, as well as the number of pixels in the pupil area, which preliminarily characterizes its size. Due to the not entirely uniform illumination of the dark area of the pupil, part of the pupil may not be included in the pupil area, so that the center and size of the pupil are not determined quite accurately at this stage (Fig. 5a).

Third stage. By analyzing the histogram of pixel brightness values in the area selected in the second stage, the binarization threshold is found for accurately constructing the pupil ellipse. Next, binarization is used to determine the boundary of the pupil, and based on the boundary of the pupil, a preliminary ellipse of the pupil is constructed.

Fourth stage. Accurate construction of the pupil ellipse. The precise construction of the pupil ellipse is carried out using the nodal points of the pupil boundary determined at the third stage, filtered so that the nodal points form a convex figure. The icon ellipse is constructed using the least squares method.

The modeling of these methods carried out by the authors according to the third and fourth stages shows a high probability of visual coincidence of the precisely constructed ellipse of the pupil with the boundary of the pupil visible in the images of the eye obtained from cameras 6.1, 6.2 of the eye at the first stage (Fig. 5b).

Fifth stage. Determining the position of highlights and numbering highlights.

When finding the position of highlights on the cornea of the eye, thresholding processing with cluster selection is used (see, for example, Suzuki, S. and Abe, K., “Topological Structural Analysis of Digitized Binary Images by Border Following.” Computer Vision, Graphics, and Image Processing 30 1, pp 32-46, 1985) and filtering them by brightness, size and deviation from roundness.

The size of the iris is calculated from the average size of the human iris and the distance from the cameras 6.1, 6.2 of the eye to the pupil (Fig. 5 c). Highlights outside the iris are filtered out.

The highlights are numbered to determine the gaze direction vector in the global coordinate system, for example, in the following order: from one of the top pair closest to the bridge of the nose, in a circle, away from the bridge of the nose (clockwise for the right eye and counterclockwise for the left), as shown in Fig. 5d.

First, the highlights of one image frame are numbered for each eye. The resulting coordinates of the highlights in the coordinate system of cameras 6.1, 6.2 are stored as test numbering of the highlights. Next, the highlights are numbered in several passes. On the first pass, the two top highlights are selected, based on them, a transition matrix from the test numbering to the current array of highlights is constructed and the quality of the overlay on the test coordinates is checked by the distance from the highlight to the nearest test coordinate of the highlight, as well as the slope of the transformation matrix. On the second pass, select the highlights farthest from the bridge of the nose and repeat the procedure. If only one of the passes is successful, choose numbering according to it, and if both, then take the average transition matrix and carry out another numbering according to it.

Sixth stage. Determination of the direction of the optical axis of the eye (Fig. 5e). Solve a well-known system of equations (see, for example, Guestrin, Elias & Eizenman, Moshe. (2006). "General Theory of Remote Gaze Estimation Using the Pupil Center and Corneal Reflections." Biomedical Engineering, IEEE Transactions on. 53. 1124 - 1133 . 10.1109/TBME. 2005.863952), in particular, using the Levenberg-Marquardt algorithm, for which, in particular, the nodal point C of the eye, the refractive point R for the center of the pupil, the position of the center P of the pupil in the local coordinate system, the nodal point O of the camera are determined , the position of the image V of the center of the pupil, as well as the reflection point Qi and the position of the image of the glare from the i-th light source 8, indicated in Fig. 4 by position 8i (see Fig. 4).

Seventh stage. Determining the direction of view. First, the direction angles of the viewing direction vector in the local coordinate system are determined. Next, they are recalculated into the global coordinate system. In this case, the individual characteristics of the user are taken into account (the angles of deviation of the area of best vision from the direction vector of the eye) and the design features of the device for determining the direction of the user’s gaze (in particular, the relative position of cameras 6.1, 6.2 and camera 7 of the scene), which are determined at the calibration stage.

Calibration stage. Calibrating the user's gaze direction consists of two steps. The first step is to perform the actual calibration, and the second step is to check the calibration. Calibration is carried out using, for example, a monitor screen or tablet along which the ArUco tag moves. Finding the specified mark occurs using an algorithm from the widely used ArUco library (S. Garrido-Jurado, R. Munoz-Salinas, F. J. Madrid-Cuevas, and M. J. Mann-Jimenez. 2014. "Automatic generation and detection of highly reliable fiducial markers under occlusion ". Pattern Recogn. 47, 6, 2280-2292. DOI=10.1016/j.patcog.2014.01.005). During the development of the calibration procedure, it was found that a moving tag was approximately three times more accurate than a stationary tag, and also better retained the user's attention.

Thus, the present invention makes it possible to achieve high accuracy in determining the direction of gaze even with small changes in the position of the system on the user, while remaining compact and consuming low power.

Claims (25)

1. A method for determining the direction of a user's gaze using a device for determining the direction of a user's gaze, comprising a left eye camera and a right eye camera for obtaining an image of the left eye and a right eye, respectively, in the eye camera coordinate system, a scene camera for obtaining an image of the surrounding scene in the scene camera coordinate system , and six left eye light sources and six right eye light sources for generating highlights on the left eye and right eye, respectively, the method comprising the following steps: a) obtaining an image of the left eye and an image of the right eye containing highlights created respectively by the light sources of the left eye and the light sources of the right eye, b) determination of the position of the pupil of each eye, at which: b1 in the eye image, a preliminary search for the pupil is performed, b2) construct a preliminary ellipse of the pupil and b3) based on the nodal points of the boundary of the preliminary ellipse of the pupil, an ellipse of the pupil is constructed, c) determination of the position and numbering of highlights on the cornea of each eye, in which: c1) search for glare on the cornea of the eye, c2) calculate the size of the iris, c3) filter out highlights located outside the iris, and c4) perform numbering of highlights to determine the vector of the direction of view in the coordinate system of the scene camera; d) determining the direction of the gaze direction vector of each eye, in which the nodal point of the eye, the refractive point for the center of the pupil, the position of the center of the pupil in the coordinate system of the scene camera are determined; And e) determining the direction of view in which: e1) determine the direction angles of the gaze direction vector in the coordinate system of the eye cameras and e2) recalculate the specified angles of direction of the optical axis into the coordinate system of the scene camera, taking into account the calibration of the direction of view. 2. The method according to claim 1, in which at step b1 the preliminary position of the center of the pupil is determined, as well as the number of pixels in the pupil area, which preliminarily characterizes its size, and the preliminary pupil area is formed. 3. The method according to claim 1, in which at step b2, a binarization threshold is found in the preliminary pupil area and binarization is performed to determine the pupil boundary for constructing a preliminary pupil ellipse. 4. The method according to claim 1, in which step b3 is performed using the least squares method. 5. The method according to claim 1, wherein in step b3, the node points are filtered so that they form a convex figure. 6. The method according to claim 1, in which step c1 is performed by threshold processing of the eye image with cluster selection and filtering of clusters by brightness, size and deviation from roundness parameter. 7. The method of claim 1, wherein step c2 is performed using information about the average size of the human iris and information about the distance from the corresponding left eye camera or right eye camera to the pupil. 8. The method according to claim 1, in which step c4 is performed from one highlight from the upper pair, closest to the bridge of the nose, in a circle, away from the bridge of the nose, clockwise for the right eye and counterclockwise for the left eye. 9. The method according to claim 1, in which when calibrating the direction of view, the relative positions of the cameras of the left and right eyes and the scene camera are taken into account. 10. The method according to claim 1, in which the calibration of the direction of view is carried out in advance at one of the specified stages. 11. Method according to paragraphs. 1 and 10, in which the individual characteristics of the user are taken into account when calibrating the direction of gaze.