RU2589623C2 - Method of monitoring eye movements and device for its implementation - Google Patents

Abstract

FIELD: optics.

SUBSTANCE: invention relates to monitoring of parameters of vision. Method of monitoring eye movement and determining the direction of sight on a projection of a limb to linear photodetectors in which are used the optical system, a processing and information transmission unit external devices, consists in the fact that uses one or several series-connected linear Photodetectors, on which an optical system is projected at least two projections of the boundary of the limb Ymin and Ymax, and treatment of movement and determining the direction of sight occur in real time at the corresponding values of projections of the limb of one or both eyes right through calculation using the formula given below or using pre-calculated at this formula data stored in memory unit for processing:

, where Ymin and Ymax are minimum and maximum value of projections of the limb on the linear part of the photodetector, corresponding to the plus and minus signs in brackets formula R is radius of eyeball and r is radius of the limb, and the direction of sight vertically determines angle φ and horizontal angle ψ.

EFFECT: using the invention increases operating speed when monitoring eye movement and determining the direction of sight.

5 cl, 2 dwg, 1 tbl

Description

References to patents and literature:

1. Ruxin Chen, Ozlem Kalinli patent US 2012/0281181, dated 08/08/2012.

2. John Howison Schroeder, Alexander D. Kiderman patent US 2010/0092049, dated April 15, 2010.

3. Roel Vertegaal, Changuk Sohn, et al. patent US 7963652, dated June 21, 2011.

4. Thomas E. Hutchinson, Ivy Va. patent US 4950069, from 08.21.1990.

5. Kingman Yee. patent US 6283954, dated 04.09.2001.

The field of technology.

The invention relates to the monitoring of vision parameters and can find application in various fields from medicine to computer control, and can also solve many other problems, for example, analysis of customer’s attention when shopping at a store or viewing a site, control of household appliances, etc. of the invention is tracking the direction and nature of the movement of a person’s gaze in real time, as well as performing certain actions that correspond to a given sequence The direction of gaze fixation or gaze at a specific time.

For medicine, the nature of saccades (fast eye movements) and / or the direction and time of viewing certain test images carries a diagnostic function and can objectively indicate certain disorders of the brain, or the emotional state of a person.

In addition, various “ah trackers” have already been implemented today that control, for example, a computer and help communicate with immobilized people (an example of Professor Stephen Hawking). The gaze control function can also be implemented within the framework of the invention.

The prior art.

Objective methods for accurately determining the direction of view have a long history and are based on different physical principles. All methods are divided into contact methods (electrodes near the eyes and analysis of emerging potentials, sensors on the eyeball, etc.) and non-contact (film or video recording of the eye).

So in the patent [1] it is proposed to use magnetic contact lenses, and in the spectacle frame mount sensors for changing the magnetic field (this is the contact method). Perhaps this is how you can determine the rotation of the eyeball, but most likely the accuracy of such a device will be small. In addition, the method is not very suitable for people with normal vision (without contact lenses).

The vast majority of devices and methods for monitoring or tracking the direction of gaze use optical, non-contact methods. For example, they analyze the image of the eye recorded by a video camera [2], or fix the direction of the reflected beam [3].

So in the patent [2] it is stated that it is enough to have a part of the image of the limb (the limb is the border of the iris and sclera) to calculate the direction of the gaze. This method is based on processing a digitized image of the eye and requires quite serious calculations. Therefore, it can be argued that the method has significant limitations both in accuracy (since the limb boundary has some “blurring”) and in processing speed due to the cumbersomeness of calculations, although the authors do not give concrete results either in accuracy or in the speed of moving the gaze.

In addition, the authors do not discuss the question of what minimum portion of the limb is necessary for the method to work, but starting with some lengths of the limb arc the error may turn out to be so large that monitoring the direction of the gaze will be impossible, therefore this patent is very far from its practical implementation.

The patent [3] records the reflection of a set of markers located on the screen (or “scene”) from the cornea of the eye. This is one of the few patents suitable for monitoring the direction of the gaze of animals (as claimed). This method and device is easy to implement, which is confirmed by calculations and graphs, but with an increase in the number of markers (the patent says: “one or more”), the problem of recognizing the markers themselves arises. In addition, the mathematical model, which is not discussed in the patent, is quite complicated and requires considerable time for image processing, so the possible speed of the tracking eye movements is limited and will not allow to register saccades, the speed of which can reach 200-450 degrees per second.

As in the previous patent, an image of at least one eye and fixation of the patient's head are necessary. In addition, both patents intend to use small video cameras, but most of the available cameras operate at a frequency of only 30-50 frames per second with a modest image resolution, which limits both the accuracy and the speed of eye movements available for recording speed.

The next patent [4] proposes to solve the above problems, as well as to increase the speed and accuracy of the method, although it was announced much earlier in 1990. For this, the eyes are illuminated with a powerful infrared diode. It is known that the retina reflects radiation well in the IR region, so the pupil of the eye looks like a bright spot. Then, using a special algorithm, the center and other parameters of the ellipse are calculated, which uniquely corresponds to the direction of the gaze. Of course, this method partially overcomes the disadvantages of the previously discussed patents, but the small frame rate of the available cameras remains an insurmountable barrier to increasing speed.

As a prototype, consider the American patent [5] "Linear eye monitoring." The main idea of the device and the method is that based on pairs of standard linear photodetectors (photo lines), the position of the limb (the border of the transition of the iris to the sclera) is determined, so the calculation algorithm is greatly simplified. In addition, the speed of the linear sensor (megahertz) is significantly higher than the speed of any video camera, which significantly increases the speed of the device.

The disadvantages of the method include the fact that to determine the direction of view requires 4 values of the coordinates of the limb (left, right, upper and lower), since the invention uses 2 mutually perpendicular linear sensors (see illustrations to the patent), although the claims refer to linear sensors located at an angle (not an orthogonal basis), and only in the second paragraph we are talking about a mutually orthogonal arrangement of linear sensors. The essence of the method does not change from this and 4 points are required to calculate the position and orientation of the eye.

Since this invention and the sensor are designed for laser operations on the cornea of the eye, it is always possible to open the eyelids for the duration of the operation and the method will be successfully implemented. But this method is of little use for monitoring the direction of gaze, since for most people, the upper (and in many cases the lower, for example, Asians) point of the limb is covered by the eyelid. Thus, the proposed method has a narrow scope when the entire limb is visible (laser eye surgery), although the use of linear sensors has significantly increased the processing speed, which allows us to use this method in devices for laser eye surgery.

Thus, the method and device proposed in [5] have a narrow scope, although the use of linear sensors and a simpler calculation algorithm significantly increase the processing speed of the image of the limb of the eye, compared with the use of video cameras.

SUMMARY OF THE INVENTION

The aim of this invention is to develop a simple, reliable method and device for monitoring eye movement in real time and tracking the direction of view with characteristic times of several milliseconds or less.

Each of the above methods for monitoring gaze [1-4] has its own advantages and specific disadvantages, but the general and main one of which is the low speed of the initial information - 30-50 frames per second. However, to study saccades and some other medical applications (for example, studying brain pathologies, assessing the patient’s emotional state, etc.), it is necessary to record movements with a characteristic time of 10-30 ms or less. Although such devices that record movements with frequencies above 1000 Hz (based on the principles of video recording) exist, they are very expensive and the area of several hundred hertz is the limit of modern systems of this class.

Linear image sensors (photo rulers) have much better temporal characteristics and a low price. The characteristic frequencies for reading photo line information are a few megahertz, they are cheap, simple and reliable, and the element base is quite diverse (the number of pixels in the line varies from 256 to more than 5 thousand, they have different areas of spectral sensitivity), etc.

In addition, it turned out that if the diameter of the eyeball (radius R) and the radius of the limb (r) are known, then the direction of view can be easily calculated analytically from the two coordinates of the projection of the image of the limb on a single linear photosensor. For example, in a cylindrical coordinate system for the photo line, lens and center of the eyeball located on the same axis (see Fig. 1), using the formula (1), you can calculate the projection on the photo line of the intersection of the limb with the YX plane. And according to these projections, you can almost unambiguously restore the direction of view:

Here, Y min / max is the minimum and maximum value of the limb projections on the photo line corresponding to different signs in brackets, the angle φ determines the vertical direction of the view, and ψ corresponds to the horizontal angle.

Also in FIG. Figure 1 shows the simplest optical system (lens) and the photo line (Sensor). Formula (1) corresponds to the case when the center of the eyeball lies in the plane, but since a single plane (XY) can be drawn through the point (center of the eyeball) and the line (photo line), then formula (1) is general, although in specific cases it can be clarified, for example, with a special limb shape or a special ruler shape.

Formula (1) does not give a strictly unambiguous solution (a two-valued solution), since when φ is replaced by -φ, the same projections are obtained. But in real cases, the position of the eyeball is asymmetric with respect to the eyelids and the gaze is mainly directed downward. In addition, the calculation of the rate of change of the angle φ, as well as the continuity property of the function of the values of the angle, allow us to refine and unambiguously determine the direction of view. In the processing unit, it is possible to programmatically calculate the rate of change of the angle φ, so the transition of the "equator" with a non-zero speed will further clarify the sign of the angle φ

A radical solution to the ambiguity problem is the development of special CCD arrays (photodetectors) with a geometry different from the line (“curved”), or the use of two or more such arrays connected in series. Their shape can be chosen such (in fact, almost any see. Fig. 2 - the lines correspond to the shapes of the photo lines) so that 3 or more projections of the limb are projected onto the photo line. Then the solution has an unambiguous correspondence with the direction of view along the set of projections of the limb. The selection of the values of several photo lines can be carried out sequentially, especially if the array of lines is made on a single crystal.

The mutual arrangement of several photo lines does not matter, but on condition that all photo lines are located in the focus of a single optical system, which greatly simplifies the design of the device.

The use in the device of two or more interconnected photo lines (not necessarily sequential) or lines of complex shape significantly increases the accuracy of determining the direction of view.

In addition, to select the sign of the solution in formula (1) and to increase the accuracy of determining the angles, you can use the correlation of the values of the projections of the limb of the left and right eye. Due to the phenomenon of divergence (the reduction of both eyes on one object), the projection values of both eyes become interconnected, which can be easily taken into account in the calculations. Preferred is the non-parallel arrangement with respect to each other of the photo line of the right and left eye.

Given that the average radius of the eyeball is approximately 12 mm, and the radius of the limb is about 6 mm, we obtain a limit for vertical angles of ± 30 degrees. That is, the proposed method allows you to determine the direction of gaze vertically in the range of about 60 degrees, which is less than 125 degrees available to the eye. But since the main range of meaningful information is perceived near the optical axis, and the saccade range in 85% of cases does not exceed 15 degrees, these characteristics of the device correspond to practical needs.

For horizontal angles, the situation is even better, since 120 degrees (30 to 150 degrees) corresponds well with the field of view of each eye 60 degrees to the nose and 90 degrees to the temple (the field of view of two eyes is about 180 degrees horizontally).

Of course, it is necessary that when turning the eyes, both projections of the limb boundaries are displayed in the photo line, which requires a large depth of field of the optical system of the device. In practice, increasing the depth of field corresponds to a small aperture and accordingly requires the use of more sensitive photodetectors.

More sophisticated optical systems are possible than the single lens shown in FIG. 1, for example, with a large number of optical elements, as well as prisms or mirrors for turning the optical axis, which is important for a specific implementation of the device, but does not affect the essence of the proposed method.

When implementing this method, calculations by formula (1) or by similar formulas can be preliminarily performed and loaded into the memory of the processing unit, for example, as a table and determining the direction of view is done by comparing the hardware values of the limb projection in real time and the values from the table. In this case, the time to determine the direction of view will be further reduced, which will allow you to track the fastest eye movements.

Estimates of the accuracy of the method strongly depend on the parameters of the optical system. For a system with a focal length of 10 mm and a CCD of a ruler containing 2048 pixels, an angle resolution estimate of better than 0.7 degrees is obtained, which is quite acceptable, although it is bad that the lowest resolution corresponds to the gaze position near the optical axis.

Therefore, although the proposed method and device for its implementation cannot determine any angles in the entire field of view of a person, the range of angles available to the method and device is in the area of greatest practical interest.

You can determine the radius of the limb in the self-tuning mode using the device itself: it is enough to start the test at first, looking in front of you and moving your eyes, for example, vertically up and down, and the device will calculate and determine the maximum projection difference, which corresponds to the diameter of the limb. Additional more complex tests of "aiming" at specially selected points on the monitor will allow you to "bind" the coordinates of the device to an external coordinate system, which increases the accuracy of the method.

If a two-eye monitoring system is used, it is possible to correct for divergence of the eyes, or to control the divergence itself. By the nature of eye movement, for example, by the nature and speed of saccades, it is possible in real time to objectively assess the emotional state of a person.

It is preferable to use photo lines with a sensitivity region extending to the near IR region (near 900 nm), which makes it possible to use limb illumination with powerful arsenide-galium LEDs with a radiation wavelength near 850-940 nm. These wavelengths are not visible to the eye, therefore, they do not affect the person and the measurements, but this is not a fundamental condition and it is possible to use LEDs in a different spectral range corresponding to the spectral sensitivity range of the line.

If the device is designed to communicate or control external devices, it is preferable to use a wireless connection (bluetooth or Wi-Fi).

Thus, a new method for monitoring the direction of view and a class of devices for implementing this method are proposed.

Description of the preferred implementation of the invention

It is preferable to use the method of monitoring the direction of gaze by placing the device in the frame of glasses (or in a helmet, etc.), even if the subject has normal vision. The gaze direction monitoring system can be supplemented by a backlight LED and an external interface for communication with other devices, etc.

Table 1 presents the calculation of the left and right projections of the limb boundaries (one above the other), performed according to formula 1 for a set of randomly selected angles φ (rows) and ψ (columns) for R = 12 and r = 6 mm.

Brief Description of the Drawings

FIG. 1 - A diagram explaining the principle of operation of the device and the placement of the main elements. Sensor - photodetector. Lens - lens, Eye - eye

FIG. 2 - An example of different geometry of photosensors (lines in the figure) in the form of their projection onto the limb of the eye (onto the external ellipse).

Examples of practical implementation.

As an example of implementation, we can consider a system consisting of a quartz lens with a focal length of 10 mm located in the frame of the glasses at an equal distance of 30 mm from the eye and a photo line with 2048 pixels of any brand, for example Sony ILX511. With these parameters (taking into account the thickness of the lens), the focus of the optical system will be at a distance of 15 mm from the lens and the total projection length of the eye in the focal plane will be about 12 mm (less than the working area of the CCD line). According to the passport data, the sampling frequency of the CCD of the 2 MHz line and the monitoring of the viewing direction at frequencies of 10-100 kHz were modeled and turned out to be quite feasible when referring to the previously calculated values (similar to Table 1). Thus, the presented example proves that the implementation of the claimed device parameters does not cause technical difficulties.

Thus, the proposed method consists in the use of one or several series-connected linear photodetectors onto which at least two projections of the limb boundary are projected by the optical system, and the movement and the determination of the direction of view are processed according to the algorithm in real time from the values of the projections of one or both eye by direct calculation and / or using pre-calculated data stored in the memory of the information processing unit.

Industrial applicability

The method of monitoring the direction of gaze can be used: for testing user interfaces, analyzing the effectiveness of advertising, for controlling household appliances, for collecting and analyzing data on a person’s actions and assessing his emotional state in real time when driving various vehicles, such as a car, airplane, etc. ., for the social adaptation of patients with amyotrophic lateral sclerosis, etc.

Claims (5)

1. A method for monitoring eye movement and determining the direction of view from the projection of the limb onto linear photodetectors containing an optical system, a processing unit and an information transmission unit for external devices, characterized in that in order to increase the speed of the method to speeds that allow you to register changes in the direction of view in milliseconds and / or register saccades, one or several series-connected linear photodetectors are used, on which at least two projections are projected by the optical system th border of the limb Ymin and Ymax, and movement processing and determination of the gaze direction occurs in real time according to the corresponding values of the limb projections of one or both eyes by direct calculation according to the formula below or using data previously stored in the processing unit that are previously calculated using this formula:

here Ymin and Ymax are the minimum and maximum values of the limb projections on the linear part of the photodetector, corresponding to the plus and minus signs in the brackets of the formula, where R is the radius of the eyeball and r is the radius of the limb, and the vertical direction determines the angle φ and the horizontal angle ψ . 2. The method according to p. 1, characterized in that to increase the sensitivity of the method, the eye is illuminated with LEDs. 3. The method according to p. 1, characterized in that to improve the accuracy of the method, self-tuning is preliminarily performed, which consists in moving the gaze vertically up and down. 4. The method according to p. 1, characterized in that the data on the speed of the eye and the type of saccades are used to evaluate in real time the psychological state of a person. 5. The method according to p. 1, characterized in that the communication with external devices is carried out wirelessly.

Images