EP2836893A1

EP2836893A1 - Method for determining the direction in which a user is looking

Info

Publication number: EP2836893A1
Application number: EP13725787.9A
Authority: EP
Inventors: Marc Massonneau; Marc SWYNGHEDAUW
Original assignee: Individual
Current assignee: Suricog
Priority date: 2012-04-12
Filing date: 2013-04-12
Publication date: 2015-02-18
Also published as: US9715274B2; FR2989482B1; FR2989482A1; WO2013153538A1; US20150042558A1

Abstract

The present invention relates to a method for determining the direction in which a user is looking, which comprises acquiring images of the eye, in particular by means of an optical sensor, the method comprising: a) a first processing of said images, yielding information on the orientation of the eye according to the observation of an area of the eye in which the aspect varies with the rotation of the eye; and b) a second processing of said images, yielding information on the kinematics of the eye by comparing at least two consecutive images; method in which information is generated relating to the direction in which the user is looking relative to the head of the user, at least according to the information supplied by the first and second processes.

Description

Method for determining the direction of a user's gaze

The present invention relates to the determination of the direction of gaze of a person by monitoring his eye movements, especially when this person visualizes a screen of a computer system.

US 6,113,237 relates to a device for detecting the horizontal and vertical movements of a person's eyes. It gives the position of the pupil, but not the direction of the gaze. It does not allow to obtain a great precision.

Patent applications US 2005/0110950 and US 2010/0283972 relate to medical applications of the detection of saccadic movements of the eye and do not teach more particularly to determine the direction of gaze.

In the international application WO 2010/083853, an image of the scene observed by the user is acquired to determine the point of observation of the latter.

The application US 2010/045933 proposes a method for identifying the direction of the gaze involving a reflection on the retina which requires a complex optical apparatus (mobile mirrors scanning the area measured from a very small point light source) associated with a method of deferred processing of information based on fixed repositories. This device does not use the synergistic and self-sustaining aspects of a dual source of real-time information with relative referencing and therefore can not satisfactorily address the issues of instantaneous information processing and self-correction of data. drifts.

The application EP 2 261 772 describes a method for reconstructing the direction of gaze from a still, remote camera facing the user. A first process identifies and then tracks the position and orientation of the head by optical flow, and a second process identifies one or both of the user's irises. This fixed device, not embedded, significantly restricts the mobility of the user and does not meet the precision constraints expected from such a system.

The application US 2012/019645 describes a device for determining the look data on an onboard display screen. It does not include SEO to the outside world.

Existing systems may require a relatively long and restrictive calibration. In addition, a drift is likely to occur, which limits in time the accuracy obtained and therefore the possibility of using the results obtained sustainably. There is a need to improve the productivity and accuracy of the interface between humans and a computer system, and in particular to simplify the calibration process.

There is still a need to speed up data processing and make the glance direction determination system more responsive and easily usable in real time.

Finally, there is a need for a method allowing the user to enjoy a certain freedom of movement, at least the head.

The invention aims to meet all or part of these needs.

Process

The object of the invention is thus, according to a first of its aspects, a method of determining the direction of the gaze of a user comprising the acquisition of images of the eye, in particular using an optical sensor. , preferably a camera, the method comprising:

a) a first treatment of these images which provides information on the orientation of the eye, from the observation of an area of the eye whose appearance varies with the rotation of the eye, and

b) a second processing of these images which provides information on the kinematics of the eye by comparing at least two successive images,

Method in which information concerning the gaze direction relative to the user's head is generated, in particular in the reference frame of the user's head from at least information delivered by the first and second treatments.

Thanks to the cumulation of these treatments, the precision is improved; this makes it possible to reduce the resolution of the camera used to acquire the images and thus to increase the processing speed and to simplify it; this reduces power consumption or lighten the system or make the system wireless and extend the autonomy.

In addition, the best accuracy reduces the drift over time compared to the initial calibration and thus extend the duration of use without the need for a new calibration. The comfort of use is increased. The first and second treatments do not necessarily take place in the order a) then b) and can also take place in order b) then a), the treatments being performed in a loop to provide at intervals of time regular information about the direction of the gaze.

The image acquisition method used in the method may advantageously be non-intrusive to the eye. In other words, the images are used to determine the position and / or the displacement of the eye are images of the outside of the eye and not images of a deep surface of the eye, such as the retina or the macula, or involving refractions and / or reflections on elements of the eye, such as the retina or cornea. Thus, the images that can be used in the invention are surface appearance images of the eye, such as at least a portion of the contour of the iris and / or sclera. This may make it possible to maintain a relatively simple acquisition lens that is easy to integrate into glasses, for example. This can also simplify the lighting integrated in the device worn by the user, because it is not necessary to penetrate a light beam within the eye in particular incidence conditions.

Diffuse illumination of the surface of the eye, especially the iris and sclera, may be appropriate.

The invention is not based on the observation of the reflection of light sources on the eye and can thus be more independent of the light environment. Acquired images may represent anatomical and / or external kinematic data of the eye.

The first and second treatments may be based on the acquisition of images at different acquisition frequencies, and preferably the first processing is performed at a first acquisition frequency less than the second processing which takes place at a second frequency, preferably with a factor of at least 10 between the first and second frequencies.

The first and second treatments may be based on the acquisition of images at different resolutions, and preferably the first processing is based on the acquisition of images at a resolution greater than the resolution of the acquisition made during the second processing, for example greater by a factor of 5 at least. The information delivered by the first treatment can make it possible to correct the drifts related to the second treatment, in the precision of the measurement.

First treatment

The first treatment provides information on the orientation of the eye, from the observation of an area of the eye whose appearance varies with the rotation of the eye. The orientation is determined statically regardless of its evolution over time.

This first treatment can be performed from a single image. It can be done for each image.

This first treatment may comprise the determination of the parameters related to the shape of the ellipse corresponding to the pupil, in particular to determine the position of a point of the eye, for example the center of the pupil. An algorithm for reconstructing the shape of the pupil on the image can be used by determining by image processing the parameters of the ellipse corresponding to the circle of the pupil observed in a direction making an angle with the normal to the plane of the pupil. this circle. We can deduce the orientation of the eye by analyzing the evolution of the shape of this ellipse.

Such an algorithm is known from the article "Matching 2-D ellipses to 3-D circles with application to vehicle identification pose" by M. Hutter et al. Image and Vision Computing New Zealand, 2009. IVCN '09.

The reconstruction of an ellipse is for example possible from five points of its contour.

The disk of the pupil, projected on a plane, is an ellipse which can be parameterized in the general form: ax ² + bxy + cy ² + d.x + e.y + f = 0, with b ² - 4.ac < 0. A limited number of contour points, in particular five, are sufficient to calculate the descriptive parameters of this ellipse (center, minor axis, major axis and rotation angle). Knowing these characteristics and those of the camera, it is thus possible to reconstruct in 3D the direction of the vector normal to the disk whose projection is the ellipse.

The reconstruction of the ellipse may require minimal resolution to obtain the necessary precision. In the case where the resolution of the image is weak, an ellipse reconstruction algorithm is used to take into account the intensity of the gray levels in the entire image, rather than focusing on the outline of the task corresponding to the pupil to trace the ellipse. We then seek to do match a continuous function of given form with gray levels. This gives a better resolution than the resolution of the image depending on the size and the number of pixels of the latter. It is thus possible to use a relatively low resolution while keeping a sufficient accuracy.

The direction of the eye in the repository of the camera is given by the normal to the disc of the pupil taken in its center. This normal and the center of the disc, different from the center of the ellipse, are obtained from the characteristics of the ellipse. The direction of gaze in the reference frame of the head is obtained by applying a rotation matrix with 3 axes which is fixed and known.

The processed image does not necessarily include a view of the iris of the eye in its entirety. The invention makes it possible to obtain the ellipse even when the image comprises only a portion of the iris of the eye, whether the eye has been partially closed when the image is taken, or that the image is taken on the eye with significant magnification. The treatment can provide the desired information, even if only a partial view of the border of the iris, or only a border of the iris with the white of the eye.

Unlike conventional methods that calculate the orientation of the eye as a non-linear function of the distance between the center of the ellipse and the center of one of the reflections on the cornea of an external illumination, called points of Purkinje), this method allows to know the orientation of the eye without using an external reference such as Purkinje points. The linearity of this solution makes it possible to get lost from the calibration methods inherent in the classical method.

Second treatment

The second treatment provides information on the kinematics of the eye by comparing at least two successive images, that is to say information on the evolution of the position of the eye in its orbit between two instants.

This second treatment can be performed from two consecutive images. It can be performed for each pair of successive images acquired by the optical sensor.

The second treatment provides information on the movement of the eye in its orbit, including information on the angle of rotation of the eye, assuming that the eye is a sphere. The second treatment involves determining the optical flux between two successive images of the eye, that is to say the apparent movement caused by the relative movement between the optical sensor that acquires images and the eye. A corresponding algorithm is known from the article "Determining Optical Flow" Berthold KP Horn et al, Artificial Intellingence, vol. 17, pp. 185-203, 1981.

The variation of the flux of pixels in an image is measured by difference between two or more images. Two successive images can be very close together in time. Two successive images are preferably of constant intensity. One can locate between two images of the same area of a moving part of the eye having contrasts, for example two successive images of the sclera comprising blood vessels, or any region including all or part of the Iris or pupil, the choice of the region is not limiting, the variation of the location of characteristic areas by their shapes or intensity.

The variation in the location of the characteristic zones is observed by the variation of the intensity flux of pixels in the plane parallel to the plane of the sensor. The measurement of this variation of the optical pixel flux does not require having to explicitly identify a reference image or features or reference patterns in the image. In particular these "images" can be taken at very high frequencies by a second sensor independent of the first treatment. It is not necessary that this second sensor has a good resolution or a good focus provided that the measured area is contrasted. The measurement of this variation of the optical flux is facilitated if the intensity of each image is constant.

Combination

The information resulting from the two treatments can be combined to generate information about the direction of the gaze, which can be information about the direction of an eye. The information concerning the direction of gaze may result from the addition of the two pieces of information, each being weighted according to a law that may depend on the two pieces of information themselves. The first treatment makes it possible to obtain information concerning a change of orientation of the direction of gaze, in the reference frame of the camera, by the measurement of the normal to the pupil. The second treatment makes it possible to know the differential of movement of one or more points of the sphere modeling the eye in the plane of the camera. These two informations come from two independent treatments. They are combined and weighted to obtain a measure of the direction of the final eye. This weighting is a function of the coherence of the movement at this moment. Under the assumption of a pure three-dimensional rotation movement along two axes and of unknown center and radius, starting from an initial angular position corresponding to the image at a time t, the solution of the rotation enabling get the final state at time t + dt, which can be the next image, depending on the two information obtained. The chosen solution is a weighting of the two results as a function of their consistency with the imposed rotation model. Both treatments can be performed simultaneously or almost simultaneously. This can make it possible to take advantage of their functional and / or topographical complementarity to accelerate or facilitate the treatments or to improve the quality. One of the information can be used to correct the other or to help determine it. One can for example proceed by interpolation to determine one of the information.

For example, it is possible, by means of this information, to dispense with corneal refraction which can produce non-linearities originating from the refraction of the pupil on the cornea, this refraction being due to the position of the optical sensor. The second treatment makes it possible to obtain information on the displacement of the eye in its orbit with a lesser influence of the refraction, insofar as the zone of the eye used is preferably situated relatively in the corner of the latter, for example on the sclera.

The second treatment can produce errors in the case where the eye rotates too fast relative to the capture speed of optical sensors, hence the advantage of using successive images close in time, so a high optical sensor acquisition frequency, and the combination with the first treatment.

One can for example perform a modeling of the location of the center of rotation by considering that the set of previously determined directions of the eye pass through the center of the eye. Thus, we can facilitate the determination of subsequent viewing directions, assuming that they must also pass through the modeled center of rotation. By the first treatment, one knows at every moment, the position of the center of the pupil and the normal one to this one. From several successive measurements, the center of rotation of the pupil in space, is the minimizing solution, for example by the method of the least squares, the equation of a sphere knowing the positions in the plane of the multi-point camera of this sphere as well as normal to the sphere in each of these points.

The synergy of these two treatments makes it possible to optimize the advantages of each one in their own regimes.

The first treatment makes it possible to precisely know the position of the pupil when the movement of the eye is slow or zero compared to the acquisition frequency, angular velocity corresponding to a movement frequency of, for example, 30 Hz. If the displacement is very fast, especially with an angular velocity corresponding to a frequency greater than or equal to 200 Hz, the image is distorted, appears blurred, scrambled and reconstruction bad.

The second treatment measures a rotation speed. The angular orientation is obtained by integrating this speed in time. The acquisition system is much faster than the movement of the eye, this method allows accurate measurement during very fast movements of the eye. Conversely it is less interesting during slow movements of the eye because the background noise of the image becomes important in the face of the small displacement.

The combination of the two treatments is therefore particularly advantageous and makes it possible to obtain both a good spatial accuracy and a good temporal accuracy of the orientation of the eye by allowing a reciprocal self-correction of the two treatments.

Device

It is advantageous to use for the acquisition of images a device carried by the user, comprising at least a first camera configured to acquire an image of all or part of an eye of the user.

The device may also comprise an electronic circuit for processing the images acquired according to the method according to the invention, so as to determine a relative movement of the eye relative to the camera and the evolution of the direction of gaze over time .

This device can be worn in whole or in part by the user.

The device may further comprise at least one information representation system including screen type or semi-transparent screen, projector, speaker or earpiece, vibratory force feedback system. The representation system can for example to represent visually, graphically, sound or other information obtained in step a) or b) or derived from such information.

The device may further comprise at least one sensor of a physiological information relating to the wearer of the device. The latter sensor is for example selected from a microphone, a motion or acceleration sensor, an image sensor, one or more electrodes, a sweat sensor.

The treatment can take place fully integrated with glasses worn by the user, for example, or remotely, thanks to remote transmission of images for example, or in a mixed way, partly at the glasses and partly in a remote manner, a pretreatment being for example performed through an electrical circuit carried by the glasses and information sent to a remote circuit by a non-wire link. Preprocessing reduces the data transfer rate.

For the acquisition of images, the eye or both eyes are preferably illuminated with infrared lighting, in particular with one or more LEDs. The light source or sources are for example relatively wide angular aperture, especially greater than 60 °, to cover a large area of the eye. The light source or sources can be arranged on the sides, especially outside, this advantageously allows to minimize the reflections observed by the camera.

The device may have only one optical sensor to acquire an image of the user's eye. In the case where the device has only one optical sensor and is devoid of a second optical sensor, the first and only optical sensor can alone to obtain the images required for both treatments.

The successive images captured by the optical sensor (s) may be images of the corner of the eye and / or the pupil and / or the sclera.

The device may comprise at least one second optical sensor configured to acquire an image of all or part of the eye.

The first and second optical sensors can allow the acquisition of images of the same eye of the user. The use of two sensors for one eye makes it less possible to depend on any significant variations in the direction of gaze.

The first and second optical sensors are each directed to a different part of the same eye of the user, for example the pupil and the sclera. The acquisition and processing chain associated with one eye can be duplicated for the other eye. Thus, the device may comprise at least a third optical sensor carried by the user and configured to acquire an image of all or part of the second eye of the user. The device may include a fourth optical sensor carried by the user and configured to acquire an image of all or part of the second eye of the user. The third and fourth optical sensors are each directed to a different part of the user's second eye, for example the pupil and the sclera. Thus, one can proceed to a simultaneous observation of both eyes, which allows to further improve the accuracy.

In addition, the frequency of closure of the eyelid of the studied eye or the studied eyes can be detected. An electromyogram and / or an electroocculogram can be used. The closing of one or both eyelids can be used to indicate that the user validates the positioning of the mouse at the target point of the screen, that is to say that he clicks with his eyes on the screen. The user may, for example, have to close his eyes strongly enough to indicate that he wishes to validate a choice of position.

sensors

The sensor or sensors, in particular the second and / or fourth sensors, may be sufficiently fast digital sensors, for example operating at a frequency of at least 200 Hz, or even at least 500 Hz, or even at least 1000 Hz, by example at a frequency of about 1500 Hz, or even a frequency of at least 2000 Hz.

The sensor or sensors, in particular the second and / or fourth sensors, may have a relatively low resolution, for example less than 500 * 500 pixels, or even less than 200 * 200 pixels, for example between 16 * 16 and 100 * 100 pixels.

An image captured may for example have a resolution of the order of 100 * 100 pixels, or even 32 * 32 pixels, or even 24 * 24 pixels.

Such a resolution may be sufficient to allow obtaining satisfactory images, while allowing not to slow down the determination of the calculation of the direction of the gaze of the user. The sensor or sensors may be thermal or infrared radiation. The sensor or sensors may be associated with one or more LEDs. The sensor or sensors may include a camera, including RGB. The camera (s) can be thermal (s).

The camera (s) can be configured so that the eye is located in the focal plane of the camera.

The sensor or sensors are preferably arranged on the device other than centered with respect to an eye. They are preferably distant from a plane passing through the pupil of the corresponding eye.

The placement of the sensors can provide enough contrast between two acquired images to visualize a difference between the two images. The sensors are preferably arranged sufficiently far from the eyelashes. The sensor or sensors can capture an area of the eye of a size of about 5 mm by 5 mm. The sensor or sensors may for example be arranged at a distance from the eye of between 1 cm and 5 cm, for example of the order of 2 cm. The choice of distance ensures that the sensor is close enough to the eye, and therefore only sees the eye, so as to improve accuracy.

The sensor or sensors may be configured to preferably provide a light localized, ie rather diffuse. Indeed, we try to avoid producing on the eye a reflection of the light source.

The first sensor and the optional third sensor may have speed and resolution characteristics different from the second and fourth sensors. In particular, the first sensor and the optional third sensor can operate at a frequency between 10 Hz and 150 Hz, in particular about 30 images / sec and have a resolution of at least 100 * 100 pixels, for example between 300 * 300 pixels and 1280 * 960 pixels.

The possible second sensor and the possible fourth sensor can operate at a frequency greater than 200Hz, in particular between 200Hz and 2000Hz, or even between 300Hz and 2000Hz, in particular equal to 500Hz and have a resolution of between 32 * 32pixels and 140 * 140 pixels.

In order to minimize the accumulation of errors in the dynamic measurement and its drift, the frequency ratio between the two sensors is preferably about 10 or more, i.e., the acquisition frequency of the second sensor is preferably greater than 10 times that of the first sensor.

Position of the user's head

The movement of the device relative to a basic module can be determined. The basic module can for example be attached to a screen of a computer system.

By "computer system" is meant any computer system comprising a screen for interacting with the user, for example a computer or laptop, a tablet, a landline or mobile phone, this list is not limiting.

For example, it is possible to use optical beacons or light sources disposed on the device, in particular fixed to the device, for example on branches of the latter, and a sensor, for example a camera, arranged on the base module.

The term "optical beacon" here designates a passive element such as an optical reflector or a material reflecting a certain wavelength when it is illuminated by an external source. Such tags offer an alternative possibility to the presence of active elements such as light sources of a certain wavelength belonging to the device.

The wavelength emitted directly by a light source of the device or reflected by a beacon can be in the visible, or the near infrared, which is preferable because being invisible to the human eye. This luminous flux emitted directly by the light source of the device or reflected by a beacon can have one or more wavelengths simultaneously or alternately in time, can be polarized linearly, circularly, or in any other way, can be continuous in amplitude or in phase, or modulated in amplitude or phase.

The movement of the device relative to the base module is determined for example by triangulation. For example, at least three points defined by beacons or light sources [see your paragraph below on possible shapes] can be used on the device and a camera on the basic module.

The at least three points are not aligned and are arranged in a plane not perpendicular to the optical axis of said camera. The at least three points may be arranged in a plane substantially parallel to the optical axis of said camera. A corresponding algorithm is described in the article "3D Pose from 3 Matching Points" under Weak-Perspective Projection by TD Alter, Massachusetts Institute of Technology, Artifical Intelligence Laboratory, AI Memo No. 1378, July 1992.

Optical beacons or light sources can be of different shapes.

They may be one-off, for example of the number LED type, which may be between 3 and 9, each point then comprising an optical beacon or a light source.

They may be of linear shape, curved, straight, continuous or non-continuous, for example comprising a lateral illumination optical fiber.

Optical beacons or light sources can form an identifiable and unambiguous pattern.

They can present combinations of several forms.

Sources or beacons are for example arranged to the outside of the head.

The light sources used to determine the movement of the device relative to the base module can be for example infrared LEDs. The camera can be an infrared camera.

A reconstruction of the six degrees of freedom of the position of the head relative to the base module, ie in relation to the screen to which it can be fixed.

Thus, we measure on the one hand the direction of the gaze, in other words the movement of the eyes relative to the head, and on the other hand the movement of the head relative to the screen. Such a method makes it possible to minimize the calibration required. Both treatments can be performed simultaneously.

In an alternative embodiment, one can for example use two gyroscopes and a distance sensor, one disposed on the device and the other disposed on the base module, which can be attached to a screen.

In an alternative embodiment, the screen is not independent of the device but integral with the latter, so that the determination of the direction of the user's gaze relative to the device is sufficient to determine the direction of gaze of the user relative to the screen. In another variant embodiment, the screen is remote from the device.

Observation point The observation point of the user can be determined on a screen on which the basic module is fixed, in particular a screen of the computer system mentioned above, from the meeting point of the two visual axes of the user determined in FIG. from the information concerning the direction of the gaze. By "visual axis" is meant the axis between the fovea and the optical center of the eye of the user. The fovea is an area of the retina slightly offset from the center of the retina, and allowing the best vision of the color. The meeting point of the two visual axes thus indicates the point of observation on the screen. The "optical axis" is the axis between the center of the retina of the user and the optical center of the eye. There is a slight difference between these two axes, about 5 ° and depends on the user. By using an accurate measurement of the relative orientation of the head relative to the screen, and a measurement of the orientation of the two eyes, in other words a binocular measurement, the invention makes it possible to obtain the measurement of the convergent visual axes. on the screen. The measurement of the direction of the optical axes does not call for the reconstruction of the corneal reflection point (glint in English) introducing a non-linearity to be corrected by a heavy calibration. Thus, the invention facilitates the determination of the observation point.

calibration

Calibration can be done by asking the user to follow a moving target on the screen, ie the eye is forced to follow the eye. In the case of eye tracking, the eye physiologically optimizes its movement to minimize it, and uses the same area of the fovea, so that this calibration makes it possible to better define the position of the fovea. An alternative is to ask the user to set one or more points on the screen at specific times. In the foregoing, the user is warned that a calibration is performed.

In the invention, it is not necessary to find points of reflection of a light source, for example sensors. We do not try to locate a point of reflection (glint) of a light source on the cornea. In the invention, there is no attempt to recognize in the captured images the image of a real object reflected on the cornea of the user.

Device

Another aspect of the invention, according to another of its aspects, independently or in combination with the foregoing, is a device for determining the the direction of a user's gaze, in particular glasses or headphones, intended to be immobilized on the user's head, for controlling a computer system, comprising:

at least one first optical sensor configured to acquire an image of all or part of an eye of the user,

an electronic circuit allowing at least a preprocessing of the data, and

- a wireless transmitter

The wireless transmission of data makes it necessary to reduce the volume of data to be transmitted.

The wireless transmitter can have receiver functions and be a transmitter / receiver. The transmitted data can be analog or digital. These are, for example, data of the audio or video type.

The processor speeds up the final processing of data through preprocessing. The use of an on-board processor makes it possible to pre-process the data obtained from the sensors and thus to reduce the amount of data to be transmitted to the basic module, thereby speeding up the process. The processor may include at least some of the previously mentioned algorithms necessary for data processing.

The device is preferably positioned on the nose and / or the ears of the user, similarly to a pair of glasses.

The device may comprise a battery giving it sufficient autonomy, for example at least several hours of operation without recharging, or even at least a day. The use of low resolution images makes it possible to reduce the power consumption and thus to increase the autonomy of the device.

The user's head may be stationary relative to the device.

Together

The subject of the invention is, according to another of its aspects, independently or in combination with the foregoing, an assembly comprising:

a device as described above, and

a basic module intended to be attached to a screen of a computer system and connected to the latter.

The device can communicate with the base module by a wireless link. The base module can be configured to be used as soon as it is connected to the computer system. In other words, it can be recognized quickly, easily and automatically by the operating system of the computer system upon connection or upon restart after the hardware installation ("Plug and Play" in English). This procedure allows the installation by requiring a minimum of intervention from the user and thus minimizing handling and parameterization errors.

The assembly may comprise several devices able to communicate with the same basic module.

Description of figures

The invention will be better understood on reading the following detailed description of an exemplary embodiment of the invention and on examining the appended drawing in which:

FIG. 1 is a schematic and partial perspective view of a device for determining the direction of gaze according to the invention;

FIG. 2 schematically and partially illustrates an assembly comprising the device of FIG. 1,

FIG. 3 is a block diagram illustrating the method for determining the gaze direction according to the invention,

FIG. 4 illustrates the capture of the images on the eye, and

- Figure 5 schematically illustrates an assembly according to the invention in a given environment.

FIG. 1 shows a user U carrying a device 10 for determining the direction of the gaze, in the form of spectacles worn by the user, comprising branches 11 resting on the ears and a central portion 12 resting on the nose, the glasses 13 of the glasses may include an anti-reflective coating.

The device comprises in the example described two infrared LEDs 16 disposed in the central portion 12 on either side of the nose and each directed towards one of the eyes of the user, and four RGB cameras 15a, 15b can detect infrared radiation. Each of the cameras 15a, 15b is arranged and oriented towards one of the eyes of the user, being disposed below the eyes on the periphery of the lenses 13 of the device and arranged each other than in a vertical plane passing through the center from the pupil of the user. For each of the eyes of the user, two cameras 15a, 15b are arranged on either side of the latter, one on the side of the nose and the other on the side of the branch resting on the corresponding ear, being oriented towards the corresponding eye of the user to acquire images of the latter. The cameras 15a are oriented towards the pupil of the user, and the cameras 15b towards the sclera of the user.

The device 10 also comprises an electronic circuit 17 housed in the example described in one of the branches 11 of the device, this electronic circuit 17 for processing the images acquired by the cameras 15a, 15b, as will be described later.

The device further comprises a battery not visible in the figure, arranged for example in the second branch of the device and giving it sufficient autonomy to avoid having to be recharged for an acceptable duration, for example several hours, or even a whole day.

The device further comprises a wireless transmitter also housed in one of the branches, transmitting the data to a base module 20 attached to a screen 25 of a computer system and connected thereto, as illustrated in FIG.

The device further comprises light sources 19 which make it possible, thanks to a camera 22 arranged on the base module 20, to determine the movement of the device 10 relative to the base module 20, so as to determine a relative displacement of the eye to the camera and the evolution of the direction of gaze over time.

Said light sources 19 are, in the example described, infrared LEDs, for example arranged in a non-aligned manner and in a plane that is not perpendicular to an axis of the camera of the base module, as illustrated. In the example, one of the light sources 19 is disposed above the nose of the user, while the others are arranged on either side of the eyes in the upper part of the device. The light sources 19 are oriented outwards, that is to say towards a camera 22 of the base module 20.

In a variant optical beacons are used instead of the light sources 19.

The steps of a method according to the invention will now be described in detail as illustrated in FIG. In a step 30, each of the cameras 15a, 15b captures images I at regular time intervals from the corresponding eye of the user or more precisely from the part of the eye of the corresponding user.

In the example described, the cameras 15a capture an image A of an area of the eye having at least partially the pupil while the sensors 15b capture an at least partial B image of the sclera of the eye, as can be seen see FIG. 4. Of course, each of the sensors could capture images of both the pupil and the sclera without going beyond the scope of the present invention.

The images I are processed in a step 40 at least partially to perform at least partly two separate treatments (a) and (b), by means of the electronic circuit 17 of the device 10, which makes it possible to carry out these two treatments at least in part, the treatments being then continued in the basic module. In the example described, only a part of the two processes can be performed in the on-board processor on the device, the data then being transmitted by a wireless link F to the base module in which the rest of the processing is performed.

The first treatment (a) provides information on the orientation of the eye from the observation of an area of the eye whose appearance varies with the rotation of the eye. This treatment can be performed for each of the captured images. It comprises the determination in step 41 of the parameters related to the shape of the ellipse corresponding to the pupil. The first processing (a) allows to deduce the optical axes in the repository of the user's head in step 42.

More precisely, the pupil orientation reconstruction steps 41 and 42 comprise the selection of the darkest zone of the processed image, the isolation of the pupil in this image, and then the obtaining of the contour of the pupil. pupil. We deduce the position of the center of the pupil, the major and minor axes of the ellipse and an angle defining the orientation of the ellipse, and finally the normal to the pupil in a three-dimensional space.

The center of rotation of the pupil is then reconstructed by determining a sphere from measurement points and previously determined normals. We deduce the center of rotation and the radius of this sphere. In addition, a second image processing (b) is performed which provides information on the kinematics of the eye by comparing at least two successive images in step 45.

For this purpose, we first select a contrasted subimage in the image I at a given moment. Then we seek a displacement of this same sub-image by searching for a corresponding subimage in the next image such that a correlation between the two sub-images is maximum. The approximation of the non-deformation of the sub-image between the two successive images is correct insofar as the frequency of image capture is sufficient, being in particular much greater than the speed of the observed motion, that is, say in the example described greater than 500 Hz.

Starting from a primary orientation of the eye, the change of orientation of the eye is calculated from the set of infinitesimal displacements determined previously. We then reconstruct the angular displacement of the eye, that is to say the two angles of rotation (pitch and yaw) around any perpendicular axes, from any direction.

In addition, the second treatment (b) is used to correct at step 47 corneal refraction which can produce non-linearities from the pupil refraction on the cornea, which make it possible to correct the information of the first treatment (a). ) at step 42. The direction of the normal obtained above makes it possible to determine a direction of the optical axis that can be tainted by error due to corneal diffraction, and noise from the reconstruction of the ellipse. The angular displacement makes it possible to obtain the variation of the angular variation of the optical axis from any initial direction. This measurement may also be marred by an error related to the resolution of the correlation measurement. The direction of the optical axis is obtained by weighting these two measurements at each iteration, taking into account the positioning obtained in the previous step, the three-dimensional reconstruction of the eye, and a parametrization resulting from the calibration.

In parallel, a processing 50 is performed to determine the position of the user's head relative to the base module, that is to say the movement of the device 10 relative to the base module 20. The camera 22 arranged on the basic module 20 captures images I of the head of the user on which is visible light from at least three points defined by the optical beacons or light sources 19 of the device 10. These images are processed to determine the position of the device relative to the base module. In a first step 51, the light spots in the image are selected. In a second step 52, the noise is eliminated so as to ensure that the light spots displayed on the images correspond to the beacons or light sources 19 intended to determine the movement of the device relative to the base module. In a third step 53, the position of the light spots in the image is determined and from these data, in a step 55, the position of the head relative to the base module is calculated.

In a step 60, the four angles of the fovea with respect to the cameras are furthermore determined from physiological data 61, preset parameters 62 and pre-calibrated screen coordinates 63. The preset parameters 62 may be device and base module construction parameters, for example the position of the cameras or lighting.

With calibration data 65, the two visual axes are deduced at a step 70.

Finally, from the intersection of the visual axes and the position of the head relative to the basic module, the step coordinates of the screen viewed by the user, which can be transmitted to the user, are calculated at step 80. step 81 to the computer system.

Examples of use of a device according to the invention.

Example 1: evolution in a given environment

The device makes it possible to establish a relationship between the direction of an operator's gaze and a work environment in order to improve the design and ergonomics of this environment. The environment can be for example an aircraft pilot cockpit, as shown in Figure 5, car, simulator or a multi-screen environment control.

The assembly 100 of FIG. 5 comprises a device 10 in the form of glasses, not illustrated, communicating by a wireless link with the base module 20, the latter connected to a computer system 90. The assembly 100 is arranged in a environment 200, here a cockpit flying plane. Thus, the base module 20 can be used as soon as it is connected to the computer system. The assembly 100 shown comprises several devices 10 and 10 ', worn for example by the pilot and the co-pilot. In addition to four image sensors and an electrical circuit, not shown, each device 10, 10 'comprises a transmitter / receiver 40 for wireless communication with the base 20, an information representation system 60 comprising in the example illustrated a semi-transparent screen 62 partially superimposed on the spectacle lenses and an atrium 63. The device 1 also comprises a sensor of a physiological information 70 in order to evaluate the psychological state of the wearer of the device 10, in particular in an emergency situation generating potential stress.

Example 2: Training and / or game

In the case of "serious games" or operator training procedures, the device makes it possible to quantify the look and the effectiveness of the training, in particular to measure compliance with the safety standards of the operator in a critical environment.

Example 3: User Interface

In the case of disability, the device according to the invention may in particular act as a substitute for the mouse and the keyboard for paraplegic upper limbs With the binocular measurement (simultaneous measurement of the direction of the two eyes in the same frame), we measure vergence of the user which is a fundamental parameter having applications in the field of ophthalmology or three-dimensional registration in space.

The illustrated implementations and examples are given for illustrative purposes only and are not limiting of the invention.

Claims

A method for determining the direction of the gaze of a user comprising the acquisition of images of the eye, using at least one optical sensor, the method comprising:

a) a first treatment (a) of these images which provides information on the orientation of the eye, from the observation of an area of the eye (A) whose appearance varies with the rotation of the eye; the eye, and

b) a second treatment (b) of these images which provides information on the kinematics of the eye by comparison of at least two successive images (B),

method in which information about the gaze direction relative to the user's head is generated from at least information from the first and second treatments.

2. Method according to the preceding claim, wherein the first treatment comprises determining the parameters related to the shape of the ellipse corresponding to the pupil, in particular to determine the position of a point of the eye.

3. Method according to the preceding claim, wherein the second treatment comprises the determination of the optical flux between two successive images of the eye.

4. Method according to any one of the preceding claims, the image acquisition method used being non-intrusive to the eye and the acquired images representing external anatomical and / or kinematic data of the eye.

5. Method according to any one of the preceding claims, the first and second processing based on the acquisition of images at first and second different acquisition frequencies, preferably the first acquisition frequency being lower than the second frequency. , in particular with a factor of at least 10 between the first and second frequencies.

6. Method according to any one of the preceding claims, the first and second processing based on the acquisition of images at different resolutions, in particular the first processing based on the acquisition of images at a single image. resolution greater than the resolution of the acquisition made during the second treatment, in particular greater by a factor of 5 at least.

7. Method according to any one of the preceding claims, the information delivered by the first treatment for correcting in the accuracy of the measurement the drifts related to the second treatment.

8. A method according to any one of the preceding claims, wherein there is used a device (10) carried by the user (U) comprising:

at least one first optical sensor being a camera (15a, 15b) configured to acquire an image of all or part of an eye of the user,

- An electrical circuit (17) allowing at least a pretreatment of the acquired images, so as to determine a relative movement of the eye relative to the camera and the evolution of the direction of gaze over time.

9. Method according to the preceding claim, wherein the device comprises at least a second optical sensor configured to acquire an image of all or part of the eye of the user.

10. Method according to the preceding claim, wherein the first and second optical sensors allow the acquisition of images of the same eye of the user.

11. Method according to one of the two preceding claims, wherein the first and the second optical sensor are each directed to a different part of the same eye of the user, including the pupil and the sclera.

12. Method according to any one of the preceding claims, further comprising at least one other optical sensor carried by the user and configured to acquire an image of all or part of the second eye of the user.

13. Method according to the preceding claim, wherein the movement of the device (10) relative to a base module (20) is determined.

14. Method according to the preceding claim, wherein the point of observation of the user is determined on a screen to which the base module (20) is fixed, from the meeting point of the two visual axes of the determined user. from the information concerning the direction of gaze.

15. A method according to any preceding claim wherein performs a calibration by asking the user to follow a moving target on the screen.

16. Device for determining the direction of the gaze of a user, in particular glasses or helmet, intended to be immobilized on the user's head, for controlling a computer system, for implementing the method according to any one claims 1 to 15, comprising:

at least one first optical sensor (15a, 15b) configured to acquire at least one image of all or part of an eye of the user,

an image processing system, including in particular an electrical circuit (17) allowing at least

a) a first treatment (a) of said at least one image that provides information on the orientation of the eye, from the observation of an area of the eye (A) whose appearance varies with the rotation of the eye, and

b) a second treatment (b) of these images which provides information on the kinematics of the eye by comparing at least two successive images (B).

17. Device according to the preceding claim further comprising a wireless transmitter

18. Device according to the preceding claim, the transmitter being a transmitter / receiver.

19. Device according to one of the two preceding claims further comprising

at least one information representation system (60, 62, 63), in particular of the screen or semi-transparent screen type, projector, loudspeaker and / or atrium, vibratory feedback system, and / or

at least one sensor of physiological information (70).

20. Set (100) comprising:

at least one device (10) according to any one of claims 16 to

19, and

a base module (20) for connection to a computer system (0).

21. An assembly according to the preceding claim, wherein the device (10) communicates with the base module (20) by a wireless link.

22. An assembly according to one of the two preceding claims, wherein the base module (20) is configured to be used as soon as it is connected to the computer system.