Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
  • G06F3/013—Eye tracking input arrangements
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/20—Analysis of motion
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
  • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/70—Determining position or orientation of objects or cameras
  • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
  • G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/90—Determination of colour characteristics
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/40—Extraction of image or video features
  • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
  • G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
  • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
  • G06V40/18—Eye characteristics, e.g. of the iris
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
  • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
  • G06V40/18—Eye characteristics, e.g. of the iris
  • G06V40/19—Sensors therefor
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10016—Video; Image sequence
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10024—Color image
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20024—Filtering details
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20048—Transform domain processing
  • G06T2207/20061—Hough transform
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30196—Human being; Person
  • G06T2207/30201—Face

Definitions

• the invention relates to the domain of eye gaze estimation with regard to a sequence of images watched by a viewer.
• Eye gaze is considered as an important cue that may reveal useful and irrefutable information from the human mind. The eye gaze is believed to reflect the attention, the behavior and somehow, the emotion of a person within a visual context. In practice, the process of interpretation of eye gaze may be involved in various applications of Human Computer Interaction (HCI) as gaze-based interactive user interfaces, adaptive and interactive content presentation, virtual reality, human behavior study and diagnostic applications, etc.
• HCI Human Computer Interaction
• eye gaze trackers can be generally classified into two categories: intrusive and remote systems according to the way the equipments make contact with the subject.
• One of the earliest intrusive gaze tracker is based on special contact lens fixed on the eyes that allow to detect its position. These contact lenses contain a sensor (a mirror or an induction coil) that is used to reflect light or to measure the eye position in a high-frequency electro-magnetic field.
• a sensor a mirror or an induction coil
• Electrooculography (EOG) based methods make use of the fact that an electrostatic field exists when eyes rotate.
• EOG technique By measuring the electric potential differences of the skin regions around the eyes (with electrodes), the position of the eye can be estimated.
• EOG technique provides a reliable measurement with simple configuration which enables recording in dark environment (where video- oculography is useless) and which doesn't require the eyes to be opened.
• the major problem is that the EOG signal suffers from noise due to eye blinking, movement of facial muscles and EOG potential drift (especially in long recording experiments).
• Video-oculography techniques can also be classified as intrusive methods if they are used in a head-mounted system. In general, an intrusive method allows a high accuracy and free head movement but the main drawback is that it requires close contact to the user that is only restricted to laboratory experiments.
• model-based For everyday applications, nonintrusive (or remote) methods are therefore much more preferred.
• video-based techniques are the most widely used.
• the former uses 3D geometric models of the eye to estimate the gaze.
• the point of regard is determined as the intersection between the 3D gaze direction (composed of the optical axis and the visual axis) and the screen plane.
• Majority of model- based methods are based on the corneal reflection technique with the use of additional light sources, generally infrared light, to illuminate the eyes. The main idea is to estimate the gaze from the relative position between the pupil center and the glint - the brightest light spot on the eye due to reflection.
• appearance based methods consider the gaze estimation as a 2D mapping problem between the image features of the eyes and the positions of the gaze on the screen.
• the mapping function can be found by training a multi-layer neural network, or a regression model like Gaussian process regression or by using a non-linear manifold embedding technique such as Locally Linear Embedding to reduce the high dimensional eye image to 2 dimensions and derive the gaze by linear combination in the low-dimensional space.
• Geometric model based approach is generally more accurate (less than one degree) and widely used in commercial eye tracker. However, it requires high resolution camera and additional light sources.
• Current appearance-based methods are known to be less accurate (with an accuracy of several degrees). More accurate appearance-based methods are known, which can achieve less than one degree of accuracy but with the expense of using extensive calibration points, e.g. disclosed by K. H. Tan, D. J. Kriegman, and N. Ahuja, "Appearance-based eye gaze estimation", Proceedings of the Sixth I EEE Workshop on Applications of Computer Vision (WACV), pages 1 91 -1 95, 2002.
• the training process of the GPR are done thanks to a Monte-Carlo approximation (i.e. samples are generated according to the averaged gaze probability map).
• this approach has some limits. Firstly, in order to go into operating mode, the system needs an off-line and time-consuming training beforehand (i.e. 10 minutes of training for a 10 minutes test). Secondly, the method makes use of many parameters that are empirically determined. Thirdly, in order for the Monte Carlo approximation to reach a desire accuracy, many samples are required at the expense of a significantly increasing computational cost. Nevertheless, the method only achieves a low accuracy of six degrees due to the fact that it is entirely based on saliency information which is not always reliable.
• the purpose of the invention is to overcome at least one of these disadvantages of the prior art.
• the purpose of the invention is to determine the location of the gaze of a viewer on a screen he/she is watching at without any calibration.
• the invention relates to a method for gaze estimation, comprising the steps of:
• the detecting step comprises the steps of:
• the at least a heat map is represented in color space YC b C r as output of the converting.
• the detecting step further comprises a Gaussian filtering of the at least a heat map, the first and second pixels being determined after the Gaussian filtering.
• the method further comprises the steps of:
• the at least a third position of the gaze of the viewer on the screen determining at least a third position of the gaze of the viewer on the screen, the at least a third position of the gaze corresponding to a fusion of the at least a first position of the gaze and the at least a second position of the gaze.
• the at least a first position of the gaze is determined by using the particle filtering method and at least another first position of the gaze previously determined in a temporal point of view.
• At least a third position of the gaze is determined by using the particle filtering method with at least another first position of the gaze and at least another second position of the gaze previously determined in a temporal point of view.
• At least a first position of the gaze of the viewer is determined by taking into account a movement of the head of the viewer.
• the invention also relates to a device configured for determining the gaze of a viewer, the device comprising at least one processor configured for:
• the at least one processor is further configured for.
• the at least one processor is further configured for filtering said at least a heat map with a Gaussian filter.
• the at least one processor is further configured for:
• the at least a third position of the gaze of the viewer on the screen determining at least a third position of the gaze of the viewer on the screen, the at least a third position of the gaze corresponding to a fusion of the at least a first position of the gaze and the at least a second position of the gaze.
• the at least one processor is further configured for executing a particle filtering method.
• the at least one processor is further configured for detecting a movement of the head of the viewer.
• the invention also relates to a computer program product comprising instructions of program code for execution by at least one processor to perform the method for estimating the gaze, when the program is executed on a computer.
• FIG. 1 shows a method for estimating the position of the gaze of a viewer, according to a particular embodiment of the invention
• - figure 2 shows an image of the eye of the viewer with an associated heat map, according to a particular embodiment of the invention
• - figure 3 shows average spatial histograms of gaze positions according to different types of video contents the viewer is watching at, according to a particular embodiment of the invention
• FIG. 4 shows a particle filtering framework applied to the estimation of the position of the gaze of the viewer of figure 1 , according to a particular embodiment of the invention
• FIG. 5 shows a graphical user interface related to the method for estimating the position of the gaze, according to a particular embodiment of the invention
• FIG. 6 diagrammatically shows a device implementing a method for estimating the position of the gaze, according to a particular embodiment of the invention
• FIG. 7 shows a method for estimating the position of the gaze, according to a particular embodiment of the invention.
• the invention will be described in reference to a particular embodiment of a method for estimating the position of the gaze of a viewer watching at one or more video images displayed on a screen.
• the location of the centre of one or both eyes of the viewer is detected by analyzing one or more images of at least a part of the viewer comprising a representation of one or both eyes of the viewer.
• the analyzed image corresponds advantageously to an image of the viewer while he/she is watching at one video image.
• a mapping function representative of the mapping between eye appearances and gaze positions on the screen, and based on centre-bias property of the human gaze distribution is used to determine the position(s) of the gaze of the viewer on the screen.
• Figure 1 illustrates a method for estimating the position of the gaze of a viewer, according to a particular and non-limitative embodiment of the invention.
• the input 10 of the process comprises data representative of one or several video images 101 and data representative of one or several eye images 102.
• An eye image 102 corresponds advantageously to an image of one or both eyes of the viewer watching at the video image(s) 102, or more generally speaking to an image of the face of viewer from which data representative of the image(s) of the eye(s) may be extracted.
• the eye image(s) 102 is (are) acquired via a camera, for example a webcam.
• the camera is for example located on the top of the screen on which the video image(s) is (are) displayed, meaning that the camera is a device not integrated into the screen and connected to the screen with a wire or wirelessly. According to a variant, the camera is integrated into the screen.
• One eye image 102 is advantageously associated with one video image 101 , meaning that one eye image 102 is shot while displaying the associated video image 101 .
• the video image 101 corresponds for example to an image of a sequence of images (e.g. a movie), to a picture, to a web page, etc.
• a centre of one eye or the centre of each of the eyes is/are detected from the eye image 102.
• the face of the viewer may be detected by using a face detection algorithm, for example the boosted cascade face detector, as described in "Robust real-time object detection" by P. Viola and M. Jones, IJCV, vol. 57, no. 2, pp. 137-154, 2002.
• the rough positions of the eye regions are then determined from the detected face based on anthropometric relations. Empirically, it is found that eye centers are always contained within two regions starting from 20% ⁇ 30% for the left eye and 60% ⁇ 30% for right eye of the detected face region, with dimensions of 25%x20% of the detected face region.
• the Hough Transform (HT) method is used for detecting the centre of the eye(s), the HT method detecting circles (and lines) in a parameter space using a voting-based algorithm.
• the HT method is for example described in the U.S. Patent US 3,069,654.
• the centre of the eye(s) is detected by using a method taking benefit of the color information available in the eye image(s) 102. According to this method:
• the eye image captured in RGB color space is first converted into the YCbCr space.
• an eye center heat map (HM) can be determined as follows:
• HM(x,y) Cb(x,yW -Cr(x,Y)). - Y(x,Y)) (1 ) where (x,y) corresponds to the coordinates of a pixel of the heat map, the heat map comprising advantageously as many pixels as the eye image from which the heat map is obtained, a pixel of the eye image with coordinates (x,y) having one corresponding pixel in the heat map with the same coordinates (x,y).
• Figure 2 shows an eye image 20 and its associated heat map
• All the sub-regions that may be the pupil region are then extracted using the region growing method. To do so, local maxima larger than a predefined threshold T1 are chosen as seed points, called first points 212. 4-connected regions around each seed point are then built by growing all pixels whose values are higher than a predefined threshold T2, these pixels being called second pixels 21 3. The selected points are then dynamically added to the set of "candidate points" and the process is continued until we go to the end of the eye region.
• PR is the set of candidate pixels.
• a second step 1 04 the location of the centre(s) of the eye(s) is converted into the position of the gaze.
• the gaze distribution is indeed biased toward the center of the screen in free viewing mode.
• a gaze tracker for example a SMI RED tracker with a sampling frequency of 50 Hz
• For the two first activities 30 and 31 for example 4 observers are asked to watch 8 video sequences (i.e., 4 movie clips and 4 television clips, each of 1 0 minutes).
• the viewers are free to choose 5 favorite web sites to browse during 1 0 minutes.
• - x g and y g are the converted gaze positions, corresponding to the first position 1 05 of the gaze;
• - x c and y c are the current eye center positions in absolute image coordinates. Since the subject's head is supposed to be fixed, we do not require an eye comers localization technique to convert these values into relative positions of eye coordinates;
• - x c , y c , o xc and o yc are respectively the mean values and the standard deviation values of x c and y c .
• a x and A y are tuning factors which describe the "scales" of the gaze distribution. They are empirically set to for example 4 that is large enough to quantify the center-bias level.
• the use of such a simple mapping model enables to get a coarse estimation of the gaze position (while giving good performance) from the eye image, independently of (i.e. without) the saliency map.
• the estimation of the gaze position maybe refined by using saliency map(s) 106 associated with the video image(s) 101 .
• a second gaze position 107 may then be obtained from the saliency map(s).
• a third gaze position 109 may be obtained, which has the advantage of being finer than the first gaze position 105 and the second gaze position 107 taken alone.
• the third gaze position 105 corresponds advantageously to the average of the first gaze position and the second gaze position.
• the third gaze position 105 corresponds advantageously to the weighted average of the first gaze position and the second gaze position, the weight assigned to the first gaze position being greater than the weight assigned to the second gaze position if the confidence in the estimation of the first gaze position is greater than the confidence in the estimation of the second gaze position, and inversely.
• the saliency map is used as to adapt the tuning factors A x and A y .
• a x and A y are adapted according to the dispersions in the saliency map, i.e. according to the variance of the saliency map.
• a particle filter 108 may be implemented, based on the first gaze position 105 and second gaze position 107, as to obtain a much finer third gaze position 109.
• Figure 4 shows a particle filtering framework applied to the estimation of the position of the gaze of the viewer, according to a particular and non-limitative embodiment of the invention.
• a gaze sensing system receives as input two sources of information, the visual content (e.g., image/video) and the viewer's appearance (e.g., head pose or eye appearance), and outputs the most probable gaze points, i.e. the third gaze position 109.
• the probabilistic relationships between the stimulus image I 40, 41 and 42 (corresponding to the video image(s) 101 at different consecutive times t-1 , t and t+1 )), the gaze position g ( x, y) (where g is a 2D vector, x and y are the gaze positions on horizontal and vertical axes, respectively) 43, 44, 45 at the different consecutive times t-1 , t and t+1 and the eye image e 46, 47, 48 (corresponding to the eye images 103 at the different consecutive times t-1 , t and t+1 ) may be illustrated via a probabilistic graphical model 4 as shown in Figure 4.
• This graphical model 4 describes a DBN (Dynamic Bayesian Network) in which the nodes in each time-frame t-1 , t and t+1 represent the relationship between the considered random variables and the directed edges represent their conditional dependencies. Nodes which are not connected are said to be "conditionally independent" of each other. The links between time frames reflect the temporal relationships.
• the gaze position is estimated as the posterior probability p(gt
• the eye appearance e t 47 is not fully independent of the stimulus l t 41 . It is clear that the stimulus somehow has an impact on the gaze position g t 44 and hence, indirectly affect the eye appearance.
• the gaze position distribution is assumed to follow a first- order Markov process, i.e., the current state only depends on the previous state. This assumption is strongly valid for fixation and smooth-pursuit eye movements.
• the current gaze position may also be considered as dependent on the previous gaze position if it is modeled by a distribution having a sufficiently large scale.
• !i t,ei:t) may be estimated via the prior probability p(g t
• the sign o means "being proportional to”.
• Equation 6 characterizes a dynamic system with one state variable g and two simultaneous measurements / and e. Under linear conditions and Gaussian assumptions on the state noise and measurement noise, an optimal solutions in closed-form expression may be obtained by using the Kalman filter method.
• the particle filtering framework is adopted as a suboptimal alternative to tackle the problem regardless of the underlying distribution.
• the particle filter provides a more multimodal framework that allows to integrate observations of different types (i.e., different distributions). Particle filtering based methods approximate the posterior probability density p(g t
• N the number of particles in the posterior distribution.
• gt-i ) is chosen as the proposal distribution resulting in a bootstrap filter with an easy implementation. In this way, weight updating is simply reduced to the computation of the likelihood.
• resampling may also been adopted according to a variant as to replace the old set of particles by the new set of equally weighted particles according to their important weights.
• eye movements there are two types of eye movements: smooth pursuit and saccade.
• the former denotes a gradual movement which typically occurs when one focuses on a moving object while the latter is a very fast jump from one eye position to another.
• Other types of eye movement such as fixation or vergence for instance, can be loosely classified into these two types.
• smooth-pursuit eye movements can be successfully modeled by a distribution whose peak is centered at the previous gaze position state g t -i (e.g., Gaussian distribution). Otherwise, for a saccadic eye movement i.e., a movement to an arbitrary position on the screen, another Gaussian distribution centered at the previous gaze position can also be used but with a much larger scale to describe the uncertainty property of the saccade.
• the state transition should be modeled by a Gaussian mixture of two densities.
• a unique distribution for both types of eye movement is adopted :
• pi -i ) »toi-i; ifo. ⁇ /( ⁇ T 2 )).
• It) represents the gaze probability given only the image frame which can be directly obtained from the saliency map.
• e t ) denotes the likelihood distribution given the current eye image. In the context of object tracking, this likelihood is often computed by a similarity measure between the current observation and the existing object model. In line with these works, in the context of gaze estimation, we model the likelihood p(gt
• ⁇ is the parameter which determines the "peaky shape" of the distribution
• d e t ⁇ e t - e t ⁇ 2 denotes a distance measure between the current observation e t and the estimated eye image e t (corresponding to the particle position g t .).
• the likelihood value p(g t le t ) given the observation e f is exponentially proportional to the distance between g t and the "observed gaze position" g et which is derived from the eye center location via equations
• FIG. 5 illustrates a graphical user interface 5 (GUI) adapted to assist a user in controlling the result of the estimation of the gaze position, according to a particular and non-limitative embodiment of the invention.
• the GUI 5 comprises a first part 51 for displaying an image of the face of the viewer who is watching at a video image, also called stimulus image.
• a frame 51 1 may be generated as to graphically show the part of the image of the face of the viewer corresponding to an eye of the viewer.
• the centre of the eye may also been graphically identifier, for example via a red point. This may enable to check that the detection of the centre of the eye works well.
• the video image the viewer is watching at is displayed into a second part 53 of the GUI 5.
• the first gaze position is advantageously showed on the video image 53 with a specific graphical identifier 531 , for example a red point.
• the user may then check with the viewer if the position of the graphical identifier corresponds actually to the part of the video image 53 the viewer is watching at.
• the user and the viewer may be one and a same person.
• a third part 52 of the GUI illustrates the saliency map corresponding to the video image displayed in the second part 53 of the GUI 5.
• the maximum salient peak is advantageously graphically identified on this third part 52, for example via a blue point.
• the second gaze position may also been identified on the video image 53, for example with the same graphical identifier that on the third part 52.
• the third gaze position corresponding to the fusion of the first gaze position and of the second gaze position, is also illustrated on the video image 53, for example via a yellow point.
• the GUI 5 also comprises a fourth part 54 showing curves representative of the evolution of the mean error over time (for example in degree per millimetre) for each configuration (i.e. detection of the centre of the right eye, detection of the centre of the right eye, estimation of the first gaze position, estimation of the second gaze position and estimation of the third gaze position), one specific color being used for each curve.
• the GUI 5 enables a user (and/or the viewer) to see directly on the screen the results of the different detection and estimation performed by the system, as well as to visually check the validity of the results.
• Figure 6 diagrammatically illustrates a hardware embodiment of a device 6 configured for determining the gaze of a viewer watching at a video image on a screen, according to a particular and non-limitative embodiment of the invention.
• the device 6 is also configured for the creation of display signals of one or several images, for example images representative of the Graphical User Interface 5.
• the device 6 corresponds for example to a personal computer (PC), a laptop, a tablet, a Smartphone, a games console or a multimedia terminal.
• PC personal computer
• laptop a laptop
• a tablet a tablet
• a Smartphone a games console or a multimedia terminal.
• the device 6 comprises the following elements, connected to each other by a bus 65 of addresses and data that also transports a clock signal:
• microprocessor 61 or CPU
• a graphics card 62 comprising:
• GPUs Graphical Processor Units
• GRAM Graphical Random Access Memory
• I/O devices 64 such as for example a keyboard, a mouse, a webcam, and
• the device 6 also comprises a display device 63 of display screen type directly connected to the graphics card 62 to display synthesized images calculated and composed in the graphics card, for example live.
• a display device is external to the device 6 and is connected to the device 6 by a cable or wirelessly for transmitting the display signals.
• the device 6, for example the graphics card 62 comprises an interface for transmission or connection (not shown in figure 6) adapted to transmit a display signal to an external display means such as for example an LCD or plasma screen or a video-projector.
• register used in the description of memories 621 , 66, and 67 designates in each of the memories mentioned, both a memory zone of low capacity (some binary data) as well as a memory zone of large capacity (enabling a whole program to be stored or all or part of the data representative of data calculated or to be displayed).
• the microprocessor 61 When switched-on, the microprocessor 61 loads and executes the instructions of the program contained in the RAM 67.
• the random access memory 67 notably comprises:
• - data 671 representative of one or several eye images acquired by a camera or received, wirelessly or via a cable, from another device;
• mapping function for example statistical properties of the human gaze distribution according to the viewed content
• the algorithms implementing the steps of the method specific to the invention and described hereafter are stored in the memory GRAM 621 of the graphics card 62 associated with the device 6 implementing these steps.
• the graphic processors 620 of the graphics card 62 load these parameters into the GRAM 621 and execute the instructions of these algorithms in the form of microprograms of "shader" type using HLSL (High Level Shader Language) language or GLSL (OpenGL Shading Language) for example.
• the random access memory GRAM 621 notably comprises:
• parameters representative of the location of the centre of the eye(s) for example the coordinates of the centre(s)
• parameters representative of the first gaze position for example the coordinates x,y of the gaze position
• parameters representative of the second and third gaze position are stored in the register 6214 or in other registers
• the GRAM also comprises in registers data representative of saliency map(s) associated with video image(s) (which is (are) associated with the eye image(s)) as well as parameters representative of the maximum salient peak(s), parameters representative of the second gaze position(s) and parameters representative of the third gaze position(s).
• the data 671 of the eye image(s) and the parameters 672 representative of the mapping function are not loaded into the GRAM 621 and are processed by the CPU 61 .
• parameters representative of the location of the centre of the eye(s) and the parameters representative of the first gaze position (as well as the parameters representative of the second and third gaze position when computed) are stored in the RAM 67 and not in the GRAM 621 .
• the power supply 68 is external to the device 6.
• Figure 7 illustrates a method for estimating the gaze position of a viewer implemented in the device 6, according to a non-restrictive advantageous embodiment of the invention.
• the different parameters of the device 6 are updated.
• the location(s) of the centre of the eye(s) of a viewer watching at a video content displayed on a screen is (are) detected.
• the video content displayed on the screen may be any video image or sequence of video images or any content comprising textual and/or graphical elements, like a web page, a picture, etc.
• the location of the centre of the eye(s) is detected by analyzing one or more images of the eye(s) of the viewer, acquired for example with a webcam while the viewer is watching at the video image(s).
• the image(s) of the eye(s) may be an image of the face of the viewer in which the eye(s) are detected in any way known by the skilled person in the art.
• the location(s) of the centre of the eye(s) is for example detected by using the Hough Transform (HT) method or any method based on edge (gradient) detection and/or machine learning algorithm.
• HT Hough Transform
• the location of the centre of the eye(s) is detected by converting the eye image(s) (i.e. the part(s) of the image(s) of the viewer comprising the eye(s)) into heat map(s), one heat map being associated with one eye image.
• a heat map corresponds advantageously to a conversion of a RGB eye image into a pixel image represented in the YC b C r color space.
• the heat map corresponds to a conversion of the RGB eye image into a pixels image represented in the YUV color space or in the RGB color space.
• a value computed for example with the equation 1 is associated with each pixel of the heat map.
• First pixels 21 2 of the heat map having an associated value greater than a first threshold value T1 (for example equal to any value comprised between 0.98 and 1 ) are selected, which coordinates are for example stored in a memory of the RAM-type or G RAM-type.
• Second pixels 21 3 belonging to the neighbourhood of the first pixels are then selected, the selected second pixels being the pixels of the neighbourhood of the first pixels having an associated value greater than a second threshold value T2 (for example equal to any value comprised between 0.90 and 0.95), which coordinates are for example stored in a memory of the RAM-type or G RAM- type.
• the coordinates of the centre of the eye(s) are then determined as being a weighted average of the coordinates of the first and second pixels, by using for example equations 3 and 4.
• Advantage of this variant based on color cues is that the method is simple and computation implied by this variant is quick, which enable for example real-time implementation.
• the heat map(s) obtained from the conversion of the eye image(s) is (are) filtered, for example with a Gaussian filter or the extended Kalman filter before the determination of the first and second pixels used for computing the location(s) of the centre of the eye(s).
• a Gaussian filter enables to eliminate some noises from the heat map which is smoothes by the filtering.
• the use of a Gaussian filter has the advantage of stabilizing the detection result(s) (i.e., avoiding wrong detection due to for example eyelid, eye glass, reflexion).
• a first position of the gaze of the viewer is determined by using the detected location of the centre of the eye as described hereinabove with regard to step 71 and by using a mapping function based on centre-bias property of human gaze distribution.
• the use of such a mapping function with the detected location of the centre of the eye enables to avoid the need of calibrating the system used for determining the gaze position, such a calibration being usually to be performed by the user before any determination of the gaze (for example with a series of test images).
• the determination of the first gaze position is refined by using a second gaze position determined by using the saliency map computed from the video image the viewer is watching at when the first position of the gaze is determined.
• a third position of the gaze resulting from a combination/fusion of the first gaze position and the second gaze position is then obtained.
• the third gaze position is for example computed by averaging the first and second gaze positions or by averaging the first and second gaze positions to which are assigned different weights.
• the variance of the saliency map is used as to adapt parameters of the equation used for determining the first gaze position.
• the expression "fusion" of the first gaze position and of the second gaze position may be interpreted as meaning averaging, weighted averaging or adapting parameters used for computing the first gaze position.
• a particle filtering is implemented as to determine the first gaze position, the particle filtering enabling to take into account the result(s) of first gaze position previously determined (in a temporal point of view) when computing a current first gaze position.
• the particle filtering method is implemented by using the determination of the first gaze position and also the determination of the second gaze position.
• the movement(s) of the head of the viewer is (are) taken into account when determining the first position of the gaze of the viewer.
• a head movement is for example detected when the difference between a current eye centre location and its mean values are sufficiently large, for example: Where T is set proportionally to the distance between the user and the display which may be implicitly derived via the size of the detected viewer's face.
• the eye centre location i.e. the means values (x c , y c ) and the standard deviation (a Xc , a yc ) is re-initialized to 0.
• Steps 71 and 72 are advantageously reiterated for each newly received or acquired eye image.
• the invention is not limited to a method for but also extends to any device implementing this method and notably any devices comprising at least one CPU and/or at least one GPU.
• the implementation of calculations necessary to the implementation of the method's steps is not limited either to an implementation in shader type microprograms but also extends to an implementation in any program type, for example programs that can be executed by a CPU type microprocessor.
• the invention also relates to a method (and a device configured) for estimating the likelihood of the gaze.
• the invention further relates to a method for adapting the content of the video image(s) watched by the viewer according to the result(s) of the determined gaze position(s) or to a method for controlling a user interface with the eye by using the determined gaze position.
• the implementations described herein may be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method or a device), the implementation of features discussed may also be implemented in other forms (for example a program).
• An apparatus may be implemented in, for example, appropriate hardware, software, and firmware.
• the methods may be implemented in, for example, an apparatus such as, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, Smartphones, tablets, computers, mobile phones, portable/personal digital assistants ("PDAs”), and other devices that facilitate communication of information between end-users.
• PDAs portable/personal digital assistants
• Implementations of the various processes and features described herein may be embodied in a variety of different equipment or applications, particularly, for example, equipment or applications associated with data encoding, data decoding, view generation, texture processing, and other processing of images and related texture information and/or depth information.
• equipment include an encoder, a decoder, a post-processor processing output from a decoder, a pre-processor providing input to an encoder, a video coder, a video decoder, a video codec, a web server, a set-top box, a laptop, a personal computer, a cell phone, a PDA, and other communication devices.
• the equipment may be mobile and even installed in a mobile vehicle.
• the methods may be implemented by instructions being performed by a processor, and such instructions (and/or data values produced by an implementation) may be stored on a processor-readable medium such as, for example, an integrated circuit, a software carrier or other storage device such as, for example, a hard disk, a compact diskette (“CD"), an optical disc (such as, for example, a DVD, often referred to as a digital versatile disc or a digital video disc), a random access memory (“RAM”), or a read-only memory (“ROM”).
• the instructions may form an application program tangibly embodied on a processor-readable medium. Instructions may be, for example, in hardware, firmware, software, or a combination.
• a processor may be characterized, therefore, as, for example, both a device configured to carry out a process and a device that includes a processor-readable medium (such as a storage device) having instructions for carrying out a process. Further, a processor-readable medium may store, in addition to or in lieu of instructions, data values produced by an implementation.
• implementations may produce a variety of signals formatted to carry information that may be, for example, stored or transmitted.
• the information may include, for example, instructions for performing a method, or data produced by one of the described implementations.
• a signal may be formatted to carry as data the rules for writing or reading the syntax of a described embodiment, or to carry as data the actual syntax-values written by a described embodiment.
• Such a signal may be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal.
• the formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream.
• the information that the signal carries may be, for example, analog or digital information.
• the signal may be transmitted over a variety of different wired or wireless links, as is known.
• the signal may be stored on a processor-readable medium.
• the present invention may be used in real-time applications.
• the device 6 described with respect to figure 6 is advantageously equipped with interaction means such as a keyboard a mouse, a joystick or any other modes for introduction of commands, vocal recognition being for instance also possible.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Physics & Mathematics (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Multimedia (AREA)
• Human Computer Interaction (AREA)
• General Engineering & Computer Science (AREA)
• Health & Medical Sciences (AREA)
• General Health & Medical Sciences (AREA)
• Ophthalmology & Optometry (AREA)
• Image Analysis (AREA)
• Eye Examination Apparatus (AREA)
• Position Input By Displaying (AREA)

Priority Applications (5)

Applications Claiming Priority (4)

Publications (1)

Family

ID=50179590

Family Applications (1)

Country Status (6)

Cited By (4)

Families Citing this family (44)

Family Cites Families (6)

• 2014
  • 2014-02-20 CN CN201480010593.8A patent/CN105027144A/zh active Pending
  • 2014-02-20 JP JP2015559469A patent/JP2016515242A/ja not_active Withdrawn
  • 2014-02-20 EP EP14706534.6A patent/EP2962251A1/en not_active Withdrawn
  • 2014-02-20 WO PCT/EP2014/053349 patent/WO2014131690A1/en not_active Ceased
  • 2014-02-20 US US14/769,814 patent/US9965860B2/en not_active Expired - Fee Related
  • 2014-02-20 KR KR1020157023391A patent/KR20150122666A/ko not_active Withdrawn

Non-Patent Citations (8)

Also Published As

Similar Documents

Legal Events