SE1950758A1

SE1950758A1 - Method and system for determining a refined gaze point of a user

Info

Publication number: SE1950758A1
Application number: SE1950758A
Authority: SE
Inventors: Geoffrey Cooper
Original assignee: Tobii Ab
Priority date: 2019-06-19
Filing date: 2019-06-19
Publication date: 2020-10-27
Also published as: CN112114659A; SE543229C2

Abstract

An eye tracking system, a head mounted device, a computer program, a carrier and a method in an eye tracking system for determining a refined gaze point of a user are disclosed. In the method a gaze convergence distance of the user is determined. Furthermore, a spatial representation of at least a part of a field of view of the user is obtained and depth data for at least a part of the spatial representation are obtained. Saliency data for the spatial representation are determined based on the determined gaze convergence distance and the obtained depth data, and a refined gaze point of the user is determined based on the determined saliency data.

Description

1l\/IETHOD AND SYSTEl\/l FOR DETERl\/IINING A REFINED GAZE POINT OF A USER TECHNICAL FIELD The present disclosure relates to the field of eye tracking. ln particular, the present disclosure relates to a method and system determining a refined gaze point of a user.

BACKGROUND Eye/gaze tracking functionality is included in increasing number of applications, such as virtualreality (VR) and augmented reality (AR) applications. By inclusion of such eye trackingfunctionality, an estimated gaze point of a user can be determined which in turn can be used as input to other functions.

When determining an estimated gaze point of a user in an eye tracking system, a jitter may arisein the signal representing the estimated gaze point of the user e.g. due to measurement errorsin the eye tracking system. Different gaze points of the user may be determined in differentmeasuring cycles over a period even though the user is actually focusing on the same point overthat period. ln US 2016/0291690 A1, saliency data for a field of view of a user are used togetherwith eye gaze direction of the user to more reliably determine a point of interest at which theuser is gazing. However, determining saliency data for a field of view of a user requiresprocessing and even if the saliency data are used, the point of interest determined may differ from the actual point of interest. lt would be desirable to provide an eye tracking technology that provides a more robust andaccurate gaze point than the known methods.

SUMMARY An object of the present disclosure is to provide a method and system, which seek to mitigate, alleviate, or eliminate one or more of the above-identified deficiencies in the art.

This object is obtained by a method, an eye tracking system, a head mounted device, a computer program and a carrier according to the appended claims. 2According to an aspect, a method in an eye tracking system for determining a refined gaze pointof a user is provided. ln the method, a gaze convergence distance of the user is determined, aspatial representation of at least a part of a field of view ofthe user is obtained, and depth datafor at least a part of the spatial representation are obtained. Saliency data are determined forthe spatial representation based on the determined gaze convergence distance and theobtained depth data, and a refined gaze point of the user is then determined based on the determined saliency data.

Saliency data provide a measure to attributes in the user's field of view and represented in thespatial representation indicating the attributes' likelihood to guide human visual attention.Determining saliency data for the spatial representation means that saliency data relating to at least a portion of the spatial representation are determined.

The depth data for the at least a part of the spatial representation indicate distances from theuser's eyes to objects or features in the field of view of the user corresponding to the at least apart of the spatial representation. Depending on the application, e.g. AR or VR, the distances are real or virtual.

The gaze convergence distance indicates a distance from the user's eyes at which a user isfocusing. The convergence distance can be determined using any method of determiningconvergence distance, such as methods based on gaze directions of the user's eyes and intersection between the directions or methods based on interpupillary distance.

Basing the determination of saliency data also on the determined gaze convergence distanceand the obtained depth data for at least a part of the spatial representation, enablesdetermining the saliency data faster and with less required processing. lt further enablesdetermining of a refined gaze point of the user that is a more accurate estimate of a point of interest of the user. ln embodiments, determining saliency data for the spatial representation comprises identifyinga first depth region of the spatial representation corresponding to obtaining depth data withina predetermined range including the determined gaze convergence distance. Saliency data are then determined for the first depth region ofthe spatial representation. 3The identified first depth region ofthe spatial representation corresponds to objects or featuresin the at least a part of the field of view ofthe user which are within the predetermined rangeincluding the determined gaze convergence distance. lt is generally more likely that the user islooking at one of these objects or features than at objects or features corresponding to regionsof the spatial representation with depth data outside the predetermined range. Consequently,it is beneficial to determine saliency data for the first depth region and to determine a refined gaze point based on the determined saliency data. ln embodiments, determining saliency data for the spatial representation comprises identifyinga second depth region of the spatial representation corresponding to obtained depth dataoutside the predetermined range including the gaze convergence distance, and refraining from determining saliency data for the second depth region of the spatial representation.

The identified second depth region of the spatial representation corresponds to objects orfeatures in the at least a part of the field of view of the user which are outside thepredetermined range including the determined gaze convergence distance. lt is generally lesslikely that the user is looking at one of these objects or features than at objects or featurescorresponding to regions of the spatial representation with depth data inside thepredetermined range. Consequently, it is beneficial to refrain from determining saliency datafor the second depth region in order to avoid processing which is likely to be unnecessary ormay even provide misleading results since the user is not likely looking at the objections and/orfeatures corresponding to regions of the spatial representation with depth data outside thepredetermined range. This will reduce used processing power for determining saliency data in relation to methods where saliency data are also determined without taking determined gaze convergence distance ofthe user and depth data for at least a part of the spatial representation. ln embodiments, determining a refined gaze point comprises determining the refined gaze pointof the user as a point corresponding to a highest saliency according to the determined saliencydata. A determined refined gaze point will thus be a point that in some respect is most likely todraw visual attention. Used together with determining saliency data for an identified first depthregion of the spatial representation corresponding to obtained depth data within a predetermined range including the determined gaze convergence distance, a determined 4refined gaze point will thus be a point that in some respect is most likely to draw visual attention within the first depth region. ln embodiments, determining saliency data for the spatial representation comprisesdetermining first saliency data for of the spatial representation based on visual saliency,determining second saliency data for the spatial representation based on the determined gazeconvergence distance and the obtained depth data, and determining saliency data based on thefirst saliency data and the second saliency data. The first saliency data may for example be basedon high contrast, vivid colour, size, motion etc. The different types of saliency data are combined after optional normalisation and weighting. ln embodiments, the method further comprises determining a new gaze convergence distanceof the user, determining new saliency data for the spatial representation based on the new gazeconvergence distance, and determining a refined new gaze point of the user based on the newsaliency data. Hence, a dynamic refined new gaze point can be determined based on new gazeconvergence distances determined over time. Several alternatives are contemplated such as forexample using only a current determined new gaze convergence distance or a mean of gaze convergence points determined over a predetermined period. ln embodiments, the method further comprises determining a plurality of gaze points of theuser, and identifying a cropped region of the spatial representation based on the determinedplurality of gaze points of the user. Preferably, determining saliency data then comprises determining saliency data for the identified cropped region of the spatial representation. lt is generally more likely that the user is looking at a point corresponding to the cropped regionthan at points corresponding to regions outside the cropped region. Consequently, it isbeneficial to determine saliency data for the cropped region and to determine a refined gaze point based on the determined saliency data. ln embodiments, the method further comprises refraining from determining saliency data forregions of the spatial representation outside the identified cropped region of the spatial representation. lt is generally less likely that the user is looking at a point corresponding to regions outside the cropped region than at points corresponding to the cropped region. Consequently, it is 5beneficial to refrain from determining saliency data for the regions outside the cropped regionin order to avoid processing which is likely to be unnecessary or may even provide misleadingresults since the user is not likely looking at points corresponding to regions outside the croppedregion. This will reduce used processing power for determining saliency data in relationmethods where saliency data are also determined without cropping based on determined gaze points of the user. ln embodiments, obtaining depth data comprises obtaining depth data for the identifiedcropped region ofthe spatial representation. By obtaining depth data for the identified croppedregion, and not necessarily depth data for regions outside the cropped region, saliency data canbe determined within the cropped region and based on the obtained depth data for theidentified cropped region only. Hence, the amount of processing needed can be reduced further for determining saliency data. ln embodiments, the method further comprises determining a respective gaze convergence distance for each of the plurality of determined gaze points of the user. ln embodiments, the method further comprises determining a new gaze point of the user. Oncondition that the determined new gaze point is within the identified cropped region,identifying a new cropped region being the same as the identified cropped region. ln alternative,on condition that the determined new gaze point is outside the identified cropped region,identifying a new cropped region including the determined new gaze point and being different from the identified cropped region. |fthe new determined gaze point ofthe user is determined to be within the identified croppedregion, the user is likely to look at a point within the cropped region. By maintaining the samecropped region in such a case, any determined saliency data based on the identified croppingregion can be used again. Hence, no further processing is needed for determining saliency based on the identified cropping region. ln embodiments, consecutive gaze points of the user are determined in consecutive timeintervals, respectively. Furthermore, for each time interval, it is determined ifthe user is fixatingor saccading. On condition the user is fixating a refined gaze point is determined. On condition the user is saccading, determining a refined gaze point is refrained from. |fthe user is fixating it 6is likely that the user is looking at a point at that time and hence, a refined gaze point is likelyrelevant to determine. lf on the other hand, the user is saccading, the user is not likely lookingat a point at that time and hence, a refined gaze point is not likely relevant to determine. Theseembodiments will enable reduction of processing whilst at the same time determine a refine gaze point if it is likely that such a determining is relevant to determine. ln embodiments, consecutive gaze points of the user are determined in consecutive timeintervals, respectively. Furthermore, for each time interval it is determined if the user is insmooth pursuit. On condition the user is in smooth pursuit, consecutive cropped regionsincluding the consecutive gaze points, respectively, are determined such that the identifiedconsecutive cropped regions follow the smooth pursuit. lf smooth pursuit is determined,consecutive cropped regions can be determined with little additional processing needed if cropped regions are determined to follow the smooth pursuit. ln embodiments, the spatial representation is an image, such as a 2D image of the real world,3D image of the real world, 2D image of a virtual environment, or 3D image of a virtualenvironment. The data could come from a photo sensor, a virtual 3D scene, or potentially another type of image sensor or spatial sensor.

According to a second aspect, an eye tracking system for determining a gaze point of a user isprovided. The eye tracking system comprises a processor and a memory, said memorycontaining instructions executable by said processor. The eye tracking system is operative todetermine a gaze convergence distance ofthe user and obtain a spatial representation if at leasta part of a field of view of the user. The eye tracking system is further operative to obtain depthdata for at least a part of the spatial representation, and determine saliency data for the spatialrepresentation based on the determined gaze convergence distance and the obtained depthdata. The eye tracking system is further operative to determine a refined gaze point of the user based on the determined saliency data.

Embodiments of the eye tracking system according to the second aspect may for exampleinclude features corresponding to the features of any of the embodiments of the method according to the first aspect. 7According to a third aspect, a head mounted device for determining a gaze point of a user isprovided. The head mounted device comprises a processor and a memory, said memorycontaining instructions executable by said processor. The head mounted device is operative todetermine a gaze convergence distance of the user, and obtain a spatial representation of atleast a part of a field of view ofthe user. The head mounted device is further operative to obtaindepth data for at least a part of the spatial representation, and determine saliency data for thespatial representation based on the determined gaze convergence distance and the obtaineddepth data. The head mounted device is further operative to determine a refined gaze point of the user based on the determined saliency data. ln embodiments, the head mounted device further comprises one of a transparent display and a non-transpa rent display.

Embodiments of the head mounted device according to the third aspect may for exampleinclude features corresponding to the features of any of the embodiments of the method according to the first aspect.

According to a fourth aspect, a computer program is provided. The computer programcomprising instructions which, when executed by at least one processor, cause the at least oneprocessor to determine a gaze convergence distance of the user, and obtain a spatialrepresentation of a field of view of the user. The at least one processor is further caused toobtain depth data for at least a pa rt ofthe spatial representation, and determine saliency datafor the spatial representation based on the determined gaze convergence distance and theobtained depth data. The at least one processor is further caused to determine a refined gaze point ofthe user based on the determined saliency data.

Embodiments ofthe computer program according to the fourth aspect may for example includefeatures corresponding to the features of any of the embodiments of the method according to the first aspect.

According to a fifth aspect, a carrier comprising a computer program according to the fourthaspect is provided. The carrier is one of an electronic signal, optical signal, radio signal, and a computer readable storage medium. 8Embodiments of the carrier according to the fifth aspect may for example include featurescorresponding to the features of any of the embodiments of the method according to the first aspect.

BRIEF DESCRIPTION OF THE DRAWINGS These and other aspects will now be described in the following illustrative and non-limiting detailed description, with reference to the appended drawings.

Figure 1 is a flowchart illustrating embodiments of a method according to the present disclosure.

Figure 2 includes images illustrating results from steps of embodiments of a method according to the present disclosure.

Figure 3 is a flowchart illustrating steps of a method according to the present disclosure.

Figure 4 is a flowchart illustrating further steps of a method according to the present disclosure.

Figure 5 is a flowchart illustrating yet further steps of a method according to the present disclosure.

Figure 6 is a block diagram illustrating embodiments of an eye tracking system according to the present disclosure.

All the figures are schematic, not necessarily to scale, and generally only show parts which arenecessary in order to elucidate the respective example, whereas other parts may be omitted or merely suggested.

DETAILED DESCRIPTION Aspects of the present disclosure will be described more fully hereinafter with reference to theaccompanying drawings. The method, the eye tracking system, the head mounted device, thecomputer program and the carrier disclosed herein can, however, be realized in many differentforms and should not be construed as being limited to the aspects set forth herein. Like numbers in the drawings refer to like elements throughout. 9Saliency data provide a measure to attributes in the user's field of view and represented in thespatial representation indicating the attributes' likelihood to guide human visual attention.Some of the most likely attributes to do so are, for example, colour, motion, orientation, andscale. A saliency model can be used to determine such saliency data. Saliency models typicallypredict what attracts human visual attention. Many saliency models determines saliency datafor a region based e.g. how different the region is from what surrounds it, based on a model of a biologically plausible set of features that mimic early visual processing. ln a spatial representation of a field of view of a user, a saliency model can be used to identifydifferent visual features that to different extent contribute to the attentive selection of astimulus, and produce saliency data indicating saliency of different points in the spatialrepresentation. Based on the determined saliency data, a refined gaze point can then be determined that more likely correspond to a point of interest at which the user is gazing.

When saliency data are determined in a saliency model, on, for example a spatial representationin the form of a 2D image, each pixel of the image may be analysed for how salient it is accordingto a certain visual attribute, and each pixel is assigned a saliency value for that attribute. Oncesaliency is calculated for each pixel, the difference in saliency between pixels is known.Optionally, salient pixels may then be grouped together into salient regions to simplify the feature result.

Prior art saliency models typically use a bottom-up approach to calculate saliency, using animage as input to the model. The inventor has realized that additional top-down, determinedinformation about a user from an eye tracking system can be used in order to achieve a moreaccurate estimate of the point of interest at which the user is gazing and/or make the saliencymodel to run faster. Top-down information provided by the eye tracker may be one or moredetermined gaze convergence distances of the user. Further top-down information provided bythe eye tracker may be one or more determined gaze points ofthe user. Saliency data are then determined for the spatial representation based on the top down information.

Figure 1 is a flowchart illustrating embodiments of a method 100 in an eye tracking system fordetermining a refined gaze point of a user. ln the method a gaze convergence distance of theuser is determined 110. The gaze convergence distance indicates a distance from the user's eyes at which a user is focusing. The convergence distance can be determined using any method of determining convergence distance, such as methods based on gaze directions ofthe user's eyesand intersection between the directions, methods based on time of flight measurements, andmethods based on interpupillary distance. The eye tracking system in which the method 100 isperformed in may e.g. be a head mounted system, such as augmented reality (AR) glasses orvirtual reality (VR) glasses, but could also be an eye tracking system that is not head mountedbut rather remote from the user. Further, the method comprises the step of obtaining 120 aspatial representation of at least a part of a field of view ofthe user. The spatial representationcould for example be a digital image of at least a part of the field of view of the user capturedby one or more cameras in or remote from the eye tracking system. Furthermore, depth datafor at least a part of the spatial representation are obtained 130. Depth data for the spatialrepresentation of the user's field of view indicate real or virtual distances from the user's eyesto points of parts of objects or features in the field of view ofthe user. The depth data are linkedto points or parts of the spatial representation corresponding to the points of parts of objectsor features of the field of view ofthe user, respectively. Hence, a point in or region ofthe spatialrepresentation which is a representation of a point on or part of an object or feature in the fieldof view ofthe user will have depth data indicating the distance from the user's eyes to the pointon or part of the object or feature. For example, the spatial representation may be two images(stereo images) taken from two outward facing cameras at a lateral distance in a head mounteddevice. A distance from the user's eyes to points or parts of objects or features in the field ofview of the user can then be determined by analysis of the two images. The thus determineddepth data can be linked to points or parts of the two images corresponding to the points ofparts of the objects or features in the field of view of the user, respectively. Other examples ofspatial representations are possible, such as 3D mesh based on Time-of-flight measurements orsimultaneous localization and mapping (SLAM). Based on the determined gaze convergencedistance and the obtained depth data, saliency data are determined 140 for the spatialrepresentation. Finally, a refined gaze point of the user is then determined 150 based on the determined saliency data.

Depending on the application, depth data for the spatial representation of the user's field ofview indicate real or virtual distances from the user's eyes to points or parts of objects orfeatures in the field of view. ln applications where the spatial representation includes representations of real world objects or features of at least part of the field of view of the user, 11the distances indicated by depth data are typically real, i.e. they indicate real distances from theuser's eyes to the real world objects or features represented in the spatial representation. lnapplications where the spatial representation includes representations of virtual objects orfeatures of at least part of the field of view of the user, the distances indicated by depth dataare typically virtual as the user perceives them, i.e. they indicate virtual distances from the user's eyes to the virtual objects or features represented in the spatial representation.

The determined gaze convergence distance and the obtained depth data can be used toenhance the determining of saliency data such that they provide refined information on whichthe determining of the refined gaze point can be based. For example, one or more regions ofthe spatial representation can be identified that correspond to parts of objects or features inthe field of view with distances from the users eyes that are consistent with the determinedgaze convergence distance. The identified one or more regions can be used to refine the saliencydata by adding information indicating which regions ofthe spatial representation are more likelyto correspond to a point of interest at which the user is gazing. Furthermore, the identified oneor more regions of the spatial representation can be used as form of filter before saliency dataare determined for the spatial representation. ln this way, saliency data are determined onlyfor such regions of the spatial representation that correspond to parts of objects or features inthe field of view with distances from the users eyes that are consistent with the determined gaze convergence distance.

Specifically, determining 140 saliency data for the spatial representation can compriseidentifying 142 a first depth region of the spatial representation corresponding to obtaineddepth data within a predetermined range including the determined gaze convergence distance.The range can be set to be broader or narrower depending on e.g. the accuracy of thedetermined gaze convergence distance, the accuracy of obtained depth data and on otherfactors. Saliency data are then determined 144 for the first depth region of the spatial representation.

The identified first depth region ofthe spatial representation corresponds to objects or featuresin the at least a part of the field of view of the user which are within the predetermined rangeincluding the determined gaze convergence distance. lt is generally more likely that the user is looking at one of these objects or features than at objects or features corresponding to regions 12of the spatial representation with depth data outside the predetermined range. Consequently,identification ofthe first depth region provides further information useful for identifying a point of interest at which the user is gazing. ln addition to determining the first depth region, determining saliency data for the spatialrepresentation preferably comprises identifying a second depth region of the spatialrepresentation corresponding to obtained depth data outside the predetermined rangeincluding the gaze convergence distance. ln contrast to the first depth region, no saliency dataare determined for the second depth region of the spatial representation. lnstead, afteridentification of the second depth region, the method explicitly refrains from determining saliency data for the second depth region.

The identified second depth region of the spatial representation corresponds to objects orfeatures in the at least a part of the field of view of the user which are outside thepredetermined range including the determined gaze convergence distance. lt is generally lesslikely that the user is looking at one of these objects or features than at objects or featurescorresponding to regions of the spatial representation with depth data inside thepredetermined range. Consequently, it is beneficial to refrain from determining saliency datafor the second depth region in order to avoid processing which is likely to be unnecessary ormay even provide misleading results since the user is not likely looking at the objections and/or features corresponding to regions of the spatial representation with depth data outside the predetermined range.

Typically, the method 100 is performed repeatedly to produce new refined gaze points overtime as the point of interest the user is gazing at is normally changed over time. The method100 thus typically further comprises determining a new gaze convergence distance of the user,determining new saliency data for the spatial representation based on the new gazeconvergence distance, and determining a refined new gaze point of the user based on the newsaliency data. Hence, a dynamic refined new gaze point is be determined based on new gazeconvergence distances determined over time. Several alternatives are contemplated such as forexample using only a current determined new gaze convergence distance or a mean of gaze convergence points determined over a predetermined period. Furthermore, if the user's field 13of view also changes over time, a new spatial representation is obtained and new depth data for at least a part of the new spatial representation are obtained.

Additional top-down information provided by the eye tracker may be one or more determinedgaze points of the user. The method 100 may further comprise determining 132 a plurality ofgaze points of the user, and identifying 134 a cropped region of the spatial representation basedon the determined plurality of gaze points ofthe user. The plurality of gaze points are generallydetermined over a period. The determined individual gaze points ofthe determined plurality ofgaze points may typically differ from each other. This may be due to the user looking at differentpoints over the period but could also be due to errors in the determined individual gaze points,i.e. the user may actually be looking at the same point over the period but the determinedindividual gaze points still differ from each other. The cropped region preferably includes all ofthe determined plurality of gaze points. The size of the cropped region may depend e.g. onaccuracy of determined gaze points such that higher accuracy will lead to a smaller cropped region. lt is generally more likely that the user is looking at a point corresponding to the cropped regionthan at points corresponding to regions outside the cropped region. Consequently, it isbeneficial to determine saliency data for the cropped region and to determine a refined gazepoint based on the determined saliency data. Furthermore, since it is more likely that the useris looking at a point corresponding to the cropped region than at points corresponding toregions outside the cropped region, determining saliency data for regions of the spatialrepresentation outside the identified cropped region of the spatial representation can berefrained from. Each region of the spatial representation outside the identified cropped regionfor which saliency data are not determined, will reduce the amount of processing needed inrelation to determining saliency data for all regions of the spatial representation. Generally, thecropped region can be made substantially smaller than the whole of the spatial representationwhilst the probability that the user is looking at a point within the cropped region is maintainedhigh. Hence, refraining from determining saliency data for regions of the spatial representation outside the cropped region can reduce the amount of processing substantially. ln addition or alternative to using the identified cropped region in determining of saliency data, the cropped region can be used when obtaining depth data. For example, since it is more likely 14that the user is looking at a point corresponding to the cropped region than at pointscorresponding to regions outside the cropped region, depth data can be obtained for theidentified cropped region, and not necessarily for regions outside the cropped region. Saliencydata can then be determined within the cropped region and based on the obtained depth datafor the identified cropped region only. Hence, the amount processing needed for obtaining depth data and determining saliency data can be reduced.

The method 100 may further comprise determining at least a second gaze convergence distanceof the user. The first depth region of the spatial representation is then identified correspondingto depth data within a range determined based on said determined gaze convergence distanceand the determined at least second gaze convergence distance. Saliency data are then determined for the first depth region of the spatial representation.

The identified first depth region ofthe spatial representation corresponds to objects or featuresin the at least a part of the field of view ofthe user which are within a range determined basedon the determined gaze convergence distance and the determined at least second gazeconvergence distance. lt is generally more likely that the user is looking at one ofthese objectsor features than at objects or features corresponding to regions of the spatial representationwith depth data outside the range. Consequently, identification of the first depth region provides further information useful for identifying a point of interest at which the user is gazing.

There are several alternatives for determining the range based on the determined gazeconvergence distance and the determined at least second gaze convergence distance. ln a firstexample, a maximum gaze convergence distance and a minimum gaze convergence distance ofthe determined gaze convergence distance and the determined at least second gazeconvergence distance may be determined. The maximum and minimum gaze convergencedistances may then be used to identify the first depth region of the spatial representationcorresponding to obtained depth data within a range including the determined maximum andminimum gaze convergence distances. The range can be set to be broader or narrowerdepending on e.g. the accuracy ofthe determined gaze convergence distances, the accuracy ofobtained depth data and on other factors. As an example, the range can be set to be from thedetermined minimum gaze convergence distance to the maximum gaze convergence distance.

Saliency data are then determined for the first depth region of the spatial representation. ln the first example, the identified first depth region of the spatial representation correspondsto objects or features in the at least a part of the field of view of the user which are within arange including the determined maximum and minimum gaze convergence distances. lt isgenerally more likely that the user is looking at one of these objects or features than at objectsor features corresponding to regions of the spatial representation with depth data outside therange. Consequently, identification of the first depth region according to the first example provides further information useful for identifying a point of interest at which the user is gazing. ln a second example, a mean gaze convergence distance of the determined gaze convergencedistance and the determined at least second gaze convergence distance of the user may bedetermined. The mean gaze convergence distance may then be used to identify the first depthregion of the spatial representation corresponding to obtained depth data within a rangeincluding the determined mean gaze convergence distances. The range can be set to be broaderor narrower depending on e.g. the accuracy ofthe determined gaze convergence distance, theaccuracy of obtained depth data and on other factors. Saliency data may then determined for the first depth region of the spatial representation. ln the second example, the identified first depth region of the spatial representationcorresponds to objects or features in the at least a part of the field of view of the user which arewithin the range including the mean gaze convergence distance. lt is generally more likely thatthe user is looking at one of these objects or features than at objects or features correspondingto regions of the spatial representation with depth data outside the range. Consequently,identification of the first depth region according to the second example provides further information useful for identifying a point of interest at which the user is gazing.

The refined gaze point ofthe user can be determined 150 as a point corresponding to a highestsaliency according to the determined saliency data. A determined refined gaze point will thusbe a point that in some respect is most likely to draw visual attention. Used together withdetermining 144 saliency data for an identified first depth region of the spatial representationcorresponding to obtained depth data within a predetermined range including the determinedgaze convergence distance, a determined refined gaze point will thus be a point that in somerespect is most likely to draw visual attention within the first depth region. This can be further combined with identifying 132 a plurality of gaze points and identifying 134 a cropped region 16comprising the determined plurality of gaze points and obtaining 130 depth data for only thecropped region. Furthermore, saliency data may be determined 146 only for the identifiedcropped region and optionally only for the identified depth region or combined with saliencydata for the identified depth region such that saliency data are produced only for the depthregion within the cropped region. A determined refined gaze point will thus be a point that insome respect is most likely to draw visual attention within the first depth region within the cropped region.

Determining saliency data for the spatial representation may comprise determining firstsaliency data for of the spatial representation based on visual saliency, determining secondsaliency data for the spatial representation based on the determined gaze convergence distanceand the obtained depth data, and determining saliency data based on the first saliency data andthe second saliency data. Visual saliency is an ability of an item or an item in an image to attractvisual attention (bottom-up, i.e. the value is not known but could be guessed from algorithms).ln more detail, visual saliency is a distinct subjective perceptual quality that makes some itemsin the world stand out from their neighbours and immediately grab our attention. The visualsaliency may be based on colour, contrast, shape, orientation, motion or any other perceptual characteristic.

Once saliency data have been computed for the different saliency features, such as the visualsaliency and depth saliency based on determined gaze convergence distance and the obtaineddepth data, they may be normalized and combined to form a master saliency result. Depthsaliency relates to the depth at which the user is looking (top-down, i.e. the value is known).Distances conforming with a determined convergence distance are considered to be moresalient. When combining saliency features, each feature can be weighted equally or havedifferent weights according to which features are estimated to have the most impact on visualattention and/or which features had the highest maximum saliency value compared to anaverage or expected value. The combination of saliency features may be determined by aWinner-Take-All mechanism. Optionally, the master saliency result can be translated into amaster saliency map: a topographical representation of overall saliency. This is a useful step forthe human observer, but not necessary if the saliency result is used as input for a computer program. ln the master saliency result, a single spatial location should stand out as most salient. 17Figure 2 includes images illustrating results from steps of embodiments of a method accordingto the present disclosure. A spatial representation of at least a part of a user's field of view inthe form of an image 210 is input to a method for determining a refined gaze point. A pluralityof gaze points are determined in the image 210, and a cropped region including the plurality ofdetermined gaze points is identified as illustrated in an image 215. Furthermore, stereo images220 is obtained for the at least part ofthe user's field of view is received and the cropped regionis identified as illustrated in an image 225 and depth data are obtained for the cropped regionbased on the stereo images 220 as illustrated in an image 230. A gaze convergence distance ofthe user is then received, which for the present example is 3.5 m, and a first depth region isdetermined as a region in the cropped region that corresponds to depth data within a rangearound the gaze convergence distance. ln the present example, the range is 3 m < x < 4 m andthe resulting first depth region is illustrated in an image 235. Visual saliency is determined forthe cropped region illustrated in 240 to produce saliency data illustrated in the form ofa saliencymap 245 of the cropped region. The saliency map 245 and the first depth region illustrated inthe image 235 are combined to a saliency map 250 for the first depth region within the croppedregion. A refined gaze point is the point identified as the point with highest saliency in the first depth region within the cropped region. This point is illustrated as a black dot in an image 255.

Figure 3 is a flowchart illustrating steps of a method according to the present disclosure.Generally, the flowchart illustrates steps in relation to identifying cropped regions over timebased on new determined gaze points, for example in relation to embodiments of a method asillustrated in Figure 1. An identified cropped region is a cropped region that has been previouslyidentified based on a plurality of previously determined gaze points. A new gaze point is thendetermined 310. On condition 320 that the determined new gaze point is within the identifiedcropped region, the identified cropped region is not changed but is continued to be used and anew gaze point is determined 310. An alternative way of seeing this is that a new cropped regionis determined to be the same as the identified cropped region. On condition 320 that thedetermined new gaze point is not inside the identified cropped region, i.e. outside the identifiedcropped region, a new cropped region is determined 330 comprising the determined new gaze point. ln this case the new cropped region will be different from the identified cropped region.

Generally, the flowchart illustrates steps in relation to determine refined gaze points over time 18based on new determined gaze points, for example in relation to embodiments of a method asillustrated in Figure 1. Consecutive gaze points of the user are determined 410 in consecutivetime intervals, respectively. Furthermore, for each time interval, it is determined 420 if the useris fixating or saccading. On condition 420 the user is fixating, a refined gaze point is determined430. On condition 420 the user is saccading, determining a refined gaze point is refrained from.|fthe user is fixating it is likely that the user is looking at a point at that time and hence, a refinedgaze point is likely relevant to determine. lf on the other hand, the user is saccading, the user isnot likely looking at a point at that time and hence, a refined gaze point is not likely relevant todetermine. ln relation to Figure 1, this could for example mean that the method 100 will only be performed in case it is determined that the user is fixating.

Figure 5 is a flowchart illustrating yet further steps of a method according to the presentdisclosure. Generally, the flowchart illustrates steps in relation to identifying cropped regionsover time based on determined gaze points, for example in relation to embodiments of amethod as illustrated in Figure 1. An identified cropped region is a cropped region that has beenpreviously identified based on a plurality of previously determined gaze points. Consecutivegaze points of the user are determined 510 in consecutive time intervals, respectively.Furthermore, for each time interval it is determined 520 if the user is in smooth pursuit. Oncondition 520 the user is in smooth pursuit, a new cropped region is determined 530 based onthe smooth pursuit. lf smooth pursuit is determined, consecutive cropped regions can bedetermined with little additional processing needed if cropped regions are determined to followthe smooth pursuit. For example, the consecutive cropped regions can have the same shapeand simply be moved in relation to each other in the same direction and speed as the smoothpursuit of the user. On condition 520 the user is not in smooth pursuit, a new cropped region including a plurality of gaze points comprising the determined new gaze point, is determined. ln embodiments, the spatial representation is an image, such as a 2D image of the real world,3D image of the real world, 2D image of a virtual environment, or 3D image of a virtualenvironment. The data could come from a photo sensor, a virtual 3D scene, or potentially another type of image sensor or spatial sensor.

Figure 1 comprises some steps that are illustrated in boxes with a solid border and some steps that are illustrated in boxes with a dashed border. The steps that are comprised in boxes with a 19solid border are operations that are comprised in the broadest example embodiment. The stepsthat are comprised in boxes with a dashed border are example embodiments that may becomprised in, or a part of, or are further operations that may be taken in addition to theoperations of the border example embodiments. The steps do not all need to be performed inorder and not all of the operations need to be performed. Furthermore, at least some of the steps may be performed in parallel.

I\/|ethods of for determining a refined gaze point of a user and steps therein as disclosed herein,e.g. in relation to Figures 1-5, may be implemented in an eye tracking system 600, e.g.implemented in a head mounted device, of Figure 6. The eye tracking system 600 comprises aprocessor 610, and a carrier 620 including computer executable instructions 630, e.g. in theform of a computer program, that, when executed by the processor 610, cause the eye trackingsystem 600 to perform the method. The carrier 620 may for example be an electronic signal,optical signal, radio signal, a transitory computer readable storage medium, and a non- transitory computer readable storage medium.

A person skilled in the art realizes that the present invention is by no means limited to theembodiments described above. On the contrary, many modifications and variations are possible within the scope ofthe appended claims.

Additionally, variations to the disclosed embodiments can be understood and effected by thoseskilled in the art in practicing the claimed invention, from a study ofthe drawings, the disclosure,and the appended claims. ln the claims, the word "comprising" does not exclude other elementsor steps, and the indefinite article "a" or "an" does not exclude a plurality. The terminology usedherein is for the purpose of describing particular aspects of the disclosure only, and is notintended to limit the invention. The division of tasks between functional units referred to in thepresent disclosure does not necessarily correspond to the division into physical units; to thecontrary, one physical component may have multiple functionalities, and one task may becarried out in a distributed fashion, by several physical components in cooperation. A computerprogram may be stored/distributed on a suitable non-transitory medium, such as an opticalstorage medium or a solid-state medium supplied together with or as part of other hardware,but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. The mere fact that certain measures/features are recited in mutually different dependent claims does not indicate that a combination of thesemeasures/features cannot be used to advantage. Method steps need not necessarily beperformed in the order in which they appear in the claims or in the embodiments describedherein, unless it is explicitly described that a certain order is required. Any reference signs in the claims should not be construed as limiting the scope.

Claims

21CLAll\/IS

1. A method in an eye tracking system for determining a refined gaze point of a usercomprising: determining a gaze convergence distance ofthe user; obtaining a spatial representation of at least a part of a field of view ofthe user; obtaining depth data for at least a part of the spatial representation; determining saliency data for the spatial representation based on the determined gazeconvergence distance and the obtained depth data; and determining a refined gaze point of the user based on the determined saliency data.

2. The method of claim 1, wherein determining saliency data for the spatialrepresentation comprises: identifying a first depth region of the spatial representation corresponding to obtaineddepth data within a predetermined range including the determined gaze convergence distance;and determining saliency data for the first depth region of the spatial representation.

3. The method of any one of claims 1 and 2, wherein determining saliency data for thespatial representation comprises: identifying a second depth region of the spatial representation corresponding toobtained depth data outside the predetermined range including the gaze convergence distance;and refraining from determining saliency data for the second depth region of the spatial representation.

4. The method of any one of claims 1-3, wherein determining a refined gaze pointcomprises:determining the refined gaze point of the user as a point corresponding to a highest saliency according to the determined saliency data.

5. 225. The method of any one of claims 1-4, wherein determining saliency data comprises:determining first saliency data for the spatial representation based on visual saliency;determining second saliency data for the spatial representation based on thedetermined gaze convergence distance and the obtained depth data; and determining saliency data based on the first saliency data and the second saliency data.

6. The method of any one of claims 1-5, further comprising: determining a new gaze convergence distance of the user; determining new saliency data for the spatial representation based on the new gazeconvergence distance; and determining a refined new gaze point ofthe user based on the new saliency data.

7. The method of any one of claims 1-6, further comprising:determining a plurality of gaze points of the user; andidentifying a cropped region of the spatial representation based on the determined plurality of gaze points of the user.

8. The method of claim 7, wherein determining saliency data comprises: determining saliency data for the identified cropped region ofthe spatial representation.

9. The method of any one of claims 7 and 8, further comprising:refraining from determining saliency data for regions of the spatial representation outside the identified cropped region ofthe spatial representation.

10. The method of any one of claims 7-9, wherein obtaining depth data comprises: obtaining depth data for the identified cropped region of the spatial representation.

11. The method of any one of claim 2, further comprising: determining at least a second gaze convergence distance of the user, wherein the first depth region of the spatial representation is identified correspondingto obtained depth data within a range based on said determined gaze convergence distance and the determined at least second gaze convergence distance of the user. 23

12. The method of any one of c|aims 7-11, further comprising: determining a new gaze point of the user; on condition that the determined new gaze point is within the identified cropped region,identifying a new cropped region being the same as the identified cropped region; or on condition that the determined new gaze point is outside the identified croppedregion, identifying a new cropped region including the determined new gaze point and being different from the identified cropped region.

13. The method of any one of c|aims 7-12, wherein consecutive gaze points of the userare determined in consecutive time intervals, respectively, further comprising, for each timeinterval: determining if the user is fixating or saccading; on condition the user is fixating, determining a refined gaze point; and on condition the user is saccading, refraining from determining a refined gaze point.

14. The method of any one of c|aims 7-12, wherein consecutive gaze points of the userare determined in consecutive time intervals, respectively, further comprising, for each timeinterval: determining if the user is in smooth pursuit; and on condition the user is in smooth pursuit, identifying consecutive cropped regionsincluding the consecutive gaze points, respectively, such that the identified consecutive cropped regions follow the smooth pursuit.

15. The method of any one of c|aims 1-14, wherein the spatial representation is an image.

16. An eye tracking system for determining a gaze point of a user comprising a processorand a memory, said memory containing instructions executable by said processor, whereby saideye tracking system is operative to: determine a gaze convergence distance of the user; obtain a spatial representation if at least a part of a field of view ofthe user; 24obtain depth data for at least a part ofthe spatial representation;determine saliency data for the spatial representation based on the determined gazeconvergence distance and the obtained depth data; and determine a refined gaze point ofthe user based on the determined saliency data.

17. A head mounted device for determining a gaze point of a user comprising a processorand a memory, said memory containing instructions executable by said processor, whereby saidhead mounted device is operative to: determine a gaze convergence distance of the user; obtain a spatial representation of at least a part of a field of view ofthe user; obtain depth data for at least a part ofthe spatial representation; determine saliency data for the spatial representation based on the determined gazeconvergence distance and the obtained depth data; and determine a refined gaze point ofthe user based on the determined saliency data.

18. The head mounted device of claim 17, further comprising one of a transparent display and a non-transparent display.

19. A computer program, comprising instructions which, when executed by at least oneprocessor, cause the at least one processor to: determine a gaze convergence distance of the user; obtain a spatial representation of a field of view ofthe user; obtain depth data for at least a part of the spatial representation; determine saliency data for the spatial representation based on the determined gazeconvergence distance and the obtained depth data; and determine a refined gaze point ofthe user based on the determined saliency data.

20. A carrier comprising a computer program according to claim 19, wherein the carrieris one of an electronic signal, optical signal, radio signal, and a computer readable storage medium.