EP4182775A1

EP4182775A1 - Computer method, device and programme for assisting in the positioning of extended reality systems

Info

Publication number: EP4182775A1
Application number: EP21746530.1A
Authority: EP
Inventors: Fabrice Malaingre
Original assignee: Theoris
Current assignee: Theoris
Priority date: 2020-07-16
Filing date: 2021-07-08
Publication date: 2023-05-24
Also published as: FR3112639A1; WO2022013492A1

Abstract

The present invention relates to positioning assistance for an extended reality system comprising a mobile part and provided with a device for locating the mobile part according to a first frame of reference defined by the location device, the extended reality system comprising a device for tracking a user's gaze. After having identified each point of a plurality of points in the real environment and the position of each point according to a second frame of reference, an estimate of the position of the point in the first frame of reference is obtained for each point of the plurality of points, the position of the point in the first frame of reference being estimated using the gaze tracking device. A correlation between the first frame of reference and the second frame of reference is subsequently obtained, according to the positions of the points of the plurality of points in the first frame of reference and in the second frame of reference.

Description

METHOD, DEVICE AND COMPUTER PROGRAM FOR POSITIONING AID FOR EXTENDED REALITY SYSTEMS

[0001] The invention relates to extended reality systems, for example to augmented reality or mixed reality systems, and to positioning assistance, to correlate real and virtual spaces and thus allow the addition of virtual objects to a user's perception of a real environment.

[0002] Extended reality (also known as extended reality), including in particular augmented reality (also known as augmented reality) and mixed reality (also known as mixed reality), aims to combine physical reality and digital content, for example by inserting one or more virtual objects into a real environment. Such an insertion typically consists of inserting virtual objects into a user's field of vision, for example by projecting a representation of these virtual objects onto a transparent surface such as spectacle lenses, or adding virtual objects to a representation of the real environment, usually in images from one or more video streams.

[0003] For these purposes, an extended reality system comprises a rendering engine (also known as a rendering engine) capable of enriching a real environment with a virtual scene, that is to say of adding a representation of virtual objects in reality, according to a particular point of view, typically the point of view of the user of the system. To manipulate the objects present in the virtual scene (virtual objects and/or models of real objects), each rendering engine uses a predetermined reference of its own, rigidly linked to the real environment to be enriched.

[0004] In many types of applications, the position and orientation of the virtual objects to be added to reality must be determined with precision according to elements of the real environment, for example according to the position of singular points of the real scene such as characteristic points of a real object to be enriched. Regardless of how the position and orientation of the virtual objects are determined, it is important to establish a link between the real and virtual environments to ensure spatial and temporal coherence between these environments, i.e. say that virtual objects are placed in the right place and at the right time in the user's perception of the real world.

[0005] It is thus necessary for an extended reality system to determine in real time the perception that their user has of the real environment. Furthermore, many extended reality systems being mobile due to the fact that, in general, at least a part of these systems is carried by their user, it is therefore necessary to determine in real time the position and orientation of the mobile part of the extended reality systems with respect to the real environment, in order to ensure spatial coherence and temporal between real scene and virtual scene. This involves, for example, determining the position and orientation of an extended reality headset in the real environment. For these purposes, a localization with 6 degrees of freedom (called 6DoF tracking in English terminology), implementing techniques of the SLAM type (acronym for Simultaneous Localization And Mapping in English terminology) or equivalent, is often used.

[0006] They are therefore often relative location devices which aim to determine a position and an orientation with respect to a previous position and orientation and not with respect to predetermined fixed points. In other words, the position and orientation of such extended reality systems are determined in a reference frame specific to the relative localization device, in which a cartography of the environment is constructed in real time by the device itself. Therefore, depending on the initialization conditions, a location relative to 6 degrees of freedom can lead to different positions and orientations even though this extended reality system is placed in the same place of a determined and stable real environment, and this the fact that the environment map generated in real time will be different.

[0007] The result is that if a location relative to 6 degrees of freedom can prove to be extremely efficient for tracking movements, it cannot be used independently, without taking into account the initialization conditions of the extended reality system. Indeed, the position and orientation coordinates determined by the relative localization device are expressed in a real reference, rigidly linked to the real environment, which is specific to it and which carries no semantics as to the real environment in which it evolves. This marker is different from the one used by the rendering engine.

[0008] It is therefore necessary, during an initialization phase, to establish a link between the marker defined and used by the rendering engine, generally called a virtual marker, and the marker defined and used by the localization device, generally called real benchmark. This may involve, for example, determining a passage matrix between, on the one hand, the predetermined virtual marker linked to the real environment and the real marker linked to this same environment but determined by the location device during its initialization. . Such a passage matrix is a correlation matrix between the real and virtual environments.

[0009] However, the pre-existing solutions for carrying out this real/virtual correlation are often unsatisfactory. By way of illustration, there are real/virtual correlation solutions using recognition of "fiduciary markers", for example solutions for identifying and locating QR codes (abbreviation for Quick Response codes in Anglo-Saxon terminology) positioned in the real environment. Although these solutions are generally robust, the placement of the fiduciary markers can prove to be problematic because it requires support, manipulations and transfers the problem of precision of the real/virtual correlation to the precision of placement of the markers themselves.

[0010] There are also solutions based on the recognition of “natural markers” according to which reference images (or patterns) are used to identify and locate patterns naturally present in the real environment. Still other solutions are based on "shape" recognition algorithms according to which elements present in the real environment are identified and located according to their 3D modelling. While these solutions have advantages in terms of ease of implementation for users of extended reality systems, they are generally not very robust with respect to changes in the environment (for example changes in shape or aspect of an object changing over time), but also with respect to variations in the lighting conditions of this same environment.

[0011]There is therefore a real need for a positioning aid system for extended reality applications, making it possible to establish a correlation between real and virtual environments.

The present invention helps in particular to solve these problems.

Disclosure of Invention

To this end, the invention proposes a positioning aid mechanism for extended reality systems, allowing a real/virtual correlation.

[0014] There is thus proposed a positioning aid method for an extended reality system, the extended reality system comprising at least one part that is mobile in a real environment and being provided with a location device for determining a position and /or a relative orientation of the moving part according to a first benchmark rigidly linked to the real environment, the first benchmark being defined by the location device, the extended reality system further comprising a device for tracking a user's gaze of the extended reality system, the method comprising:

- obtaining an identification of each point of a plurality of points of the real environment and of the position of each point of the plurality of points according to the same second reference frame rigidly linked to the real environment, different from the first mark; - for each point of the plurality of points, obtaining an estimate of the position of the point in the first marker, the position of the point in the first marker being estimated using the gaze tracking device; and

- the determination of a correlation between the first marker and the second marker, according to the positions of the points of the plurality of points in the first marker and in the second marker, said correlation being used to display at least one virtual object at a position and according to an orientation defined in the second marker.

The method according to the invention thus makes it possible to establish a link, in a simple, rapid way, and without particular knowledge, between a real marker linked to a real environment and defined by a localization device on the one hand, and a predefined virtual benchmark and also linked to this real environment on the other hand. The virtual marker is for example a marker used by a rendering engine.

[0016] According to particular embodiments, the mobile part comprises the location device, the location device being a simultaneous location and mapping device of the SLAM type.

[0017] Still according to particular embodiments, the gaze tracking device is rigidly linked to the location device.

[0018] Still according to particular embodiments, several estimates of the position of a point fixed by the user are obtained, the method further comprising the calculation of a position of a point fixed by the user from estimates obtained.

[0019] Still according to particular embodiments, the obtaining of several estimates of the position of a point fixed by the user comprises an evaluation of dispersion, the position estimates obtained being such that the dispersion of the position estimates obtained is below a threshold.

Still according to particular embodiments, obtaining several estimates of the position of a point fixed by the user comprises filtering of estimated positions, the filtering being based on a distribution of estimated positions.

Still according to particular embodiments, at least one position estimate is obtained from at least two lines of sight and, if the at least two lines of sight are not coplanar, the at least one estimated position is the position of the center of a segment perpendicular to the at least two lines of sight.

[0022] Still according to particular embodiments, the second marker is a marker used by a rendering engine of the extended reality system. Another aspect of the invention relates to a processing unit configured to execute each of the steps of the method described above. The advantages and variants of this processing unit are similar to those mentioned above.

[0024] A computer program, implementing all or part of the method described above, installed on pre-existing equipment, is in itself advantageous, since it makes it possible to establish in a simple and rapid manner a link between a real marker linked to a real environment, defined by a localization device, and a predefined virtual marker also linked to this real environment.

Thus, the present invention also relates to a computer program comprising instructions for implementing the method described above, when this program is executed by a processor.

[0026] This program can use any programming language (for example, an object language or other) and be in the form of interpretable source code, partially compiled code or fully compiled code. .

Another aspect relates to a non-transitory storage medium for a computer-executable program, comprising a set of data representing one or more programs, said one or more programs comprising instructions for, when executing said or several programs by a computer comprising a processing unit operationally coupled to memory means and to an input/output interface module, to execute all or part of the method described above.

Brief description of the drawings

Other characteristics, details and advantages of the invention will appear on reading the detailed description below. This is purely illustrative and should be read in conjunction with the attached drawings, in which:

Fig. 1

[0029] [Fig. 1] illustrates an example of an environment in which the invention can be implemented according to particular embodiments of the invention;

Fig. 2

[0030] [Fig. 2], comprising Figures 2a and 2b, schematically illustrates the estimation of the coordinates of a point fixed by a user, according to a projection on a transverse plane and according to a projection on a sagittal plane;

Fig. 3 [0031] [Fig. 3] schematically illustrates the estimation of the coordinates of a point fixed by a user from several lines of sight obtained from several positions of the user, when the latter moves while keeping his gaze on the same fixed point;

Fig. 4

[0032] [Fig. 4] illustrates an example of estimation of the coordinates of a point fixed by a user, from two lines of sight relative to this same point fixed by the user from two different positions;

Fig. 5

[0033] [Fig. 5], comprising FIGS. 5a and 5b, schematically illustrates the results of measurements carried out to estimate the coordinates of a singular point of a real environment, fixed by a user, using a gaze tracking device providing an estimate of the coordinates d a fixed point and a gaze tracking device providing a characterization of the line of sight passing through the fixed point, respectively;

Fig. 6

[0034] [Fig. 6] illustrates an example of steps for determining a passage matrix between a real marker defined by a localization device and a predetermined virtual marker, the real and virtual markers being rigidly linked to the same real environment;

Fig. 7

[0035] [Fig. 7] illustrates a first example of steps for estimating the coordinates of points measured by a gaze tracking device when the user is aiming at a singular point in a real environment from several distinct observation positions, in a real frame linked to a localization device of an extended reality system, the gaze tracking device used being capable of directly supplying the coordinates of the point fixed by the user;

Fig. 8

[0036] [Fig. 8] illustrates a second example of steps for estimating the coordinates of points measured by a gaze tracking device when the user is aiming at a singular point in a real environment from several distinct observation positions, in a real frame linked to a localization device of an extended reality system, the gaze tracking device used being limited to providing the characterization of the line of sight passing through the point fixed by the user; Fig. 9

[0037] [Fig. 9] illustrates an example of steps for calculating the coordinates of a set of measured points, representing the same point fixed by a user, from a set of lines of sight; Fig. 10

[0038] [Fig. 10] illustrates an example of steps for estimating the coordinates of a point fixed by a user from the coordinates of several measured points (cloud of measured points); and

Fig. 11 [0039] [Fig. 11] illustrates an example of a device that can be used to implement, at least partially, one embodiment of the invention, in particular the steps described with reference to Figures 6 to 10.

detailed description

[0040] Human beings have extremely powerful biomechanical capacities, in particular that of being able to visually fix a precise point in space in an extremely stable manner for a short time, even when the body is in motion. So, for example, when a person is driving and stares briefly at an object such as a roadside sign, eye movement is slaved to converge on the stared object despite the speed. The inventors have determined that such properties could advantageously be used in an extended reality system, in particular during its initialization.

[0041] According to embodiments of the invention, a positioning aid for an extended reality system comprising a location device is provided by establishing a relationship between a predetermined reference point rigidly linked to the real environment, for example a marker used by a rendering engine, called a virtual marker, and a marker defined by the location device of this extended reality system, also rigidly linked to this same real environment, called a real marker. For these purposes, the extended reality system comprises a gaze tracking device making it possible to estimate the coordinates of a point in the real environment, fixed by a user, in the real frame defined by the location device. By establishing a relationship between the coordinates of singular points fixed by the user through the extended reality system, in the real frame defined by the localization device, and the coordinates of these points in the predetermined virtual frame, it is possible to establish a relationship between the real environment and the virtual environment of the extended reality system. This relation makes it possible to precisely determine the position and the orientation of virtual objects to be inserted into the real environment as it is perceived by the user.

Figure 1 illustrates an example of an environment in which the invention can be implemented according to particular embodiments.

As illustrated, a user 100 provided with an extended reality headset 105 can move around in a real environment 110. The extended reality headset 105 here comprises a gaze tracking device (not shown), rigidly linked to the headset of extended reality. The extended reality headset 105 further comprises, in particular, a localization device as well as a rendering engine (not represented), or is connected to such means, to estimate the movements of the extended reality headset 105 (changes of position and/or orientation) and display virtual objects according to particular positions and orientations. The rendering engine can be implemented in information processing means such as the device described with reference to FIG. 11. The location device also comprises information processing means, the same as those used by the rendering engine or others, as well as one or more motion sensors.

The user can move freely in the real environment 110. Although the real environment here is an interior space, it could also be an exterior space. In particular, the user can move his head in three distinct directions (forward-backward, left-right and up-down) and turn it along three distinct axes (generally called yaw, pitch and roll), i.e. 6 degrees of freedom.

[0045] The real environment 110 includes fixed elements, for example the windows 115-1 and 115-2 and the table 120. It can also include fixed elements that can be moved, for example the plant 125 and the armchair 130.

The invention makes it possible to establish a relationship between the position of fixed elements of a real environment, for example the windows 115-1 and 115-2 and the table 120, according to a predetermined virtual reference point and used by a rendering engine and the position of these elements in a real frame defined by the localization device. For these purposes, it is necessary to identify at least three different singular points of the real environment and to determine the coordinates of these points in the virtual frame in the real frame. Knowledge of the coordinates of these three points in the virtual and real coordinate systems makes it possible to establish a transition matrix of coordinates expressed in the virtual system and in the real system (and/or vice versa). These singular points are preferably singular points of the real environment, chosen so as to be easily identifiable by a user, for example the user 100, but also easily identifiable on a corresponding digital model. It may be, for example, the upper left and lower right corners of the windows 115-1 and 115-2.

It is recalled here that a gaze tracking device makes it possible to estimate the coordinates of a point fixed by a user in a reference frame linked to the gaze tracking device or in any other reference frame including a passage matrix with a marker linked to the gaze tracking device is known. Indeed, the position of the center of the pupil in relation to the eye makes it possible to determine an optical axis. The intersection of the optical axes from the two eyes represents the fixed point.

Figure 2, comprising Figures 2a and 2b, schematically illustrates the estimation of the coordinates of a point fixed by a user, according to a projection on a transverse plane and according to a projection on a sagittal plane.

As illustrated in Figures 2a and 2b, a first optical axis 200-1 can be defined from a first eyeball 205-1, by the line passing through the center 210-1 of the eyeball and the center of the pupil 215-1. Similarly, a second optical axis 200-2 can be defined from a second eyeball 205-2, by the line passing through the center 210-2 of the eyeball and the center of the pupil 215-2. The intersection of the optical axes 200-1 and 200-2 corresponds to the point fixed by the user, referenced 220.

It is observed here that FIG. 2b being a view according to a projection on a sagittal plane, the optical axes 200-1 and 200-2 are superimposed. Only the optical axis 200-1 is "visible" here.

Thus, knowing the coordinates of the center of the eyeballs and of the center of the pupils in any frame rigidly linked to the eyeballs, it is possible to calculate the coordinates of a target point, in this frame.

[0053] It is observed here that while certain gaze tracking devices are capable of estimating the coordinates of a point fixed by a user, others are limited to estimating, still in any reference frame rigidly linked to the eyeballs, the line of sight to which this same point fixed by the user belongs. This line of sight 225, which corresponds to the direction of the user's gaze, can be defined as being the straight line passing through the point fixed by the user and an optical center. The latter is a point rigidly linked to the user which can be defined in different ways, in particular depending on the gaze tracking device. It can for example correspond to the center of the globe eyepiece of the dominant eye of the user or at the center of the interpupillary segment, the ends of which correspond to the center of the eyeballs (referenced 230 in FIG. 2a).

Furthermore, it is recalled here that a SLAM-type location device is capable of defining its position and its orientation in a real reference defined by this location device, this reference being rigidly linked to the real environment. Thus, when such a location device is rigidly linked to the gaze tracking device, it is possible to determine, depending on the capacities of this gaze tracking device, either the coordinates of a point fixed by a user, or a characterization of the line of sight passing through the point set by the user, in the real reference defined by the localization device, this reference being rigidly linked to the real environment.

[0055] By way of illustration, the gaze tracking device implemented in the mixed reality headset marketed by Microsoft under the name HoloLens 2 (Microsoft and HoloLens are trademarks) makes it possible to obtain in real time a characterization of the line of sight passing through the point fixed by the gaze of the user in a real benchmark defined by the localization device implemented in this mixed reality headset (this real benchmark being rigidly linked to the real environment ).

However, even in the case where the gaze tracking device is limited to providing a characterization of the line of sight passing through the point fixed by a user, it is possible to estimate the coordinates of the point fixed by asking the user to move while continuing to stare at the same point.

[0057] FIG. 3 schematically illustrates the estimation of the coordinates of a point fixed by a user from several lines of sight obtained from several positions of the user, when the latter moves while keeping his gaze on the same fixed point.

As illustrated, a gaze tracking device capable of determining a line of sight can be used to determine the line of sight 300-1 of a user located at a first position 305-1 whose gaze is fixed on a point 310. When the user moves to a position 305-2 while continuing to stare at the point 310, the gaze tracking device can then determine the line of sight 300-2. The coordinates of the target point 310 can then be calculated from the lines of sight 300-1 and 300-2. It is observed that if the lines of sight do not belong to the same plane, the target point can, for example, be defined as the middle of a segment perpendicular to the two lines of sight and of which each of the ends belongs to the one of the two lines of sight, as illustrated in FIG. 4, the length of this segment being the shortest possible distance between the two lines of sight. The aiming, from several distinct locations, of a point fixed by a user, in order to estimate the coordinates of this point, can be called a moving aiming.

Thus, FIG. 4 illustrates an example of estimation of the coordinates of a point fixed by a user, from two lines of sight relating to this same point fixed by the user from two different positions. As illustrated, the point 400 located in the middle of the segment 405 perpendicular to the lines of sight 300-1 and 300-2, one end of which belongs to the line of sight 300-1 and the other end belongs to the line of sight 300- 2, can be considered as the point fixed by the user as a point equidistant from the two lines of sight 300-1 and 300-2. It is recalled here that if the lines of sight are coplanar, the point fixed by the user is the point of intersection of these lines of sight.

[0061] It is observed that, according to the examples described with reference to Figures 1, 2 and 3, the relationship between the real environment and the virtual environment is determined on the basis of the stereoscopic vision of a user equipped with an extended reality system (for example an extended reality headset). When this extended reality system allows direct vision (called optical see-through in Anglo-Saxon terminology or OST), the user's vision is not distorted and the calculations described above can be implemented directly. When the vision is indirect (called video see-through in Anglo-Saxon terminology or VST), for example when the environment is acquired using sensors, for example stereoscopic video sensors, then reproduced, for example displayed on screens, it is necessary to verify that the perception of the environment is on a 1:1 scale to directly implement the calculations described previously. If the scale is not 1:1, a corresponding correction is necessary.

Examples of OST-type extended reality headsets that can be used to implement the invention are the headset known as HoloLens 2 from the company Microsoft and the headset known as Magic Leap One from the company Magic Leap (Magic Leap One and Magic Leap are trademarks). Examples of VST-type extended reality headsets that can be used to implement the invention are the headset known as XR-1 from the company Varjo (XR-A and Varjo are trademarks) and the headset known as the name Lynx-R1 of the company Lynx (Lynx-R1 and Lynx are trademarks).

According to other embodiments, the coordinates of a point fixed by a user are estimated using monocular vision, its coordinates then being, for example, estimated from two determined lines of sight by observing the same point using the same eye from two different locations, as described with reference to Figure 3. According to particular embodiments, the establishment of a relationship between a real marker defined by a localization device and a predetermined virtual marker (for example a marker of a rendering engine), the real and virtual markers being rigidly linked to the same real environment, is carried out by a mobile sighting of several singular points. For these purposes, a user provided with an extended reality device comprising a gaze tracking device aims at a singular point in the real environment by staring at it for a certain time, for example a few seconds, while moving, while continuing to fix the singular point. It does so for at least three singular points. In order not to disturb the ocular servoing, no virtual object is preferably displayed in the user's field of vision during this correlation phase.

FIG. 5, comprising FIGS. 5a and 5b, schematically illustrates the results of measurements carried out to estimate the coordinates of a singular point of a real environment, fixed by a user, using a gaze tracking device providing the coordinates of a fixed point and a gaze tracking device providing a characterization of the line of sight passing through the fixed point, respectively. For the sake of clarity, only a view according to a projection on a transverse plane is presented.

As illustrated by FIGS. 5a and 5b, this estimation of the coordinates of the point set by the user is based on several measurements from the gaze tracking device, this plurality of measurements possibly making it possible to overcome the limitations of this device.

In the case of a gaze tracking device providing the coordinates of the fixed point, the main limitation lies in the imprecision of the coordinates provided. This imprecision results in particular from the imprecision of the gaze tracking device itself, but also from the imprecision of the location device, as well as their combination. The use of a VST-type extended reality device can also increase this inaccuracy because a user may have to adapt his vision to a distance inconsistent with the convergence distance of his eyes (the distance of the device display in relation to the distance from the targeted singular point).

[0068] Thus, as illustrated in FIG. 5a, a mobile sighting carried out using a gaze tracking device providing the coordinates of a point fixed by a user can produce as results the measured points 510-1 at 510-4 when the user aims at the singular point 500 from positions 505-1 to 505-4, respectively.

In the case of a gaze tracking device providing a characterization of the line of sight passing through the point fixed by a user, the main limitation resides in the fact that it is necessary to have at least two lines sight to be able to estimate the coordinates of the target point. To this limitation can also be added an inaccuracy linked to the gaze tracking device itself, or linked to the location device, or even to their combination.

[0070] Thus, as illustrated in FIG. 5b, a gaze tracking device providing a characterization of the line of sight passing through a point fixed by a user can produce as results the measured lines of sight 515-1 to 515-4, when the user fixes the singular point 500' from the positions 505'-1 to 505'-4. These measured lines of sight can then be used to estimate the coordinates of measured points such as measured point 520.

[0071] Consequently, whatever the gaze tracking device, the results of a moving sight can be formalized in the form of a cloud of measured points which are distributed around the singular point fixed by a moving user, and whose coordinates in the reference frame of the gaze tracking device have been estimated. By way of illustration, the coordinates of this singular point can then be determined as being the coordinates of the center of gravity of the cloud of points measured resulting from the moving sight.

To limit the risk of error, it is also possible to filter the measured points relating to the same singular point set by the user, to keep only the measured points closest to each other. By way of illustration, the distribution of these measured points can be taken into account to ignore the most distant points and consider only the remaining points for the purpose of calculating the coordinates of the center of gravity which can be considered as being the fixed singular point .

[0073] After having described, with reference to FIGS. 2 to 5, the determination of the coordinates of a point fixed by a user in a reference frame linked to a gaze tracking device and the mobile sight, implemented in modes of embodiment of the invention, examples of process steps according to embodiments of the invention are described with reference to Figures 6 to 10.

FIG. 6 illustrates an example of steps for determining a passage matrix between a real marker defined by a location device and a predetermined virtual marker, the real and virtual markers being rigidly linked to the same real environment. The virtual marker is, for example, a marker used by the rendering engine of the extended reality system. [0075] The steps illustrated in FIG. 6 are implemented in a computer connected or integrated into an extended reality system, at least one part of which comprises a gaze tracking device and a location device rigidly linked to one to the other.

[0076]The purpose of these steps is in particular to estimate the coordinates of singular points of a real environment in a real reference defined by a localization device and to determine a transition matrix between a predetermined virtual reference and the real reference. The coordinates of the singular points in the predetermined virtual frame can, for example, be obtained from an external system or determined according to standard measurement techniques. As described previously, the number of singular points, to be fixed by a user, denoted n, is greater than or equal to 3.

As illustrated, a first step 600 includes the initialization of a variable i, representing an index of singular points to be fixed in order to establish a relationship between the real benchmark defined by the location device and a predetermined virtual benchmark, here at the value 1.

The coordinates of the singular point having index i, in the predetermined virtual frame, are then obtained (step 605). They can, for example, be obtained from the rendering engine of the extended reality system. These coordinates in the virtual coordinate system are noted here (xf,y,z).

The user is then invited to stare at the singular point having the index i (step 610). Such an invitation may take the form of a voice message or a visual indication in the extended reality system. A next step consists in inhibiting the display of virtual elements (step 615) in order to avoid disturbing the gaze of the user when he fixes the singular point of index i. A following step has as its object the sighting of the singular point itself. Embodiments of this step 620 are described in more detail with reference to Figures 7 and 8 which illustrate two different embodiments of this sighting, either by using a gaze tracking device providing the coordinates of the fixed point (Figure 7) , or by using a gaze tracking device providing a characterization of the line of sight passing through the fixed point (figure 8).

[0080] Whatever the embodiment of the sighting and in order to overcome the limitations of the gaze tracking device used, step 620 allows the recording of a set of points measured from several different observation positions of the user, for example from a number of observation positions denoted p (the result of step 620 is thus a cloud of p measured points). For each observation position k (k varying from 1 to p), the estimated coordinates, in the real coordinate system linked to the localization device of the extended reality system, of the point measured when the user aims at the singular point of index i , denoted ( xZ _k> yï _,k ' ^z ï _,k ) _> are obtained. It is observed that the number of positions p can vary from the estimation of the coordinates of a singular point to the estimation of the coordinates of another singular point. It is also observed that if the number of observation positions p is greater than or equal to 1 for a gaze tracking device capable of directly providing the coordinates of the fixed point, the number p is greater than or equal to 2 in the case d a gaze tracking device providing characterization of the line of sight (at least 2 measured lines of sight are needed to estimate the coordinates of 1 measured point).

In a next step, the coordinates of the singular point having the index i, in the real coordinate system defined by the location device, denoted (x[,y[,z[), are calculated (step 625). The estimated coordinates (x[,y[,z[) are here calculated from the coordinates (xf _'k yf _,k ' ^z I _,k ) of the cloud of measured points recorded when the user fixed the singular point having l 'index i from the different observation positions k. An example of implementation of this step is described in more detail with reference to Figure 10.

A test is then carried out to determine if all the singular points have been targeted by the user, that is to say if the value of the index i is less than the number of singular points to be targeted, denoted n (step 630).

If the value of index i is strictly less than the number n of singular points to be fixed, index i is incremented by 1 (step 635) and the previous steps (steps 605 to 625) are repeated to determine the coordinates of the singular point having the new value of the index i.

Otherwise, if the value of the index i is not strictly less than the number n of singular points to target, indicating that the coordinates of all the singular points have been estimated, the passage matrix between the real landmark defined by the location device and the predetermined virtual landmark is calculated (step 640). This matrix makes it possible in particular to determine a position and an orientation to which virtual objects must be rendered in order to be spatially coherent with the real environment in which the extended reality system evolves.

/xf\ /xf\

This matrix, denoted here M _v®r , is such that l yf = M _v®r x yf , whatever the

\yi ^r l \yï/ the value of index i.

An example of calculating such a matrix is presented in the article entitled "Least-Squares Fitting of Two 3-D Point Sets", Arun, Huang and Blostein (IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 9 Issue 5, May 1987). [0087] FIG. 7 illustrates a first example of steps for estimating the coordinates of points measured by a gaze tracking device when the user is aiming at a singular point in a real environment from several distinct observation positions, in a real landmark linked to a localization device of an extended reality system, the gaze tracking device used being capable of directly supplying the coordinates of the point fixed by the user.

The singular point targeted by the user is here the singular point having the index i. The estimated coordinates of the point measured when the user fixes the singular point of index i from the observation position k, in the real reference linked to the localization device, are here noted (xZ _k ,y[ _k ,z [ _k ).

As illustrated, the purpose of a first step here is to initialize the value of the index k of the positions, to the value 1 (step 700). The coordinates ([ _k> y[ _k ' ^z l _k ) of the point measured when the user aims at the singular point having the index i are then obtained from the gaze tracking device (step 705). As described previously, these coordinates are obtained in the real frame linked to the location device of the extended reality system comprising the gaze tracking device, the two devices being rigidly linked.

In a next step, a test is carried out to determine whether a sufficient number of measurements have been carried out (step 710), that is to say, according to the example of FIG. 7, whether an estimate has been made for a number p of observation positions, by comparing the value of the index k with the number p. By way of illustration, this number p can be determined as a function of a predetermined duration, for example a duration comprised between 1 and 7 seconds, so as to perform as many measurements as possible in the predetermined time. Alternatively, this number p can be chosen between two values, for example between 1 and 300. If the value of the index k is strictly lower than the value p, the value of the index k is incremented by 1 and a pause d a predetermined duration is performed so that the user has time to move (step 715). This duration is, for example, a few tens of milliseconds. The previous steps (steps 705 and 710) are then repeated to obtain new measurements.

Otherwise, if the value of the index k is greater than or equal to the value p, an estimation of the dispersion of the cloud of measured points is preferably carried out (step 720), to check the consistency of the measurements. carried out. By way of illustration, the dispersion can be estimated as being the difference between the smallest abscissa and the largest abscissa, between the smallest ordinate and the largest ordinate and between the largest dimension smallest and the largest dimension (which can result in the calculation of the bounding box containing all the measured points). These values are compared with a threshold (or with several thresholds, for example an abscissa threshold, an ordinate threshold and a dimension threshold) to determine whether the dispersion of the points measured is acceptable or not (step 725).

If the dispersion is too great, the previous steps are re-executed to obtain new measurements. Otherwise, if the dispersion is acceptable, the measured coordinates (*[ _¾ , y _¾ ,¾), with k varying here from 1 to p, are memorized to be then used to estimate the coordinates of the targeted singular point of index i in the real reference linked to the localization device.

[0093] FIG. 8 illustrates a second example of steps for estimating the coordinates of points measured by a gaze tracking device when the user is aiming at a singular point in a real environment from several distinct observation positions, in a real landmark linked to a localization device of an extended reality system, the gaze tracking device used being limited to providing the characterization of the line of sight passing through the point fixed by the user.

Again, the singular point targeted by the user is here the singular point having the index i. The characterization of the line of sight measured when the user fixes the singular point of index i from the observation position k, in the real reference linked to the localization device, formalized by a point and a direction is noted here (o[ _fc ;d[ _fc ), or more simply the index i not being recalled in order to improve readability.

As illustrated, the purpose of a first step is to initialize the value of the index k, representing an index of observation positions, to the value 1 (step 800). In a following step, a characterization ( 0 ;d ^ of the line of sight corresponding to the gaze of the user when he fixes the singular point having the index i is obtained from the gaze tracking device (step 805). This characterization of the line of sight obtained is expressed in the real reference linked to the localization device of the extended reality system comprising the gaze tracking device (which is rigidly linked to it).

In a next step, a test is carried out to determine whether a sufficient number of measurements have been carried out (step 810), that is to say, according to the example of FIG. 8, whether a line of sight was obtained for a number p of observation positions, by comparing the value of the index k with the number p. By way of illustration, this number can be determined according to a predetermined duration, for example a duration between 1 and 7 seconds, so as to perform as many measurements as possible over time. predetermined. Alternatively, this number p can be chosen between two values, for example between 2 and 300. If the value of the index k is strictly lower than the value p, the value of the index k is incremented by 1 and a pause d a predetermined duration is performed so that the user has time to move (step 815). This duration is, for example, a few tens of milliseconds. The previous steps (steps 805 and 810) are then repeated to obtain new measurements.

Otherwise, if the value of the index k is greater than or equal to the value p, a cloud of measured points, the coordinates of which are expressed in the real reference linked to the location device, is calculated (step 820). This cloud, for example composed of r measured points, is calculated from the p lines of sight previously obtained. An example of steps to determine the coordinates of each of these r measured points, noted here j varying here from 1 to r, is described with reference to FIG.

In a next step, an estimation of the dispersion of the cloud of measured points obtained is preferably carried out (step 825), to check the consistency of the measurements. Again and by way of illustration, the dispersion can be estimated as being the difference between the smallest abscissa and the largest abscissa, between the smallest ordinate and the largest ordinate and between the smallest dimension and the largest dimension (calculation of the bounding box containing all the measured points). These values are compared with a threshold (or with several thresholds, for example an abscissa threshold, an ordinate threshold and a dimension threshold) to determine whether the dispersion is acceptable or not (step 830).

If the dispersion is too great, the previous steps are re-executed to obtain new measurements. Otherwise, if the dispersion is acceptable, the measured coordinates (x ,y ,zij) _> ^with J varying here from 1 to r, are memorized to be then used to estimate the coordinates of the targeted singular point of index i in the real landmark linked to the localization system.

[0100] FIG. 9 illustrates an example of steps for calculating the coordinates of a set of measured points, representing the same point set by a user, from lines of sight. These steps can be performed when performing step 820 of Figure 8.

[0101] The estimated coordinates of the measured points ^with J varying here from 1 to r, corresponding to the same singular point of index i, are here obtained from p lines of sight characterized in the form \ O _k ,-d ^r _k J, with k varying here from 1 to p, in a real frame linked to the extended reality system.

[0102] As illustrated, the purpose of a first step is to initialize the values of the indexes u, v and j (step 900). The indexes u and v are line of sight indexes. They are initialized here to values 1 and 2, respectively. The index j is an index of measured points whose value is here initialized to 1.

[0103] The lines of sight having for index u and v, that is to say the lines of sight and , are then selected (step 905). In a next step, it is determined whether these lines of sight are coplanar (step 910). By way of illustration, this test can be carried out by determining whether the vector formed by the points (¾ and Oz is orthogonal to the vector resulting from the vector product of the vectors d _u ^r and that is, if the dot product (¾0 . (t¾A¾) is zero.

[0104] If the lines of sight are coplanar, a test is performed to determine if they are parallel (step 915). This test may consist, for example, to determine if the vector product of vectors _u and ^r let d _u ^r Ad _v ^r , be zero.

If the lines of sight are parallel, no measured point can be deduced from these two lines of sight.

A test is then performed to determine whether the value of the index v is strictly less than the number p of lines of sight considered (step 920). If the value of the index v is strictly less than the number p, this value is incremented by 1 (step 925) and a new pair of lines of sight is processed to estimate, if necessary, the coordinates of a measured point. Otherwise, if the value of the index v is not strictly less than the number p, a test is performed to determine whether the value of the index u is strictly less than the number p minus 1 of lines of sight considered (step 930).

If the value of index u is strictly less than the number (p - 1), this value is incremented by 1, the value of index v is modified to be equal to that of index u plus 1 (step 935), and a new pair of lines of sight is processed to estimate, if necessary, the coordinates of a measured point. If, on the contrary, the value of the index u is greater than or equal to the number (p-1), the search for measured points ends, all the lines of sight having been taken into consideration. [0108] If the lines of sight are coplanar but not parallel

(steps 910 and 915), the coordinates (¾, yf, z[ _j ) of the point of intersection of these lines are calculated (step 940) then stored as those of a measured point having the index j (step 945) . The value of the index j is incremented by 1 then the test(s) described previously on the values of the indices u and v are carried out to, if necessary, process a new pair of lines of sight.

[0109] If the lines of sight are not coplanar (step 910), the segment perpendicular to these two lines of sight is calculated here (step 950). If point S is considered to be the end of the segment belonging to the line of sight and that point S is the end of the segment belonging to the line of sight and the coordinates of these points can be determined according to the following relations:

[0110] Coordinates of the middle of the segment are then calculated (step

955) then stored as those of a measured point having the index j (step 945). The value of the index j is incremented by 1 then the test(s) described previously on the values of the indices u and v are carried out to, if necessary, process a new pair of lines of sight.

After performing the steps illustrated in FIG. 9, a set of measured points, comprising (j-1) points, corresponding to the same point targeted by a user, is obtained. The coordinates of the points making up this cloud of measured points can be used to estimate the coordinates of the point targeted by the user.

FIG. 10 illustrates an example of steps for estimating the coordinates of a point fixed by a user from coordinates of several measured points (cloud of measured points). The measured points are for example the points of the set of points obtained from a gaze tracking device during step 620 of FIG. 6 or the points of the set of points determined from lines sight, as described with reference to Figure 9.

As illustrated, the purpose of a first step is to filter the points measured according to their abscissa (step 1000). By way of illustration, this filtering may consist in selecting the points according to their distribution on the abscissa axis in order to eliminate the extreme points, for for example the points belonging to the upper and lower c percentiles where the value c is for example chosen between 1 and 10.

Similarly, a second step has the purpose of filtering the points according to their ordinate (step 1005). Again, this filtering can consist in selecting the points according to their distribution on the y-axis in order to eliminate the extreme points.

[0116] Again similarly, the purpose of a third step is to filter the points according to their rating (step 1010). Again, this filtering can consist of selecting the points according to their distribution on the dimension axis in order to eliminate the extreme points.

It is observed here that according to particular embodiments, the steps 1000 to 1010 can be combined in order to simultaneously consider the three dimensions of space and filter the points measured according to the distances which separate them in order to eliminate the points located at the periphery of the point cloud. It is also possible to filter the points by eliminating the points which would be in the neighborhood of the smallest convex hull (or convex hull in English terminology) encompassing the cloud of measured points.

In a next step, the coordinates (x[,y[,z[) of the center of gravity of the cloud of measured points considered, after filtering, are calculated (step 1015). These coordinates can be estimated as follows:

As described above, the center of gravity here represents an estimate of the singular point with index i set by the user.

According to particular embodiments, this center of gravity could be projected onto the nearest point of a geometric surface determined by the location device, this geometric surface corresponding to a virtual reconstruction of the real surface on which finds the singular point set by the user. [0121] Figure 11 illustrates an example of a device that can be used to implement, at least partially, an embodiment of the invention, in particular the steps described with reference to Figures 6 to 10.

The device 1100 is for example a computer or a calculator.

The device 1100 preferably comprises a communication bus 1102 to which are connected:

[0124] a central processing unit or microprocessor 1104 (CPU, acronym for Central Processing Unit in English terminology); [0125] a read only memory 1106 (ROM, acronym for Read Only Memory in English terminology) which may include the operating system and programs such as "Prog";

a random access memory or cache memory 1108 (RAM, acronym for Random Access Memory in English terminology) comprising registers adapted to record variables and parameters created and modified during the execution of the aforementioned programs;

[0127] an input/output interface 1110 connected to one or more sensors 1112, in particular one or more gaze tracking devices, one or more video sensors, one or more localization devices (eg 6DoF) and/or one or more multiple motion sensors; and

[0128] a graphics card 1114 connected to one or more rendering devices 1116 (e.g. screens, projectors, etc.).

[0129] Optionally, the device 1100 can also have the following elements:

[0130] a hard disk 1120 which may include the aforementioned "Prog" programs and data processed or to be processed according to the invention; and

[0131] any device 1122 allowing a user to interact with the device 1100 such as a voice recognition system, an object tracking device (e.g. hand tracking), a joystick or a remote control; and

[0132] a communication interface 1126 connected to a distributed communication network 1128, for example a wireless communication network and/or a local communication network.

The communication bus allows communication and interoperability between the different elements included in the device 1100 or connected to it. The representation of the bus is not limiting and, in particular, the central unit is capable of communicating instructions to any element of the device 1100 directly or via another element of the device 1100.

The executable code of each program allowing the programmable device to implement the processes according to the invention can be stored, for example, in the hard disk 1120 or in ROM 1106. According to a variant, the executable code of the programs could be received via the communication network 1128, via the interface 1126, to be stored in the same way as described previously.

More generally, the program(s) can be loaded into one of the storage means of the device 1100 before being executed.

The central unit 1104 will control and direct the execution of the instructions or portions of software code of the program or programs according to the invention, instructions which are stored in the hard disk 1120 or in the ROM 1106 or else in the other aforementioned storage elements. When the power is turned on, the program or programs which are stored in a non-volatile memory, for example the hard disk 1120 or the ROM 1106, are transferred to the random access memory 1108 which then contains the executable code of the program or programs according to the invention, as well as registers for storing the variables and parameters necessary for the implementation of the invention.

[0138] Depending on the chosen embodiment, certain acts, actions, events, or functions of each of the methods described herein may be performed or occur in a different order from that in which they were described, or may be added, merged or otherwise not made or not occurring, as the case may be. Further, in some embodiments, certain acts, actions or events are performed or occur concurrently and not sequentially.

[0139] Although described through a certain number of detailed exemplary embodiments, the proposed method and the equipment for the implementation of the method include various variants, modifications and improvements which will be apparent to those skilled in the art. art, it being understood that these various variants, modifications and improvements form part of the scope of the invention, as defined by the following claims. Additionally, various aspects and features described above may be implemented together, or separately, or substituted for each other, and all of the various combinations and sub-combinations of aspects and features are within the scope of the 'invention. In addition, some systems and equipment described above may not incorporate all of the modules and functions described for the preferred embodiments.

Claims

[Claim 1] Positioning aid method for an extended reality system, the extended reality system comprising at least one part that moves in a real environment and being provided with a location device for determining a position and/or an orientation relative to the mobile part according to a first benchmark rigidly linked to the real environment, the first benchmark being defined by the localization device, the extended reality system further comprising a device for tracking the gaze of a user of the reality system extended, the method comprising:

- obtaining an identification of each point of a plurality of points of the real environment and of the position of each point of the plurality of points according to the same second reference frame rigidly linked to the real environment, different from the first mark;

- for each point of the plurality of points, obtaining an estimate of the position of the point in the first marker, the position of the point in the first marker being estimated using the gaze tracking device; and

[Claim 2] Method according to claim 1, according to which the mobile part comprises the localization device, the localization device being a simultaneous localization and mapping device of the SLAM type.

[Claim 3] A method according to claim 2, wherein the gaze tracking device is rigidly connected to the tracking device.

[Claim 4] A method according to any one of claims 1 to 3 wherein several estimates of the position of a point fixed by the user are obtained, the method further comprising calculating a position of a point fixed by the user from the estimates obtained.

[Claim 5] A method according to claim 4, wherein the obtaining of several estimates of the position of a point fixed by the user comprises an evaluation of dispersion, the position estimates obtained being such that the dispersion of the position estimates obtained is below a threshold.

[Claim 6] A method according to claim 4 or claim 5 wherein obtaining multiple estimates of the position of a point fixed by the user comprises filtering of estimated positions, the filtering being based on a distribution of estimated positions.

[Claim 7] A method according to any one of claims 1 to 6 wherein at least one position estimate is obtained from at least two lines of sight and wherein, if the at least two lines of sight are not coplanar, the at least one estimated position is the position of the center of a segment perpendicular to the at least two lines of sight.

[Claim 8] A method according to any of claims 1 to 7 wherein the second marker is a marker used by a rendering engine of the extended reality system.

[Claim 9] Computer program comprising instructions for implementing each of the steps of the method according to one of Claims 1 to 8, when this program is executed by a processor.

[Claim 10] Device comprising a processing unit configured to execute each of the steps of the method according to one of Claims 1 to 8.