DE202018006796U1 - System for predicting gaze-related parameters - Google Patents

Abstract

System having:
- a headwear device comprising a left camera for capturing a left image of at least part of a left eye of a user wearing the headwear device and a right camera for capturing a right image of at least part of a right eye of the user wearing the head-worn device; and
- a processing unit configured to:
◯ Obtaining the left image;
◯ Obtaining the right image;
◯ inputting the left and the right image together into a convolutional neural network; and
◯ Obtain one or more gaze-related parameters from the convolutional neural network as a result of inputting the left and right images together.

Description

TECHNICAL AREA

Embodiments of the present invention relate to a system for predicting gaze-related parameters and a head-wearable (eyeglass) device.

BACKGROUND ART

Current head-mounted eye trackers often rely on the explicit extraction of features such as pupil center, infrared (IR) spot position, or pupil contours. Feature extraction is performed by classical computer vision and image processing algorithms. Then the line of sight z. B. estimated by a regression function with the extracted features as inputs.

Feature extraction typically requires high-resolution cameras that are relatively bulky and difficult or impossible to integrate fully and invisibly into an eyeglass frame. The lack of integration and obscuration of the user's field-of-view (FOV) limit the acceptance and thus the usability of the system and can also negatively affect the user's gaze behavior, i. H. make this behavior unnatural. In addition, current systems require infrared illumination of the eye to facilitate the task of detecting the pupil. Therefore, IR LEDs must be placed close to the eye and camera, further adding to the bulkiness of the system. Furthermore, processing high-resolution images is computationally expensive. These systems therefore require a laptop or at least a small portable computer to be carried by the user for image processing and real time estimation of the gaze. Considerable bandwidth is also required for data transmission. Combined with the weight of the cameras and the rest of the parts of the device (e.g. cables, camera/USB adapters, transmission/storage modules, etc.), all of which weigh heavily on the user's head, longer recordings in everyday life are justified uncomfortable or even painful. High-resolution sensors and sophisticated calculations, in turn, result in high power consumption, which either sets a fixed limit to the recording time due to battery capacity or restricts the user's freedom of movement, since the system requires an additional or even wired power supply.

In addition, the camera or cameras must be carefully adapted to each individual facial geometry, e.g. B. to enable reliable pupil feature detection, which requires additional flexibility at the hardware level and possibilities for manual camera adjustment. For the same reason, eye-tracking headset displacements can currently negatively impact gaze estimation accuracy or require frequent (and tedious) recalibration by the user.

Also, known head-worn eye trackers suffer from the disadvantage that stray light from the environment reflected by the test user's eyes can adversely affect eye tracking performance. In shape-based eye-tracking approaches, cameras that monitor the test user's eyes may not be able to distinguish between features of the eye that are actually used to track eye movement and features such as reflections that result from ambient lighting conditions. In general, the lighting situation is not very well defined. Reliable eye tracking is often hampered by environmental conditions and unwanted stray light interfering with the tracking mechanism. Known head-worn eye tracking devices typically suffer from limited accuracy and reliability.

Existing eye gaze sensing systems are also less than commonly used in consumer home applications for two main reasons. Their costs remain high and most systems require a tedious and time-consuming calibration process. Current calibration processes may be required whenever the user's head moves more than a threshold distance relative to the eye-tracking device from an original position at which the system was calibrated. Current head-worn eye-tracking systems with feature- and/or shape-based approaches are difficult to use in the consumer environment due to the need to keep the head still, or to maintain a rigid spatial relationship between the eye-tracking device and the head, or to frequently recalibrate.

Accordingly, there is a need for further improvement in the detection of gaze direction and other gaze-related parameters.

EXECUTIVE SUMMARY

• ◯ Erhalten des linken und des rechten Bildes (von der jeweiligen Kamera);
• ◯ typischerweise gemeinsames Eingeben (Einspeisen) des linken und des rechten Bildes in ein neuronales Faltungsnetzwerk; und
• ◯ Erhalten eines oder mehrerer blickbezogener Parameter von dem neuronalen Faltungsnetzwerk als Ergebnis des Eingebens des linken und rechten Bildes.

According to one embodiment, a system (for predicting one or more gaze-related parameters) comprises a wearable head-mounted device A device comprising a first camera for generating a left image of at least a portion of a user's left eye and a second camera for generating a right image of at least a portion of a user's right eye when the user wears the head-mounted device . In addition, the system has a processing unit which can be connected to the first and second cameras and which, in the connected state, is set up for

• ◯ Obtaining the left and right image (from the respective camera);
• ◯ typically joint inputting (feeding) of the left and the right image into a neural convolution network; and
• ◯ Obtain one or more gaze-related parameters from the convolutional neural network as a result of inputting the left and right images.

• ◯ Präsentieren eines ersten Stimulus für den Benutzer, der die am Kopf tragbare Vorrichtung trägt, wobei der erste Stimulus den Benutzer dazu anleitet, auf ein Objekt zu blicken, das eine Blickrichtung relativ zu einem mit der am Kopf tragbaren Vorrichtung fixierten Koordinatensystem und/oder einem Blickpunkt in dem Koordinatensystem definiert;
• ◯ Verwenden der ersten Kamera der am Kopf tragbaren Vorrichtung, um, wenn erwartet wird, dass der Benutzer auf den ersten Stimulus reagiert oder erwartet wird, dass er auf den ersten Stimulus reagiert hat, ein erstes linkes Bild von mindestens einem Teil des linken Auges des Benutzers zu erzeugen, und Verwenden der zweiten Kamera der am Kopf tragbaren Vorrichtung, wenn erwartet wird, dass der Benutzer auf den ersten Stimulus reagiert oder erwartet wird, dass er auf den ersten Stimulus reagiert hat, um ein erstes rechtes Bild von mindestens einem Teil des rechten Auges des Benutzers zu erzeugen;
• ◯ Erzeugen eines ersten Datensatzes, der das erste linke Bild, das erste rechte Bild und eine erste Repräsentation eines blickbezogenen Parameters des Benutzers umfasst, wobei die erste Repräsentation mit dem durch das Objekt definierten Blickpunkt und/oder der durch das Objekt definierten Blickrichtung korreliert ist; und/oder
• ◯ Hinzufügen des ersten Datensatzes zur Datenbank, um eine aktualisierte Datenbank zu erstellen.

According to one embodiment, a system for improving the prediction of gaze-related parameters comprises a head-mounted device having a first camera for generating a left image of at least a portion of a user's left eye and a second camera for generating a right image of at least one Part of a right eye of the user when the user wears the head-worn device. The system also includes a processing unit connectable to the first camera and the second camera and configured to determine a predicted value of a gaze-related parameter of the user using the left image and the right image as input to a trained neural network. In addition, the system has a computing system that can be connected to the processing unit and is configured to host a database and to train a neural network using the database in order to maintain and/or improve the trained neural network. The system is set up for:

• ◯ Presenting a first stimulus to the user wearing the headwearable device, the first stimulus directing the user to gaze at an object having a gaze direction relative to a coordinate system fixed with the headwearable device and/or a viewpoint defined in the coordinate system;
• ◯ Using the first camera of the head-worn device, when the user is expected to respond or is expected to have responded to the first stimulus, to acquire a first left image of at least a portion of the user's left eye user, and using the second camera of the head-worn device when the user is expected to respond or is expected to have responded to the first stimulus to generate a first right image of at least a portion of the generate the user's right eye;
• ◯ generating a first data set comprising the first left image, the first right image and a first representation of a gaze-related parameter of the user, wherein the first representation is correlated with the gaze point defined by the object and/or the gaze direction defined by the object; and or
• ◯ Adding the first record to the database to create an updated database.

According to one embodiment of a head-wearable eyewear device for determining one or more gaze-related parameters of a user, the head-wearable eyewear device has a left camera for capturing a left image of at least a portion of a left eye, a right camera for capturing a right image of at least a portion of a user's right eye, and a goggle body having a middle plane and configured to be wearable on a user's head and having at least left and right eyepiece openings (eye openings). Both eyepiece openings define a bounding box having a top surface, a bottom surface, a left side surface and a right side surface. The top surface of the bounding box is oriented perpendicular to the median plane of the goggle body. The cameras are arranged on the goggle body in inner eye camera arrangement zones and/or in outer eye camera arrangement zones. The left and right inner eye camera assembly zones are located in a nose bridge portion of the eyeglass body and extend between 2 and 14 mm, particularly between 3 and 12 mm, in a respective direction perpendicular to the median plane. The left outer eye camera arrangement zone is located in a left lateral portion of the glasses body and extends by 6 mm, specifically 5 mm, more precisely 4 mm from the left side surface of the bounding box towards the middle plane and extends by 7 mm, specifically 6 mm, even more particularly 5 mm, starting from the left side surface away from the middle plane. The right outer eye camera assembly zone is located in a right lateral portion of the goggle body and extends 6 mm, particularly 5 mm, particularly 4 mm from the right lateral surface of the bounding box Ders in the direction of the middle plane and extends by 7 mm, in particular 6 mm, more especially 5 mm, starting from the right side surface away from the middle plane.

According to an embodiment of a head-wearable eyeglass device for determining one or more vision-related parameters of a user, the head-wearable eyewear device has a goggle body which has a median plane, a nose bridge portion, a left lateral portion, a right lateral portion, a left eyepiece opening and has a right eyepiece opening. The eyeglass device is configured to be wearable on a user's head. The median plane intersects the nose bridge section and the respective eyepiece opening is located between the nose bridge section and the respective lateral section. The eyeglass device further includes at least one of a left camera for capturing a left image of at least a portion of a user's left eye, wherein the left camera is disposed in the nose bridge portion, and wherein an optical axis of the left camera with respect to the median plane is in one angle of 150° to 142° (or -30° to -38°), in particular 144° (or -36°), a right camera for capturing a right image of at least a portion of a user's right eye, wherein the right camera is arranged in the nose bridge portion, and wherein an optical axis of the right camera is inclined with respect to the median plane at an angle of 30° to 38°, particularly 36°, a left camera for taking a left image of at least a portion of a user's left eye, wherein the left camera is disposed in the left lateral portion, and wherein an optical axis of the left camera ra is inclined with respect to the median plane at an angle of 55° to 70°, in particular 62°, and/or a right camera for capturing a right image of at least a portion of a user's right eye, the right camera is arranged in the right lateral section, and wherein an optical axis of the right camera with respect to the median plane is at an angle of 125° to 110° (or -55° to -70°), in particular from 118° (or -62 °), inclined on.

According to an embodiment of a method for creating and updating a database for training a neural network, in particular a convolutional neural network, the method comprises presenting a first stimulus to a first user wearing a head-worn device. The head-worn device has a first camera and a second camera. The first camera is located near a left eye of the first user and the second camera is located near a right eye of the first user when the first user wears the headwear device. When the first user is expected to respond to the first stimulus, or is expected to have responded to the first stimulus, the first camera of the head-worn device is used to capture a first left image of at least a portion of the left eye of the first user, and a second camera of the head-worn device is used to generate a first right image of at least a portion of the first user's right eye. A data connection is established between the head-worn device and the database. A first data set is created that includes the first left image, the first right image, and a first representation of a gaze-related parameter. The first representation is correlated with the first stimulus. The first record is added to the database.

According to one embodiment of a method for training a neural network, in particular a neural convolutional network, the method comprises providing a database which includes a plurality of data sets, the data sets being a respective left image, a respective right image and a respective corresponding representation of a gaze-related parameter , in particular a respective corresponding value of the gaze-related parameter. A neural network with a given architecture is provided. Neural network parameters are determined using the respective left images and respective right images of a subset or all datasets as input and the respective corresponding representations of the gaze-related parameters of the subset or all datasets as the desired neural network output.

According to one embodiment of a method for calibrating a headwear device, the method includes presenting a stimulus to a user wearing the headwear device. The stimulus is correlated to a desired value of a gaze-related parameter. The head-worn device includes a first camera and a second camera. The first camera is placed near a user's left eye and the second camera is placed near a user's right eye when the user wears the head-mounted device. The first camera is used to generate a left image of at least a portion of the user's left eye and the second camera is used to generate a right image of at least a portion of the right eye of the user wearing the headworn device and who is expected to respond to the stimulus or who is expected to have responded to the stimulus. A trained neural network, specifically a trained convolution neural network, is used to determine a predicted value of the gaze-related parameter using the right image and the left image as an input to the trained neural network. The desired value of the gaze-related parameter and the predicted value of the gaze-related parameter are used to determine a correction function for the user. Optionally, the correction function is saved for the user.

According to one embodiment of a system for improving the prediction of gaze-related parameters, the system comprises at least one head-mountable device, a processing unit and a computing system connectable to the processing unit and configured to host a first database and a neural network, in particular train a convolutional neural network using the first database. The at least one head-mountable device includes a first camera for generating a left image of at least a portion of a user's left eye and a second camera for generating a right image of at least a portion of a user's right eye when the user uses the am carries head attachable device. The processing unit is connectable to the first camera and the second camera and configured to determine a predicted value of a gaze-related parameter of the user using the left image and the right image as input to the neural network. The system is configured to add records to the first database and transmit neural network parameters from the computing system to the processing unit. The data sets include a respective left image, a respective right image, and a respective representation of the user's gaze-related parameter.

According to an embodiment of a method for detecting one or more gaze-related parameters of a user, the method comprises creating a left image of at least a portion of a left eye of the user using a first camera of a head-mounted device worn by the user a right image of at least a portion of a user's right eye using a second camera of the head-worn device, feeding the left and right images together as an input to a convolution neural network, and obtaining the one or more gaze-related parameters from the convolution neural network as a result of inputting the left and right images.

Other embodiments include appropriate computer systems, computer-readable storage media or devices, and computer programs recorded on one or more computer-readable storage media or computer storage devices, each configured to perform the processes of the methods described herein.

A system of and/or including one or more computers may be configured to perform certain operations or processes by means of software, firmware, hardware, or any combination thereof installed on one or more computers and capable of causing the system, when in operation, to do so carry out processes. One or more computer programs may be configured to perform particular operations or processes by including instructions that, when executed by a single or multiple processors of the system, cause the system to perform the processes.

Other features and advantages will become apparent to those skilled in the art upon reading the following detailed description and considering the accompanying drawings.

character list

• 1A eine Draufsicht auf eine verallgemeinerte Ausführungsform einer am Kopf tragbaren Brillenvorrichtung zum Bestimmen von einem oder mehreren blickbezogenen Parametern gemäß mindestens einer Ausführungsform;
• 1B eine Vorderansicht der Brillenvorrichtung gemäß 1A;
• 1C eine Seitenansicht der Brillenvorrichtung gemäß 1A
• 2A ein Ablaufdiagramm eines Verfahrens zum Erstellen und Aktualisieren einer Datenbank zum Trainieren eines neuronalen Netzes gemäß Ausführungsformen;
• 2B ein Ablaufdiagramm eines Verfahrens zum Erstellen und Aktualisieren einer Datenbank zum Trainieren eines neuronalen Netzes gemäß Ausführungsformen;
• 2C ein Ablaufdiagramm eines Teils des Verfahrens aus 2A und 2B, jeweils gemäß Ausführungsformen;
• 3A ein Ablaufdiagramm eines Verfahrens zum Trainieren eines neuronalen Netzes gemäß Ausführungsformen;
• 3B ein Ablaufdiagramm eines Verfahrens zum Kalibrieren einer am Kopf tragbaren Vorrichtung;
• 3C ein Ablaufdiagramm eines Verfahrens zum Erfassen von einem oder mehreren blickbezogenen Parametern gemäß Ausführungsformen;
• 3D ein System zum Verbessern der Vorhersage von blickbezogenen Parametern gemäß einer Ausführungsform;
• 4 ein System zum Verbessern der Vorhersage von blickbezogenen Parametern gemäß einer Ausführungsform;
• 5 ein System zum Verbessern der Vorhersage von blickbezogenen Parametern gemäß einer Ausführungsform; und
• 6 ein System zum Verbessern der Vorhersage von blickbezogenen Parametern gemäß einer Ausführungsform.

The components in the figures are not necessarily to scale; instead, emphasis is placed on illustrating the principles of the invention. Moreover, in the figures, like reference characters indicate corresponding parts. Show it:

• 1A 14 is a plan view of a generalized embodiment of a head-wearable eyeglass device for determining one or more vision-related parameters, in accordance with at least one embodiment;
• 1B a front view of the spectacle device according to FIG 1A ;
• 1C a side view of the spectacle device according to FIG 1A
• 2A 12 is a flow chart of a method for creating and updating a database for training a neural network according to embodiments;
• 2 B a flowchart of a method for creating and updating a data bank for training a neural network according to embodiments;
• 2C a flowchart of part of the process 2A and 2 B , each according to embodiments;
• 3A FIG. 12 shows a flow diagram of a method for training a neural network according to embodiments; FIG.
• 3B FIG. 12 is a flowchart of a method for calibrating a head-worn device; FIG.
• 3C a flowchart of a method for detecting one or more gaze-related parameters according to embodiments;
• 3D a system for improving prediction of gaze-related parameters according to an embodiment;
• 4 a system for improving prediction of gaze-related parameters according to an embodiment;
• 5 a system for improving prediction of gaze-related parameters according to an embodiment; and
• 6 a system for improving prediction of gaze-related parameters according to an embodiment.

DETAILED DESCRIPTION

In the following detailed description, reference is made to the accompanying drawings which form a part hereof, and in which is shown by way of illustration specific embodiments in which the invention may be practiced. In this regard, directional terminology such as "top," "bottom," "front," "back," "leading," "trailing," etc. is used with reference to the orientation of the character(s) being described. Because components of embodiments can be arranged in a number of different orientations, the directional terminology is used for purposes of illustration and is in no way limiting. It is understood that other embodiments may be utilized and that structural or logical changes may be made therein without departing from the scope of the present invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is defined in the appended claims.

It is an object of the invention to provide methods, systems and devices that enable the determination of one or more gaze-related parameters of a user with improved performance and/or improved functional and/or ergonomic properties.

These objects are met by the subject matter of the intended claims.

In particular, an eyeglass device includes an eyeglass body configured to be worn on a user's head, for example in a manner in which ordinary eyeglasses are worn. In particular, therefore, when the spectacle device is worn by a user, it can be worn at least partially through a nasal area of the user's face.

This state of use of the eyeglass device placed on the user's face is further defined as the "intended use" of the eyeglass device, with reference to direction and position, for example horizontal and vertical, parallel and perpendicular, left and right, front and rear, up and down etc., is a reference to that intended use. Thus, lateral positions such as left and right, up and down, and front/forward and back/back positions are to be understood from the user's perspective. This applies equally to horizontal and vertical orientation, with the user's head being in a normal and thus upright, non-tilting and non-nodding position during intended use.

According to one embodiment, the eyeglass body includes a left ocular opening and a right ocular opening, which primarily serve to allow the user to see through these ocular openings. The ocular openings may be embodied, without limitation, as a sun shield, optical lenses, or non-optical transparent eyeglass lenses, or an intangible, optical path that allows light rays to pass therethrough.

According to a further aspect of the invention, the spectacle body forms the eyepiece openings at least partially or entirely by delimiting them with respect to the environment. In this case, the eyeglass body serves as a frame for the optical openings. This frame does not necessarily have to form a complete and closed environment around the eyepiece openings. Furthermore, it is possible for the optical openings themselves to have a frame-like configuration, for example by providing a support structure with the aid of transparent spectacle lenses. In the latter case, the spectacle device has a shape similar to that of rimless spectacles, with only a nose bridge section and ear loops attached to the glass panels, which therefore double as an integrated frame and optical openings.

In addition, a middle plane of the glasses body can be identified. In particular, the median plane describes a structural median plane of the eyeglass body, with respective structural components or portions that are comparable or similar to one another being arranged in a similar manner on either side of the median plane. When the eyeglass device is worn correctly in the intended use, the median plane coincides with a median plane of the user.

According to a particular embodiment of the invention, the eyeglass body comprises a nose bridge portion (also referred to as a nose bridge portion), a left lateral portion and a right lateral portion, wherein the median plane intersects the nose bridge portion and the respective Eyepiece opening is located between the nose bridge section and the respective lateral section.

For orientation purposes, a plane perpendicular to the central axis is defined as the middle plane, which is in particular vertically oriented, with this vertical plane not necessarily being arranged fixedly at a defined front or rear position of the spectacles device.

With the help of the two eyepiece openings, a limiting cuboid (limiting cuboid / as an enveloping body) is defined, which is to serve as an artificial reference system for geometric design data in the following; thus, the bounding box is a virtual, non-embodied structure defined as having a top surface, a bottom surface, a left side surface, and a right side surface; Consequently, the cuboid cannot be recognized in the form of a real cuboid body on the spectacle device. The bounding box is oriented with respect to the median plane of the eyeglass body such that at least the top surface is perpendicular to the median plane.

According to one embodiment, the bounding box is a rectangular box; consequently, the top and bottom faces are both perpendicular to the median plane, and the left and right side faces are both oriented parallel with respect to the median plane.

In the following supplementary clarifications and explanations of the artificial reference system, the "limiting cuboid" is provided for the person skilled in the art in a non-restrictive manner: The definition of the limiting cuboid through the eyepiece openings is carried out by virtually covering the volume of both eyepiece openings with a cuboid, with the left side surface of the bounding box touches the left eyepiece opening from the left side, the right side surface of the box touches the right eyepiece opening from the right side, the top surface of the box touches at least one of the eyepiece openings from above and the bottom surface of the box touches at least one of the eyepiece openings touched from below. As a result, the ocular openings do not protrude from the boundary surfaces of the bounding box and the size of the bounding box does not extend beyond the maximum extent of the ocular openings.

According to the invention, the eyewear device comprises a left and a right camera, the left camera for taking a left image or a sequence of images of at least a portion of the user's left eye and the right camera for taking an image or a sequence of images of at least a portion of a user's right eye. The recorded images do not necessarily have to be an image visible to the human eye, but can also be a suitable representation of the filmed eye in a light range that is not visible to humans.

The cameras can be arranged on the spectacle body in inner eye camera arrangement zones and/or in outer eye camera arrangement zones, these zones being determined in particular in such a way that a suitable image of at least a portion of the respective eye can be recorded for the purpose of determining one or more of a gaze-related parameter ; In particular, the cameras are arranged in a nose bridge section and/or in a lateral edge section of the spectacle frame in such a way that an optical field of a respective eye is not covered by the respective camera. The optical field is defined as occluded when the camera forms an explicitly visible area/section within the optical field, for example when the camera protrudes into the field from the boundaries of the visible field, or by protruding from the boundaries into the field. For example, the cameras can be integrated into a frame of the eyeglass body and thus cannot be concealed. In the context of the present invention, a restriction of the visible field caused by the eyeglass device itself, in particular the eyeglass body or frame, is not considered to be an occlusion of the optical field.

According to an embodiment of the eyeglass device, the cameras are only arranged in the inner eye camera arrangement zones, in particular no more than a single camera is provided in each inner eye camera arrangement zone; as a result, only one left camera is dedicated to the left eyepiece opening and only one right camera is dedicated to the right eyepiece opening.

The inner eye camera assembly zones may be distributed in a nose bridge portion of the goggle body. In detail, it has been determined through extensive experimentation and user testing that the respective inner eye camera array zone must extend between 2mm and 14mm, particularly between 3mm and 12mm, in a horizontal direction perpendicular to the median plane in order to obtain optimal images for determining one or of the several gaze-related parameters. In particular, the inner eye camera array zones can lie in and extend in the vertical plane, wherein - starting from the median plane in the left and right horizontal directions - the respective inner eye camera array zone in a horizontal direction at a distance of 2 mm, in particular 3 mm, starts from the middle plane and extends in the horizontal direction to a distance of 14 mm, in particular 12 mm, from the middle plane.

According to an alternative or supplement, the at least one camera or both cameras can be arranged in an outer eye camera arrangement zone or outer eye camera arrangement zones. In particular, the eyeglass device comprises no more than two cameras, the right camera being arranged in the right outer eye camera arrangement zone and the left camera being arranged in the left outer eye camera arrangement zone.

According to one embodiment, the left camera is placed in the left outer eye camera placement zone and the right camera is placed in the right outer eye camera placement zone, or the left camera is placed in the left inner eye camera placement zone and the right camera is placed in the right inner eye camera placement zone. This symmetrical camera placement strategy results in an improved ability to determine a user's gaze-related parameter, particularly when the lighting conditions of an environment surrounding the eyewear device are not optimal. Furthermore, the symmetrical use of cameras can also entail advantages with regard to the use of neural networks, in particular neural convolution networks, for determining a gaze-related parameter. According to a further embodiment, more than one left and/or more than one right camera can be used.

The left outer eye camera arrangement zone is located in a left lateral portion of the spectacle body in projection on the vertical plane and extends by 6 mm, in particular 5 mm, more typically 4 mm, from the left lateral surface of the bounding box towards the median plane and extends by 7 mm, in particular 6 mm, more typically 5 mm, measured from the left side face in a direction away from the median plane, away from the median plane.

Also, the right outer eye camera arrangement zone is in projection on the vertical plane in a right lateral portion of the glasses body and extends by 6 mm, particularly 5 mm, more typically 4 mm, from the left lateral surface of the bounding box towards the median plane and extends by 7 mm, in particular 6 mm, more precisely 5 mm, measured from the right side surface in a direction away from the middle plane, away from the middle plane.

According to a further embodiment, the inner eye camera arrangement zone extends in the vertical direction between 9 and 31 mm, in particular 12 and 28 mm, from the upper surface to the lower surface of the bounding box in projection onto the vertical plane.

Additionally or alternatively, the outer eye camera arrangement zone, when projected onto the vertical plane, has a size of 25 to 33 mm, especially 28 to 30 mm, in a perpendicular direction with respect to the upper surface, and, when the spectacle device is intended to be used, in a vertical direction on.

It has been found that the above object is achieved independently, alternatively or additionally, by a spectacle device according to claim 6.

Definitions, directions and basic information of the eyeglass device described above also apply in the following. In particular, this spectacle device according to claim 6 comprises a spectacle body configured to be worn on a user's head, for example in a manner in which ordinary spectacles are worn.

According to an embodiment of the eyeglass device, the latter comprises an eyeglass body having a middle plane (see above), a nose bridge portion, a left lateral portion, a right side portion, a left eyepiece opening and a right eyepiece opening.

As discussed above, the nose bridge portion is a portion of the eyeglass device that serves as a support for the device on the user's nasal region, being located between the left and right eyepiece openings. The left/right lateral section represents a structural area of the device located to the left/right of the left/right eyepiece opening. Thus, the respective left/right eyepiece opening lies between the nose bridge section and the respective left/right side section.

In addition, the eyeglass device includes at least one left camera for capturing a left image of at least a portion of a user's left eye and at least one right camera for capturing a right image of at least a portion of a user's right eye.

According to a first embodiment, the left camera is arranged in the nose bridge portion, with an optical axis of the left camera with respect to the median plane at an angle of 150° to 142° (or -30° to -38°), in particular 144° ( or -36°), is inclined. Generally, tilt angles are provided positive counter-clockwise and are to be measured as such.

According to a second configuration, the right camera is disposed in the nose bridge portion, with an optical axis of the right camera being inclined at an angle of 30° to 38°, particularly 36°, with respect to the median plane.

Both configurations have the advantage of allowing the cameras to have an optimal view of an eyeball of the user, with the user essentially not being able to see the user's field of vision being obscured and therefore not present; first, the human brain largely ignores occlusions in the nasal area, since the nose itself forms a permanent occlusion of the optical field; secondly, the nose bridge portion normally serves as a mount for the eyeglass device on a user's nose, and a microcamera may be integrated into support members of the nose portion.

According to a third configuration, the left camera is disposed in the left lateral portion, with an optical axis of the left camera being inclined at an angle of 55° to 70°, particularly 62°, with respect to the median plane.

According to a fourth embodiment, the right camera is placed in the right lateral section, with an optical axis of the right camera being oriented at an angle of 125° to 110° (or -55° to -70°), in particular 118°, with respect to the median plane ° (or -62°).

According to a first preferred embodiment, the eyeglass device comprises no more than two cameras, one camera - the left camera - being arranged according to the first layout and the other camera - the right camera - being arranged according to the second layout.

In the context of a second preferred embodiment, the eyeglass device comprises no more than two cameras, one camera - the left camera - being arranged according to the third interpretation and the other camera - the right camera - being arranged according to the fourth interpretation.

However, a combination of the first and the third design or the second and the fourth design is also possible.

Furthermore, but not limited thereto, a combination of at least three of the configurations or even all configurations is disclosed in the context of the invention, thus removing the limitation of no more than two cameras in this particular case.

All mentioned configurations, in particular combinations thereof, provide an eyeglass device for obtaining suitable image data for determining one or more gaze-related parameters of the user, in particular when using a convolutional neural network for this determination.

Preferred embodiments of the invention are described below, these embodiments relating to all the embodiments of the spectacles device discussed above, in particular to the spectacles device according to claim 1 and/or according to claim 6 .

It is also expressly included that an eyeglass device according to claim 6 may also include one or a plurality of features of an eyeglass device according to claims 1 to 5. For example, the spectacle device according to claim 6 - including the angles of the optical axis of the cameras - can have a description of the left/right, outer/inner camera arrangement zone and associated distances in full or in part. Conversely, the glasses device according to claim 1, which is mainly determined by the camera position in Kameraanord tion zones is defined, have one or more specific features of the inclination of the optical axis of one or more cameras.

According to a non-limiting embodiment, the spectacle body of the spectacle device according to one or more embodiments as described above has at least partially a symmetrical structure with respect to the median plane. In particular, at least one functional-structural area of a nose section and/or a left and right side section of the spectacle body is symmetrical; as an example, the functional-structural area can be a support element of the nose section or a plurality of the same, which support the eyeglass device on a nose region of the user, a respective inner, lateral, upper and/or lower part of the eyeglass body or a holder or a joint element in the left and right side portions for wearing the glasses body on the user's ear.

Furthermore, according to a particular embodiment, a frame of the eyeglass body is substantially symmetrical about the median plane, with only minor areas, portions or elements of the frame being non-symmetrical.

According to the specific embodiment, the cameras are arranged in the lower 75%, in particular in the lower 50%, preferably in the lower 25%, of the volume of the bounding box.

Furthermore, according to an embodiment in which at least one camera is arranged in a lateral portion of the body and/or in one of the outer eye camera arrangement zones, the camera is in a frame of the eyeglass body, in a respective left or right holder, which acts as a carrier for the eyeglass device serves on an ear of the user, or arranged in/on a hinge element connecting the holder to the frame of the eyeglass body. By applying these teachings, the camera(s) can be placed very discreetly and unobtrusively within the goggle body while gaining a beneficial view of the user's eyeball.

According to the embodiments, the spectacles device can have lighting devices for illuminating the left and/or right eye of the user, in particular when the lighting conditions in an area surrounding the spectacles device are not optimal.

According to one embodiment, the eyewear device includes a scene camera for capturing images of a field of view of the user wearing the eyewear device. The scene camera is therefore not to be understood as a camera for capturing an image of a user's eye. In this way, an advantageous calibration process for determining the gaze-related parameter can be made possible. The calibration process is described below with reference to 3C described.

In order to reliably and efficiently analyze the images from the left and right cameras with regard to the gaze-related parameters, it is desirable to train the neural network architecture used with a large number of training examples (training examples). To this end, a respective database for storing the teaching examples obtained by means of the eyeglass device and/or other head-worn device as described herein has proven advantageous.

According to one embodiment, a method for creating and updating a database for training a neural network, in particular a convolutional neural network, is provided. The method includes presenting a first stimulus to a first user wearing a headwear device. The head-worn device has a first camera and a second camera. The first camera is located near a left eye of the first user and the second camera is located near a right eye of the first user when the first user wears the headwear device. When the first user is expected to respond to the first stimulus, or is expected to have responded to the first stimulus, the first camera of the head-worn device is used to capture a first left image of at least a portion of the left eye of the first user, and a second camera of the head-worn device is used to generate a first right image of at least a portion of the first user's right eye. A data connection is established between the head-worn device and the database. A first data set is created that includes the first left image, the first right image, and a first representation of a gaze-related parameter. The first representation is correlated with the first stimulus. The first record is added to the database.

The method enables maintenance and improvement of the database with the help of the users. Because the database can be used to improve the performance of an instance of the neural network used at the user's site (e.g., on the head-worn device designed leads), users are interested in allowing data exchange between the user's head-worn device, typically owned, and a computing system for hosting the database and training or post-training the neural network using the database.

In the following, the database is also referred to as first database and image database.

In this description, the terms “first camera” and “left camera” are used interchangeably. Likewise, the terms “second camera” and “right camera” are used synonymously in this document.

The term "neural network" (NN) when used in this specification is intended to describe an artificial neural network (ANN) or connectionist system (connectionist system) that includes a plurality of connected entities or nodes referred to as artificial neurons become. The output signal of an artificial neuron is calculated by a (non-linear) activation function of the sum of its input signal or signals. The connections between the artificial neurons typically have respective weightings (gain factors for the transmitted output signal(s) that are adjusted in one or more learning phases. Other parameters of the NN that may or may not be modified during learning can be parameters of the activation function of the artificial neurons such as a threshold. Often, the artificial neurons are organized into layers, also known as modules. The simplest NN architecture, called a "multilayer perceptron", is a sequence of so-called fully connected layers. A layer consists of several separate units (neurons), each of which computes a linear combination of the input followed by a non-linear activation function.Different layers (of neurons) can perform different types of transformations on their respective inputs.Neural N Networks can be implemented in software, firmware, hardware, or any combination thereof. A machine learning method, in particular a monitored, non-monitored or semi-monitored (deep) learning method, can be used in the learning phase or phases. For example, a deep learning technique, particularly a gradient descent technique such as back propagation, can be used to train (forward coupling) NNs with a layered architecture. Modern computer hardware, e.g. B. GPUs, make back propagation efficient for multi-layered neural networks. A convolutional neural network (CNN) is an artificial feed-forward neural network that includes an input (neural network) layer, an output (neural network) layer, and one or more hidden (neural network) layers located between the input layer and the output layer are arranged. The peculiarity of CNNs is the use of convolution layers that perform the mathematical operation of convolving the input with a kernel. The hidden layers of a CNN can include convolution layers, as well as optional pooling layers (for downsampling the output of a previous layer before its input to the next layer), fully connected layers, and normalization layers. At least one of the hidden layers of a CNN is a convolutional neural network layer, hereinafter also referred to as the convolutional layer. Typical convolution kernel sizes are, for example, 3x3, 5x5 or 7x7. Using a convolutional layer or layers can help compute repetitive features in the input more efficiently than fully connected layers. Accordingly, the memory footprint can be reduced and the performance can be improved. Because of the architecture with shared weights and translation invariance properties, CNNs are also referred to as shift or space invariant artificial neural networks (SIANNs). In the following, the term “neural network model” is intended to describe a set of data required to define a neural network operable in software and/or hardware. The model typically includes data relating to the architecture of the NN, in particular the network structure, including the arrangement of the layers of the neural network, the sequence of information processing in the NN, and data representing or consisting of parameters of the NN, in particular the connection weights within the fully connected layers and core weights within the convolution layers.

Traditional eye-tracking algorithms extract handcrafted features from the eye images, such as the pupil contour. The observed features are then used to either fit an eye model to the observations or to perform direct regression on the output.

In contrast, the systems described in this paper perform gaze estimation based on end-to-end learning, with the input image directly connected to a learning algorithm is passed without previously extracting handcrafted features.

In a training phase, the network learns to automatically interpret the input image or images in order to output the correct result. Supervised learning algorithms, such as those using neural networks (NN), take advantage of the existence of so-called labeled data, i. H. a type of input data combined with ground truth associated with that input data. For example, an image captured by a camera represents the input data. The ground truth value or values can be any ancillary or superordinate information, encoded in any form in the input data and available at the time the data is generated input data is known.

For example, a user wearing a head-worn device, as described in 1A until 1C shown, are asked to look at a specific landmark or object in space, the coordinates of which can be precisely determined in the video images recorded by a scene camera connected to the device. The image or images recorded by one or more optical sensors (cameras) facing the person's eye or eyes then represent the input data encoding the person's gaze direction information, while the coordinates representing ground truth. Thus, by asking the person to look at markers in many different directions and distances, the ground truth is generated for all possible viewing directions. Collecting large amounts of labeled data, also referred to as training data, thus forms the basis for training a learning algorithm.

A NN typically implements a mathematical model with a large number of parameters arranged to take a unit of input data (e.g. the pixel values of an image extending into a 1-dimensional vector) and calculates a predicted output in one Format according to Ground Truth (e.g. a pair of (x/y) coordinates). The parameters (weights) of the network are usually initialized randomly. The goal of the training process is then to optimize the parameters so that when a training sample is entered, its ground truth value is correctly reported. In addition, when training is complete, a new unit of labeled input data, also referred to as test data, that was not used for training should produce a correct prediction when fed into the network. The training process is carried out by entering training examples into the network and, for example, summing the absolute deviations of the output predictions from the ground truths, which results in a cost value or a cost function. Numerical methods are used to iteratively minimize these costs, updating the parameters of the network model (back propagation, gradient descent). A learning rate parameter is part of such methods to update the parameters. During the training of a neural network, the learning rate, i. H. the step size of the gradient descent algorithm, typically gradually reduced (learning rate decay). Reducing the learning rate after a cosine curve has been found to be beneficial in significantly reducing training time without sacrificing accuracy.

For example, the training process can be stopped when a desired prediction accuracy is achieved based on a set of test data samples or when costs no longer decrease significantly. The final values of the parameters can then be stored digitally as a "model" and applied to a new input data sample to produce a prediction. Depending on the amount of training data and the complexity of the model, training can be a process that requires several hours or days of computation time, even with parallelization on many GPUs, while applying the model in a forward pass on a single input data item should be almost instantaneous .

In a regression task, i. H. a task with continuous output space such as predicting a pair of (x/y) coordinates, neural networks are typically laid out as follows:

Input -> some layers of the neural network -> Intermediate feature representation -> Linear combination -> Output in R2 (each a small subset of R2).

Another approach that has been shown to increase accuracy works as follows: Discretize the output space, i.e. H. a small subset of R2, in k different values. Instead of directly regressing the output (e.g., predicting a pair of (x/y) coordinates), a probability distribution over the possible output values k is computed. As a final output, the weighted mean of all values weighted by their respective probability is calculated. The flow of information is thus:

Input -> some layers of the neural network -> Intermediate feature representation -> Linear combination -> output in {1,...,k} -> softmax -> probability distribution P over {1,...,k} -> weighted mean of {1,...,k} with P -> Output in small subset of R2.

Enforcing the representation as a probability distribution as an intermediate result seems to have a regulatory effect and in this way reduces the prediction error of the network.

Establishing the data connection typically involves connecting the headworn device to a computing system that operates as a server that hosts the database. The headwear device can be connectable directly, for example via WiFi and/or an internet connection, or indirectly, for example via a desktop computer, a tablet, a laptop or a smartphone (mobile phone) that can be connected to the headwear device and the computing system is to be connected to the computing system (server).

Typically, a plurality of data sets relating to the first user and including a respective left image, a respective right image, and a respective representation of the gaze-related parameter are created and added to (stored in) the database.

Correspondingly, the database includes data sets of the user, which allow a sufficiently good feature generalization of the neural network in one or more training phases as well as a high accuracy of the prediction of gaze-related parameters using the trained neural network.

For example, at least 10 records, more typically at least 50 records, and more typically at least 100 records of the first user may be stored in the database.

To this end, a second stimulus may be presented to the first user wearing the head-worn device and the first camera of the head-worn device may be used to detect when the first user is expected to respond to the second stimulus , or is expected to have responded to the second stimulus, generate a second left image of at least a portion of the first user's left eye, and if the first user is expected, or expected, to respond to the second stimulus that he has responded to the second stimulus, the second camera of the head-worn device is used to generate a second right image of at least a portion of the first user's right eye. A second data set including the second left image, the second right image, and a second representation of the gaze-related parameter, where the second representation correlates to the second stimulus, can be created and added to the database.

In addition, the images of the first user's data sets may be generated under different conditions, for example for a rested user and a tired user, or at different times and/or under different lighting conditions. Furthermore, the images of the first user's data sets may be generated for different distances between a presented stimulus and the user (resulting in different refractive powers of the eyes).

Thus, one or more sequences of stimuli can be presented to the first user and respective datasets created and added to the database.

For statistical reasons, some of the stimuli may be the same.

For example, the first and/or second stimulus may be presented to the first user twice or more.

Depending on the stimulus, there may be a certain time delay between the stimulus and the user's desired response. The time delay can range from 100 ms or 200 ms to about half a second or even a second.

For example, if a user is asked to look at a non-moving object or mark in their field of view, the user can be expected to have responded after the time delay and then looked at the object or mark for a period of about half a second mark looks.

Also, when a user looks at one of their moving fingertips in response to a particular stimulus, the time delay during the movement may be insignificant. Accordingly, the user can be expected to respond to the stimulus during fingertip movement.

Additionally, generating the respective data set typically includes selecting a left and right image pair from respective video streams or image sequences obtained using the respective first and second cameras.

For example, if a user is asked to look at a non-moving object or a marker in his field of view, selecting the image pair, each ensuring its validity, can automatically determine that the user's gaze has stabilized, or analyze the trajectory of the predicted one Include gaze or use of a separate mechanism, such as a separately trained neural network, to detect saccades or fixation.

In addition, a video stream or image sequence can be used to generate multiple datasets. For example, datasets relating to different viewpoints/viewpoints may be captured using a video stream or image sequence acquired (e.g., recorded) with the first camera using a video stream or image sequence obtained with the second camera and a video stream or image sequence obtained with a scene camera of a respective head-mounted device are generated.

Typically, respective records relating to multiple users are created and added to (stored in) the database.

Accordingly, the feature generalization of the neural network in the training phase as well as the accuracy of predicting gaze-related parameters using the trained neural network can be further improved as more and more datasets are added from an increasing number of users and recorded under varying conditions.

Typically, records pertaining to at least 10 users, more typically at least 100 users, and more typically at least several hundred, several thousand, or even several tens of thousands or more users may be stored in the database.

For example, a third stimulus may be presented to a second user wearing the headworn device. The first camera of the head-worn device can be used when the second user is expected to respond to the third stimulus or is expected to have responded to the third stimulus, to capture a third left image of at least a portion of a left To generate the eye of the second user, and the second camera of the head-worn device can be used when the second user is expected to respond to the third stimulus or is expected to have responded to the third stimulus to generate a third generate a right image of at least a portion of a right eye of the second user. A third data set including the third left image, the third right image, and a third representation of the gaze-related parameter, the third representation correlating to the third stimulus, may then be created and added to the database.

Likewise, a fourth stimulus may be presented to the first user, the second user, or a third user wearing another headwear device. The other head-worn device includes a first camera positioned adjacent the left eye of the respective user and a second camera positioned adjacent the right eye of the respective user when the respective user wears the additional head-worn device . The first camera of the head-worn device can be used when the respective user is expected to respond or is expected to have responded to the fourth stimulus, to capture a fourth left image of at least a portion of a left eye of the respective user, and the second camera of the head-worn device can be used to generate a fourth right image of at least a portion of a right eye of the respective user. A data connection can be established between the further head-worn device and the database. A fourth data set including the fourth left image, the fourth right image, and a fourth representation of the gaze-related parameter, where the fourth representation correlates to the fourth stimulus, can be created and added to the database.

The data set(s) can be generated by the user, for example using a processing unit of the respective head-worn device and/or a connected desktop computer, tablet, laptop or smartphone.

The data set or data sets can also be stored locally until a connection to the computing system (server) is available.

In addition, the first camera of the head-mounted device or the other head-worn device can be used to generate another left image of at least a portion of the left eye of the first or second user and the second camera of the respective head portable device can be used to take another right image of at least a portion of the respective user's right eye if the respective user is expected to respond to the further stimulus or is expected to have responded to the further stimulus. Another data set including the other left image and the other right image can be created and added to the database.

The authors found that a certain proportion of weakly labeled datasets, i. H. Datasets with less precisely known gaze-related parameters, or datasets not labeled at all, i. H. Records for which no valid representations, e.g. B. values of the gaze-related parameter(s) are known, can nevertheless improve the performance of the NN trained with the database. The reason for this is that even unlabeled datasets include information about the contained image diversity.

For example, the database may include up to 70%, more typically up to 80%, and even more typically up to 90% or more of weakly and/or unlabeled records.

In one embodiment, more than one gaze-related parameter, for example two or three gaze-related parameters, is added to the database with at least one of the data sets.

The term "gaze-related parameter" as used in this specification shall mean a gaze direction, a cyclopean gaze direction, a 3D viewpoint, a 2D viewpoint, an eye pose as a 3D position and orientation, a pair of 3D gaze directions (left and right eye), an orientation of a visual axis, an orientation of an optical axis, an orientation of a pupil axis, an orientation of a line of sight, an orientation and/or a position and/or a closure of an eyelid, a pupil area, a pupil size, a pupil diameter, a sclera property, an iris diameter , a property of a blood vessel, a corneal property of at least one eye, a corneal radius, an eyeball radius, a distance from the center of the pupil to the center of the cornea, a distance from the center of the cornea to the center of the eyeball, a distance from the center of the pupil to the center of the limbus, a keratometric corneal refractive index, a corneal refractive index, a vitreous fluid br refractive index, a distance from the crystal lens to the center of the eyeball, to the center of the cornea and/or to the corneal apex, a crystal lens refractive index, a degree of astigmatism, an orientation angle of a flat and/or a steep axis, an orientation of the major and/or minor axis of the limbus, an ocular cyclotorsion, a distance between the eyes, eye vergence, eye adduction and/or eye abduction statistics, eye elevation and/or eye deepening statistics, cognitive load data, blink events, user drowsiness and/or alertness, and an iris assessment and/or detection parameter include the user. For example, points and directions may be specified in a scene camera image, an eye camera coordinate system, a scene camera coordinate system, a device coordinate system, head coordinate system, world coordinate system, or any other suitable coordinate system.

According to one embodiment, a given or resulting value of the gaze-related parameter(s) is determined for the given user who is expected to respond or has responded to the given stimulus.

The specific given or resulting value of the gaze-related parameter or parameters can be used as the respective representation of the gaze-related parameter or parameters of the data set or data sets.

The particular given or resulting value may be considered the actual or ground truth value of the gaze-related parameter.

The following is a data set that includes a left image of at least a portion of the left eye, a right image of at least a portion of the right eye, and a corresponding actual or ground truth value of one or more gaze-related parameters, such as gaze point or line of sight, also referred to as a labeled data set. Typically, the two images of a data set are taken at substantially the same time, i. H. within a period of at most 50 ms or even at most 10 ms.

Labeled datasets are particularly valuable for training NNs.

According to embodiments, a gaze-related parameter determining unit, in particular a respective unit of the respective head-worn device, is used to determine the respective given or resulting value of the gaze-related parameter, in particular the respective given or resulting gaze direction and/or the respective given or resulting point of view for the respective user.

In one embodiment, a scene camera arranged to capture a given object in the field of view of the respective user wearing the respective head-worn device is used to capture images that are used as a basis for determining a respective resulting value of the gaze-related parameter can become.

For example, in response to a particular stimulus, e.g. a visual stimulus and/or an acoustic stimulus, upon a given respective object defining a respective given gaze direction relative to a coordinate system specified for the respective head-worn device and/or a respective given gaze point in the coordinate system, or he is made to do so.

According to one embodiment, the respective user is prompted by a visual stimulus or an acoustic stimulus to look at a mark or an object, for example the tip of one of his fingers.

As already explained above, the respective user can be asked to look at his fingertip in different positions or to follow his moving fingertip with his eyes. Even a single stimulus can therefore be used to generate a sequence of data sets each including a respective left image, a respective right image and another resulting value of the respective gaze-related parameter(s) as a representation of the gaze-related parameter(s).

A respective left image and a respective right image may be generated when the respective user is expected to be looking at the respective given object, direction and/or viewpoint.

Then the resulting value(s) of the given gaze direction and/or viewpoint in the coordinate system specified for the respective head-worn device are determined using a scene image of the user's field of view when the respective User is looking at the given object, direction, and/or viewpoint. The scene image can be obtained using the scene camera.

Furthermore, determining and storing the resulting value(s) in the coordinate system specified for the particular head-worn device allows for their later use.

Determining the resulting value(s) may be accomplished using state-of-the-art machine learning, computer vision, or image recognition techniques.

Of course, it is assumed that the user or users behave cooperatively. Such behavior is to be expected since each user has an interest in improving the accuracy and reliability of the predictions of the NN being trained/post-trained using the database, and therefore of the quality of the data provided by the user(s). data sets provided.

This also applies to another embodiment in which a desired value of the gaze-related parameter is used to determine the position and/or appearance of an object to be displayed in the user's field of view. In the following, the desired value of the gaze-related parameter is also referred to as the given value of the gaze-related parameter.

For example, the user may be prompted to look at a given marker or object displayed at a given (desired) location on a screen associated with or attachable to the head-worn device.

For the sake of accuracy, the scene camera can have a resolution of at least 640 x 480 pixels, or at least 800 x 600 pixels, more typically at least 1024 x 768 pixels, and more typically at least 1280 x 1024 pixels, or at least 1920 x 1080 pixels (at least VGA or even SVGA).

Unlike the scene image or images, the resolution of the left and right images is typically comparatively low. The number of pixels of the left images and the typically equally large right images can be at most 40000, in particular at most 10000, in particular at most 5000 and more precisely at most 2500 or even 1000.

Even with left and right grayscale images with low resolution of 64 by 64 pixels, 50 by 50 pixels or even only 32 by 32 pixels, a surprisingly high reliability of the gaze direction/viewpoint prediction can be achieved using trained NNs.

In fact, in many cases the gaze direction/point of view can be accurately recognized even if the left image or the right image or even both images contain no pupil or only a portion of each pupil.

In embodiments relating to left and right eye IR cameras, a right IR light source of each head wearable device may be used to illuminate the right eye of each user and a left IR light source of each head mounted device portable device can be used to illuminate the left eye of the respective user. IR Illumination can only be used/invoked when the image quality is too low or expected to be low, for example in a dark environment. IR illumination can also be on permanently or always on and only off to save power and/or when image quality is acceptable without illumination.

Creating or adding a data set may include concatenating the respective left image and the respective right image. It should be noted that concatenated images can be presented directly to a 2-dimensional input layer of the NN.

In addition, creating or adding the respective data set may include storing a respective representation of another gaze-related parameter different from the gaze-related parameter, a respective user identifier, a respective user group identifier, and/or a device identifier of the respective headwearable device. The respective user group ID can also be part of the user ID.

Storing the respective user identifier, the respective user group identifier and/or the device identifier of the respective headworn device in the records of the database can enable the training of the neural network in a device in a specific, user-specific and/or user-group-specific manner.

For example, the neural network may be specific to children, adults, the elderly, people of the same ethnicity, women, men, a group of people from a common background, a group of people receiving drugs or an active substance such as alcohol, or the visually impaired or in otherwise disabled people, a particular device, a device class, a user identifier, a user group, and the like.

Typically, the database contains respective records of different users, conditions of the users, lighting conditions (indoor and/or outdoor lighting conditions), slip conditions of the worn head-worn device, and/or different distances between the user and a presented object at which the user is looking.

According to one embodiment of a method for training a neural network, in particular a neural convolutional network, the method includes providing a database which has a plurality of data sets, each of which has a respective left image, a respective right image and a respective corresponding representation of a gaze-related parameter , in particular a respective corresponding value of the gaze-related parameter. A neural network with a given architecture is provided. Neural network parameters are determined using the respective left images and respective right images of a subset or all datasets as input and the respective corresponding representations of the gaze-related parameters of the subset or all datasets as the desired neural network output.

For the sake of clarity, the method for training the neural network is also referred to as a training method.

According to one embodiment, a trained neural network, i. H. uses a neural network, which has been trained with all or selected data sets of the database, to generate, from a left image generated by the first camera and a right image generated by the second camera of the respective head-mounted device, predict a user's gaze-related parameter, typically in real time, particularly a user's eye-gaze direction and/or a user's eye-eye point when the user is wearing a respective head-worn device.

As already mentioned above, the training method can be user or user group specific.

Correspondingly, predictions of the neural network of particularly high reliability and/or accuracy can be achieved for the user or the user group.

In addition, the trained neural network can be used to determine another gaze-related parameter, an eye parameter and/or a physiological parameter of the user.

For example, the trained neural network may have been trained to recognize whether the user is (or is becoming) tired, mentally distracted, drunk, or the like based on the left and right images.

Instead of or in addition to using a user-specific trained neural network, a calibration method at the user's location can be used.

According to one embodiment, a method for calibrating a headwear device includes presenting a stimulus to a user wearing the headwear device, as discussed herein. The first camera of the head-worn device is used to generate a left image of at least a portion of the user's left eye and the second camera of the head-worn device is used to generate a right image of at least a portion of the right eye of the user wearing the headworn device and who is expected to respond to the stimulus or who is expected to have responded to the stimulus. A trained neural network, specifically a trained convolution neural network, is used as discussed herein to determine a predicted value of the gaze-related parameter using the right image and the left image as an input to the trained neural network. The desired value of the gaze-related parameter and the predicted value of the gaze-related parameter are used to determine a correction function for the user.

The correction function for the user can be stored locally, for example in a memory of the headwear device and/or in the database.

Differences between corresponding desired and predicted values of the gaze-related parameter can e.g. B. due to a peculiarity of a new user that is not yet sufficiently reflected in the database, a learning error or a change in the user over time.

In any case, the correction function can be used to immediately improve the network predictions for the specific user.

On the other hand, the corresponding left and right images and the desired (ground truth) values of the gaze-related parameter can be added as new datasets to the database.

In other words, one or more new datasets containing a respective left image, a respective right image, a respective desired gaze-related parameter, or other representation of the gaze-related parameter may be created and added to the database to provide an updated database generate.

Accordingly, the updated database covers a larger range of data sets and/or the performance of the post-trained network can be improved.

In addition, the updated database can be used to retrain the neural network. Accordingly, the performance of the post-trained network can be further improved.

The determined correction function can be used to correct predictions of the trained or post-trained NN for the user.

According to one embodiment, the first camera is used to generate a left image of at least a portion of the user's left eye and the second camera is used to generate a right image of at least a portion of the user's right eye showing the wears a wearable device on the head. The trained or post-trained neural network is used to determine another predicted value of the gaze-related parameter using the other right image and the other left image as an input to the trained neural network. The further predicted value is corrected by applying the correction function to the further predicted value to obtain a corrected value of the gaze-related parameter.

According to a further embodiment, the trained or post-trained neural network is used to determine a predicted value of another gaze-related parameter, different from the gaze-related parameter, using the right image and the left image as an input for the respective neural network, and the predicted value of the other gaze-related parameter and a desired value of the other gaze-related parameter associated with the stimulus or a far corresponding to the stimulus is used to determine a further correction function for the user.

The correction function or correction functions can be implemented as a polynomial function. For example, a polynomial of two input variables can be used as a respective correction function for a 2D gaze direction or a 2D viewpoint.

Typically, the correction function or functions are determined using different pairs of desired and predicted values of the gaze-related parameter or parameters.

The desired value can correspond to a given value (predetermined value) or a resulting value, and/or can be determined in a similar manner as above in relation to the method for creating and updating the database.

For example, the unit for determining gaze-related parameters described above can be used to determine the resulting value as a given value (ground truth value) of the respective gaze-related parameter, in particular when the user points to his fingertip or another object in his field of view that is being monitored by a scene camera as described, is watching and/or following her. Furthermore, the unit for determining gaze-related parameters is typically configured to determine a respective given value or resulting value as a respective representation of the gaze-related parameter in order to store them in a respective data set.

In embodiments where the user is asked to look at a given object on a screen, before displaying the object, the unit for determining gaze-related parameters may determine the position of the object on the screen according to a given value (predetermined value) of the determine a gaze-related parameter. Alternatively, the gaze-related parameter determination unit may determine the gaze-related parameter according to a randomly chosen position of the object on the screen.

The calibration method described may be performed at least in part by one or more processors of the respective head-worn device, such as an eyewear device as described herein, goggles, a head-wearable AR display and a head-wearable VR display, or by a or multiple processors of a local computer connected to the headworn device are performed and/or controlled.

The calibration method described may be invoked when a new user wears the headwear device for the first time, from time to time when headwear device slippage is likely, expected or detected, or at the request of the user.

According to one embodiment, a method for detecting one or more gaze-related parameters of a user includes creating a first image of a left eye of the user using a first camera of a head-worn device worn by the user, creating a right image of a right eye of the user using a second camera of the head-worn device, feeding the left and right images together as an input to a (trained) convolutional neural network, and obtaining the one or more gaze-related parameters from the convolutional neural network as a result of inputting the left one and right picture.

In the following, the method for detecting one or more gaze-related parameters of the user is also referred to as a recognition method.

The convolutional neural network is typically a trained neural network as discussed in this paper. Using such a trained convolutional neural network to obtain (determine) respective predicted values of the gaze-related parameter or parameters using the right image and the left image as input enables a particularly high accuracy and reliability of the prediction in real time with comparatively little computing resources.

Therefore, the processes of the recognition method can be partially or even fully executed by a processing unit or computer integrated in a wearable device, for example in a (standard) eyeglass frame of the headwear device.

Alternatively, the processing unit or computer may be partially or fully integrated into a desktop computer, local server, smartphone, tablet, or laptop connected to the head-worn device.

The left and right images can be concatenated before being input into the convolutional neural network become. Typically, the concatenated images are input to a two-dimensional input layer of the convolutional neural network.

In embodiments relating to equal-sized, square-shaped left and right images each having NxN pixels, the input layer is typically an Nx2N matrix.

N is typically less than or equal to 50, preferably less than or equal to 30, in particular less than or equal to 20.

In other embodiments, the left and right images are not square rectangles, but are typically the same size and shape as well.

Furthermore, the first and second images may not be pre-processed to obtain spatial and/or temporal patterns or configurations prior to being input to the convolutional neural network. This can further reduce the computation cost.

For example, the left and right images may not undergo feature extraction before being input to the convolution neural network.

Also, the output of the convolutional neural network is typically not post-processed to obtain (the predicted values of) the one or more gaze-related parameters.

The predicted value(s) of the gaze related parameter(s) can be used as input to another module, e.g. B. a consumer household related (software) module, a business (software) module and/or a medical (software) module.

Alternatively, the predicted value(s) are corrected using a custom correction function, as discussed herein, before being used as input to another module.

For example, the predicted or corrected value(s) may be used by other modules to determine which object on a screen the user is looking at, how long the user is looking at the object, whether the user is blinking while looking at the object, etc. This information can be used to enhance the user's interaction with a control device or computer connected to the display and/or head-worn device, or for advertising.

In embodiments relating to a headwear device with a display, the display of the headwear device may serve as the screen.

The convolutional neural network typically has at least 6 layers and preferably more than 10 layers.

The convolutional neural network can have between 12 and 30 layers, preferably between 16 and 20 layers.

The convolutional neural network typically uses a filter kernel or kernels of size M, where M ranges from 1 to 7, preferably 3 to 5.

It should be noted that filter kernel sizes are often specified as a single integer M. This means that the matrix of the filter is actually of size MxMxNc, where Nc is the number of channels in the corresponding layer of the network.

For example, if left and right RGB color images (3 channels) are used as input, the first convolution layer for M=5 can have exemplary 16 filter matrices or kernels of size 5x5x3, resulting in the next layer data with 16 channels, one from each Convolution with one of the sixteen 5x5x3 filters of the previous layer. Similarly, when left and right grayscale (1 channel) images are used as input, the respective filter matrices or kernels for M=5 are 5x5(x1) in size.

Any of the head-worn devices described herein for sensing one or more gaze-related parameters may be used.

In one embodiment, the first and second cameras may be located within a range of 32 to 40, preferably 34 to 38, more preferably 36 degrees with respect to the centerline plane of the head-worn device.

In another embodiment, the first and second cameras may be located within a range of 114 to 122, preferably 116 to 120, preferably 118 degrees with respect to the centerline plane of the head-worn device.

In both embodiments, the typically tiny cameras are used by the user not noticed at all when wearing the device.

For example, the first and second cameras can each have a volume of less than approximately 40 mm ³ or even 10 mm ³ .

The head-worn device may be part of a system capable of improving users' parameter prediction over time.

According to one embodiment, a system for improving the prediction of gaze-related parameters comprises at least one head-mountable device, a processing unit, and a computing system. The computing system is connectable to the processing unit and is configured to host a first database and to train the neural network, in particular a convolutional neural network, using the first database. The at least one head-mountable device includes a first camera for generating a left image of at least a portion of a user's left eye and a second camera for generating a right image of at least a portion of a user's right eye when the user carries head attachable device. The processing unit is connectable to the first camera and the second camera and configured to determine a predicted value of a gaze-related parameter of the user using the left image and the right image as input to the neural network. The system is configured to add records to the first database and transfer neural network parameters (or even a complete model of the NN) from the computing system to the processing unit. The added data sets include a respective left image, a respective right image, and a respective representation of the user's gaze-related parameter.

Typically, the system includes a variety of head mountable devices as discussed herein.

Accordingly, data sets from many users can contribute to updating the database and thus improving (by training or post-training using the database) the neural network transmitted to the user's location and used to predict the gaze-related parameter(s).

In one embodiment, a gaze-related parameter determination unit of the respective head-worn device is used to determine a given or resulting user gaze direction and/or viewpoint as the desired value of the respective gaze-related parameter.

The unit for determining gaze-related parameters may include a scene camera arranged to capture a field of view of the user wearing the respective head-mounted device.

It should be noted that not only the instance, but the implementation of the NN in the training phase (in the computing system) and the execution phase (at the user's site) can be different.

Reference will now be made in detail to various embodiments, one or more examples of which are illustrated in the figures. Each example is provided for illustrative purposes and is not intended to limit the invention. For example, features illustrated or described as part of one embodiment can be used on or in conjunction with other embodiments to yield a still further embodiment. It is intended that the present invention includes these modifications and variations. The examples are described using specific terms that should not be construed as limiting the scope of the appended claims. The drawings are not to scale and are for illustrative purposes only. For the sake of clarity, like elements or manufacturing steps have been given like reference numerals in the different drawings, unless otherwise indicated.

Referring to 1A until 1C A generalized embodiment of a headwear eyewear device for determining one or more gaze-related parameters of a user is shown. Actually through 1A and 1C a variety of embodiments are shown, these embodiments differing from each other with respect to the position of the cameras 14,24. Therefore, the eyeglass device is 1 in 1A shown with more than one camera 14, 24 per eyepiece opening 11, 21 solely for the purpose of representing the individual embodiments. In this embodiment, however, the spectacle device comprises no more than one camera 14, 24 associated with each eyepiece opening 11, 21.

1A Fig. 12 is a plan view of the eyeglass device 1 with the left side 10 of the eyeglass device 1 on the right side of the drawing sheet from 1A is shown and the right side 20 of the eyeglass device 1 on the left side of the drawing sheet of FIG 1A is shown. The eyeglass device 1 has a median plane 100 which coincides with a median plane of the user of the eyeglass device 1 when worn according to the intended use of the eyeglass device 1 . In relation to the intended use of the eyeglass device 1 by the user are a horizontal direction 101, a vertical direction 102, 100, an “up” direction 104, a “down” direction 103, a front direction 105 and a back direction 106 Are defined.

The eyeglass device 1 as in 1A , 1B , and 1C shown comprises a spectacle body 2 which has a frame 4 , a left holder 13 and a right holder 23 . In addition, the eyeglass body 2 defines a left eyepiece opening 11 and a right eyepiece opening 21 which serve to provide an optical window for the user to look through, similar to a frame or body of ordinary eyeglasses. A nose bridge portion 3 of the eyeglass body 2 is located between the eyepiece openings 11,21. With the help of the left and right holders 13, 23 and support members of the nose bridge portion 3, the eyeglass device 1 can be worn through the ears and nose of the user.

According to the through 1A In the embodiments shown, a left camera 14 and a right camera 24 can be arranged in the spectacle body 2 . In general, the nose bridge portion 3 or a lateral portion 12 and/or 22 of the goggle body 2 is a preferred location for arranging/integrating a camera 14, 24, particularly a micro camera.

The specific position of the respective camera 14, 24 can be provided by defining an angle of an optical axis 15, 25 of the respective camera 14, 24 with respect to the median plane 100.

If a camera 14 or 24 is arranged in the nose bridge section 3 of the spectacle body 2, the optical axis 15 of the left camera 14 is at an angle a of 142° to 150°, preferably 144°, measured counterclockwise (or -30° to -38°, preferably -36°) with respect to the median plane 100. Correspondingly, the optical axis 25 of the right camera 24 has an angle of inclination β of 30° to 38°, preferably 36°, in relation to the middle plane 100 .

When a position of a camera 14, 24 is in one of the side portions 12, 22 of the eyeglass body 2, the optical axis 15 of the left camera 14 has an angle γ of 55° to 70°, preferably 62°, with respect to the median plane and/or the optical axis 25 of the right camera 24 is inclined by an angle δ of 125° to 110° (or -55° to -70°), preferably 118° (or -62°).

The glasses device 1 off 1A represents various embodiments that have different positions of the cameras 12, 24, for example that both cameras 14, 24 are located in the nose bridge section 3, the left camera 14 is located in the left lateral section 12 and the right camera 24 is arranged in the right lateral section 22, or that the left/right cameras 14/24 are arranged in the nose bridge section 3 and the right/left cameras 24/14 are arranged in the right/left side sections 22/12.

In order to provide an additional or alternative specification for a position of a camera 14, 24 in the eyeglass body 2, specific camera arrangement zones 17, 18, 27, 28 are defined, a related technical teaching agreeing that the upper specification is the angles α, β, γ, δ of the optical axis 15, 25 is used. These camera array zones 17, 27, 18, 28 are virtual areas projected in a vertical plane perpendicular to the median plane 100. FIG.

In addition, a boundary cuboid 30 - in particular a rectangular cuboid - be defined by the optical openings 11, 21, which serve as four default positions of the camera arrangement zones 17, 27, 18, 28. As in 1A , 1B , and 1C shown, the bounding cuboid 30, represented by a dashed line, encloses a volume of both eyepiece openings 11, 21 and touches the left eyepiece opening 11 with a left side surface 31 from the left side 10, the right eyepiece opening 21 with a right side surface 32 from the right Side 20, at least one of the eyepiece openings 11, 21 with an upper surface 33 from above and a lower surface 34 from below.

In case a left/right camera 14, 24 is arranged in the nose bridge portion 3, a projected position of the left camera 14 is fixed in a left inner eye camera arrangement zone 17 and the right camera 24 is in the right inner eye camera arrangement zone 27 (will be translated into this projected).

When located in the left/right lateral section 12, 22, the left camera 14 is located in the left outer eye camera array zone 18 and the right camera 24 is located in the right outer eye camera array zone 28 when projected in the plane of the camera array zones.

Using the front view on the eyeglass device 1 shown in 1B 1, the positions of the eye camera array zones 17, 18, 27, 28 will be explained. In 1B rectangular squares represent the eye camera assembly zones 17, 18, 27, 28 in a vertical plane perpendicular to the median plane 100. The two inner eye camera assembly zones 17, 27 start at a distance of 2 mm from the median plane 100 to a distance of 14 mm in horizontal direction 101 in the left/right direction.

In a vertical direction 102, the inner eye camera assembly zones 17, 27 extend over 22 mm, starting at a distance of 9 mm from the upper surface 33 and down to a distance of 31 mm. Thus, the inner eye camera array zones 17, 27 have a size of 12 mm in the horizontal direction 101 and 22 mm in the vertical direction 102.

The left and right outer eye camera arrangement zones 18, 28 reference the respective left and right side surfaces 31, 32 of the bounding box 30. In this way, the respective outer eye camera arrangement zones 18, 28 start from a distance of 5 mm from the respective side surface 31, 32 in the direction of the middle plane 100 and extends to a distance of 6 mm from the respective side surface 31, 32 in the opposite direction away from the middle plane 100. Therefore, the respective side surface 31, 32 intersects the respective outer eye camera assembly zone 18, 28.

The size of the outer eye camera array zones 18, 28 in the vertical direction is preferably 28mm to 30mm.

As a preferred option, all embodiments of the spectacle device 1 as in 1A until 1C shown the commonality that no more than one camera 14/24 is assigned to one of the optical openings 11, 21; thus the spectacle device 1 comprises only two cameras 14, 24 for recording an image of a left and a right eyeball 19, 29.

The spectacle device 100 as in FIG 1A comprises a processing unit 7 configured to process the left and right images from the respective camera 14, 24 to determine the gaze-related parameter using at least one convolutional neural network. According to the present embodiments, the processing unit 7 is integrated in the holder in a non-visible way, for example inside the right holder 23 or the left holder 13 of the spectacle device 1. According to an embodiment not shown, a processing unit can be arranged in the left holder.

Referring to 2A embodiments of a method 1000 for creating and updating a database are explained. The database is typically used to train a neural network, in particular a convolutional neural network. This will be referred to below 3A explained in more detail.

In a first block 1100, a first stimulus S is presented to a user wearing a head-worn device. The headwear device may be any of the headwear devices described herein. It can be implemented as an eyeglass device, e.g. B. goggles, goggles, a head wearable AR display and a head wearable VR display. The headwear device has a first camera that is placed near a left eye of the user and a second camera that is placed near a right eye of the user when the first user wears the headwear device.

The database can be used to train the neural network to generate a gaze-related image from a pair of a left image generated by the first camera and a right image generated by the second camera as input to the neural network predict parameters of the user, in particular an instantaneous direction of the user and/or an instantaneous point of the user when the user wears the head-worn device.

Accordingly, the database may include a plurality of data sets that include pairs of left and right images and one or more corresponding values of gaze-related parameters that are considered respective actual values or ground truth values.

The database can be user specific.

More typically, the database includes records from a variety of users, i. H. a first user, a second user, a third user, a fourth user, and so on and so forth. For the sake of clarity, the following description is primarily based on an example user or first user.

Using datasets from a variety of users to train the neural networks can aid in training (generalizing) and thus improve the accuracy and/or reliability of the NN's predictions.

Also, the database typically includes records obtained using multiple head-worn devices, typically of the same type. At least the positions and orientations of the left and right cameras of the head-worn devices are typically at least substantially the same. Alternatively, the positions and orientations of the left and right cameras of the head-worn devices are also stored (encoded) in the datasets and used as inputs to train the neural network.

In a subsequent block 1200, the first and second cameras of the head-worn device record a respective image P _l , P _r , typically a respective photograph, of the user's left and right eyes when the user is expected to respond to the first stimulus responds or is expected to have responded to the first stimulus. The image Pi, P _r can also be selected from a respective video stream or sequence of images recorded using the first and second cameras.

In a subsequent block 1500, a data set consisting of or including the left image Pi, the right image P _r and a representation of a (desired) gaze-related parameter is filed (stored) in the database. The representation of the gaze-related parameter can be a given or specified respective (ground truth) value of the gaze-related parameter, but also any other representation such as a corresponding scene camera image that correlates with the first stimulus and is used to determine the (ground-truth) truth) value of the gaze related parameter is appropriate.

In one embodiment, the user wearing the head-worn device is prompted at block 1100 by an auditory stimulus, a visual stimulus, or a combination or sequence of auditory and visual stimuli to look (gaze) at a given object in their field of view ). The given object may be a real object in the user's field of view, such as a user's fingertip, an object in space, or an object displayed on a screen of the head-worn device.

The size of the given object should be sufficiently small and well defined to provide an unambiguous gaze target.

Accordingly, the coordinates and direction of the given object are sufficiently well-defined in a coordinate system established with respect to the head-worn device, and hence the user's head, to represent actual (or ground-truth) values of each viewpoint and the line of sight of the user wearing the head-worn device.

Cooperative behavior of the user can be assumed, since the user has an interest in improving the accuracy and reliability of the predictions of the trained NN, i. H. a NN that was trained using the records in the database and is therefore dependent on their quality.

In one embodiment, in block 1500, more than one gaze-related parameter, such as two or three gaze-related parameters, is added with the dataset to the database.

Instead of or in addition to actual values or other representations of a cyclopean gaze direction, in block 1500, for example, values or other representations of respective gaze directions from one or both eyes, the orientation of the pupil axis of one or both eyes, a 3D viewpoint or a 2D viewpoint with added to the record in the database.

As indicated by the left and right dashed arrows in 2A indicated, the method 1000 may then return to block 1100 to present another stimulus or the same stimulus again. It should be noted that in block 1500 multiple records may be added.

Presenting the same stimulus may be useful for statistical reasons, to obtain datasets under different conditions, for example for a rested and a tired user, or at different times and/or in different lighting conditions.

In addition, the stimulus need not correlate one-to-one with the resulting value of the gaze-related parameter(s).

For example, the user may hold the finger in different positions with respect to the device-defined coordinate system when prompted to look at a tip of one of their fingers held in their field of view. Although the fingertip, among other things, defines a respective resultant gaze direction relative to the coordinate system established with respect to the headwear device, the actual value of the resultant gaze direction may have yet to be determined for storage in the database. In these embodiments, a unit for Gaze related parameter determination, for example a gaze related parameter determination unit including a scene camera arranged to capture the given object in the user's field of view, typically a scene camera of the head wearable device, are used to determine an actual value of the gaze related parameter , for example to determine a resulting line of sight and/or the respective resulting point of view of the user.

Based on the scene camera image(s), the actual value(s) of the gaze-related parameter(s) may be determined using known machine learning, computer vision, or image recognition techniques. This can be done locally, i. H. using one or more processors of a processing unit of the head-worn device or a typically more powerful local computer connected to the head-worn device, or using one or more processors of a typically even more powerful computing system that also hosts the database . In the latter case, the dataset(s) may include the scene camera image(s) as representations of the gaze-related parameter(s).

For the sake of clarity, the following description focuses on the (2D or 3D) gaze direction and the (2D or 3D) gaze point (determined in the coordinate system fixed with respect to the head-worn device) as the gaze-related parameters . However, this is not to be construed as limiting. Those skilled in the art will understand that other gaze related parameters can be treated as well. For example, the system can be used to assess cognitive load. Relative changes in pupil diameter reflected in the recorded images may be associated with some sort of measure of cognitive stress. The cognitive load itself can be determined by means of an additional device, such as an EEG, which serves as the label (ground truth). The dataset then consists of the respective images and the measure of cognitive load.

Using a gaze-related parameter recognition or determination unit of the head-worn device to determine the actual value g of the gaze-related parameter is indicated by a block 1300 in FIG 2 B 10 is shown illustrating a method 1001 that is typically similar to method 1000 except for block 1300. FIG.

In embodiments where a screen of the wearable device is used to display objects as stimuli to the user, the actual value(s) g of the gaze-related parameter is typically known or given.

For example, for a given head-worn device, the object's viewpoint and/or line of sight can be calculated from the on-screen coordinates of the object. Alternatively, the viewpoint and/or line of sight can be given in the coordinates of the coordinate system established with respect to the head-worn device and the screen, respectively, and the desired (2D) coordinates of the displayed object on the screen can be calculated before the object appears on the screen.

In relation to 2C An embodiment of block 1500 used in methods 1000 and 1001 is discussed in more detail.

In a first sub-block 1510, a data connection is established between the head-worn device and the database.

Subsequent to or prior to block 1510, in a sub-block 1520, one or more data sets {P _l , P _r , {g}} are created. Each data set can have a respective left image, a respective right image and a respective representation of one or more gaze-related parameters {g} and be correlated with a respective stimulus S .

In subsequent sub-block 1550, the one or more records {P _l , P _r , {g} } are added to the database.

The data sets may include the left and right image pairs as a respective concatenated image. Concatenated images can be presented to a 2-dimensional input layer of the NN. Accordingly, typically in block 1520, the pairs of left and right images may be concatenated.

Unlike the scene image or images, the resolution of the left and right images can typically be comparatively low. The number of pixels of the left images and the typically equally large right images can be at most 10000, in particular at most 5000 and more precisely at most 2500.

Even with left and right grayscale images with a low resolution of 64 by 64 pixels, 50 by 50 pixels or even only 32 by 32 pixels or only 24 by 24 pixels, training can be performed using ter NNs a surprisingly high accuracy (e.g. with a mean angle prediction error of less than 2°) and reliability of the gaze direction/viewpoint prediction can be achieved.

In relation to 3A a method 2000 for training a neural network is explained.

In block 2100, a database DB is provided, which is as above in relation to 2A until 2C created and/or updated. The database DB has a large number of data sets, each of which contains a respective left image, a respective right image and a respective corresponding representation of a gaze-related parameter, e.g. B. a respective corresponding value of the gaze-related parameter.

Furthermore, in a block 220, a neural network NNW is provided with a given network architecture. The neural network NNW provided is typically a convolutional neural network (CNN).

The provided neural network NNW can be a previously trained NN.

Alternatively, the parameters (weights) of the provided neural network NNW may have been initialized with random values.

In block 2300, neural network parameters are modified using the respective left image and right image of a subset or all data sets as input and the respective corresponding gaze-related parameter representations of the subset or all data sets as the desired neural network output.

Block 2300 may include multiple learning cycles, each using one or more records of the database DB.

Typically, in block 2300, a deep learning technique, particularly a gradient descent technique such as back propagation, will be used to train the neural network NNW.

Finally, a trained or post-trained neural network tNNW can be output and/or stored.

In particular, the determined weights of the tNNW may be stored and typically later transmitted to a processing unit associated with or connectable to a head-worn device.

More typically, the determined weights of the tNNW are (later) transmitted to a plurality of respective processing units and/or head-worn devices.

Subsequently, one or more local instances of the trained neural network tNNW can be used to predict one or more gaze-related parameters of a respective user from the left and right images that are generated by the respective cameras of the head-worn device, typically in real time. when the user is wearing a head-worn device.

As in 3B 11 illustrating a detection method 3000, in blocks 3010, 3020, a left image P _l of at least a portion of a user's left eye may be captured by a first camera of a head-worn device worn by the user, and a right image P _r of at least a portion of a user's right eye may be captured by a second camera of the head-mounted device.

The left image P ₁ and the left image P ₁ are typically recorded substantially simultaneously.

In a subsequent block 3030, the left image P _l and the right image P _r can be input together as input to a trained convolutional neural network tNNW, typically as a concatenated image in an input layer of the trained convolutional neural network tNNW.

In a subsequent block 3040, a predicted value g _pr of one or more gaze-related parameters can be obtained from the trained convolutional neural network tNNW as a result of the input left and right images, typically as an output of an output layer of the trained convolutional neural network tNNW.

The predicted value(s) g _pr can be corrected in an optional block 3050 using a user specific correction function F _corr .

The predicted value(s) or the corrected predicted value(s) g _cpr can be output and/or used as input to an assessment module, e.g. B. a user interface module using gaze related user parameters.

Upon exiting block 3040 or block 3050, method 3000 may return to blocks 3010, 3020 as indicated by the dashed arrows.

In relation to 3C An embodiment of a method 4000 for calibrating a head-worn device is illustrated.

In a block 4010, a stimulus S is presented to a user wearing the headworn device. The stimulus S is correlated with a desired (expected) value ge of a gaze-related parameter.

In a subsequent block 4020, the first camera and second camera of the head-worn device are used to capture a left image P _l of at least a portion of the user's left eye and a right image P _r of at least a portion of the user's right eye , which is expected to respond (has responded) to the stimulus S.

Subsequently, in a block 4030, a trained neural network tNNW, in particular a trained convolutional neural network, is used to calculate a predicted value of the gaze-related parameter using the right and left images P _l , P _r as an input to the trained neural network to determine tNNW.

In a subsequent block 4040, a difference between the desired value g _e and the predicted value g _pr can be calculated and used to determine a correction function F _corr for the user.

As indicated by the dashed arrow in 3C specified, the correction function F _corr is typically determined after several cycles.

In relation to 3D an embodiment of a system 500 for improving the prediction of gaze-related parameters is discussed.

The system 500 includes a computing system 510 that hosts a database as discussed herein. The computing system 510 may be implemented/operate as or include a server for hosting the database. Computing system 510 may consist of a single computer or a plurality of interconnected computers.

The computing system 510 is configured to train a neural network using the database. To this end, the computing system 510 may include one or more hardware components particularly suited for obtaining and training NNs, particularly CNNs, such as GPUs.

In the exemplary embodiment, the computing system 510 is coupled to a processing unit 525, which is a portion of a head-mountable device 520 that includes a first camera for generating a left image of at least a portion of a user's left eye and a second camera for generating a right image of at least a portion of a right eye of the user when the user wears, is connectable or even forms part of the head-mountable device.

When connected to the left and right cameras, the processing unit 525 is configured to receive the left and right images.

In addition, the processing unit 525 is configured to determine a predicted value of a gaze-related parameter of the user using the left image and the right image as input to a neural network instance, particularly a convolutional neural network instance when executed in the processing unit.

The processing unit 525 also typically includes one or more hardware components particularly suited to executing NNs, particularly CNNs, such as a GPU.

The processing unit 525 is typically configured to: 3B control and/or execute explained method 3000 .

In addition, the processing unit 525 is typically configured to: 3C to control and/or execute the method 4000 explained.

In the illustrated connected state, processing unit 525 and computing system 510 are configured to transmit data sets from processing unit 525 to computing system 510, add data sets to the database, and transmit neural network parameters from computing system 510 to processing unit 525.

The computing system 510 is typically connected to one or more further processing units 535, which are connected to or even form part of a respective further head-mountable device 530, each having a respective first camera and respective second camera.

Processing unit 525 may operate as a client when connected to computing system 510 operating as a server.

Client(s) and server(s) are typically remote from each other and typically interact through a communications network such as a TCP/IP data network. The client-server relationship results from software running on the respective devices.

Typically, communication between processing unit 525 and computing system 510 uses encryption.

System 500, at least in a connected state, is capable of performing any of the methods discussed herein, particularly methods 1000 through 4000.

In addition, the system 500 is also typically configured to perform any of the processes discussed below.

In relation to 4 An embodiment of a system 600 for improving the prediction of gaze-related parameters is discussed. The system 600 is similar to the system 500 referred to above 3D and also includes a computing system 610 . For clarity, only one head-worn device 620 in 4 shown.

In the exemplary embodiment, the head-worn device 620 can connect to an exemplary companion device such as a tablet or smartphone 627 for control, power supply and (temporary) storage of generated data sets. The companion smartphone 627 can also be used to upload collected labeled datasets to the computing system 610 hosting the database, downloading (updating) a neural (convolutional) network if desired, as more fully described with reference to FIG 5 and 6 is described and to interact with the user. The communication link between the smartphone 627 and the head-worn device 620 can be a magnetically attached USB 2.0 (or higher) cable that exits the frame behind the wearer's left or right ear and connects to the USB-C port of the wearer phone is connected.

A companion app, i. H. a computer program designed to run on a mobile device such as a phone/tablet or watch (mobile app) running on the companion smartphone 627 may be the primary point of interaction with the user. The user can also control recordings, user profiles, calibrations and validations using the companion app. The user can also update and manage personal profiles, mesh models and calibrations with the app. These interactions can be small or minimal. The smartphone 627 can typically operate autonomously in a fully automated manner. The companion app can control the device and send firmware and model updates.

The head-worn device 620 may also include components that enable the orientation of the device in 3D space to be determined, accelerometers, GPS functions, and the like.

The headwear device 620 may further include any type of power source, such as a removable or rechargeable battery or a solar cell.

According to one embodiment, the calculation of the user's viewpoint is performed by a processing unit or controller 625 that is fully and invisibly integrated into a standard eyeglass frame of the device 620, an example of which is the Intel/Movidius Myriad2 VPU with CNN inference capabilities. In this case, a trained CNN model can be loaded via companion 627 and executed on integrated controller 625 .

The controller 625 can receive respective images from the left and right eye cameras of the device 620, perform inferences using the trained CNN, and sends the predicted gaze position data (or predicted values of other gaze-related parameters) to the connected companion smartphone 627 for display and storage.

The controller 625 may also forward the input image data from the eye cameras upon request, for example for data collection and/or use in a "refinement" as described below with reference to FIG 5 and 6 is described.

The controller 625 may also send the scene camera images of the device 620 to the companion smartphone for display and storage. The latter function can be achieved by a separate dedicated controller using the same USB connection.

Alternatively, a device-specific control device of the device 620 only controls the cameras and forwards the image data to the accompanying device 627, or even further to the computing system 610, for example a remote server and/or a cloud-based structure or the like without inferring the eye parameter(s) at the user's location.

In the companion smartphone 620, one or more DSPs, one or more special purpose chips, and/or one or more GPUs can be used for gaze estimation using CNN-based inference.

Regardless of which device performs the task of determining the eye parameter(s), the user can have real-time access to gaze data and scene video for their use cases. The companion phone 627 or other mobile device can also serve as an intermediary station, allowing the user to upload their recorded data to a desired location as well as upload the labeled training data to one or more servers, as described below.

Regarding the 5 and 6 Embodiments of a system 700 for improving the prediction of gaze-related parameters are discussed. The system 700 is typically similar to the systems 500 and 600 referred to above 3D or. 4 , and also includes a computing system 710 .

For the sake of clarity are included in the two 5 and 6 system 700 and its operation are shown. 5 refers to the computing system 710. 6 refers to a head-worn device 720 of the system 700 connectable to the computing system 710 (see also arrows a through d depicting an information flow).

An object of the present disclosure is to create, provide, apply and improve a so-called "universal neural network" (also referred to as universal NN model and "universal NN model") for determining eye parameters, in particular gaze direction information of users . Such a universal NN is suitable for the application in its basic embodiment in the unmodified state, without further calibration or setup steps by a "new" user, ie a user whose data sets were not used to train the NN.

Therefore, it is assumed that a so-called core data corpus of labeled datasets was collected and stored in a database DB, and a training algorithm based on NNs was trained based on this data.

The core data corpus can consist, for example, of data sets of image pairs showing the left and right eyes (or respective portions thereof) and corresponding ground truth viewpoints in scene camera image coordinates.

The core data corpus typically covers a large number of naturally occurring appearances of the eyes. Therefore, while recording training data, care can be taken to cover the entirety of the possible appearances of the eye images. This ensures that the universal NN works reliably on all images in practice.

Examples of factors that can be covered are differences in lighting conditions (including but not limited to outdoor lighting conditions), different positions of the headset (the head-mountable device) on the user's head (to reflect headset movement), differences in the user's physiological appearance, for example due to gender or ethnicity, short-term physiological changes (e.g., a user's eyes are more swollen or more open on one day than on others), and distance between the gaze target and the user, as described below (different gaze distances lead to different refractive powers of the eyes). In addition, the ground truth viewpoints should correspond to objects at different distances from the user to ensure that a large number of different states of binocular powers are present in the data corpus.

In addition, data augmentation techniques can be used to further diversify the core data corpus and increase the amount of data available.

To further improve the scope of the headset's slip configurations for a given user, slip simulation has proven beneficial. Using a pair of left and right eye images along with the pair's ground-truth gaze label, samples of random movements are generated generated that the headset might experience due to slipping/slipping. Given these movements and the fixed geometric relationships between all the cameras in the headset, projective transformations that produce the effect implied by the movements can be calculated. This allows further valid samples to be computed from labeled datasets without actually recording them.

Likewise, random erasure, in which randomly chosen areas in an image are erased and the corresponding pixels are replaced with random values, or an average image pixel value produces images with different levels of simulated occlusion, can generate additional "artificial" training datasets, which incurs additional costs brings, but promotes the diversity of the core data corpus.

A universal NN trained on such a core data corpus has then learned to generate or predict an output encoding an eye parameter such as the gaze direction of any user. In one possible example, the output is given as 2D pixel coordinates of the viewpoint in the scene camera image. Other encodings such as the 3D position of the viewpoint in a 3D coordinate system; or the 3D gaze ray in eye camera coordinates can also be implemented.

In a preferred embodiment, the system 700 is binocular, i. H. his cameras capture both eyes (or their condition). Binocular acquisition is important (necessary in most cases) to determine the exact position of the viewpoint, which can only be deduced from the binocular powers of the eyes. With a one-eyed system, only the user's 3D gaze direction can be inferred, while the precise depth of viewpoint cannot be inferred. The 3D gaze direction can also be used to calculate the exact gaze position by causing the gaze ray to intersect the real geometry, provided that geometry is known.

Regarding the NN model to be trained for use in the system and methods of the present disclosure, it was surprisingly found that convolutional neural networks (CNNs) despite the inherently challenging mobile or handheld setup of the device resulting in low resolution of the input images, and the difficult aspect of the optical sensor(s) placed inconspicuously are able to achieve sufficient accuracy.

While CNNs have been used for tasks such as object recognition where this can be beneficial, no attempt has been made to use them for full gaze prediction in a mobile environment. Given the wide variability in appearance of images in such an environment, resulting in images sometimes not even showing the pupil or showing only a small portion of the pupil, with large variations in lighting conditions and skin tone, it is unexpected that CNNs are trainable, to learn enough meaningful convolution kernels to provide a universal cross-user model for accurate gaze prediction based only on two low-resolution input images.

Thus, in accordance with an important aspect of the present disclosure, exploiting CNNs in a mobile eye tracking device enables a more stealthy device design and more reliable cross-user gaze detection (e.g., a mean angle prediction error of less than 2°).

It was found that the exact internal architecture of the network, i. H. the precise number, dimensions, and sequence of convolution and other layers is not critical to sufficiently good operation of the systems.

However, the network should preferably be highly efficient to allow for real-time interaction and embedded computation. Likewise, the precise individual values of the parameters resulting from the training process affect the final accuracy of the trained network only slightly.

In particular, network architectures optimized for small sizes or running on mobile or embedded platforms have proven to be suitable. Thus, candidates using CNNs include, without limitation, LeNet, SqueezeNet, MobileNet, Darknet, Resnet18, and any adaptations thereof. These network architectures are well known and therefore do not need to be described in detail here.

In a preferred embodiment, the neural network also uses one or more so-called "squeeze and excitation" (SE) blocks (layers). These blocks perform feature recalibration. Input data or features U (WxHxC, which corresponds to image width x image height x number of channels) are first passed through a squeeze operation that aggregates the feature maps across spatial dimensions WxH to produce a channel descriptor (1x1xC). The descriptor embeds the global distribution of feature responses per channel and allows information from the network's global receptive field to be utilized by its lower layers. This is followed by an excitation operation in which sample-specific activations, learned for each channel through a self-gating mechanism based on channel dependency, govern the excitation of each channel. The feature mappings U are then re-weighted per channel by these additional learned parameters to produce the output of the SE block that can be input directly into subsequent layers.

Once a core data corpus has been created and a CNN has been trained using the database DB (and the methods described herein), the parameters {tCNN} of the trained convolutional neural network tCNN can be uploaded to the memory of the head-worn device 720 .

6 shows an example of a viewing direction prediction process. The cameras of the device 720 record a live video stream of eye images, which is fed as an input data stream to the (integrated) processing unit 725, which uses a CNN module 7251 to apply the trained network tCNN to the images (the to forward images through the network tCNN) and thus calculates the predicted gaze direction g _pr . The optional scene camera can simultaneously record a video stream of images corresponding to a portion of the user's field of view wearing the device 720 . This video stream can be output by the device 720 along with the predicted gaze direction g _pr , or the gaze direction g _pr alone can be output via any of the device's data interfaces or stored in device memory. This results in a compact, portable, and fully embedded device 720 for end-to-end gaze estimation (end-to-end gaze direction determination) that enables accurate, real-time inference of gaze direction directly at the device by using a pre-trained and optimized neural network model is used.

The simplest usage scenario imaginable is that a user picks up the device 720, puts it on, and can start tracking without any further setup steps. For example, the results can be relayed and visualized in real-time to a local or remote device such as a cell phone, tablet, laptop, or any type of visualization device. At its simplest, the visualization consists of displaying the scene camera's video stream with the specified predicted viewpoint.

In another embodiment, a calibration (left in 6 , see also 3B) and performing a correction procedure (block 7252) as described herein for the user. It should be understood that a calibration step is optional since using a trained CNN allows the handheld device 720 to operate in a calibration-free mode with sufficient accuracy. However, performing an optional calibration procedure can produce other beneficial effects.

First, such a process can provide a simple user-specific correction function F _corr that can be applied on-the-fly to the predicted gaze g _pr output by the network tCNN to further and immediately improve the accuracy achieved for that specific user. A possible example of correction functions are simple bivariate polynomials with an adjustable degree. This correction can, for example, either only be applied during the current usage session or saved and also applied during a subsequent usage session by that user. The motive behind such an operation is that a universal mesh can hardly achieve 100% accuracy due to the physiology of the human eye. For example, a person-specific angle exists between the optical axis of each eye and that eye's actual visual axis or line of sight. This angle is usually different in a person's left and right eyes, varies across populations, and is unknown per se. In other words, a trained NN performs an averaging of this variation. Accordingly, a person-specific calibration can further increase the accuracy with regard to predictions.

Second, datasets from one or more calibration procedures performed by individual users can be used in different ways in conjunction with the existing core data corpus to improve the accuracy of the predictions.

For example, labeled calibration data from a specific user can be used to fine-tune the universal network into a personal network, which then makes more accurate predictions for that specific user. This can be done, for example, by retraining or reoptimizing parts of the neural network, i. H. only a subset of all parameters of the universal web, to better reproduce the ground truth of that specific user.

Third, labeled calibration datasets can be added to the core data corpus itself over time by a variety of different users (see dashed-dotted arrow in 5 ) as more users use the devices 720 and perform calibration procedures. The volume of the core data corpus can grow dynamically in this way, and the universal Network can be refined or retrained at intervals to provide an even more accurate universal neural network. This can be done from the start using the increased training data volume but reinitializing all model parameters, or retraining the universal NN can use the previous version of the parameters or part of them as initialization.

Thus, there is a triple additional benefit: the calibration procedure can immediately improve the accuracy for a specific user by generating a correction function F _corr to apply to the prediction of the universal NN, one or more of these operations can also turn the universal NN into a more accurate personal model, and calibration procedures performed by one or many users can even serve to improve the database DB and thus the universal NN, from which all users globally benefit immediately.

In one embodiment, the calibration method includes instructing a user wearing device 720 to look at a particular known landmark, pattern, or object in space whose coordinates are determined by a scene camera associated with or through device 720 is provided, recorded video images can be precisely determined automatically using state-of-the-art machine learning, computer vision, or image recognition techniques (block 7253 in 6 ). The image or images recorded by the cameras facing the user's eye(s) are used in block 7251 to predict the user's gaze direction (point of view). The offset of the predicted gaze direction (gaze point) g _pr and the expected gaze direction (gaze point) g _e defined by the marker position can then be calculated in a block 7254 and used to generate a correction map or function F _corr for now applying ( block 7252) upon the prediction of the universal NN to determine a calibrated gaze value _gcpr .

As an alternative to such an explicit calibration method, an implicit calibration method, such as based on previous viewpoints (e.g., mouse cursors on a computer screen) or salience maps, can be applied.

Possible application scenarios involving calibration are therefore as follows.

When a new user uses the device 720 for the first time, the user can be given the opportunity to perform a calibration, the result of which will be used in all subsequent usage sessions (one-time calibration). Alternatively, the user can be given the opportunity at the beginning of each new usage session or at specific time intervals, e.g. B. daily to perform a calibration.

Alternatively, users can invoke the calibration at their own will. In addition, each time a user performs a calibration, the corresponding calibration records can be automatically sent to the computing system 710 (next time a connection is made). Correspondingly, the computing system 710 can perform one of the refinement methods (updating the database DB, retraining the network tCNN) as described in this document. This can be done in an anonymous way since only eye images and ground truth coordinates need to be transmitted.

The calibration methods described yield pairs of images labeled with the ground truth gaze position that can be used to improve the universal NN as described above.

In addition to these labeled images, however, it is optionally possible to also collect and store images in data sets that are obtained in everyday use. These datasets cannot be labeled with the ground truth gaze position as this information is not available outside of calibration mode. However, a large corpus of unlabeled data still provides information on the image diversity it contains. Techniques from the fields of unsupervised or semi-supervised learning can be used to obtain this information to enhance the universal model or an individual-specific model in a manner similar to that described above.

In order to make it easier for a new user of the device 720 to use his person-specific model for gaze estimation together with other hyperparameters that may be set in his profile, it is possible to use automatic user identification and load the user's profile without the who must actively select it.

The user identification can be based on a separate learning-based algorithm. As training data for this algorithm, the training data from all known users of a particular instance of a handheld device, either by calibr ments or collected in everyday use.

Since the number of regular users of a specific instance of a handheld device 720 is small, for example under 50, a simple off-the-shelf learning algorithm such as a support vector machine or random forest suffices to train the user identifier.

Furthermore, identifying the user also allows detecting a) new users who have not yet been calibrated and b) when the current user last calibrated. In both cases, the user can then decide to perform a calibration, or the device can suggest a calibration to the identified user.

The device may also be embodied in configurations other than eyeglasses, such as integrated into the nosepiece or frame assembly of a head-mounted AR or VR display (HMD) or goggles or similar device, or as a separate nose clip supplement or module for use with such devices. In this case, the optical sensor feed can be fed into a general-purpose computer vision processing chip that handles the CNN calculation for gaze estimation among other tasks related to the AR/VR experience. This could also be a dedicated chip that only performs (CNN-based) gaze estimation, but with the chip residing on (or driven by) a PCB that also hosts other HMD components. The system can thus also include a display device or screen for presenting information to the user. The methods described herein apply without adaptation to such alternative apparatus and/or system

According to one embodiment, a system for improving the prediction of gaze-related parameters comprises at least one head-mountable device and a computing system connectable to the at least one head-mountable device and configured to host a database. The at least one head-mountable device includes a first camera for generating a left image of at least a portion of a user's left eye and a second camera for generating a right image of at least a portion of a user's right eye when the user uses the am carries head attachable device. The computing system and the at least one head-mountable device are configured to upload data sets from the at least one head-mountable device to the first database and the neural network parameters, in particular parameters of a convolutional neural network, from the computing system to the at least one head-mountable device download attachable device. The data sets include a respective left image, a respective right image, and a respective representation of a gaze-related parameter. The computing system is typically further configured to use the database to train the neural network and/or determine the parameters of the neural network.

While various embodiments of the invention have been disclosed, it will be apparent to those skilled in the art that various changes and modifications can be made that will achieve some of the advantages of the invention without departing from the spirit and scope of the invention. Those of ordinary skill in the art will recognize that other components that perform the same functions may be appropriately substituted. It should be mentioned that features explained with reference to a specific figure can be combined with features of other figures, even if this is not explicitly mentioned. Such modifications to the inventive idea are intended to be covered by the appended claims.

Although processes may have been presented in the figures in a particular order, unless otherwise noted, it is not to be construed that these operations must be performed in the particular order shown or sequentially in order to obtain desirable results. In certain circumstances, multitasking and/or parallel processing may be beneficial.

Relative spatial terms such as "below," "beneath," "lower," "above," "above," and the like are used for ease of description to explain the positioning of one element relative to a second element. These terms are intended to encompass different orientations of the device, as well as different orientations than those depicted in the figures. Further, terms such as "first," "second," and the like are also used to describe various elements, regions, sections, etc., and are also not intended to be limiting. Throughout the description, the same terms are used for the same elements.

As used herein, the terms “comprising,” “including,” “including,” “comprising.” send" and similar open-ended terms that indicate the presence of the named elements or features but do not exclude any other elements or features. The articles "a", "an", "the", "the", "the" are intended to include both the plural and the singular, unless the context clearly dictates otherwise.

With the various variations and applications in mind, it should be understood that the present invention is not limited to the foregoing description, nor is it limited by the accompanying drawings. Instead, the present invention is to be limited only by the following claims and their legal equivalents.

Reference List

1

glasses device

2

glasses body

3

nose bridge section

4

Frame

5

lighting device

6

scene camera

7

processing unit

10

left side

11

left eyepiece opening

12

left lateral section

13

left holder

14

left camera

15

optical axis (left camera)

16

17

left inner eye camera placement zone

18

left outer eye camera placement zone

19

left eye

20

right side

21

right eyepiece opening

22

right lateral section

23

right holder

24

right camera

25

optical axis (right camera)

26

27

right inner eye camera placement zone

28

left outer eye camera placement zone

29

right eye

30

bounding box

31

left side face

32

right side face

33

upper surface

34

lower surface

100

middle level

101

horizontal direction

102

vertical direction

103

down

104

upwards

105

in front

106

rear

a

Inner left camera angle 14

β

Inner right camera angle 24

g

Left outer camera angle 24

δ

Right outer camera angle 24

500-502

system

510

computing system/server

520, 530

head wearable eyewear device

525, 535

processing unit

>= 1000

Process, process steps

Claims (19)

A system comprising: - a head-mounted device having a left camera for capturing a left image of at least a portion of a left eye of a user wearing the head-worn device and a right camera for capturing a right image of at least a portion a right eye of the user wearing the head-mounted device; and - a processing unit configured to: ◯ obtain the left image; ◯ Obtaining the right image; ◯ inputting the left and the right image together into a convolutional neural network; and ◯ obtaining one or more gaze-related parameters from the convolutional neural network as a result of inputting the left and right images together. system after claim 1 , wherein the respective gaze-related parameter is related to at least one element of a list and/or is selected from the list, the list consisting of: a gaze direction, a cyclopean gaze direction, a 3D viewpoint, a 2D viewpoint, a viewing axis orientation, an optical axis orientation, a pupil axis orientation, a line of sight orientation, an orientation and/or a position and/or a lid closure, a pupil area, a pupil size, a pupil diameter, a sclera feature, an iris diameter, a feature of a blood vessel, a corneal feature of at least one eye, a corneal radius, an eyeball radius, a distance from the center of the pupil to the center of the cornea, a distance from the center of the cornea to the center of the eyeball, a distance from the center of the pupil to the center of the limbus, a keratometric refractive index of the cornea, a refractive index of the cornea, a refractive index of the vitreous humor, a distance of the crystalline n lens to the center of the eyeball, to the center of the cornea and/or to the corneal apex, a refractive index of the crystalline lens, a degree of astigmatism, an orientation angle of a horizontal and/or a vertical axis, a major and/or minor axis orientation of the limbus, a cyclotorsion of the eye, an eye intraocular distance, eye vergence, eye adduction and/or eye abduction statistics, and eye elevation and/or eye depression statistics, data on blink events, drowsiness and/or awareness of the user, and a parameter for verification and/or Identification of the user's iris. system after claim 1 or 2 wherein the first and second cameras are oriented in a range of 32 to 40 degrees, preferably 34 to 38 degrees, more preferably 36 degrees with respect to a median plane of the head-worn device. system after claim 1 or 2 wherein the first and second cameras are oriented in a range of 114 to 122, preferably 116 to 120, preferably 118 degrees with respect to a median plane of the head-worn device. A system as claimed in any preceding claim, wherein the processing unit is configured to concatenate the left and right images before inputting them to the convolution neural network. System according to any one of the preceding claims, wherein the processing unit is configured to feed the left and right images to the convolutional neural network as common input, in particular without preprocessing the left and right images to spatial and/or temporal patterns or arrangements receive. system after claim 6 , wherein the processing unit is configured to supply the left and right images as input to the convolutional neural network without prior feature extraction. System according to any one of the preceding claims, wherein the processing unit is configured to receive the result as an output of the convolutional neural network without post-processing and/or as an output of an output layer of the convolutional neural network. The system according to any one of the preceding claims, wherein the convolutional neural network comprises at least 6 layers and preferably more than 10 layers. system after claim 9 , wherein the convolutional neural network comprises between 12 and 30 layers, preferably between 16 and 20 layers. A system according to any one of the preceding claims, wherein the convolutional neural network comprises a two-dimensional input layer. system after claim 11 , where the input layer is an Nx2N matrix. system after claim 12 , where N is less than or equal to 50, preferably less than or equal to 30, in particular less than or equal to 20. The system of any preceding claim, wherein the convolutional neural network has a filter kernel of size M, where M is in the range 2-6, preferably 3-5. The system of any preceding claim, wherein the processing unit is configured to implement the foldable neural network, and/or wherein the processing unit comprises one or more hardware components particularly suited for the operation of foldable neural networks, such as a GPU. The system of any preceding claim, wherein the processing unit is integrated into the head-worn device. system after Claim 16 , wherein the head-worn device is a corresponding eyeglass device and/or wherein the processing unit is integrated into a standard eyeglass frame. The system according to one of the Claims 1 until 15 , wherein the processing unit is integrated into a desktop computer, a server, a smartphone, a tablet or a laptop. A computer program product or a computer-readable storage medium comprising instructions which, when executed by one or more processors of a processing unit of a system comprising a head-worn device connected to the processing unit and/or providing the processing unit, wherein the am A headwear device comprises a first camera and a second camera for generating eye images of a user when the user is wearing the headwear device, causing the system to: ◯ generating a left image of at least a portion of a left eye of the user using the first camera; ◯ generating a right image of at least part of a right eye of the user using the second camera; ◯ inputting the left and the right image together into a neural convolution network, in particular into a two-dimensional input layer of the neural convolution network; and ◯ Obtain one or more gaze-related parameters from the convolutional neural network as a result of inputting the left and right images together.

Images