CN113283329A

CN113283329A - Eye tracker, eye tracking method, eye tracking device, and recording medium

Info

Publication number: CN113283329A
Application number: CN202110544572.6A
Authority: CN
Inventors: 薛亚冲; 张�浩; 陈丽莉; 孙建康; 闫桂新; 吕耀宇; 张梦蕾; 李鑫恺
Original assignee: BOE Technology Group Co Ltd; Beijing BOE Optoelectronics Technology Co Ltd
Current assignee: BOE Technology Group Co Ltd; Beijing BOE Optoelectronics Technology Co Ltd
Priority date: 2021-05-19
Filing date: 2021-05-19
Publication date: 2021-08-20
Anticipated expiration: 2041-05-19
Also published as: CN113283329B

Abstract

The invention discloses a sight tracking system, an eye tracker, a sight tracking method, equipment and a medium, wherein the sight tracking system of one embodiment comprises a glasses type head-mounted device, a positioning device and a controller, wherein the glasses type head-mounted device comprises a positioning identifier arranged on the glasses type head-mounted device and a second eye image collector used for collecting and wearing a left eye image and a right eye image of a user; the positioning device is arranged corresponding to the external display device and is used for acquiring a pose image collector of the glasses type head-mounted device; the controller acquires the pose coordinates of the glasses type head-mounted device based on a world coordinate system according to the pose images and the positioning marks acquired by the positioning device, acquires the pupil coordinates of the left eye and the right eye based on the head-mounted coordinate system according to the left eye image, the right eye image and the positioning marks acquired by the glasses type head-mounted device, performs coordinate fusion on the pose coordinates, the pupil coordinates of the left eye and the pupil coordinates of the right eye, and acquires the sight line position of a wearing user staring at an external display device according to a preset sight line acquisition model.

Description

Eye tracker, eye tracking method, eye tracking device, and recording medium

Technical Field

The invention relates to the field of virtual reality interaction, in particular to a sight tracking system, an eye tracker, a sight tracking method, a sight tracking device and a sight tracking medium.

Background

The sight tracking device in the prior art is limited in use distance, the head movement range of a user wearing corresponding head-mounted equipment is limited, and the sight tracking precision of the prior sight tracking device is inaccurate in the sight tracking process. On the other hand, the current sight tracking device on the market is expensive, the basic cost is over ten thousand, and the higher price even reaches 10 to 30 ten thousand.

Disclosure of Invention

In order to solve at least one of the above problems, a first embodiment of the present invention provides a gaze tracking system comprising a glasses-type head-mounted device, a positioning device, and a controller, wherein:

the glasses type head-mounted device comprises a positioning identifier arranged on the glasses type head-mounted device, a first eye image collector used for collecting a left eye image of a wearing user and a second eye image collector used for collecting a right eye image of the wearing user;

the positioning device is arranged corresponding to an external display device and comprises at least two pose image collectors for collecting pose images of the glasses type head-mounted device;

the controller is used for acquiring a pose coordinate of the glasses type head-mounted device based on a world coordinate system according to the pose image acquired by the positioning device and the positioning identifier, acquiring a left eye pupil coordinate and a right eye pupil coordinate based on the head-mounted coordinate system according to a left eye image, a right eye image and the positioning identifier acquired by the glasses type head-mounted device, performing coordinate fusion on the pose coordinate, the left eye pupil coordinate and the right eye pupil coordinate, and acquiring a sight line position of the wearing user staring at the external display device according to a preset sight line acquisition model.

Further, the glasses type head-wearing device comprises a first glasses frame corresponding to a left eye, a second glasses frame corresponding to a right eye and a glasses frame, wherein,

the first spectacle frame comprises a first frame body, a first positioning mark arranged on the first frame body, a first bracket arranged on the first frame body and a first eye image collector fixed on the first bracket,

the second spectacle frame comprises a second frame body, a second positioning mark arranged on the second frame body, a second bracket arranged on the second frame body and a second eye image collector fixed on the second bracket,

the spectacle frame comprises a nose bridge and a third positioning mark arranged on the nose bridge, and the origin of coordinates of the head-mounted coordinate system is the central point of the third positioning mark; wherein

The first eye image comprises the first positioning identification and the third positioning identification, and the second eye image comprises the second positioning identification and the third positioning identification.

Furthermore, the positioning device is arranged at the bottom of the external display device and comprises a first position image collector, a first lamp group arranged corresponding to the first position image collector, a second position image collector and a second lamp group arranged corresponding to the second position image collector; wherein

The first position and posture image collector is arranged corresponding to the first frame body, and the origin of coordinates of the world coordinate system is the central point of the first position and posture image collector;

the second position and posture image collector is arranged corresponding to the second frame body, and the first position and posture image collector and the second position and posture image collector are symmetrically arranged relative to the external display device.

Furthermore, the first position posture image collector and the second position posture image collector are first infrared cameras, have a first resolution and a first frame rate, and are infrared lamp sets;

the first eye image collector and the second eye image collector are second infrared cameras, have second resolution and second frame rate, and are black; wherein

The first resolution is greater than or equal to the second resolution, and the first frame rate is less than or equal to the second frame rate.

Further, the controller comprises a coordinate fusion unit and a model calibration unit, wherein

The coordinate fusion unit comprises a coordinate transformation matrix which is used for respectively transforming the left-eye pupil coordinate and the right-eye pupil coordinate based on the head-mounted coordinate system into a left-eye pupil transformation coordinate and a right-eye pupil transformation coordinate based on the world coordinate system;

the model calibration unit is used for adjusting parameters of the sight line acquisition model according to a preset number of pupil calibration coordinates, wherein the pupil calibration coordinates are left eye pupil conversion coordinates and right eye pupil conversion coordinates which are obtained according to a preset number of calibration points displayed by the external display device and are based on the world coordinate system.

A second embodiment of the invention provides an eye tracker comprising the gaze tracking system as described above.

A third embodiment of the present invention provides a method for tracking a gaze by using the gaze tracking system, including:

receiving pose images acquired by at least two pose image collectors of a positioning device, and acquiring a pose coordinate of the glasses type head-mounted device based on a world coordinate system according to the pose images and a positioning mark on the glasses type head-mounted device;

receiving a left eye image collected by a first eye image collector of the glasses type head-mounted device and a right eye image collected by a second eye image collector of the glasses type head-mounted device, and respectively obtaining left eye pupil coordinates and right eye pupil coordinates based on a head-mounted coordinate system according to the left eye image, the right eye image and a positioning identifier on the glasses type head-mounted device;

and carrying out coordinate fusion on the pose coordinate, the left eye pupil coordinate and the right eye pupil coordinate, and acquiring the sight line position of the wearing user staring at the external display device according to a preset sight line acquisition model.

Furthermore, the glasses type head-mounted device comprises a first frame corresponding to a left eye, a second frame corresponding to a right eye and a frame, wherein the first frame comprises a first frame body, a first positioning mark arranged on the first frame body, a first support arranged on the first frame body and a first eye image collector fixed on the first support, the second frame comprises a second frame body, a second positioning mark arranged on the second frame body, a second support arranged on the second frame body and a second eye image collector fixed on the second support, the frame comprises a bridge and a third positioning mark arranged on the bridge, and the origin of the head-mounted coordinate system is the central point of the third positioning mark; wherein the first eye image comprises the first positioning identifier and a third positioning identifier, and the second eye image comprises the second positioning identifier and a third positioning identifier;

the positioning device is arranged at the bottom of the external display device and comprises a first posture image collector, a first lamp group, a second posture image collector and a second lamp group, wherein the first lamp group and the second lamp group are arranged corresponding to the first posture image collector; the first position and posture image collector is arranged corresponding to the first frame body, and the origin of coordinates of the world coordinate system is the central point of the first position and posture image collector; the second position and posture image collector is arranged corresponding to the second frame body, and the first position and posture image collector and the second position and posture image collector are symmetrically arranged relative to the external display device;

the position and orientation coordinates of the glasses type head-mounted device based on the world coordinate system further comprise three-dimensional coordinates and a yaw angle of the glasses type head-mounted device based on the world coordinate system, the position and orientation images collected by at least two position and orientation image collectors of the positioning device are received, and the position and orientation coordinates of the glasses type head-mounted device based on the world coordinate system are obtained according to the position and orientation images and the positioning marks on the glasses type head-mounted device further comprise:

carrying out image processing on a first posture image acquired by a first posture image acquisition device;

identifying and screening the third positioning identifier from the first position image after image processing according to the shape and the area of the third positioning identifier, and acquiring a first relative coordinate of the third positioning identifier in the first position image and a first three-dimensional coordinate of the third positioning identifier based on a world coordinate system;

identifying a first contour of the first positioning identifier from the first posture image after image processing and acquiring a first relative coordinate set of a first boundary of the first contour;

acquiring a first distance between the first mirror frame and the positioning device according to the first relative coordinate set and the resolution of the first position and posture image collector;

carrying out image processing on a second posture image acquired by a second posture image acquisition device;

identifying and screening the third positioning identifier from the second posture image after image processing according to the shape and the area of the third positioning identifier, and acquiring a second relative coordinate of the third positioning identifier in the second posture image and a second three-dimensional coordinate of the third positioning identifier based on a world coordinate system;

identifying a second contour of the second positioning identifier from the second posture image after the image processing and acquiring a second relative coordinate group of a second boundary of the second contour;

acquiring a second distance between the second mirror frame and the positioning device according to the second relative coordinate set and the resolution of the second position and posture image collector;

and acquiring the width of the first lens frame according to the first relative coordinate set, and acquiring the yaw angle of the glasses type head-mounted device according to the first distance, the second distance and the width of the first lens frame.

Further, the receiving a left-eye image collected by a first eye image collector of the glasses-type head-mounted device and a right-eye image collected by a second eye image collector of the glasses-type head-mounted device, and respectively obtaining left-eye pupil coordinates and right-eye pupil coordinates based on a head-mounted coordinate system according to the left-eye image, the right-eye image and the positioning identifier on the glasses-type head-mounted device further includes:

performing image processing on the left-eye image;

identifying a first positioning identifier and a third positioning identifier from the left eye image after image processing, detecting a left eye pupil and acquiring left eye pupil coordinates of the left eye pupil based on the head-mounted coordinate system;

performing image processing on the right eye image;

and identifying a second positioning identifier and a third positioning identifier from the right eye image after image processing, detecting a right eye pupil and acquiring a right eye pupil coordinate of the right eye pupil based on the head-mounted coordinate system.

Further, the controller comprises a coordinate fusion unit and a model calibration unit, wherein the coordinate fusion unit comprises a coordinate transformation matrix;

the coordinate fusion of the pose coordinate, the left-eye pupil coordinate and the right-eye pupil coordinate, and the acquisition of the sight position of the user gazing at the external display device according to the preset sight acquisition model further comprise:

converting the left-eye pupil coordinate into a left-eye pupil conversion coordinate based on the world coordinate system and converting the right-eye pupil coordinate into a right-eye pupil conversion coordinate based on the world coordinate system by using the coordinate conversion matrix; adjusting parameters of the sight line acquisition model by using the model calibration unit according to a preset number of pupil calibration coordinates, wherein the pupil calibration coordinates are left eye pupil conversion coordinates and right eye pupil conversion coordinates which are acquired according to a preset number of calibration points displayed by the external display device and are based on the world coordinate system;

and responding to the real-time received pose image collected by the positioning device and the left eye image and the right eye image collected by the glasses type head-mounted device, and acquiring the sight line position of the wearing user staring at the external display device by using the adjusted sight line acquisition model.

A fourth embodiment of the invention provides a computer-readable storage medium, on which a computer program is stored, which program, when being executed by a processor, is adapted to carry out the method as described above.

A fifth embodiment of the invention provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method as described above when executing the program.

The invention has the following beneficial effects:

aiming at the existing problems, the invention sets a sight tracking system, the embodiment of the invention respectively collects clear pose images and eye images through a positioning device and a glasses type head-wearing device which are designed in a split mode, converts a positioning identifier and binocular pupil coordinates of the eye images based on a head-wearing coordinate system into a world coordinate system of the pose images through a controller, and performs fusion of the binocular pupil coordinates and the pose coordinates of the binocular eye images to realize the fixation point detection of a long-distance wearing user, and realizes the sight tracking of the wearing user under the long-distance and large head action state through a sight acquisition model preset in the controller.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

FIG. 1 illustrates a block diagram of a gaze tracking system provided by one embodiment of the present invention;

fig. 2 is a schematic structural diagram of a glasses-type head-mounted device according to an embodiment of the invention;

FIG. 3 is a schematic structural diagram of a positioning device provided in an embodiment of the present invention;

FIG. 4 is a schematic diagram of an embodiment of the gaze tracking system;

FIG. 5 is a schematic diagram of a first pose image provided by an embodiment of the invention;

fig. 6 is a schematic view illustrating a center point of a glasses-type headset and a positioning device formed by using a binocular vision principle according to an embodiment of the present invention;

FIG. 7a illustrates an edge contour coordinate calculation region of a first contour provided by an embodiment of the present invention;

FIG. 7b is a schematic diagram showing a set of relative coordinates of the first and second profiles provided by an embodiment of the present invention;

FIG. 8 is a schematic top view of a positioning device and a glasses-type headset according to an embodiment of the present invention;

fig. 9 is a schematic diagram illustrating a preset calibration point displayed by an external display device according to an embodiment of the present invention;

fig. 10 is a flowchart illustrating gaze tracking using the gaze tracking system according to another embodiment of the present invention;

fig. 11 is a schematic structural diagram of a computer device according to another embodiment of the present invention.

Detailed Description

In order to more clearly illustrate the invention, the invention is further described below with reference to preferred embodiments and the accompanying drawings. Similar parts in the figures are denoted by the same reference numerals. It is to be understood by persons skilled in the art that the following detailed description is illustrative and not restrictive, and is not to be taken as limiting the scope of the invention.

It is to be noted that the relational terms such as first and second, and the like, described herein are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

For the sight line tracking device currently on the market, the use distance corresponding to the external display device is generally 40-80cm, and the head movement range in the up-down direction is about ± 12 ° and the head movement range in the left-right direction is about ± 20 ° after the user wears the device. Therefore, the sight line tracking device in the prior art has the problems of short use distance and limited action range. On the other hand, the current sight tracking device has a limited application range due to high manufacturing cost.

Based on the above problems of the gaze tracking device, the inventors have made extensive research and experiments to find out that the conventional gaze tracking device has the problems of short use distance, limited action range, high cost, etc. the following reasons:

the main hardware influencing the performance of the sight tracking device is an image acquisition machine, in order to acquire clear pupil images of a wearing user, the existing sight tracking device generally adopts an image acquisition device with a small field of view (FOV) and an ultrahigh resolution performance requirement, so that the use distance and the head movement range of the sight tracking device cannot be considered, and the image acquisition device with the high performance requirement further improves the overall cost of the sight tracking device. Further, when the using distance is far, the pupil of the wearing user is difficult to clearly shoot in the pupil image of the wearing user collected by the image collector, and the requirement of pupil detection is difficult to achieve, so that the accurate pupil position cannot be obtained, and the accuracy of sight tracking is greatly influenced.

Therefore, in light of the above problems and the causes of the problems, the inventors propose a gaze tracking system, an eye tracker, a gaze tracking method, an apparatus, and a medium.

As shown in fig. 1, fig. 2 and fig. 3, a first embodiment of the present invention provides a gaze tracking system, comprising a glasses-type head-mounted device, a positioning device and a controller, wherein:

In the embodiment of the invention, the glasses type head-mounted device and the positioning device are designed in a split mode, a user wears the head-mounted device, and the positioning device is arranged corresponding to the external display device. According to the embodiment of the invention, clear pose images and eye images are respectively acquired by the glasses type head-mounted device and the positioning device which are designed in a split mode, the positioning identification and the binocular pupil coordinates of the eye images based on the head-mounted coordinate system are converted into the world coordinate system of the pose images by the controller, the binocular pupil coordinates and the pose coordinates of the binocular eye images are fused, so that the fixation point detection of a user wearing the glasses at a long distance is realized, and the sight tracking of the user wearing the glasses under the long-distance and large-head-motion state is realized by the sight acquisition model preset in the controller.

In an alternative embodiment, as shown in fig. 2, the glasses-type headset 6 comprises a first frame corresponding to the left eye, a second frame corresponding to the right eye, and a frame, wherein,

the first frame comprises a first frame body, a first positioning mark 61 arranged on the first frame body, a first bracket arranged on the first frame body and a first eye image collector 64 fixed on the first bracket,

the second frame comprises a second frame body, a second positioning mark 62 arranged on the second frame body, a second bracket arranged on the second frame body, and a second eye image collector 65 fixed on the second bracket,

the spectacle frame comprises a nose bridge and a third positioning mark 63 arranged on the nose bridge, and the origin of coordinates of the head-mounted coordinate system is the central point of the third positioning mark; wherein

In this embodiment, the user wears the glasses-type head mount device, the first positioning identifier is disposed on the first frame, the second identifier is disposed on the second frame, the third identifier is disposed on the bridge of the nose between the first identifier and the second identifier, and the origin of coordinates of the head coordinate system is a central point of the third positioning identifier. When the eye test is performed, the external display devices sequentially display contents, the user is located at a position opposite to the external display devices, the eyes of the user can receive the display contents of the display pictures of the external display devices, the sight line correspondingly moves to form a fixation point, when the sight line of the eyes moves according to the change of the display contents, the first eye image collector 64 fixed on the first support collects the left eye image of the user, and the second eye image collector 65 fixed on the second support collects the right eye image of the user. In one specific example, the left eye image captured by the first eye image capture device includes the third positioning indicator, the first positioning indicator, and the left eye of the user. The right eye image collected by the second eye image collector comprises a third positioning identifier, a second positioning identifier and a right eye of the user.

According to the embodiment of the invention, the clear eye image is acquired by the image acquisition device arranged on the glasses frame, so that the controller can process according to the clear eye image, and the sight tracking precision is improved. In an alternative embodiment, the first and second eye image collectors are second infrared cameras having a second resolution and a second frame rate. The eye image collector provided by the embodiment of the invention is arranged on the glasses frame of the glasses, is suitable for the wide action range of a user, can quickly capture the pupil movement of the user in a close range, collects clear eye images, and is convenient for obtaining accurate eye pupil coordinates, thereby effectively improving the accuracy. In addition, the embodiment of the invention has lower performance requirements on the first image collector and the second image collector, can greatly reduce the cost of the sight tracking system of the embodiment of the invention on the basis of realizing high-precision sight tracking, and has wide application prospect.

In a specific example, the first eye image collector and the second eye image collector are both low frame rate, large field angle and high frame rate infrared cameras, which can further collect the eye images of the user under different actions with high efficiency, and improve the collection precision. In one specific example, the performance parameters of the infrared camera include: the resolution was 1920 x 1080, the diagonal FOV was 90 °, the frame rate was 60Hz, and clear head-mounted images were taken at a distance of 240 cm. The using distance of the sight line tracking system can reach 240cm, the head moving range transversely reaches 78.4 degrees, the longitudinal direction reaches 44.1 degrees, and the using distance and the head moving range of the sight line tracking system far exceed those of the sight line tracking system on the current market.

In a specific example, as shown in fig. 2, a first image collector of the glasses-type headset according to the embodiment of the present invention is located on a first bracket extending from a middle of a lower edge of a first frame, and a second image collector is located on a second bracket extending from a middle of a lower edge of a second frame, so as to improve accuracy of eye images collected by the two image collectors. In another specific example, the first image collector is located at a first frame top edge and the second image collector is located at a second frame top edge. Or, the first image collector may be located on an edge of the first frame body away from the third positioning identifier, and the second image collector may be located on an edge of the second frame body away from the third positioning identifier.

The invention does not limit the specific positions of the first image collector and the second image collector, and the technicians in the field set the positions of the first image collector and the second image collector according to the practical application, and take the eye image collected by each image collector and realize the acquisition of the pupil coordinate as the specific design criterion.

In an alternative embodiment, as shown in fig. 3, the positioning device 7 is disposed at the bottom of the external display device 8, and includes a first position image collector 71, a first lamp set 73 disposed corresponding to the first position image collector 71, a second position image collector 72, and a second lamp set 74 disposed corresponding to the second position image collector; wherein

In the embodiment of the invention, the positioning device is arranged at the bottom of the external display device. In a specific example, as shown in fig. 3, a first lamp set 73 surrounds a first posture image collector 71, a second lamp set 74 surrounds a second posture image collector 72, and under the irradiation of an infrared lamp emitted by each lamp set, the first posture image collector 71 collects a first posture image, and the second posture image collector 72 collects a second posture image. Further, the controller processes the first posture image and the second posture image, obtains a posture coordinate based on a world coordinate system from the processed first posture image and the second posture image, performs coordinate fusion according to the posture coordinate, the left eye pupil coordinate obtained based on the eye image and the right eye pupil coordinate obtained based on the eye image, and obtains a sight line position where the wearing user gazes at the external display device according to a preset sight line obtaining model, thereby realizing sight line tracking.

In one specific example, the controller can be integrated on the positioning device, for example, disposed in a space between the first and second attitude image collectors, which effectively reduces space occupation.

According to the embodiment of the invention, the pose image collector can collect the user pose image in a long distance, and the eye image collector can collect the pupil coordinates of the user in a short distance, so that the sight tracking can be accurately carried out, the use distance is effectively increased, the action amplitude of the user is increased, the use experience of the user is enhanced, and the eye image collector has a wide application prospect.

In an optional embodiment, the first and second position image collectors are first infrared cameras having a first resolution and a first frame rate, and the first and second lamp sets are infrared lamp sets. The first eye image collector and the second eye image collector are second infrared cameras, have second resolution and second frame rate, and are black; wherein the first resolution is greater than or equal to the second resolution, and the first frame rate is less than or equal to the second frame rate.

In the embodiment, the eye image collector and the pose image collector have low performance requirements, are common image collectors in the market, and can effectively reduce the overall cost of the sight tracking system on the basis of ensuring the sight tracking performance. In one specific example, the first and second position and orientation image collectors are high-resolution, large field angle, low frame rate infrared cameras. The first eye image collector and the second eye image collector are infrared cameras with low frame rate, large field angle and high frame rate. In one specific example, the performance parameters of the infrared camera employed include: the resolution was 640 x 480, the diagonal FOV was 100 °, and the frame rate was 180 Hz. According to the embodiment of the invention, the first lamp group and the second lamp group emit infrared light, so that the first position posture image collector, the second position posture image collector, the first eye image collector and the second eye image collector perform image collection in response to the infrared light of the corresponding lamp groups.

Furthermore, the first positioning mark 61, the second positioning mark 62 and the third positioning mark 63 in the embodiment of the invention are all black, so that the outline characteristics of the glasses type head-mounted device are more obvious under an infrared lamp, and the positioning accuracy is improved.

It should be noted that, in the embodiment of the present invention, the number and the setting positions of the first lamp group and the second lamp group are not limited, and a person skilled in the art sets the corresponding number and the corresponding positions according to actual requirements, so that the first position posture image collector, the second position posture image collector, the first eye image collector, and the second eye image collector perform image collection in response to infrared light as a design criterion, which is not described herein again.

In another specific example, the head-mounted glasses apparatus includes a third lamp group disposed on the first frame and a fourth lamp group disposed on the second frame, so that the first image collector and the second image collector can collect the eye image of the user in response to infrared light emitted from the corresponding lamp groups. The embodiment of the invention does not limit the number and the positions of the lamp groups corresponding to the image collector, and the image collector receives infrared light to collect the eye images of the user as a design criterion, which is not described herein again.

In the embodiment of the invention, the central point of the first posture image collector is used as the central point of the positioning device, and a corresponding world coordinate system is established. In a specific example, the first position image collector and the second position image collector of the embodiment of the present invention are symmetrically arranged, so that the coordinates of the second position image collector are conveniently located, and the tracking speed of the sight tracking device is increased.

As shown in fig. 4, a world coordinate system (Xw, Yw, Zw) is established with a central point of the first posture image collector 71 of the positioning device 7 as an origin, an extending direction from the first posture image collector 71 to the second posture image collector 72 is an Xw axis forward direction, a direction perpendicular to the external display device is a Yw axis forward direction, a direction toward the user is a Zw axis forward direction, an optical axis of the first posture image collector 71 is along a Z axis direction, if a distance between the central point of the first posture image collector 71 and the central point of the second posture image collector 72 is L, a central coordinate of the second posture image collector 72 is (L,0,0), and the optical axis of the second posture image collector is parallel to the Zw axis. Based on the first posture image, the second posture image and the positioning identifier, the controller can obtain the posture coordinate of the glasses type head-mounted device based on the world coordinate system.

Correspondingly, as shown in fig. 2, a head-wearing coordinate system is established by using the central point of the third positioning identifier 63 of the glasses-type head-wearing device 6, and based on the left-eye image, the right-eye image and the positioning identifier collected by the glasses-type head-wearing device, the controller obtains left-eye pupil coordinates and right-eye pupil coordinates of the glasses-type head-wearing device based on the head-wearing coordinate system.

In a specific example, the process of determining the pose coordinates and the eye image coordinates is described using the above-described world coordinate system and head-mount coordinate system as examples:

and S1, the controller receives the pose images acquired by the two pose image collectors, and the pose coordinates of the glasses type head-mounted device based on the world coordinate system are acquired according to the pose images and the positioning marks on the glasses type head-mounted device.

The positioning device 7 is arranged at the bottom of the external display device 8 as shown in fig. 4, after the user wears the glasses type head-mounted device 6, the controller outputs a collecting instruction, the first lamp set 73 and the second lamp set 74 emit infrared light, and the first posture image collector 71 and the second posture image collector 72 respectively collect the first posture image and the second posture image of the user in response to the infrared light. In one specific example, the parameters of the pose image collector include: the image resolution was 1920 x 1080, the diagonal FOV was 90 °, the frame rate was 60Hz, and a clear spectacle headset was photographed at a distance of 240 cm. Therefore, the use distance of the tracking system can reach 240cm, the head movement range of the user transversely reaches 78.4 degrees and the longitudinal direction reaches 44.1 degrees, the use distance and the head movement range of the sight tracking system far exceed those of the sight tracking system on the current market, and the sight tracking system has wide application prospect.

In an optional embodiment, the step S1 specifically includes:

and S11, processing the first posture image collected by the first posture image collector.

In one specific example, the controller performs Gaussian filtering on the acquired large-range first bit posture image, and filters noise in the first bit posture image to improve image definition. Further, thresholding extraction is performed on the first pose image, and the pose image including the glasses-type head mount is extracted from a wide range of first pose images, in one specific example, the threshold may be an empirical value of 50 for a 1920 x 1080 image resolution, and the thresholded first pose image is shown in fig. 5, and includes the complete first location indicator 61, second location indicator 62, third location indicator 63, and the pupil of the user (not shown in fig. 5).

S12, identifying and screening the third positioning identifier from the first posture image after image processing according to the shape and the area of the third positioning identifier, and acquiring a first relative coordinate of the third positioning identifier in the first posture image and a first three-dimensional coordinate of the third positioning identifier based on a world coordinate system.

In an embodiment of the present invention, the position and posture coordinates of the glasses type head device based on the world coordinate system include three-dimensional coordinates of the glasses type head device based on the world coordinate system, where the three-dimensional coordinates include the first relative coordinates of the third positioning identifier and the first three-dimensional coordinates of the third positioning identifier based on the world coordinate system.

In a specific example, the step specifically includes:

taking the example that the third positioning mark 63 on the glasses-type head-mounted device shown in fig. 2 is a black circular mark, the controller searches for a circular black image in the pose image collected by each pose image collector through an ellipse fitting and contour detection method to determine the third positioning mark, so as to position the coordinate of the central point of the glasses-type head-mounted device based on the world coordinate system.

To avoid false detection, in one specific example, the controller screens the detected circular black regions to improve the detection accuracy of the third positioning mark, considering that the controller can detect a plurality of circular black images. Specifically, the screening rule of the third positioning identifier is as follows: black strip regions are present on both left and right sides and lower sides of the filtered circular black image and are symmetrical with respect to the filtered circular black region, as indicated by the dashed line frame in fig. 5 as the correct filtered region.

In a specific example, the specific screening method is as follows: the left and right sides and the lower side of the detected circular black are imaged with a preset detection area for pixel value detection.

Because the size of the circular black image in the collected pose image is related to the distance between the glasses type head-mounted device and the positioning device, when the glasses type head-mounted device is close to the positioning device, the circular black image formed by the third positioning mark on the image is larger, and vice versa.

Thus, in one specific example, the preset detection area is K times the circular black imaging. For example, if the radius of the circular black image on the pose image to be currently detected is R, the preset detection area should be KR, that is, KR/2 pixels are detected leftward, KR/2 pixels are detected rightward, and (K-2) R pixels are detected downward by taking the current circular black image as the center.

In one specific example, the following detection results may be obtained:

a) the left side and the right side of the imaging area are not provided with black strip-shaped areas, and the lower side of the imaging area is not provided with black strip-shaped areas, so that the areas are interference areas and the current circular black imaging is filtered;

b) the left side and the right side of the imaging area are provided with black strip-shaped areas, and the lower side of the imaging area is not provided with the black strip-shaped areas, so that the areas are interference areas and the current circular black imaging is filtered;

c) the left side and the right side of the imaging device are not provided with black strip-shaped areas, and the lower side of the imaging device is provided with a black strip-shaped area which is an interference area for filtering the current circular black imaging;

d) the left side and the right side are provided with black strip-shaped areas, the lower side is provided with black strip-shaped areas, but the black strip-shaped areas on the left side and the right side are asymmetric relative to the current circular black imaging center, and the areas are interference areas to filter the current circular black imaging;

e) the left side and the right side are provided with black strip-shaped areas, the lower side is provided with black strip-shaped areas, the left side and the right side of the black strip-shaped areas are symmetrical about the current circular black imaging center, the areas are correct screening areas, and the circle center of the circular black imaging represents the position of a third positioning mark on the glasses type head-mounted device;

f) adjusting the preset detection area to be 1.2K times of the current circular black imaging in the area which does not meet the conditions, and returning to detect again; if the detection is not detected, the preset detection area is continuously adjusted until the detection is successful.

Further, after the detection is successful, the center coordinates of the circular black imaging are used for representing the center coordinates of the third positioning mark. For example, in the world coordinate system, the first circular black imaging center coordinate screened out in the first posture image of the first posture image collector is (a0, b 0). Since the resolution of the pose image is 1920 × 1080, the pose image is based on the pose center coordinates in the world coordinate system of (960,540). It follows that the first relative coordinates between the first circular black imaging center in the first pose image and the first pose image center are (a0-960, b0-540), i.e. the first relative coordinates (xL, yL) between the third location identity and the first pose image center are (a0-960, b 0-540).

Similarly, the controller can obtain a second relative coordinate (xR, yR) between the third positioning indicator and the center of the second pose image as (a1-960, b1-540) based on the second pose image.

Further, the three-dimensional coordinates of the glasses type head-mounted device in the first posture image under a world coordinate system are calculated by using a binocular vision principle. Specifically, a center point (third positioning mark) of the glasses-type head-mounted device and a visual schematic diagram of the positioning device formed by using a binocular vision principle are shown in fig. 6, a world coordinate system (Xw, Yw, Zw) is established by using the center point CL0 of the first posture image collector as an origin of the world coordinate system, fig. 6 shows an Xw direction and a Zw direction in the world coordinate system, and a distance between the second posture image collector CR0 and the first posture image collector CL0 is L. The filled black circle represents a first three-dimensional coordinate in space, labeled (x, y, z), of the center of the third location marker of the eyeglass headset. The hollow black circle between the glasses type head-mounted device and the positioning device is the central point of the pose image actually acquired by each pose image acquirer in the space.

According to the similarity law of triangles, the following principle is known:

considering the Yw direction of the world coordinate system not shown in fig. 6, it follows from the triangle similarity law:

namely, it is

In the above formula, (x, y, z) represents a three-dimensional coordinate of the third positioning mark center in the space based on the world coordinate system, f represents a focal length of the image collector and is a constant, (xL, yL) represents a first relative coordinate between the third positioning mark center and the first position posture image center in the first position posture image collected by the first position posture image collector, and (xR, yR) represents a second relative coordinate between the third positioning mark center and the second position posture image center in the second position posture image collected by the second position posture image collector.

And further sorting to obtain a third positioning identifier in the first posture image based on a first three-dimensional coordinate under a world coordinate system as follows:

namely:

therefore, the three-dimensional coordinates of the third positioning identifier in the first posture image under the world coordinate system can be obtained through the steps, and the three-dimensional coordinates comprise the first relative coordinates of the third positioning identifier and the first three-dimensional coordinates of the third positioning identifier based on the world coordinate system.

S13, recognizing the first contour of the first positioning mark from the first posture image after image processing, and acquiring a first relative coordinate set of a first boundary of the first contour.

Based on the detected center of the third positioning identifier of the glasses-type headset, the controller detects the first contour of the first positioning identifier by using a contour detection technology, specifically, the method comprises the following steps:

and carrying out contour detection in a preset contour detection area of M x N on the first posture image by taking the detected third positioning mark as a mark point. In one particular example, the size of M and N is related to the distance between the eyeglass-style headgear to the positioning device. The known headgear has a length and width of: (2w + e) × (h), wherein w is the width of the first frame or the second frame, h is the height of the first frame or the second frame, and e is the distance at the edge of the bridge of the nose glasses. In one specific example, the resolution of the binocular eye images acquired by the two image collectors on the positioning device is 1920 x 1080, and the transverse FOV and the longitudinal FOV of the two image collectors are: 78.4 ° 44.1 °.

In one specific example, at the vertical distance z between the center point of the third location marker and the center point of the location device, the range M × N of the head-mounted device on the image is calculated by the formula:

wherein g is a scaling multiple of the preset contour detection area, and the value is less than 1, for example, an empirical value of 0.6 is taken. In one specific example, the preset contour detection area is enlarged to avoid the condition that the detection of the contour of the head-mounted device is not complete, so that the accuracy of contour detection is improved.

The first positioning identification contour in the first posture image can be obtained after the contour detection is carried out, and the contour data of the contour can be stored in a pixel coordinate mode, so that first contour coordinate data of the first positioning identification based on a world coordinate system can be obtained through the pixel coordinate of the contour data.

As shown in fig. 7a, in one specific example, in order to improve the profile accuracy of the first profile, the profile of each side is provided with an edge profile coordinate calculation region. Taking the example of calculating the top edge profile of the first profile shown in fig. 7a, part of the coordinates are removed at a predetermined ratio to improve the coordinate accuracy of the edge profile. For example, the Yw direction coordinates of the edge contour coordinates in the top contour calculation region are sorted in sequence, the smaller value of 1/4 and the larger value of 1/4 in all the Yw direction coordinates are eliminated, the average value of the remaining 1/2Y coordinates is calculated, and the average value is used as the top contour coordinates, so that points with larger errors in the contour are eliminated, and the contour positioning accuracy is effectively improved.

The other edge contour coordinates of the first positioning mark are calculated in the same manner, wherein the left side contour coordinate and the right side contour coordinate calculate the average value of the X coordinates in the Xw direction, so as to obtain all the first relative coordinate sets of the first boundary of the first contour according to the embodiment of the present invention. In one specific example, as shown in fig. 7b, the four first relative coordinate sets of the first positioning identifier are: top profile coordinates (U1x, U1y), bottom profile coordinates (D1x, D1y), left side profile coordinates (L1x, L1y), right side profile coordinates (R1x, R1 y).

And S14, acquiring a first distance between the first lens frame and the positioning device according to the first relative coordinate set and the resolution of the first position and orientation image collector.

Considering that when a user wears the glasses type head-mounted device, the distances between the first frame and the positioning device and the distances between the second frame and the positioning device are different due to the fact that the head rotates, the positioning accuracy of the glasses type head-mounted device based on the world coordinate system is further improved by determining the distance between each frame and the positioning device. Specifically, the method comprises the following steps:

as shown in fig. 2, if the width of the first frame corresponding to the left eye of the user is w, the height is h, and the aspect ratio is w/h, the height h is | U1y-D1y | actually calculated,

as shown in fig. 8, when the eyeglass type headgear 6 is in the tilted state, the vertical positioning distance z from the center point of the third positioning mark to the positioning device 7 is a known state, and the first distance from the frame middle position of the first frame to the positioning device is (z-d1), and the second distance from the frame middle position of the second frame to the positioning device is (z + d2) due to symmetry. Knowing that the longitudinal field angle of the first position image collector is 44.1 degrees, for the first lens frame, at a first distance (z-d1), the longitudinal height of the shooting area of the first position image collector is H1, the longitudinal resolution of the first position image collector is N, and then:

a first distance (z-d1) of the first frame from the positioning device can thus be obtained.

Based on the above steps S11 to S14, attitude coordinate data such as the first relative coordinate in the first attitude image based on the third positioning mark, the first three-dimensional coordinate in the world coordinate system based on the third positioning mark, the first relative coordinate set of the first boundary of the first contour, and the first distance between the first frame and the positioning device can be obtained from the first attitude image.

Similarly, similar image processing and calculation are performed on the second pose image, so that pose coordinate data such as a second relative coordinate of the third positioning identifier based on the second pose image, a second three-dimensional coordinate of the third positioning identifier based on the world coordinate system, a second relative coordinate group of a second boundary of the second contour, a second distance between the second mirror frame and the positioning device, and the like are obtained, and the method specifically comprises the following steps:

and S15, carrying out image processing on the second posture image collected by the second posture image collector. In one specific example, the image processing may be the previously described gaussian filtered image processing.

S16, identifying and screening the third positioning identifier from the second posture image after image processing according to the shape and the area of the third positioning identifier, and acquiring a second relative coordinate of the third positioning identifier in the second posture image and a second three-dimensional coordinate of the third positioning identifier based on a world coordinate system.

The processes and principles of contour detection and coordinate positioning of the third positioning identifier in the second position and posture image are similar to those of the previous steps, and are not described herein again. In one specific example, the process obtains a second relative coordinate between the center of the second black circle region in the second pose image and the pose image center as (a1-960, b1-540), i.e., a second relative coordinate (xR, yR) between the third location identity and the second pose image center as (a1-960, b 1-540).

And a third positioning identifier in the second posture image is based on a second three-dimensional coordinate under the world coordinate system and comprises:

namely:

and the controller obtains a second relative coordinate of a third positioning identifier in the second pose image and a second three-dimensional coordinate of the third positioning identifier based on the world coordinate system.

And S17, recognizing a second contour of the second positioning identifier from the second posture image after the image processing, and acquiring a second relative coordinate group of a second boundary of the second contour.

In this step, the process of detecting the second positioning identification profile is similar to the process and principle of detecting the first positioning identification profile, and is not repeated here.

In one specific example, as shown in fig. 7b, all of the second relative coordinate sets of the second boundary of the second contour of the embodiment of the present invention may be: top profile coordinates (U2x, U2y), bottom profile coordinates (D2x, D2y), left side profile coordinates (L2x, L2y), right side profile coordinates (R2x, R2 y).

And S18, acquiring a second distance between the second lens frame and the positioning device according to the second relative coordinate set and the resolution of the second position and posture image collector.

As shown in fig. 2, if the aspect ratio of the second frame corresponding to the left eye of the user is w/h, h is calculated from the pixel coordinates as | U2y-D2y |. As shown in fig. 8, when the eyeglass type headgear is in the tilted state, the positioning distance z from the center point of the third positioning mark to the positioning device is a known state, the first distance from the middle position of the frame of the first frame to the positioning device is (z-d1), and the second distance from the middle position of the frame of the second frame to the positioning device is (z + d 2). Knowing that the longitudinal field angle of the second position posture image collector is 44.1 degrees, for the first lens frame, at the first distance (z-d1), the longitudinal height of the shooting area of the second position posture image collector is H1, the longitudinal resolution of the second position posture image collector is N, and then:

a second distance (z + d2) between the second lens frame and the positioning device can thus be obtained.

It should be noted that, in the process of determining pose coordinate data such as the second relative coordinate of the third positioning identifier based on the second pose image, the second three-dimensional coordinate of the third positioning identifier based on the world coordinate system, the second relative coordinate group of the second boundary of the second contour, and the second distance between the second lens frame and the positioning device from the second pose image, the step S11 to step S14 are referred to, and those skilled in the art can perform the above steps according to actual applications to obtain corresponding coordinate data from the second pose image.

And S19, acquiring the width of the first lens frame according to the first relative coordinate set, and acquiring the yaw angle of the glasses type head-mounted device according to the first distance, the second distance and the width of the first lens frame.

In one specific example, the position and posture coordinates of the glasses type head-mounted device based on the world coordinate system further comprise a yaw angle of the glasses type head-mounted device based on the world coordinate system. In this step, the actual first frame width w, i.e., w ═ R1x-L1x |, can be determined based on the first relative coordinate set, and as can be seen from fig. 8, the yaw angle α when the eyeglass type head set is tilted is:

in conclusion, the glasses type head-mounted device determined by the relative coordinates and the yaw angle of the glasses type head-mounted device is based on the position and posture coordinates (x, y, z and alpha) in the world coordinate system, the glasses type head-mounted device can be accurately positioned in the world coordinate system through the position and posture coordinates, the use distance is effectively increased, the action amplitude of a user is increased, the use experience of the user is enhanced, and the glasses type head-mounted device has a wide application prospect.

And S2, receiving the left eye image collected by the first eye image collector of the glasses type head-mounted device and the right eye image collected by the second eye image collector of the glasses type head-mounted device by the controller, and respectively obtaining left eye pupil coordinates and right eye pupil coordinates based on a head-mounted coordinate system according to the left eye image, the right eye image and the positioning identification on the glasses type head-mounted device.

In a specific example, the step specifically includes:

and S21, performing image processing on the left-eye image. In one specific example, the controller performs gaussian filtering on the acquired large-range first bit position image, and removes noise in the first bit position image to improve image definition. Further, the step also extracts an image including the glasses type head-mounted device and the eyes of the user as a left eye image by using a threshold extraction technology. In one specific example, the extraction threshold may be the aforementioned 50.

And S22, recognizing the first positioning mark and the third positioning mark from the left eye image after image processing, detecting the left eye pupil and acquiring the left eye pupil coordinate of the left eye pupil based on the head-mounted coordinate system.

As shown in fig. 2, a head-mounted coordinate system of the glasses-type head-mounted device is established with a third positioning identifier at the center of the frame of the head-mounted device as a coordinate origin, in a specific example, coordinates of a first image collector and coordinates of a second image collector on the glasses-type head-mounted device are respectively:

in one specific example, the controller performs detection in the left eye image by ellipse fitting and contour detection, and takes an image with a large contour area and a small major and minor axes as the detected pupil. The contour data of the pupil in the left-eye image is stored in a pixel coordinate mode, so that the left-eye pupil coordinate of the pupil of the user based on the head-mounted coordinate system can be obtained through the pixel coordinate of the contour data.

And S23, performing image processing on the right eye image. In one specific example, the image processing may be the aforementioned gaussian filtering processing.

And S24, identifying a second positioning identifier and a third positioning identifier from the right eye image after image processing, detecting a right eye pupil and acquiring right eye pupil coordinates of the right eye pupil based on the head-mounted coordinate system. In a specific example, the procedures and principles of identifying the contour manner and determining the right-eye pupil coordinates are as in step S22, and are not described herein again.

In one specific example, in steps S22 and S24, the boundary contour of the eyeglass type head mount based on the world coordinate system can be determined based on step S1, and thus, when pupil detection of the left-eye image and the right-eye image is performed, pupil detection is performed based on the detected boundary contour of each frame, thereby improving detection efficiency.

Based on the steps, the controller performs coordinate fusion on the pose coordinate, the left-eye pupil coordinate and the right-eye pupil coordinate, and acquires the sight line position of the wearing user staring at the external display device according to a preset sight line acquisition model.

In an optional embodiment, the controller includes a coordinate fusion unit and a model calibration unit, wherein the coordinate fusion unit includes a coordinate transformation matrix, and is configured to respectively transform left-eye pupil coordinates and right-eye pupil coordinates based on the head-mounted coordinate system into left-eye pupil transformation coordinates and right-eye pupil transformation coordinates based on the world coordinate system;

the model calibration unit is used for adjusting parameters of the sight line acquisition model according to a preset number of pupil calibration coordinates, wherein the pupil calibration coordinates are left eye pupil conversion coordinates and right eye pupil conversion coordinates which are acquired according to a preset number of calibration points displayed by the external display device and are based on the world coordinate system.

In this embodiment, coordinate transformation is performed by the coordinate transformation unit, coordinate transformation is performed on the head coordinate system of the glasses-type head-mounted device and the world coordinate system of the positioning device, and the model calibration unit adjusts the pupil calibration coordinate generated after transformation of the left-eye pupil transformation coordinate and the right-eye pupil transformation coordinate to obtain the sight line acquisition model, so that the sight line tracking of the wearing user is performed by using the split glasses-type head-mounted device and the positioning device, the sight line tracking of the wearing user in a long-distance state and a large head motion state is realized, and the tracking precision of the sight line tracking is effectively improved.

By way of a specific example, the process will be described on the basis of the foregoing steps S1 to S2:

and S31, converting the left eye pupil coordinate into a left eye pupil conversion coordinate based on the world coordinate system and converting the right eye pupil coordinate into a right eye pupil conversion coordinate based on the world coordinate system by using the coordinate conversion matrix.

Specifically, as shown in fig. 4, when the optical axis of the first image collector in the glasses type head-mounted device points to the center of the first frame of the positioning device, the optical axis of the first image collector is set to rotate by θ ° around the X axis, and then the conversion relationship between the world coordinate system and the head-mounted coordinate system is:

wherein, (Xc, Yc, Zc) represents pupil transformation coordinates in the world coordinate system, (Xt, Yt, Zt) represents left eye pupil coordinates or right eye pupil coordinates in the head coordinate system, and (x, y, z) is a three-dimensional coordinate in the world coordinate system based on the third positioning identifier obtained in the previous step.

In one specific example, the pupil calibration coordinates (Xc, Yc, Zc) are calculated as:

wherein f is_x、f_y、u₀、v₀Is constant and can be determined by using the Zhang camera calibration method; u, v represent head coordinatesAnd the horizontal and vertical coordinates of the tied pupil on the eye image acquired by the image acquisition device.

And S32, adjusting the parameters of the sight line acquisition model according to the preset number of pupil calibration coordinates by using the model calibration unit.

Based on the pose coordinate of the third positioning identifier in the world coordinate system and the pupil transformation coordinate of the pupil in the world coordinate system, the coordinate of the pupil in the glasses type head-mounted device can be mapped to the image shot by the positioning device.

In a specific example, in the embodiment of the present invention, the gaze tracking of the user is performed by establishing a gaze acquisition model, taking the gaze acquisition model as a polynomial regression model as an example, the model formula is as follows:

wherein a0, a1, …, a6, b0, b1, … and b6 in the sight line acquisition model are adjustment parameters, (X, Y) are pupil calibration coordinates based on a world coordinate system displayed by an external display device, and (X, Y) are three-dimensional coordinates based on the world coordinate system obtained after a user gazes the calibration points.

According to the formula of the sight line acquisition model, the number of the preset calibration points corresponding to the sight line acquisition model is at least six calibration points. In a specific example, as shown in fig. 9, the number of the preset calibration points 81 is nine, and the preset calibration points are uniformly distributed on the display screen of the external display device 8, so as to obtain accurate adjustment parameters, and effectively improve the tracking accuracy of the sight tracking.

It should be noted that, the number and the display order of the calibration points are not limited in the embodiments of the present invention, and those skilled in the art can select the corresponding number and the display order according to the practical application to adjust the sight line acquisition model to the optimal tracking state as the design criterion, which is not described herein again.

And S33, responding to the real-time received pose image collected by the positioning device and the left eye image and the right eye image collected by the glasses type head-mounted device, and acquiring the sight line position of the wearing user staring at the external display device by using the adjusted sight line acquisition model.

Furthermore, the external display device displays different contents, the eye image collector collects a left eye image and a right eye image of the user at a short distance in real time, the pose image collector collects a pose image of the glasses-type head-mounted display device worn by the user at a long distance in real time, and the controller receives the images in real time and acquires the sight position of the user in the current state in real time by using the sight line acquisition model adjusted to the optimal state, so that sight line tracking is realized.

In the whole tracking realization process, the pose image collector can collect the user pose images at a long distance, and the eye image collector can collect the pupil coordinates of the user at a short distance, so that the sight tracking can be accurately carried out, the use distance is effectively increased, the action amplitude of the user is increased, the use experience of the user is enhanced, and the eye image collector has a wide application prospect.

Corresponding to the gaze tracking system provided in the above embodiment, as shown in fig. 10, an embodiment of the present invention further provides a method for performing gaze tracking by using the gaze tracking system, the method including:

s1, receiving pose images acquired by at least two pose image collectors of the positioning device, and acquiring a pose coordinate of the glasses type head-mounted device based on a world coordinate system according to the pose images and a positioning mark on the glasses type head-mounted device;

s2, receiving a left eye image collected by a first eye image collector of the glasses type head-mounted device and a right eye image collected by a second eye image collector of the glasses type head-mounted device, and respectively obtaining left eye pupil coordinates and right eye pupil coordinates based on a head-mounted coordinate system according to the left eye image, the right eye image and positioning marks on the glasses type head-mounted device;

and S3, carrying out coordinate fusion on the pose coordinate, the left eye pupil coordinate and the right eye pupil coordinate, and acquiring the sight line position of the user staring at the external display device according to a preset sight line acquisition model.

In the process of tracking the sight line by using the method of the embodiment of the invention, clear pose images and eye images are respectively collected by the glasses type head-mounted device and the positioning device which are designed separately, the positioning identifier and the binocular pupil coordinates of the eye images based on the head-mounted coordinate system are converted into the world coordinate system of the pose images by the controller, and the binocular pupil coordinates and the pose coordinates of the binocular eye images are fused to realize the gaze point detection of a user wearing the eye at a long distance, and the sight line tracking of the user wearing the eye at a long distance and under a large head action state is realized by the sight line acquisition model preset in the controller.

In an alternative embodiment, as shown in fig. 2, the glasses-type head-mounted device includes a first frame corresponding to a left eye, a second frame corresponding to a right eye, and a frame, where the first frame includes a first frame body, a first positioning mark disposed on the first frame body, a first bracket disposed on the first frame body, and the first eye image collector fixed on the first bracket, the second frame includes a second frame body, a second positioning mark disposed on the second frame body, a second bracket disposed on the second frame body, and the second eye image collector fixed on the second bracket, the frame includes a bridge and a third positioning mark disposed on the bridge, and an origin of coordinates of the head-mounted coordinate system is a central point of the third positioning mark; wherein the first eye image comprises the first positioning identifier and a third positioning identifier, and the second eye image comprises the second positioning identifier and a third positioning identifier;

the positioning device is arranged at the bottom of the external display device, and as shown in fig. 3, the positioning device comprises a first posture image collector, a first lamp group arranged corresponding to the first posture image collector, a second posture image collector and a second lamp group arranged corresponding to the second posture image collector; the first position and posture image collector is arranged corresponding to the first frame body, and the origin of coordinates of the world coordinate system is the central point of the first position and posture image collector; the second position and posture image collector is arranged corresponding to the second frame body, and the first position and posture image collector and the second position and posture image collector are symmetrically arranged relative to the external display device;

the position and orientation coordinates of the glasses type head-mounted device based on the world coordinate system further comprise three-dimensional coordinates and a yaw angle of the glasses type head-mounted device based on the world coordinate system. Step S1, the receiving pose images collected by at least two pose image collectors of the positioning device, and acquiring the pose coordinates of the glasses-type head mount based on the world coordinate system according to the pose images and the positioning identifiers on the glasses-type head mount further includes:

In the embodiment, the glasses type head-mounted device can be accurately positioned in a world coordinate system based on the yaw angle and the three-dimensional coordinate, the use distance is effectively increased, the action amplitude of a user is increased, the use experience of the user is enhanced, and the glasses type head-mounted device has a wide application prospect.

In an optional embodiment, the step S2 "receiving the left-eye image collected by the first eye image collector of the glasses type head-mounted device and the right-eye image collected by the second eye image collector of the glasses type head-mounted device, and respectively acquiring left-eye pupil coordinates and right-eye pupil coordinates based on a head-mounted coordinate system according to the left-eye image, the right-eye image and the positioning identifier on the glasses type head-mounted device" further includes:

performing image processing on the left-eye image;

performing image processing on the right eye image;

In an optional embodiment, the controller comprises a coordinate fusion unit and a model calibration unit, wherein the coordinate fusion unit comprises a coordinate transformation matrix;

converting the left-eye pupil coordinate into a left-eye pupil conversion coordinate based on the world coordinate system and converting the right-eye pupil coordinate into a right-eye pupil conversion coordinate based on the world coordinate system by using the coordinate conversion matrix;

adjusting parameters of the sight line acquisition model by using the model calibration unit according to a preset number of pupil calibration coordinates, wherein the pupil calibration coordinates are left eye pupil conversion coordinates and right eye pupil conversion coordinates which are acquired according to a preset number of calibration points displayed by the external display device and are based on the world coordinate system;

By utilizing the sight tracking method of the embodiment of the invention, the use distance of sight tracking can reach 240cm, the head movement range of a user transversely reaches 78.4 degrees and longitudinally reaches 44.1 degrees, the use distance and the head movement range of the sight tracking system on the market are far beyond the use distance and the head movement range of the sight tracking system on the current market, and the sight tracking method has wide application prospect.

Since the gaze tracking method provided by the embodiment of the present invention corresponds to the gaze tracking systems provided by the above embodiments, the foregoing embodiments are also applicable to the gaze tracking method provided by the embodiment, and will not be described in detail in the embodiment.

Accordingly, the present invention further provides an eye tracker, which includes the gaze tracking system described in the above embodiments, and the foregoing embodiments are also applicable to the eye tracker provided in this embodiment, which will not be described in detail in this embodiment.

The foregoing embodiments and the advantages thereof are also applicable to the present embodiment, and therefore, the description of the same parts is omitted.

Another embodiment of the present invention provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements: receiving pose images acquired by at least two pose image collectors of a positioning device, and acquiring a pose coordinate of the glasses type head-mounted device based on a world coordinate system according to the pose images and a positioning mark on the glasses type head-mounted device; receiving a left eye image collected by a first eye image collector of the glasses type head-mounted device and a right eye image collected by a second eye image collector of the glasses type head-mounted device, and respectively obtaining left eye pupil coordinates and right eye pupil coordinates based on a head-mounted coordinate system according to the left eye image, the right eye image and a positioning identifier on the glasses type head-mounted device; and carrying out coordinate fusion on the pose coordinate, the left eye pupil coordinate and the right eye pupil coordinate, and acquiring the sight line position of the wearing user staring at the external display device according to a preset sight line acquisition model.

In practice, the computer-readable storage medium may take any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present embodiment, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

As shown in fig. 11, another embodiment of the present invention provides a schematic structural diagram of a computer device. The computer device 12 shown in fig. 11 is only an example and should not bring any limitation to the function and the scope of use of the embodiments of the present invention.

As shown in FIG. 11, computer device 12 is embodied in the form of a general purpose computing device. The components of computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.

Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, micro-channel architecture (MAC) bus, enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

Computer device 12 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by computer device 12 and includes both volatile and nonvolatile media, removable and non-removable media.

The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)30 and/or cache memory 32. Computer device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 11, and commonly referred to as a "hard drive"). Although not shown in FIG. 11, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.

A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of the described embodiments of the invention.

Computer device 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), with one or more devices that enable a user to interact with computer device 12, and/or with any devices (e.g., network card, modem, etc.) that enable computer device 12 to communicate with one or more other computing devices. Such communication may be through an input/output (I/O) interface 22. Also, computer device 12 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via network adapter 20. As shown in FIG. 11, the network adapter 20 communicates with the other modules of the computer device 12 via the bus 18. It should be appreciated that although not shown in FIG. 11, other hardware and/or software modules may be used in conjunction with computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.

The processor unit 16 executes various functional applications and data processing, such as implementing a gaze tracking method provided by embodiments of the present invention, by executing programs stored in the system memory 28.

It should be noted that, the sequence of the steps of the gaze tracking method provided in the embodiment of the present invention may be appropriately adjusted, and the steps may also be increased or decreased according to the circumstances, and any method that can be easily changed by a person skilled in the art within the technical scope of the present invention should be covered within the protection scope of the present invention, and therefore, the details are not described again.

It should be understood that the above-mentioned embodiments of the present invention are only examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention, and it will be obvious to those skilled in the art that other variations or modifications may be made on the basis of the above description, and all embodiments may not be exhaustive, and all obvious variations or modifications may be included within the scope of the present invention.

Claims

1. A gaze tracking system comprising a spectacle headset, a positioning device, and a controller, wherein:

2. The gaze tracking system of claim 1, wherein the eyeglass-style headgear comprises a first eyeglass frame corresponding to a left eye, a second eyeglass frame corresponding to a right eye, and a eyeglass frame, wherein,

3. The gaze tracking system of claim 2, wherein the positioning device is disposed at a bottom of the external display device and comprises a first pose image collector, a first light group disposed corresponding to the first pose image collector, a second pose image collector, and a second light group disposed corresponding to the second pose image collector; wherein

4. The gaze tracking system of claim 3,

the first position posture image collector and the second position posture image collector are first infrared cameras and have first resolution and first frame rate, and the first lamp set and the second lamp set are infrared lamp sets;

5. The gaze tracking system of any of claims 1-4, wherein the controller comprises a coordinate fusion unit and a model calibration unit, wherein

6. An eye tracker comprising the gaze tracking system of any one of claims 1-5.

7. A gaze tracking method using the gaze tracking system of any one of claims 1-5, comprising:

8. The gaze tracking method of claim 7,

the glasses type head-mounted device comprises a first frame corresponding to a left eye, a second frame corresponding to a right eye and a frame, wherein the first frame comprises a first frame body, a first positioning mark arranged on the first frame body, a first support arranged on the first frame body and a first eye image collector fixed on the first support; wherein the first eye image comprises the first positioning identifier and a third positioning identifier, and the second eye image comprises the second positioning identifier and a third positioning identifier;

9. The gaze tracking method of claim 8, wherein the receiving a left eye image collected by a first eye image collector of the glasses-type head mounted device and a right eye image collected by a second eye image collector of the glasses-type head mounted device, and the obtaining left eye pupil coordinates and right eye pupil coordinates based on a head-mounted coordinate system according to the left eye image, the right eye image and the positioning identifier on the glasses-type head mounted device further comprises:

performing image processing on the left-eye image;

performing image processing on the right eye image;

10. The gaze tracking method of claim 9, wherein the controller comprises a coordinate fusion unit and a model calibration unit, the coordinate fusion unit comprising a coordinate transformation matrix;

11. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 7-10.

12. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 7-10 when executing the program.