CN110263657B - Human eye tracking method, device, system, equipment and storage medium - Google Patents

Abstract

The embodiment of the invention discloses a human eye tracking method, a human eye tracking device, a human eye tracking system, human eye tracking equipment and a storage medium, wherein the method comprises the following steps: calling a first camera to acquire user images in a preset viewing area, and determining three-dimensional head position information corresponding to each user in the preset viewing area according to the user images; determining a first preset number of target second cameras from the plurality of second cameras according to the three-dimensional head orientation information; calling each target second camera to collect a face image of the user, and determining two-dimensional binocular pupil orientation information of the user according to each face image; and determining the three-dimensional binocular pupil orientation information of the user according to the at least two-dimensional binocular pupil orientation information. By the technical scheme of the embodiment of the invention, the calculation speed and the calculation precision can be simultaneously improved, and the method and the device can be suitable for a multi-user human eye tracking scene.

Description

Human eye tracking method, device, system, equipment and storage medium

Technical Field

Embodiments of the present invention relate to image processing technology, and more particularly to a method, apparatus, system, device and storage medium for tracking human eyes.

Background Art

Eye tracking technology is mainly used in human-computer interaction, naked-eye 3D display, virtual reality and other fields. It obtains the viewing point of a person by tracking the movement of the eyeball. In the current naked-eye 3D display, eye tracking is used to determine the current viewing position and modify the displayed image to reduce the left-eye crosstalk of the 3D image.

Existing eye tracking is mainly based on PCCR (Pupillary Corneal Reflection Technology) plus image processing and recognition, such as the Tobii eye tracker. The Tobii eye tracker uses a near-infrared light source to generate a reflected image on the cornea and pupil of the user's eye, and then uses two image sensors to collect the eye and reflected images. Then, based on the image processing algorithm and a three-dimensional eyeball model, the eye's spatial position and gaze direction are accurately calculated.

However, the existing eye tracking method does not have the ability to identify users and is only applicable to scenarios with a single user at close range, such as using a computer, VR glasses, eye examinations, etc. In addition, the existing eye tracking method usually first identifies the face area in the user image, and then calculates the pupil position of the user's eyes based on the face area. Since the face area in the user image may occupy a small pixel area, when the pupil position of the eyes is directly calculated in the user image, it will not be possible to improve the calculation accuracy and calculation speed at the same time.

It can be seen that there is an urgent need for a human eye tracking method that can improve both calculation speed and calculation accuracy.

Summary of the invention

Embodiments of the present invention provide a human eye tracking method, apparatus, system, device and storage medium, which can improve both calculation speed and accuracy, and can be applicable to multi-user human eye tracking scenarios.

In a first aspect, an embodiment of the present invention provides a method for tracking human eyes, comprising:

Calling the first camera to capture user images in a preset viewing area, and determining three-dimensional head orientation information corresponding to each user in the preset viewing area according to the user images;

Determining a first preset number of target second cameras from a plurality of second cameras according to the three-dimensional head orientation information;

Calling each of the target second cameras to capture a facial image of the user, and determining two-dimensional binocular pupil position information of the user according to each of the facial images;

The three-dimensional binocular pupil position information of the user is determined based on at least two of the two-dimensional binocular pupil position information.

In a second aspect, an embodiment of the present invention further provides an eye tracking device, comprising:

A three-dimensional head position information determination module, used to call the first camera to collect user images in a preset viewing area, and determine the three-dimensional head position information corresponding to each user in the preset viewing area according to the user images;

a target second camera determination module, configured to determine a first preset number of target second cameras from a plurality of second cameras according to the three-dimensional head orientation information;

A two-dimensional binocular pupil position information determination module, used for calling each of the target second cameras to collect the user's facial image, and determining the user's two-dimensional binocular pupil position information according to each of the facial images;

The three-dimensional binocular pupil position information determining module is used to determine the three-dimensional binocular pupil position information of the user based on at least two of the two-dimensional binocular pupil position information.

In a third aspect, an embodiment of the present invention further provides an eye tracking system, comprising: a first camera, multiple second cameras and an eye tracking device; wherein the eye tracking device is used to implement the eye tracking method provided in any embodiment of the present invention.

In a fourth aspect, an embodiment of the present invention further provides a device, the device comprising:

one or more processors;

A memory for storing one or more programs;

An input device, for collecting images;

Output device, used for displaying screen information;

When the one or more programs are executed by the one or more processors, the one or more processors implement the human eye tracking method provided by any embodiment of the present invention.

In a fifth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the eye tracking method provided by any embodiment of the present invention.

The embodiment of the present invention determines the three-dimensional head position information corresponding to each user in the preset viewing area based on the user image captured by the first camera, thereby realizing the tracking of the user's head. Based on the user's three-dimensional head position information, a first preset number of target second cameras are determined from multiple second cameras, so that the pixel area occupied by the pupil area of both eyes in the facial image captured by each target second camera is relatively large, so that the user's three-dimensional binocular pupil position information can be accurately and quickly determined based on multiple facial images, realizing high-speed tracking of the human eye and improving the calculation accuracy at the same time. In addition, the embodiment of the present invention uses multiple second cameras to capture the facial images of users in the preset viewing area, so that different second cameras can be called to capture facial images of different users at the same time, so that it can be applicable to multi-user human eye tracking scenarios.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a human eye tracking method provided by Embodiment 1 of the present invention;

FIG2 is a schematic diagram of a second camera matching method according to Embodiment 1 of the present invention;

FIG3 is an example of the distance sensitivity of light to the eye position in the depth direction according to the first embodiment of the present invention;

FIG4 is an example of displaying second three-dimensional head position information involved in Embodiment 1 of the present invention;

FIG5 is a schematic diagram of receiving facial image data according to Embodiment 1 of the present invention;

FIG6 is a flow chart of a method for tracking human eyes provided by Embodiment 2 of the present invention;

FIG7 is an example of a spiral search graph involved in Embodiment 2 of the present invention;

FIG8 is a schematic diagram of a parallel layout of second cameras at a second layout position involved in Embodiment 2 of the present invention;

FIG9 is an example of a three-layer data table corresponding to a second camera involved in Embodiment 2 of the present invention;

FIG10 is a schematic diagram of the structure of an eye tracking device provided by Embodiment 3 of the present invention;

FIG11 is a schematic diagram of the structure of an eye tracking system provided by Embodiment 4 of the present invention;

FIG12 is a layout example of a first camera when the preset viewing area is a circular inner area according to the fourth embodiment of the present invention;

FIG13 is a layout example of a first camera when the preset viewing area is a circular outer area according to the fourth embodiment of the present invention;

FIG14 is a layout example of a first camera when the preset viewing area is a straight line single-sided viewing area according to the fourth embodiment of the present invention;

FIG15 is an example of an intersection area of two adjacent second cameras involved in Embodiment 4 of the present invention;

FIG16 is a layout example of a second camera when the preset viewing area is a circular inner area according to the fourth embodiment of the present invention;

FIG17 is a layout example of a second camera when the preset viewing area is a circular outer area according to the fourth embodiment of the present invention;

FIG18 is a layout example of a second camera when the preset viewing area is a straight line single-sided viewing area according to the fourth embodiment of the present invention;

FIG19 is a schematic diagram of the structure of another eye tracking system provided by Embodiment 4 of the present invention;

FIG20 is a schematic diagram of the structure of a device provided in Embodiment 5 of the present invention.

DETAILED DESCRIPTION

The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

Embodiment 1

FIG1 is a flow chart of an eye tracking method provided by Embodiment 1 of the present invention. This embodiment is applicable to tracking and locating the pupils of both eyes of a user when the user is watching a 3D display screen. The method can be performed by an eye tracking device, which can be implemented by software and/or hardware and integrated into a device with a 3D display function, such as a naked-eye 3D advertising machine, a naked-eye 3D display, etc. As shown in FIG1 , the method specifically includes the following steps:

S110, calling a first camera to capture user images in a preset viewing area, and determining three-dimensional head orientation information corresponding to each user in the preset viewing area according to the user images.

Among them, the first camera may refer to a camera used to track the user's head. The first camera in this embodiment may be a 3D camera or a plurality of 2D cameras. Since the first camera is responsible for tracking the user's head, it does not require high facial details and tracking speed, so a first camera with a large viewing angle may be selected. The preset viewing area may refer to the area where the user is located when viewing the 3D display screen, which may be predetermined according to the shape and position of the 3D display screen. For example, if the 3D display screen is a circle and faces the center of the circle for display, the preset viewing area may be within the circular area formed by the circle. One or more users may be simultaneously viewing in the preset viewing area. In this embodiment, the number and shooting positions (i.e., layout) of the first cameras may be pre-set according to the shape and position of the 3D display screen, so that the total detection area of each first camera may cover the preset viewing area, so that the first camera may be used to track the head of each user in the preset viewing area. In this embodiment, at least 10 users' heads may be tracked simultaneously, and the specific number may be determined according to the shooting performance of the first camera. The three-dimensional head orientation information may include the user's three-dimensional head position information and head orientation information, wherein the head orientation information may be used to reflect the user's head state, such as looking up, lowering or tilting the head.

Specifically, each first camera can be called to collect user images in a preset viewing area, and the user's head can be tracked and located by using the visual positioning principle and other methods, and the user image can be processed by face matching and other technologies to identify different user heads, so that the three-dimensional head position information corresponding to each user in the preset viewing area can be calculated. In this embodiment, the first camera can preferably be a high-definition color camera, so that the user's hair color, skin color, etc. can be identified based on the RGB information in the collected user image, so that the user's three-dimensional head position information can be determined more accurately.

Exemplarily, the three-dimensional head position information corresponding to each user in the preset viewing area can be stored in the following data structure:

hid is the identification number corresponding to a tracked user's head, so as to distinguish different user heads. (x, y, z) is the three-dimensional head position coordinate of the user, and the unit can be set to mm. (angle_x, angle_y, angle_z) is the orientation vector corresponding to the user's head orientation. In this embodiment, the accuracy of the user's head rotation in the horizontal direction (i.e., the rotation angle along the y-axis) is greater than the accuracy of the other two directions (the rotation angles along the x-axis and z-axis).

S120. Determine a first preset number of target second cameras from a plurality of second cameras according to the three-dimensional head orientation information.

Among them, the second camera may refer to a camera used to track the pupils of both eyes of the user. In this embodiment, the second camera may be a 2D camera. In this embodiment, the specific number and shooting positions (i.e., layout) of the second cameras can be pre-set according to the shape and position of the 3D display screen, so that the total detection area of each second camera can cover the preset viewing area, so that the second camera can be used to track the eyes of each user in the preset viewing area, thereby realizing multi-user eye tracking. The target second camera may refer to the second camera that is selected from multiple second cameras to capture the best user facial image. The first preset number can be set according to business needs and scenarios. The first preset number in this embodiment may be at least two.

Specifically, for each user, the target second camera corresponding to the user can be determined from multiple second cameras based on the user's three-dimensional head orientation information, so that the pixel area occupied by the pupil area of both eyes in the facial image captured by each target second camera is larger, thereby ensuring the resolution and improving the accuracy of human eye tracking. Exemplarily, in this embodiment, the second camera can be a black and white camera, and an infrared light source for illumination is set at the installation position of the second camera so that image acquisition can be performed. In this embodiment, the resolution of the second camera can be smaller than the resolution of the first camera to further improve the image processing speed.

Exemplarily, S120 may include: determining a second preset number of candidate second cameras corresponding to the user and a matching degree corresponding to each candidate second camera based on the three-dimensional head orientation information and the orientation configuration information of each second camera; and screening out a first preset number of target second cameras from each candidate second camera based on the matching degrees and the current number of calls corresponding to each candidate second camera.

The position configuration information of the second camera may include but is not limited to the installation position, resolution, depth of field parameter and viewing angle range of the second camera. Exemplarily, the position configuration information of each second camera may be stored in the following data structure:

Wherein, cid is the identification number corresponding to the second camera, so as to distinguish different second cameras; (x, y, z) is the installation position coordinate corresponding to the second camera, and the unit can be mm; (angle_x, angle_y, angle_z) is the direction vector corresponding to the shooting center of the second camera. Width and height represent the resolution of the second camera respectively; fov_h and fov_v are the viewing angles of the second camera in the horizontal and vertical directions respectively, and the unit is degree. Dof is the depth of field parameter of the second camera. Type is the layout mode of the second camera, such as circular inward type, circular outward type and plane type, wherein the circular inward type may mean that the shooting direction of the second cameras distributed on the circle is all toward the center position of the circle; the circular outward type may mean that the shooting direction of the second cameras distributed on the circle is all away from the center position of the circle; the plane type may mean that the second cameras distributed on the straight line are all shooting towards the viewing area on one side of the straight line. In this embodiment, the corresponding screening optimization strategy can be selected based on the layout mode. It should be noted that the installation position coordinates (x, y, z) corresponding to each second camera stored in this embodiment refer to the coordinates in the world coordinate system for data matching.

Specifically, this embodiment matches the three-dimensional head orientation information with the orientation configuration information of each second camera, determines a second preset number of candidate second cameras that can capture the user's facial image, and can determine the matching degree corresponding to each candidate second camera according to the angle between the user's head orientation direction and the center line of the shooting angle of each candidate second camera. For example, the larger the angle between the head orientation direction and the center line of the shooting angle of view, the smaller the matching degree corresponding to the candidate second camera. This embodiment can also detect whether there is a shooting occlusion according to the positional relationship between multiple users in the preset viewing area, so as to adjust the matching degree corresponding to the candidate second camera. For example, if a user B blocks user A when a candidate second camera is used to shoot user A, the matching degree corresponding to the candidate second camera can be reduced, or the matching degree corresponding to the candidate second camera can be set to the minimum to avoid using the candidate second camera to collect images of user A. In this embodiment, when tracking multiple users at the same time, each second camera may correspond to multiple shooting tasks, and each shooting task corresponds to a call of the second camera, so that shooting can be performed for different users. The current number of calls corresponding to the candidate second camera can refer to the number of shooting tasks to be executed corresponding to the candidate second camera at the current moment. In this embodiment, the first preset number is less than or equal to the second preset number. The size of the second preset number can be determined according to the business scenario and the actual operation situation. When the first preset number is less than the second preset number, the best second cameras of the first preset number, i.e., the target second cameras, can be further screened out from the second preset number of candidate cameras based on the matching degree and the current call count corresponding to each candidate camera. When the first preset data is equal to the second preset number, each determined candidate second camera can be directly determined as the target second camera.

Exemplarily, FIG2 shows a schematic diagram of a second camera matching. As shown in FIG2, the layout of the second camera is circular inward, that is, multiple layout positions are evenly distributed on the circle, multiple second cameras can be installed at each layout position, and the shooting direction of each camera is toward the center of the circle. In FIG2, the dotted line indicates the direction of the user's head, c1 refers to the shooting angle corresponding to the second camera C1 located at layout position 1; c2 refers to the shooting angle corresponding to the second camera C2 located at layout position 2; c3 refers to the shooting angle corresponding to the second camera C3 located at layout position 2; c4 refers to the shooting angle corresponding to the second camera C4 located at layout position 3. The head direction of user A in FIG2 is between layout position 1 and layout position 2. For user A, an optimal second camera can be matched at both layout position 1 and layout position 2. According to the shooting angle of each second camera at the layout position, the optimal second camera at the layout position can be determined. As shown in FIG2, the two target second cameras matched for user A are the second camera C1 and the second camera C2. Similarly, two target second cameras are matched for user B, namely second camera C3 and second camera C4.

S130, calling each target second camera to collect a facial image of the user, and determining two-dimensional binocular pupil position information of the user according to each facial image.

The two-dimensional binocular pupil position information may include the user's two-dimensional binocular pupil position information and eye gaze direction information, wherein the two-dimensional binocular pupil position information includes two-dimensional left eye pupil position information and two-dimensional right eye pupil position information. Exemplarily, the user's two-dimensional binocular pupil position information may be stored in the following data structure:

Among them, (left_x, left_y), (right_x, right_y) represent the two-dimensional position coordinates of the left and right pupils, wherein the x-axis direction can be from left to right, and the y-axis direction can be from top to bottom; (angle_x, angle_y) represents the angle of the eye gaze direction, wherein the x-direction can be divided into four levels: "none", "left", "middle" and "right", and the corresponding y-direction can be four levels: "none", "up", "middle" and "down", wherein "none" represents an uncertain direction. Time can be the time point corresponding to the calculation of the pupil position of the eye, and the unit can be ms, so that the time point can be used to distinguish different pupil positions of the eye. Mode represents the detected eye mode, such as the four modes of binocular mode, left eye mode, right eye mode and monocular mode, wherein the monocular mode can refer to the detection of the position of only one eye pupil and the inability to identify whether the eye is the left eye or the right eye. The two-dimensional position coordinates corresponding to the monocular mode can be stored at the left eye pupil position (left_x, left_y) or at the right eye pupil position (right_x, right_y). The accuracy of the two-dimensional position coordinates of the pupils of both eyes calculated in this embodiment is ±1 pixel.

Specifically, this embodiment uses the selected target second camera to specifically collect the user's facial image, so that when the facial image is collected by the target second camera with a lower resolution, it can ensure that the pixel area occupied by the pupil area of both eyes in the facial image is larger, thereby improving the calculation accuracy while improving the calculation speed. After determining the first preset number of target second cameras, each facial image corresponding to the user is obtained by calling each target second camera, and each collected facial image can be processed based on the image processing algorithm, so that the two-dimensional pupil orientation information of both eyes in each facial image can be quickly and accurately calculated.

S140: Determine the user's three-dimensional binocular pupil position information based on at least two pieces of two-dimensional binocular pupil position information.

Among them, the three-dimensional binocular pupil orientation information may include the user's three-dimensional binocular pupil position information and eye gaze direction information. In the light field display, due to the light angle problem, the light of pixels in different directions has different distance sensitivities to the eye position in the depth direction. FIG3 shows an example of the distance sensitivity of light to the eye position in the depth direction. The light field display screen in FIG3 is a circle, and the user can watch in the circular area. In FIG3, the accuracy of the light emitted by "pixel 1" at the eye pupil position in the y-axis direction is d1, and the accuracy of the light emitted by "pixel 2" at the eye pupil position in the y-axis direction is d2. It can be seen that d2<d1, that is, the accuracy of the light emitted by "pixel 2" at the eye pupil position in the y-axis direction is higher. If the accuracy of the eye pupil position in the y-axis direction is less than d2, the light emitted by "pixel 2" cannot irradiate the user's eye pupil position, resulting in the phenomenon of missing pixels, so it is necessary to improve the accuracy in the depth distance.

Specifically, this embodiment can perform three-dimensional reconstruction of the user's eye pupils based on at least two two-dimensional eye pupil position information, and calculate the user's three-dimensional eye pupil position information, thereby improving the accuracy in depth distance and avoiding the above-mentioned phenomenon of missing pixels. This embodiment can input the user's three-dimensional eye pupil position information into the 3D display screen driver, so that the 3D display screen can determine corresponding display data based on the three-dimensional eye pupil position information, so that the user can view the corresponding three-dimensional picture.

The technical solution of this embodiment determines the three-dimensional head position information corresponding to each user in the preset viewing area based on the user image captured by the first camera, thereby realizing the tracking of the user's head. Based on the user's three-dimensional head position information, a first preset number of target second cameras are determined from multiple second cameras, so that the pixel area occupied by the pupil area of both eyes in the facial image captured by each target second camera is relatively large, so that the user's three-dimensional binocular pupil position information can be accurately and quickly determined based on multiple facial images, realizing high-speed tracking of the human eye and improving the calculation accuracy at the same time. In addition, the embodiment of the present invention uses multiple second cameras to capture the facial images of users in the preset viewing area, so that different second cameras can be called to capture facial images of different users at the same time, thereby realizing multi-user human eye tracking.

On the basis of the above technical solution, "screening out a first preset number of target second cameras from each candidate second camera according to each matching degree and the current call number corresponding to each candidate second camera" may include: screening out candidate second cameras whose current call number is less than or equal to the preset call number as the second cameras to be selected according to the current call number corresponding to the candidate second cameras; arranging each of the second cameras to be selected in descending order based on the matching degrees corresponding to the second cameras to be selected, and determining the first first preset number of second cameras to be selected after the arrangement as the target second cameras.

The preset call number may refer to the maximum number of shooting tasks to be executed corresponding to the second camera, which may be preset according to business requirements and scenarios. For example, the preset call number may be set to 5. Specifically, in this embodiment, the candidate second cameras whose current call number is less than or equal to the preset call number may be first screened out from each candidate second camera, and the screened candidate second cameras may be used as the second cameras to be selected, and then the matching degree of each second camera to be selected may be arranged from large to small, and the first preset number of second cameras to be selected after the arrangement may be determined as the target second cameras.

On the basis of the above technical solution, the device for human eye tracking in this embodiment can periodically call the first camera to periodically collect user images, and determine the corresponding user's three-dimensional head orientation information based on each collected user image, and use the three-dimensional head orientation information to determine the target second camera, and determine the user's two-dimensional binocular pupil orientation information based on the facial image collected by each target second camera. That is to say, in this embodiment, the corresponding two-dimensional head orientation information can be determined based on the periodically collected user images. If at least two two-dimensional binocular pupil orientation information cannot be determined based on the collected user images each time in the preset historical number of times, it indicates that the user's head has been in an obstructed state or the user has been in a moving state, so when determining the target second camera of the time, all of the second preset number of candidate second cameras can be determined as the target second camera to increase the probability of tracking the human eye and improve the tracking efficiency.

Alternatively, before S140, the method may further include: if only one two-dimensional binocular pupil position information is determined according to each facial image, at least one target second camera is screened out again from the remaining candidate second cameras after the target second camera is screened out, and the target second camera screened out again is called to capture the user's facial image, and the corresponding two-dimensional binocular pupil position information is determined according to the facial image collected again; or, if the two-dimensional binocular pupil position information cannot be determined according to each facial image, at least two target second cameras are screened out again from the remaining candidate second cameras after the target second camera is screened out, and each target second camera screened out again is called to capture the user's facial image, and the corresponding two-dimensional binocular pupil position information is determined according to each facial image collected again.

Specifically, after collecting a first preset number of facial images of the user by calling a first preset number of target second cameras, if the user suddenly turns his head or is at the junction of the second cameras, the collected facial images may not contain the user's eye pupil information, so that the corresponding two-dimensional eye pupil orientation information cannot be determined based on the facial images. If only one two-dimensional eye pupil orientation information is determined based on each facial image, at least one target second camera can be screened out again from the remaining candidate second cameras based on the same screening rule, and the user's facial image can be re-collected using the re-screened target second camera, and the corresponding two-dimensional eye pupil orientation information can be determined again based on the facial image, so as to obtain at least two two-dimensional eye pupil orientation information. The screening rule may refer to a rule for screening based on the current number of calls and matching degrees of the candidate cameras. Similarly, if the two-dimensional binocular pupil position information cannot be determined based on each facial image, that is, each facial image does not contain binocular pupil information, at least two target second cameras can be screened out again from the remaining second cameras, and the user's facial image can be re-captured using the re-screened target second cameras, and the corresponding two-dimensional binocular pupil position information can be determined again based on the facial image, so as to obtain at least two two-dimensional binocular pupil position information.

On the basis of the above technical solution, after S140 and before S110, it also includes: estimating the second three-dimensional head orientation information of the user at the next moment based on the three-dimensional binocular pupil orientation information of the user at the current moment and the three-dimensional binocular pupil orientation information at the historical moment; estimating the three-dimensional binocular pupil orientation information of the user at the next moment based on the second three-dimensional head orientation information.

Among them, the first camera in this embodiment collects the user image in the preset viewing area frame by frame according to the preset frame rate, so that the three-dimensional head position information corresponding to the user can be periodically determined according to the periodically collected user image. Compared with the delay time consumed by collecting the user image and determining the three-dimensional head position information according to the user image, the delay time consumed by calculating the three-dimensional binocular pupil position information according to the three-dimensional head position information is shorter. That is to say, after the three-dimensional binocular pupil position information of the user at the current moment is determined according to the current frame user image collected by the first camera, the three-dimensional head position information of the user at the next moment determined according to the next frame user image cannot be immediately obtained, that is, the head positioning tracking speed is slower than the human eye positioning tracking speed. In this embodiment, after the three-dimensional binocular pupil position information of the user at the current moment is determined according to the current frame user image, that is, after the operation of steps S110-S140 is performed, and before the three-dimensional head position information of the user at the next moment is determined according to the next frame user image, the three-dimensional head position information of the user at the next moment can be estimated, thereby solving the problem of slow head positioning tracking speed, so as to further improve the tracking speed. The second 3D head orientation information in this embodiment may refer to the 3D head orientation information estimated based on the user's existing 3D eye pupil orientation information. The 3D eye pupil orientation information at a historical moment may refer to the 3D eye pupil orientation information accurately determined based on the user image at a historical moment.

Specifically, after S140, if the three-dimensional head orientation information at the next moment determined according to the user image is not obtained, the second three-dimensional head orientation information of the user at the next moment can be more accurately estimated based on the three-dimensional binocular pupil orientation information calculated at the current moment and the three-dimensional binocular pupil orientation information calculated at the historical moment, so that the three-dimensional binocular pupil orientation information of the user at the next moment can be estimated based on the second three-dimensional head orientation information, thereby solving the problem of slow head positioning and tracking speed. When accurate three-dimensional head orientation information determined according to the user image is obtained, steps S120-S140 are performed according to the three-dimensional head orientation information, thereby reducing the delay time and further improving the tracking speed.

Exemplarily, the second three-dimensional head orientation information includes a three-dimensional head position and a rotation angle; accordingly, the second three-dimensional head orientation information of the user at the next moment can be estimated according to the following formula:

Wherein, ( _Xp1 , _Yp1 , _Zp1 ) and _α1 are the three-dimensional eye pupil position and the angle of the gaze direction of the user at the current moment P1; ( _Xp2 , _Yp2 , _Zp2 ) and _α2 are the three-dimensional eye pupil position and the angle of the gaze direction of the user at the historical moment P2; ( _Xp3 , _Yp3 , _Zp3 ) and _α3 are the three-dimensional eye pupil position and the angle of the gaze direction of the user at the historical moment P3; (X, Y, Z) and α are the estimated three-dimensional head position and rotation angle of the user at the next moment. Wherein, the three-dimensional eye pupil position at the current moment P1, the historical moment P2 and the historical moment P3 can be the three-dimensional right eye pupil position or the three-dimensional left eye pupil position of the user. FIG4 shows an example of a second three-dimensional head orientation information display. Points A and a in Figure 4 are respectively the three-dimensional eye pupil position and the angle of the gaze direction of the user at the next moment; points A1 and a1 are respectively the three-dimensional eye pupil position and the angle of the gaze direction of the user at the current moment P1; points A2 and a2 are respectively the three-dimensional eye pupil position and the angle of the gaze direction of the user at the current moment P2; points A3 and a3 are respectively the three-dimensional head position and the rotation angle of the user at the current moment P3. This embodiment can estimate the three-dimensional head position and the rotation angle of the user at the next moment according to the accurate three-dimensional eye pupil position and the angle of the gaze direction corresponding to the current moment P1 and the previous two moments P2 and P3 according to the proportional change prediction mechanism, so that the operations of steps S120-S140 can be continued according to the estimated second three-dimensional head orientation information to improve the tracking speed.

On the basis of the above technical solution, after S130, it also includes: determining the position and size of the target facial area in the facial image collected by each target second camera according to the three-dimensional head orientation information. Wherein, the target facial area may refer to an image area composed of a facial contour in the facial image. This embodiment can define the position and size of the target facial area in the facial image based on the three-dimensional head orientation information of the user, so as to reduce the calculation area, and by performing image processing only on the target facial area, the user's two-dimensional binocular pupil orientation information can be calculated more quickly, thereby further improving the calculation speed of human eye positioning.

Based on the above technical solution, "determining the user's two-dimensional binocular pupil position information according to each facial image" in S130 may include: determining the time for receiving data through the scanning line according to the position and size of the target facial area, and receiving the target facial area data sent by the target second camera according to the time, and determining the user's two-dimensional binocular pupil position information according to the received target facial image data.

Among them, the target second camera can use CSI (CMOS Sensor Interface, camera serial interface) to send the collected facial image data to the human eye tracking device in a row manner. The target second camera can use a continuous memory module to store image data, and can transmit facial image data by directly transmitting a row pointer list of facial images, that is, a scan line method, to improve the data transmission speed. This embodiment can calculate the time to receive data through the scan line according to the position and size of the target facial area, so as to receive less useless data on the premise of receiving the facial area, thereby reducing the delay time caused by the target second camera collecting facial images and further improving the tracking speed. Exemplarily, FIG5 shows a schematic diagram of receiving facial image data. The facial image resolution in FIG5 is 640×480. When the target second camera starts to transmit a frame of facial image, it will first send a frame start synchronization signal (the corresponding transmission time is recorded as Ts), and then send a frame end synchronization signal (the corresponding transmission time is recorded as Te) at the end of the transmission. According to these two signals, the row transmission speed of the target second camera can be determined as: (Te-Ts)/480. The eye tracking device can determine that the last row of the target facial area is at a height of 300 according to the three-dimensional head orientation information, so that the data receiving time corresponding to the target facial area can be calculated as: 300×(Te-Ts)/480. The timing starts after receiving the frame start synchronization signal. When the receiving time reaches 300×(Te-Ts)/480, it indicates that the target facial image data containing the facial area has been received. At this time, there is no need to receive the remaining facial image data, thereby increasing the data transmission speed, and based on the received target facial image data, the user's two-dimensional binocular pupil orientation information can be determined more quickly, reducing the delay time, and further improving the tracking speed and calculation speed.

Embodiment 2

FIG6 is a flow chart of a human eye tracking method provided by Embodiment 2 of the present invention. Based on the above embodiment, this embodiment optimizes “determining the user's three-dimensional binocular pupil position information based on at least two two-dimensional binocular pupil position information”. The explanations of the terms that are the same or corresponding to the above embodiment are not repeated here.

Referring to FIG. 6 , the eye tracking method provided in this embodiment specifically includes the following steps:

S210, calling the first camera to collect user images in the preset viewing area, and determining three-dimensional head orientation information corresponding to each user in the preset viewing area according to the user images.

S220. Determine a first preset number of target second cameras from a plurality of second cameras according to the three-dimensional head orientation information.

S230: Call each target second camera to capture a facial image of the user.

S240: If at least two two-dimensional binocular pupil position information corresponding to the user at the current moment cannot be determined based on each facial image, a spiral search rule is determined based on the three-dimensional binocular pupil position information at historical moments.

Specifically, when the user suddenly turns his head or is at the junction of the second camera, the user's eye pupil information cannot be detected by calling the target second camera, that is, at least two two-dimensional eye pupil orientation information cannot be determined based on each facial image. At this time, it indicates that the three-dimensional head orientation information determined based on the user image is inaccurate, and it is necessary to predict the three-dimensional head orientation information corresponding to the user at the current moment, so as to adjust the three-dimensional head orientation information determined based on the user image. Usually, when the user watches the screen, the angular velocity of the head rotation is faster than the moving speed, so in addition to establishing the head kinematic model, it is also possible to search outward in a spiral shape to improve the prediction speed. In this embodiment, the radius value and motion trajectory of the spiral at each node can be determined according to the three-dimensional eye pupil orientation information determined at the historical moment, so as to determine the spiral search rule. FIG. 7 shows an example of a spiral search graph. The nodes "0", "1", "2", "3", "4" and "5" in FIG. 7 represent different head positions respectively, and the straight line connecting each node represents the head orientation corresponding to the node. The spiral search rule in this embodiment may be to start at node "0" and to search outward in the order of nodes "1" to "5".

S250: Using the three-dimensional head orientation information determined according to the user image as the three-dimensional head orientation information of the user at the current moment.

Specifically, when using the spiral search rule to predict the user's three-dimensional head orientation information, the three-dimensional head orientation information determined based on the user's image is inaccurate, that is, the target second camera called based on the three-dimensional head orientation information cannot detect the user's eye pupil information. This embodiment first uses the three-dimensional head orientation information as the user's three-dimensional head orientation information at the current moment, so as to adjust the three-dimensional head orientation information.

S260: Adjust the three-dimensional head orientation information at the current moment according to the spiral search rule, and use the adjusted three-dimensional head orientation information at the current moment as the first three-dimensional head orientation information.

Specifically, the three-dimensional head orientation information at the current moment can be adjusted according to the spiral search graph corresponding to the spiral search rule. Exemplarily, as shown in FIG7 , node "0" represents the three-dimensional head orientation information determined according to the user image, that is, the three-dimensional head orientation information at the current moment. When re-searching outward in the order of nodes "1"-"5", the three-dimensional head position and head orientation corresponding to the next node "1" of node "0" can be determined as the adjusted three-dimensional head orientation information at the current moment, that is, the first three-dimensional head orientation information, so as to reasonably adjust and predict the head orientation information.

S270. Determine at least two first two-dimensional binocular pupil position information of the user at the current moment according to the first three-dimensional head position information.

Specifically, after predicting the first three-dimensional head information, a first preset number of target second cameras can be determined from multiple second cameras based on the first three-dimensional head information, and each target second camera can be called to capture the user's facial image, and at least two first two-dimensional binocular pupil position information of the user at the current moment can be determined based on each facial image.

S280: Determine the user's three-dimensional binocular pupil position information based on at least two first two-dimensional binocular pupil position information.

Specifically, this embodiment can perform three-dimensional reconstruction of the user's binocular pupils according to at least two first two-dimensional binocular pupil position information, and calculate the user's three-dimensional binocular pupil position information, thereby improving calculation accuracy and speed.

The technical solution of this embodiment, when the three-dimensional head orientation information determined according to the user image cannot detect the user's binocular pupil information, can use a spiral search rule to adjust and predict the three-dimensional head orientation information so as to obtain more accurate first three-dimensional head orientation information, and based on the first three-dimensional head orientation information, at least two first two-dimensional binocular pupil orientation information can be obtained so as to determine the user's three-dimensional binocular pupil orientation information, thereby solving the problem of being unable to track the human eyes due to sudden rotation of the user, and further improving the speed and accuracy of human eye tracking.

Based on the above technical solution, if at least two first two-dimensional binocular pupil orientation information at the current moment cannot be determined based on the first three-dimensional head orientation information, and the current test duration is less than the preset duration, the first three-dimensional head orientation information is updated to the three-dimensional head orientation information at the current moment, and step S260 is entered.

Among them, the preset duration can be determined according to the frame rate of the first camera. For example, the time interval between two adjacent frames of images taken by the first camera can be determined as the preset duration to ensure that the prediction adjustment operation is performed before the three-dimensional head orientation information determined according to the user image is obtained. Specifically, after determining a first preset number of target second cameras from multiple second cameras according to the first three-dimensional head information, and calling each target second camera to capture the user's facial image, if at least two first two-dimensional binocular pupil orientation information of the user at the current moment cannot be determined according to each facial image, and the current test duration is less than the preset duration, it indicates that the estimated first three-dimensional head orientation information at this time is wrong, and the test time is short. At this time, the first three-dimensional head orientation information can be used as the three-dimensional head orientation information at the current moment, and the operation of steps S260-S280 is returned to be executed, and the three-dimensional head orientation information at the current moment is adjusted again to update the first three-dimensional head orientation information. For example, node "1" in Figure 7 is used as the three-dimensional head orientation information at the current moment. When re-searching outward in the order of nodes "1"-"5", the three-dimensional head position and head orientation corresponding to the next node "2" of node "1" can be determined as the adjusted three-dimensional head orientation information at the current moment, that is, the updated first three-dimensional head orientation information, so that the head orientation information can be reasonably adjusted and predicted again.

Based on the above technical solution, if at least two first two-dimensional binocular pupil position information at the current moment cannot be determined based on the first three-dimensional head position information, and the current test duration is equal to the preset duration, the three-dimensional binocular pupil information determined at the previous moment of the user will be used as the three-dimensional binocular pupil position information of the user at the current moment.

Specifically, if at least two first two-dimensional binocular pupil orientation information at the current moment cannot be determined based on the first three-dimensional head orientation information, and the current test duration is equal to the preset duration, it indicates that the three-dimensional head orientation information determined based on the user image will be obtained soon, and the prediction operation can be stopped at this time to avoid indefinite prediction, and the three-dimensional binocular pupil information determined at the previous moment is directly determined as the three-dimensional binocular pupil orientation information at the current moment. Exemplarily, in this embodiment, when the current test duration is equal to the preset duration, if only one first two-dimensional binocular pupil orientation information at the current moment is determined, the depth distance can be calculated using the two-dimensional binocular pupil orientation information determined at the previous moment, and the three-dimensional binocular pupil orientation information at the current moment can be more accurately calculated based on the depth distance and the first two-dimensional binocular pupil orientation information at the current moment.

On the basis of the above technical solution, before calling the first camera to capture the user image in the preset viewing area, it also includes: determining the number of first layout positions corresponding to the first camera according to the viewing angle range corresponding to the preset viewing area and the first preset orientation error corresponding to the first camera; determining the number of first cameras corresponding to each first layout position according to the first viewing angle of the first camera.

Among them, the first preset orientation error may refer to the degree of rotation of the user's head when matching a new first camera, which can be preset according to business needs and scenarios. For example, if the first camera is arranged on a circle, the first preset orientation error can be set to 60 degrees, that is, a new first camera can capture the front image of the user every 60 degrees of rotation of the user's head. The viewing angle range may refer to the viewing angle corresponding to the preset viewing area. For example, if the preset viewing area is a circular area, the corresponding preset viewing angle is 360 degrees. The first viewing angle of the first camera may refer to the range of the shooting angle of the first camera. The preset detection distance corresponding to the preset viewing area may be the maximum value of the user's distance from the first camera in the preset viewing area. The first depth of field of each first camera is greater than the preset detection distance corresponding to the preset viewing area, so that the first camera can be used to clearly capture the user image in the preset viewing area.

Specifically, in this embodiment, the result obtained by dividing the viewing angle range corresponding to the preset viewing area by the first preset orientation error corresponding to the first camera can be determined as the number of first layout positions. And according to the viewing angle range required by each first layout position and the first viewing angle of the first camera, the number of first cameras corresponding to each first layout position is determined so that the entire preset viewing area can be covered. Exemplarily, if the viewing angle range required by the first layout position is 150 degrees and the first viewing angle of the first camera is 150 degrees, it can be determined that only one first camera needs to be installed at each first layout position.

On the basis of the above technical solution, before calling the first camera to capture the user image in the preset viewing area, it also includes: determining the number of second layout positions corresponding to the second camera according to the viewing angle range corresponding to the preset viewing area and the second preset orientation error corresponding to the second camera; determining the number of depth of field layers corresponding to each second layout position according to the second depth of field of the second camera and the preset detection distance corresponding to the preset viewing area; determining the number of second cameras corresponding to each layer of depth of field according to the second viewing angle of the second camera.

Among them, the second preset orientation error may refer to the degree of rotation of the user's head when matching to an optimal second camera, which can be preset according to business needs and scenarios. For example, if the first camera is arranged on a circle, the first preset orientation error can be set to 30 degrees, so that every time the user's head rotates 30 degrees, there can be an optimal second camera to capture the front image of the user, and the user's left and right eyes can be distinguished. The viewing angle range may refer to the viewing angle corresponding to the preset viewing area. For example, if the preset viewing area is a circular area, the corresponding preset viewing angle is 360 degrees. The preset detection distance corresponding to the preset viewing area may be the maximum value of the user's distance from the second camera in the preset viewing area. The second depth of field of the second camera may be less than the preset detection distance, so that at least two layers of depth of field layout can be performed to ensure the resolution of the image and improve the contrast. The second viewing angle of the second camera may refer to the range of the shooting angle of the second camera, which can be obtained through the lens parameters of the second camera. In this embodiment, the first camera is used to track the user's head, and has lower requirements for facial details and tracking speed, while the second camera is used to track the position of the human eyes, and has higher requirements for tracking speed and positioning accuracy. Therefore, the first preset orientation error can be set to be greater than or equal to the second preset orientation error, and the first viewing angle can be greater than the second viewing angle.

Specifically, in this embodiment, the result obtained by dividing the viewing angle range corresponding to the preset viewing area by the second preset orientation error corresponding to the second camera can be determined as the number of second layout positions. According to the second depth of field of the second camera and the preset detection distance corresponding to the preset viewing area, an appropriate number of depth of field layers is selected so that the entire preset viewing area can be covered, and clear imaging can be ensured at different shooting distances, thereby improving the resolution and contrast of the image. Exemplarily, in this embodiment, three layers of depth of field (i.e., near-middle-far) can be used for multi-layer layout, and each layer can increase the tracking coverage by arranging the second camera in parallel. According to the viewing angle range required by each layer of depth of field and the second viewing angle of the second camera, the number of second cameras corresponding to each layer of depth of field is determined so that the entire preset viewing area can be covered. Exemplarily, if the viewing angle range required by each layer of depth of field is 150 degrees in the horizontal direction and the second viewing angle of the second camera is 30 degrees, it can be determined that 6 second cameras correspond to each layer of depth of field to ensure that the coverage range in the horizontal direction is 150 degrees, as shown in FIG8. In order to increase the coverage angle range in the vertical direction, at least two second camera groups can be superimposed at each second layout position, and each second camera group includes at least two layers of second cameras with different depths of field, so as to track the user's lowering and raising of the head. For example, when the second layout position corresponds to three layers of depth of field, the three layers of depth of field (i.e., one second camera group) correspond to 18 second cameras. If two second camera groups are superimposed at each layout position, it can be determined that 36 second cameras need to be installed at each second layout position.

In this embodiment, after a corresponding number of second cameras are installed at each second layout position, the orientation configuration information of each second camera can be stored in the form of a three-layer data table, so as to improve the speed of the second camera matching search, and then improve the speed of determining the target second camera. Exemplarily, the three-layer data table may include a second layout position pointer table, multiple structure pointer tables and multiple data structure tables. Figure 9 shows an example of a three-layer data table corresponding to a second camera. As shown in Figure 9, the second layout position pointer table can arrange each second layout position in order according to the layout direction of the second layout position (such as clockwise or counterclockwise). Each second layout position corresponds to a structure pointer table for storing the identification code cid of each second camera corresponding to the second layout position. The data structure table can be used to store the orientation configuration information structcamera_position corresponding to the second camera.

The following is an embodiment of a human eye tracking device provided by an embodiment of the present invention. The device and the human eye tracking methods of the above embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the human eye tracking device, reference can be made to the embodiment of the above-mentioned human eye tracking method.

Embodiment 3

FIG10 is a schematic diagram of the structure of an eye tracking device provided in Embodiment 3 of the present invention. This embodiment is applicable to the case where the user's eye pupils are tracked and located when the user is watching a 3D display screen. The device includes: a 3D head orientation information determination module 310, a target second camera determination module 320, a 2D eye pupil orientation information determination module 330, and a 3D eye pupil orientation information determination module 340.

Among them, the three-dimensional head orientation information determination module 310 is used to call the first camera to capture the user image in the preset viewing area, and determine the three-dimensional head orientation information corresponding to each user in the preset viewing area based on the user image; the target second camera determination module 320 is used to determine a first preset number of target second cameras from multiple second cameras based on the three-dimensional head orientation information; the two-dimensional binocular pupil orientation information determination module 330 is used to call each target second camera to capture the user's facial image, and determine the user's two-dimensional binocular pupil orientation information based on each facial image; the three-dimensional binocular pupil orientation information determination module 340 is used to determine the user's three-dimensional binocular pupil orientation information based on at least two two-dimensional binocular pupil orientation information.

Optionally, the target second camera determination module 320 includes:

a candidate second camera determining unit, configured to determine a second preset number of candidate second cameras corresponding to the user and a matching degree corresponding to each candidate second camera according to the three-dimensional head orientation information and the orientation configuration information of each second camera;

The target second camera determination unit is used to screen out a first preset number of target second cameras from each candidate second camera according to each matching degree and the current call count corresponding to each candidate second camera; wherein the first preset number is less than the second preset number.

Optionally, the target second camera determination unit is specifically used to: screen out candidate second cameras whose current call times are less than or equal to a preset call times as the second cameras to be selected according to the current call times corresponding to the candidate second cameras; arrange the second cameras to be selected in descending order based on the matching degrees corresponding to the second cameras to be selected, and determine the first first preset number of second cameras to be selected after the arrangement as the target second cameras.

Optionally, the three-dimensional binocular pupil orientation information determination module 340 is specifically used to: if at least two two-dimensional binocular pupil orientation information corresponding to the user at the current moment cannot be determined based on each facial image, then determine a spiral search rule based on the three-dimensional binocular pupil orientation information at historical moments; use the three-dimensional head orientation information determined based on the user image as the three-dimensional head orientation information of the user at the current moment; adjust the three-dimensional head orientation information at the current moment according to the spiral search rule, and use the adjusted three-dimensional head orientation information at the current moment as the first three-dimensional head orientation information; determine at least two first two-dimensional binocular pupil orientation information of the user at the current moment based on the first three-dimensional head orientation information; determine the user's three-dimensional binocular pupil orientation information based on at least two first two-dimensional binocular pupil orientation information.

Optionally, the device further comprises:

a second three-dimensional head orientation information determination module, configured to estimate the second three-dimensional head orientation information of the user at a next moment according to the three-dimensional eye pupil orientation information of the user at a current moment and the three-dimensional eye pupil orientation information of the user at a historical moment, after determining the three-dimensional eye pupil orientation information of the user according to at least two two-dimensional eye pupil orientation information and before determining the three-dimensional head orientation information corresponding to each user in a preset viewing area according to the user image;

The three-dimensional binocular pupil position information estimation module is used to estimate the three-dimensional binocular pupil position information of the user at the next moment according to the second three-dimensional head position information.

Optionally, the second three-dimensional head orientation information includes a three-dimensional head position and a rotation angle; accordingly, the second three-dimensional head orientation information of the user at the next moment is estimated according to the following formula:

Among them, ( _Xp1 , _Yp1 , _Zp1 ) and _α1 are the three-dimensional eye pupil position and the angle of the gaze direction of the user at the current moment P1; ( _Xp2 , _Yp2 , _Zp2 ) and _α2 are the three-dimensional eye pupil position and the angle of the gaze direction of the user at the historical moment P2; ( _Xp3 , _Yp3 , _Zp3 ) and _α3 are the three-dimensional eye pupil position and the angle of the gaze direction of the user at the historical moment P3; (X, Y, Z) and α are the estimated three-dimensional head position and rotation angle of the user at the next moment.

Optionally, the device further comprises:

The target facial region determination module is used to determine the position and size of the target facial region in the facial image captured by each target second camera according to the three-dimensional head orientation information after determining a first preset number of target second cameras from multiple second cameras.

Optionally, the two-dimensional binocular pupil position information determination module 330 is specifically used to: determine the time for receiving data through the scanning line according to the position and size of the target facial area, receive the target facial area data sent by the target second camera according to the time, and determine the user's two-dimensional binocular pupil position information according to the received target facial image data.

Optionally, the first camera is a color camera or a 3D camera, the second camera is a black and white camera, and an infrared light source is arranged at the installation position of the second camera.

Optionally, the device further includes: a first layout position number determination module, configured to determine the number of first layout positions corresponding to the first camera according to a viewing angle range corresponding to the preset viewing area and a first preset orientation error corresponding to the first camera before calling the first camera to capture a user image in the preset viewing area;

The first camera quantity determination module is used to determine the number of first cameras corresponding to each first layout position according to the first viewing angle of the first camera, wherein the first depth of field of each first camera is greater than the preset detection distance corresponding to the preset viewing area.

Optionally, the device further comprises: a second layout position number determination module, configured to determine the number of second layout positions corresponding to the second camera according to a viewing angle range corresponding to the preset viewing area and a second preset orientation error corresponding to the second camera before calling the first camera to capture a user image in the preset viewing area;

A depth of field layer number determination module, used to determine the depth of field layer number corresponding to each second layout position according to the second depth of field of the second camera and the preset detection distance corresponding to the preset viewing area, wherein the second depth of field is less than the preset detection distance;

The second camera quantity determination module is used to determine the number of second cameras corresponding to each layer of depth of field according to the second viewing angle of the second camera.

The human eye tracking device provided in the embodiment of the present invention can execute the human eye tracking method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the human eye tracking method.

It is worth noting that in the embodiment of the above-mentioned human eye tracking device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

Embodiment 4

Fig. 11 is a schematic diagram of the structure of an eye tracking system provided by Embodiment 4 of the present invention. Referring to Fig. 11, the system comprises: a first camera 410, a plurality of second cameras 420 and an eye tracking device 430; wherein the eye tracking device 430 can be used to implement the eye tracking method provided by any embodiment of the present invention.

Among them, the first camera 410 is used to collect the user image in the preset viewing area so as to realize the tracking of the user's head. The first camera 410 in this embodiment can be a 3D camera or a plurality of 2D cameras. Since the first camera 410 is responsible for tracking the user's head, the facial details and tracking speed are not required to be high, so the first camera 410 with a large viewing angle can be selected. The second camera 420 is used to collect the user's facial image so as to realize high-speed tracking of the human eye. The second camera 420 in this embodiment can be a 2D camera. Exemplarily, the first camera 410 can be a high-definition color camera, and the second camera 420 can be a black and white camera. The resolution of the first camera in this embodiment can be greater than the resolution of the second camera. In this embodiment, by using a second camera with a lower resolution, the calculation speed can be improved while ensuring the calculation accuracy.

The layout requirements of the first camera 410 in this embodiment can be set but not limited to: (1) The depth of field of the first camera 410 is greater than the preset detection distance corresponding to the preset viewing area, so that only one first camera with a depth of field is needed within the preset detection distance range, that is, the depth of field of each first camera is the same. (2) The area occupied by the head area in the user image captured by the first camera 410 can be greater than the accuracy requirement, such as 60×60 pixels, so that the photographed head area is clearer. (3) A sufficient number of first cameras are set according to the first preset orientation error, and can cover the entire preset viewing area, wherein the first preset orientation error can be set to 60 degrees.

In this embodiment, the first camera can be laid out based on the layout requirements of the first camera 410 and the preset viewing area. Specifically, at least one first camera is arranged at each first layout position in the preset viewing area, and the total detection area corresponding to each first camera is the preset viewing area, that is, the total shooting range of each first camera can cover the preset viewing area. Exemplarily, when the preset viewing area is an inner circular area, each first layout position can be distributed on the circle corresponding to the preset viewing area, and at least one first camera is arranged at each first layout position, and the shooting direction of each first camera is toward the center position of the circle. Among them, the distance between each adjacent two first layout positions can be equal, that is, each first layout position is evenly distributed on the closed shape corresponding to the preset viewing area; it can also be within the preset allowable error range. For example, FIG. 12 shows an example of the layout of the first camera when the preset viewing area is an inner circular area. The dotted line in FIG. 12 represents the optical axis of the first camera, and the solid lines on both sides of the dotted line represent the viewing angle range of the first camera. As shown in FIG12 , six first layout positions are evenly distributed on the circle, and a first camera is set at each first layout position. That is, by setting six first cameras, the entire inner circular area can be covered, so that user images at various positions in the inner circular area can be collected. This layout of the first cameras can be called a circular inward type.

Exemplarily, when the preset viewing area is a circular outer area, each first layout position can be distributed on the circle corresponding to the preset viewing area, and at least one first camera is set at each first layout position, and the shooting direction of each first camera is away from the center position of the circle. Among them, the distance between each two adjacent first layout positions can be equal, that is, each first layout position is evenly distributed on the closed shape corresponding to the preset viewing area; it can also be within the preset allowable error range. For example, Figure 13 shows an example of the layout of the first camera when the preset viewing area is a circular outer area. The dotted line in Figure 13 represents the optical axis of the first camera, and the solid lines on both sides of the dotted line represent the viewing angle range of the first camera. As shown in Figure 13, 6 first layout positions are evenly distributed on the circle, and a first camera is set at each first layout position, that is, by setting 6 first cameras, the entire circular outer area can be covered, so that the user image in the circular outer area can be collected. This layout of the first camera can be called a circular outward type.

Exemplarily, when the preset viewing area is a linear one-sided viewing area, such as a viewing area in a movie theater, each first layout position can be distributed on the straight line of the linear one-sided viewing area, and at least one first camera is set at each first layout position, and the shooting direction of each first camera is toward the linear one-sided viewing area. Among them, the distance between each two adjacent first layout positions can be equal, that is, each first layout position is evenly distributed on the straight line of the linear one-sided viewing area; it can also be within the preset allowable error range. For example, FIG. 14 shows an example of the layout of the first camera when the preset viewing area is a linear one-sided viewing area. The dotted line in FIG. 14 represents the optical axis of the first camera, and the solid lines on both sides of the dotted line represent the viewing angle range of the first camera. As shown in FIG. 14, three first layout positions are evenly distributed on the straight line, and a first camera is set at each first layout position, and the three first cameras can be shot toward the linear one-sided viewing area in a fan-shaped direction so as to cover the entire linear one-sided viewing area. This layout of the first camera can be called a planar type.

The layout requirements of the second camera 420 in this embodiment can be set but not limited to: (1) At least two layers of depth of field are used at each second layout position to improve image resolution and contrast. (2) The frame rate of the second camera is greater than 60 frames per second. (3) The area occupied by the face area in the face image captured by the second camera can be greater than the accuracy requirement, such as 100×100 pixels, to improve the calculation accuracy of the two-dimensional binocular pupil orientation information. (4) The shortest distance d in the intersection area of the depth of field range of two adjacent second cameras at each second layout position (such as the shaded area in Figure 15) is greater than the distance between the pupils of the two eyes, so as to ensure that a face image containing both the left and right pupils can be captured. Where d can be set to 6.5 cm. (5) A sufficient number of second layout positions are set according to the second preset orientation error, and can cover the entire preset viewing area, where the second preset orientation error can be set to 30 degrees. (6) When the second camera is a black and white camera, an infrared light source is set at the center of each second layout position for fill light illumination.

Exemplarily, at least one second camera group can be set at each second layout position, and each second camera group includes at least two layers of second cameras, and each layer of second cameras is a plurality of second cameras with the same depth of field, and the depth of field of the second cameras in different layers is different. As shown in FIG8 , when the user is within the detection range of 1 meter to 5 meters from the second camera, the second camera can use a low-resolution (640×480) high-speed (greater than 90 frames per second) black and white camera, and the second viewing angle of each second camera is 30 degrees, each second camera group can include three second cameras with different focal lengths to ensure clear imaging of users at different distances, and the use of 6 cameras arranged in parallel allows the maximum horizontal coverage to reach 150 degrees.

This embodiment can layout the second camera based on the layout requirements of the second camera 420 and the preset viewing area. Specifically, at least one second camera is arranged at each second layout position in the preset viewing area, and the total detection area corresponding to each second camera is the preset viewing area, that is, the total shooting range of each second camera can cover the preset viewing area. Exemplarily, when the preset viewing area is an inner circular area, each second layout position can be distributed on the circle corresponding to the preset viewing area, and at least one second camera is arranged at each second layout position, and the shooting direction of each second camera is toward the center position of the circle. Among them, the distance between each adjacent second layout position can be equal, that is, each second layout position is evenly distributed on the closed shape corresponding to the preset viewing area; it can also be within the preset allowable error range. For example, Figure 16 shows an example of the layout of the second camera when the preset viewing area is an inner circular area. As shown in FIG16 , 12 second layout positions are evenly distributed on the circle. If each second camera group is arranged in the manner of FIG8 , 18 second cameras can be arranged in the horizontal direction at the second layout position, that is, each second camera group includes 18 second cameras, so as to ensure that the shooting angle of each second layout position in the horizontal direction can be 150 degrees. At this time, the total number of second cameras required is: 12×18=216. In order to increase the coverage angle range in the vertical direction, at least two second camera groups can be superimposed at each second layout position in this embodiment, so as to track the situation when the user lowers his head and looks up. At this time, at least 12×18×2=432 second cameras should be arranged at each second layout position, so that every time the user's head turns 30 degrees, there is an optimal target second camera to shoot the front face image of the user, thereby reducing the occlusion phenomenon and improving the tracking accuracy. This layout of the second camera can be called a circular inward type.

Exemplarily, when the preset viewing area is an outer circular area, each second layout position can be distributed on the closed shape corresponding to the preset viewing area, and at least one second camera is arranged at each second layout position, and the shooting direction of each second camera is away from the center position of the circle. Among them, the distance between each two adjacent second layout positions can be equal, that is, each second layout position is evenly distributed on the closed shape corresponding to the preset viewing area; it can also be within the preset allowable error range. For example, FIG17 shows an example of the layout of the second camera when the preset viewing area is an outer circular area. As shown in FIG17, 12 second layout positions 1-12 are evenly distributed on the circle, and each second layout position is provided with 18 second cameras in the horizontal direction to ensure that the shooting angle of each second layout position in the horizontal direction can be 150 degrees.

Exemplarily, when the preset viewing area is a linear single-sided viewing area, such as a viewing area in a movie theater, each second layout position can be distributed on the straight line of the linear single-sided viewing area, and at least one second camera is set at each second layout position, and the shooting direction of each second camera is toward the linear single-sided viewing area. Among them, the distance between each two adjacent second layout positions can be equal, that is, each second layout position is evenly distributed on the straight line of the linear single-sided viewing area; it can also be within the preset allowable error range. For example, Figure 18 shows an example of the layout of the second camera when the preset viewing area is a linear single-sided viewing area. As shown in Figure 18, three second layout positions are evenly distributed on the straight line. If each second layout position is arranged in the manner of Figure 8, 18 second cameras are set to ensure that the shooting angle of each second layout position in the horizontal direction can be 150 degrees, and the shooting direction corresponding to the three second layout positions can be a fan-shaped, so as to cover the entire linear single-sided viewing area of the second layout position in the horizontal direction. This layout of the second camera can be called a plane type.

The working process of the eye tracking system in this embodiment is as follows: the eye tracking device 430 calls the first camera 410, so that the first camera 410 collects the user image in the preset viewing area, and transmits the collected user image to the eye tracking device 430, the eye tracking device 430 determines the three-dimensional head orientation information corresponding to each user in the preset viewing area according to the user image, and determines a first preset number of target second cameras from multiple second cameras according to the three-dimensional head orientation information, and calls each target second camera, so that each target second camera can collect the user's facial image, and transmits the collected facial image to the eye tracking device 430, the eye tracking device 430 determines the user's two-dimensional binocular pupil orientation information according to each facial image, and determines the user's three-dimensional binocular pupil orientation information according to at least two two-dimensional binocular pupil orientation information, thereby achieving high-speed tracking of the user's binocular pupils and improving the calculation accuracy.

The eye tracking system provided in this embodiment realizes head recognition and high-speed eye tracking respectively by using the first camera and the second camera, and uses the eye tracking device 430 to schedule and manage the second camera, thereby realizing high-speed tracking of the pupils of both eyes of the user and improving the calculation accuracy.

On the basis of the above technical solution, the eye tracking device 430 can be integrated on a server to implement the eye tracking method provided by any embodiment of the present invention, or the eye tracking method provided by any embodiment of the present invention can be implemented by using a first client, multiple second clients and a central server. FIG19 shows a schematic diagram of the structure of another eye tracking system provided by this embodiment. As shown in FIG19 , the system includes: a first camera 410, multiple second cameras 420, a first client 440, multiple second clients 450 and a central server 460.

The first camera 410 is connected to the first client 440, and is used to collect user images in the preset viewing area and send the user images to the first client 440; the first client 440 is connected to the central server 460, and is used to determine the three-dimensional head orientation information corresponding to each user in the preset viewing area according to the user images sent by the first client 440, and send the three-dimensional head orientation information to the central server 460; each second camera 420 is respectively connected to a corresponding second client 450, and is used to collect the user's facial image, and send the facial image to the corresponding second client 450; The second client 450 is connected to the central server 460, and is used to determine the user's two-dimensional binocular pupil position information based on each facial image, and send the two-dimensional binocular pupil position information to the central server 460; the central server 460 is used to determine a first preset number of target second cameras from multiple second cameras based on the three-dimensional head position information sent by the first client 440, and call the second client connected to each target second camera to obtain the two-dimensional binocular pupil position information sent by each called second client, and determine the user's three-dimensional binocular pupil position information based on at least two two-dimensional binocular pupil position information.

It should be noted that the first client may be, but is not limited to, a high-performance PC (Personal Computer). The second client may be, but is not limited to, an embedded computer to increase the speed of the sound. In this embodiment, the number of the first camera 410 may be one or more to increase the head tracking range and improve the head tracking accuracy. When there are multiple first cameras 410, each first camera 410 is connected to the first client 440, so that the first client 440 performs image processing on the user images captured by each first camera 410, and more accurately determines the three-dimensional head orientation information of each user in the preset viewing area.

Specifically, the working process of the human eye tracking system provided in FIG. 19 is as follows: the first client 440 calls the first camera 410 so that the first camera 410 captures the user image in the preset viewing area and sends the user image to the first client 440; the first client 440 determines the three-dimensional head orientation information corresponding to each user in the preset viewing area according to the user image sent by the first client 440, and sends the three-dimensional head orientation information to the central server 460; the central server 460 is used to determine a first preset number of target second cameras from a plurality of second cameras 420 according to the three-dimensional head orientation information sent by the first client 440, and call the second client connected to each target second camera; the second client 450 connected to each target second camera calls the corresponding target second camera so that each target second camera captures the user's facial image and sends the facial image to the corresponding second client 450; the second client 450 determines the user's two-dimensional binocular pupil orientation information according to each received facial image, and sends the two-dimensional binocular pupil orientation information to the central server 460; the central server 460 determines the user's three-dimensional binocular pupil orientation information according to at least two two-dimensional binocular pupil orientation information. The central server 460 can also be connected to the 3D display driver to input the user's three-dimensional binocular pupil position information into the 3D display driver, so that the 3D display can determine the corresponding display data according to the three-dimensional binocular pupil position information, so that the user can view the corresponding three-dimensional picture. In this embodiment, the first client, the second client and the central server are responsible for processing the three links in the human eye tracking process, that is, the first client is responsible for processing the user image, the second client is responsible for processing the facial image, and the central server is responsible for matching and scheduling the second camera, and calculating the user's three-dimensional binocular pupil position information, so that the system runs faster and has higher processing efficiency, thereby further improving the human eye tracking speed.

Embodiment 5

FIG20 is a schematic diagram of the structure of a device provided in Embodiment 5 of the present invention. Referring to FIG20 , the device includes:

one or more processors 510;

Memory 520, used to store one or more programs;

An input device 530, used for collecting images;

Output device 540, used to display screen information;

When one or more programs are executed by one or more processors 510 , the one or more processors 510 implement the human eye tracking method proposed in any of the above embodiments.

FIG20 takes a processor 510 as an example; the processor 510, memory 520, input device 530 and output device 540 in the device may be connected via a bus or other means, and FIG20 takes the connection via a bus as an example.

The memory 520, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions/modules corresponding to the eye tracking method in the embodiment of the present invention (for example, the three-dimensional head orientation information determination module 310, the target second camera determination module 320, the two-dimensional binocular pupil orientation information determination module 330 and the three-dimensional binocular pupil orientation information determination module 340 in the eye tracking device). The processor 510 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 520, that is, realizing the above-mentioned eye tracking method.

The memory 520 mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function; the data storage area can store data created according to the use of the device, etc. In addition, the memory 520 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 520 may further include a memory remotely arranged relative to the processor 510, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The input device 530 may include a camera or other acquisition device for acquiring user images and facial images, and inputting the acquired facial images and facial images into the processor 510 for data processing.

The output device 540 may include a display device such as a display screen for displaying screen information.

The device proposed in this embodiment and the human eye tracking method proposed in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as executing the human eye tracking method.

Embodiment 6

This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the human eye tracking method provided in any embodiment of the present invention is implemented.

The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for performing the operations of the present invention may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims (20)

1. A method of eye tracking, comprising:

calling a first camera to collect user images in a preset watching area, and determining three-dimensional head direction information corresponding to each user in the preset watching area according to the user images;

determining a first preset number of target second cameras from the plurality of second cameras according to the three-dimensional head orientation information;

calling each target second camera to collect the face image of the user, and determining the two-dimensional binocular pupil orientation information of the user according to each face image;

determining three-dimensional binocular pupil orientation information of the user according to the at least two pieces of two-dimensional binocular pupil orientation information;

if at least two pieces of two-dimensional binocular pupil orientation information corresponding to the current moment of the user cannot be determined according to each facial image, determining a spiral search rule according to three-dimensional binocular pupil orientation information at historical moments; taking the three-dimensional head position information determined according to the user image as the three-dimensional head position information of the user at the current moment; adjusting the three-dimensional head position information at the current moment according to the spiral search rule, and taking the adjusted three-dimensional head position information at the current moment as first three-dimensional head position information; determining at least two first two-dimensional binocular pupil position information of the user at the current moment according to the first three-dimensional head position information; and determining the three-dimensional binocular pupil position information of the user according to at least two first two-dimensional binocular pupil position information.

2. The method of claim 1, wherein determining a first preset number of target second cameras from a plurality of second cameras based on the three-dimensional head position information comprises:

determining a second preset number of candidate second cameras corresponding to the user and a matching degree corresponding to each candidate second camera according to the three-dimensional head position information and the position configuration information of each second camera;

screening a first preset number of target second cameras from the candidate second cameras according to the matching degrees and the current calling times corresponding to the candidate second cameras;

wherein the first preset number is less than or equal to the second preset number.

3. The method of claim 2, wherein the step of screening a first preset number of target second cameras from the candidate second cameras according to the matching degrees and the current calling times corresponding to each candidate second camera comprises:

screening out candidate second cameras with the current calling times smaller than or equal to preset calling times according to the current calling times corresponding to the candidate second cameras, and taking the candidate second cameras as second cameras to be selected;

and based on the matching degree corresponding to the second cameras to be selected, performing descending order arrangement on the second cameras to be selected, and determining the arranged first preset number of second cameras to be selected as target second cameras.

4. The method according to claim 1, further comprising, after determining three-dimensional binocular pupil orientation information of the user according to at least two of the two-dimensional binocular pupil orientation information, and before determining three-dimensional head orientation information corresponding to each user in the preset viewing area according to the user image:

according to the three-dimensional binocular pupil position information of the user at the current moment and the three-dimensional binocular pupil position information of the user at the historical moment, second three-dimensional head position information of the user at the next moment is estimated;

and estimating the three-dimensional binocular pupil orientation information of the user at the next moment according to the second three-dimensional head orientation information.

5. The method of claim 4, wherein the second three-dimensional head position information comprises a three-dimensional head position and a rotation angle;

correspondingly, second three-dimensional head orientation information of the user at the next moment is estimated according to the following formula:

wherein (X) _p1 ，Y _p1 ，Z _p1 ) And alpha ₁ The three-dimensional eye pupil position and the gaze direction angle of the user at the current moment P1 are obtained; (X) _p2 ，Y _p2 ，Z _p2 ) And alpha ₂ The three-dimensional eye pupil position and the gazing direction angle of the user at the historical moment P2 are obtained; (X) _p3 ，Y _p3 ，Z _p3 ) And alpha ₃ The three-dimensional eye pupil position and the gazing direction angle of the user at the historical moment P3 are obtained; (X, Y, Z) and α are estimated three-dimensional head position and rotation angle of the user at the next time instant.

6. The method of claim 1, after determining a first preset number of target second cameras from the plurality of second cameras, further comprising:

and determining the position and the size of a target face area in the face image acquired by each target second camera according to the three-dimensional head orientation information.

7. The method of claim 6, wherein determining two-dimensional binocular pupil orientation information of the user from each of the facial images comprises:

and determining the time for receiving data through a scanning line according to the position and the size of the target face area, receiving target face area data sent by the target second camera according to the time, and determining the two-dimensional binocular pupil orientation information of the user according to the received target face image data.

8. The method according to any one of claims 1 to 7, wherein the first camera is a color camera or a 3D camera, the second camera is a black and white camera, and an illuminating infrared light source is arranged at the installation position of the second camera.

9. The method of claim 1, prior to invoking the first camera to capture the user image in the preset viewing area, further comprising:

determining the number of first layout positions corresponding to the first camera according to the viewing angle range corresponding to the preset viewing area and the first preset orientation error corresponding to the first camera;

and determining the number of the first cameras corresponding to each first layout position according to the first visual angle of the first cameras, wherein the first depth of field of each first camera is greater than the preset detection distance corresponding to the preset viewing area.

10. The method of claim 1, prior to invoking the first camera to capture the user image in the preset viewing area, further comprising:

determining the number of second layout positions corresponding to the second camera according to the viewing angle range corresponding to the preset viewing area and a second preset orientation error corresponding to the second camera;

determining the number of depth-of-field layers corresponding to each second layout position according to a second depth of field of the second camera and a preset detection distance corresponding to the preset viewing area, wherein the second depth of field is smaller than the preset detection distance;

and determining the number of the second cameras corresponding to each layer of depth of field according to the second visual angle of the second cameras.

11. An eye tracking device, comprising:

the three-dimensional head orientation information determining module is used for calling a first camera to acquire a user image in a preset viewing area and determining three-dimensional head orientation information corresponding to each user in the preset viewing area according to the user image;

the target second camera determining module is used for determining a first preset number of target second cameras from the plurality of second cameras according to the three-dimensional head position information;

the two-dimensional binocular pupil orientation information determining module is used for calling each target second camera to acquire a face image of the user and determining the two-dimensional binocular pupil orientation information of the user according to each face image;

the three-dimensional binocular pupil orientation information determining module is used for determining the three-dimensional binocular pupil orientation information of the user according to at least two pieces of two-dimensional binocular pupil orientation information;

the three-dimensional binocular pupil orientation information determining module is specifically configured to: if at least two pieces of two-dimensional binocular pupil orientation information corresponding to the current moment of the user cannot be determined according to each face image, determining a spiral search rule according to three-dimensional binocular pupil orientation information at historical moments; taking the three-dimensional head position information determined according to the user image as the three-dimensional head position information of the user at the current moment; adjusting the three-dimensional head position information at the current moment according to the spiral search rule, and taking the adjusted three-dimensional head position information at the current moment as first three-dimensional head position information; determining at least two first two-dimensional binocular pupil position information of the user at the current moment according to the first three-dimensional head position information; and determining the three-dimensional binocular pupil position information of the user according to at least two first two-dimensional binocular pupil position information.

12. An eye tracking system, the system comprising: the system comprises a first camera, a plurality of second cameras and a human eye tracking device; wherein the eye tracking apparatus is used to implement the eye tracking method of any one of claims 1-10.

13. The system of claim 12, wherein the first camera is a color camera or a 3D camera and the second camera is a black and white camera.

14. The system according to claim 12, wherein at least one of the first cameras is disposed at each first layout position within a preset viewing area, and a total detection area corresponding to each of the first cameras is the preset viewing area;

at least one second camera is arranged at each second layout position in a preset viewing area, and a total detection area corresponding to each second camera is the preset viewing area.

15. The system of claim 14, wherein an illuminating infrared light source is disposed at the center of each of the second layout positions.

16. The system according to any one of claims 12-15, wherein a depth of field of each of the first cameras is greater than a predetermined detection distance corresponding to a predetermined viewing area.

17. The system according to claim 14, wherein at least one second camera group is disposed at each second layout position, the second camera group comprises at least two layers of second cameras, each layer of second cameras is a plurality of second cameras with the same depth of field, and the depth of field of the second cameras in different layers is different.

18. The system according to claim 14, wherein the shortest distance in the intersection region of the depth of field ranges of two adjacent second cameras at each second layout position is greater than the distance between pupils of both eyes.

19. An apparatus, characterized in that the apparatus comprises:

one or more processors;

a memory for storing one or more programs;

an input device for acquiring an image;

output means for displaying screen information;

the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the eye tracking method of any one of claims 1-10.

20. A computer-readable storage medium, on which a computer program is stored, which program, when being executed by a processor, carries out the eye tracking method according to any one of claims 1 to 10.

Images