CN109508679B - Method, device and equipment for realizing three-dimensional eye gaze tracking and storage medium - Google Patents

Abstract

The invention discloses a method, a device, equipment and a computer readable storage medium for realizing eyeball three-dimensional sight tracking, which comprises the following steps: inputting a human face image to be detected into a pre-constructed head posture detection network to obtain a head posture in the human face image; inputting the face image into a pre-constructed eyeball action detection network to obtain the eyeball action of the face image; and inputting the head posture and the eyeball motion into a pre-constructed three-dimensional sight line vector detection network to obtain a three-dimensional sight line direction vector of the eyeball in the face image. The method, the device, the equipment and the computer readable storage medium provided by the invention can extract the three-dimensional sight direction vector of the eyeball of the shot person from the two-dimensional face image, and have wide application scenes.

Description

Method, device, equipment and storage medium for realizing eyeball three-dimensional gaze tracking

technical field

The present invention relates to the technical field of eyeball tracking, in particular to a method, device, device and computer-readable storage medium for realizing eyeball three-dimensional line-of-sight tracking.

Background technique

The research on eye tracking algorithm has achieved relatively mature results, and has been successfully implemented in many commercial applications, such as VR/AR technology. Although traditional eye tracking technology can achieve high precision, the current eye tracking algorithm is basically It is based on the traditional image processing method, relies on expensive infrared equipment, and needs to install special detection equipment on the head to detect the characteristics of the eyeball. The detection accuracy of traditional image processing methods is affected by light changes, and the detection distance is severely restricted. Therefore, there is an urgent need for an algorithm that can realize eye tracking through a RGB image captured by a common camera. In the field of computer vision, deep convolutional neural networks have achieved significant results in many aspects, such as object detection, instance segmentation, and so on.

There is also a corresponding eye tracking technology based on deep learning in the prior art. The specific steps are as follows: obtain the image data of retinopathy; perform data labeling on the image data of retinopathy to obtain the labeled data; establish an initial deep learning network; input the image data of retinopathy In the initial deep learning network, the corresponding predicted data is output; use the loss function to compare the corresponding labeled data and predicted data of the retinopathy image data, and obtain the comparison result; according to the comparison result, adjust the parameters in the initial deep learning network until the comparison The result reaches the preset threshold, and the final deep learning network model is obtained; the image data of the retinopathy to be tested is processed by using the deep learning network model, and the corresponding eyeball center coordinates and eyeball diameter are obtained.

Therefore, in the existing eye tracking technology, one is based on the traditional image processing algorithm to realize the eye tracking technology. Although this type of algorithm has relatively mature commercial applications, the detection accuracy of the traditional image processing algorithm is affected by the change of light, and Relying on expensive head-mounted infrared devices, the convenience experience of the head is poor, and the detection distance is also limited. The other is an eye tracking algorithm based on a deep learning algorithm. However, the existing eye tracking algorithm based on a deep learning algorithm in the technology can only detect the center position of the eye and the diameter of the eye, and only contains two-dimensional information of eye movements. constraint.

From the above, it can be seen that how to obtain the three-dimensional line-of-sight direction vector of the eyeball through the two-dimensional face image is a problem to be solved at present.

Contents of the invention

The purpose of the present invention is to provide a method, device, device and computer-readable storage medium for realizing eyeball three-dimensional sight tracking, so as to solve the problem that the eye tracking algorithm based on deep learning in the prior art can only detect two-dimensional information of the eyeball .

In order to solve the above-mentioned technical problems, the present invention provides a method for realizing eyeball three-dimensional line of sight tracking, including: inputting the face image to be detected into a pre-built head pose detection network, and obtaining the head pose in the face image ; Input the human face image to the pre-built eye movement detection network to obtain the eye movement of the human face image; input the head posture and the eye movement to the pre-built three-dimensional line of sight vector detection network to obtain The three-dimensional line-of-sight direction vector of the eyeball in the face image.

Preferably, the input of the human face image to be detected to the pre-built head posture detection network, before obtaining the head posture in the human face image includes:

Gather multiple face images with head posture and three-dimensional labels of eyeballs, construct a face image data set, wherein the face image is an RGB image;

Build an initial head pose detection network and an initial eye movement detection network;

Using the face image data set to train the initial head pose detection network and the initial eye movement detection network respectively, to obtain the trained head pose detection network and eye movement detection network.

Preferably, the collection of multiple face images with three-dimensional labels of head posture and eyeball sight, and constructing a face image data set includes:

Using each camera in the area array camera array to separately collect the face images of the data provider to obtain the first subset of face images;

Each row of cameras in the area array camera array collects multiple face images, representing different head postures of the data provider in the y direction;

The multiple face images collected by each row of cameras in the area array camera array represent different head postures of the data provider in the p direction;

Rotating the face images collected by the area camera array clockwise and counterclockwise, respectively, to obtain a second subset of face images representing different head postures of the data provider in the r direction;

Combining the first subset of face images and the second subset of face images to obtain the face image data set.

Preferably, said using each camera in the area array camera array to respectively collect the face image of the data provider includes:

When collecting each face image, record the moving point on the display screen where the data provider's eyes are facing up, so as to determine the three-dimensional vector label of the data provider's eye sight, and record the moving points in each face image at the same time. head pose.

Preferably, said building an initial head pose detection network includes:

Taking the Alex NET model as the basic structure, construct the initial head detection network, the network structure of the preliminary head detection network is:

C(3,1,6)-BN-PReLU-P(2,2)-C(3,1,16)-BN-PReLU-P(2,2)-C(3,1,24)-BN -PReLU-C(3,1,24)-PReLU(3,1,16)-BN-PReLU-P(2,2)-FC(256)-FC(128)-PReLU-FC(3);

Among them, C(k,s,c) means that the convolution kernel size is k, the convolution step size is s, and the number of channels is c convolutional layer, and P(k,s) means that the kernel size is k, and the step size is s The maximum pooling layer, BN means batch normalization, PReLU means activation function, FC(n) means fully connected layer, and the number of neurons is n.

Preferably, using the face image data set to train the initial head pose detection network and the initial eye movement detection network respectively includes:

Using the face image data set to train the head pose detection network and the initial eye movement detection network;

和所述初步眼球动作检测网络损失函数

之和。Wherein, the loss function Loss ₁ =Loss _h +Loss _e is the loss function of the preliminary head pose detection network

and the preliminary eye movement detection network loss function

Sum.

Preferably, the head posture and the eyeball movement are input to a pre-built three-dimensional line of sight vector detection network, and before obtaining the three-dimensional line of sight direction vector of the eyeball in the face image includes:

Using the head posture detection network and the eye movement detection network to detect the face images in the face data set respectively, to obtain the head posture and eye movements of each face image;

Utilizing the head postures and eye movements of each of the face images to train the pre-established initial three-dimensional line of sight vector detection network, thereby obtaining the three-dimensional line of sight vector detection network that has completed the training;

之和。The current loss function Loss ₂ = Loss ₁ + Loss _g = Loss _h + Loss _e + Loss _g is the loss function Loss ₁ and the initial three-dimensional sight vector detection network loss function

Sum.

The present invention also provides a device for realizing eyeball three-dimensional line-of-sight tracking, including:

Head posture detection module, for inputting the human face image to be detected to the pre-built head posture detection network to obtain the head posture in the human face image;

An eye movement detection module, configured to input the face image to a pre-built eye movement detection network to obtain the eye movement of the face image;

The 3D line of sight detection module is configured to input the head posture and the eyeball movement into a pre-built 3D line of sight vector detection network to obtain the 3D line of sight direction vector of the eyeballs in the face image.

The present invention also provides a device for realizing eyeball three-dimensional line of sight tracking, including:

The memory is used to store the computer program; the processor is used to implement the steps of the above-mentioned method for realizing eyeball three-dimensional sight tracking when executing the computer program.

The present invention also provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned method for realizing eyeball three-dimensional gaze tracking are realized.

In the method for realizing eyeball three-dimensional line of sight tracking provided by the present invention, the face image to be detected is input to a pre-built head pose detection network, and the head pose in the face image is obtained. The human face image is input into the pre-built eye movement detection network to obtain the eye movement in the human face image. The head pose and the eye movement are input to a pre-built 3D line of sight vector detection network, so as to obtain the 3D line of sight direction vector of the eyeball in the face image according to geometric constraints and through a line of sight conversion network. The eye tracking method provided by the present invention is based on a deep learning network, extracts the head posture and eye movement of the person being photographed from the two-dimensional face image, and inputs the head posture and the eye movement into pre-training In a good three-dimensional line of sight vector detection network, the three-dimensional line of sight direction vector of the subject's eyeballs in the face image is obtained. The method provided by the present invention has a wide range of application fields. Obtaining the three-dimensional line of sight vector direction of the eyeball through the face image can be used in the monitoring field of safe driving, the field of human-computer interaction, and the field of psychological research; When the neural network implements eye tracking technology, it can only detect the center position and diameter of the eyeball, which does not have the problem of wide application scenarios. Correspondingly, the device, equipment and computer-readable storage medium provided by the present invention all have the above-mentioned advantageous effects.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only For some embodiments of the present invention, those skilled in the art can also obtain other drawings based on these drawings without creative work.

Fig. 1 is the flow chart of the first specific embodiment of the method for realizing eyeball three-dimensional line-of-sight tracking provided by the present invention;

Fig. 2 is the flow chart of the second specific embodiment of the method for realizing eyeball three-dimensional line-of-sight tracking provided by the present invention;

Fig. 3 is a structural block diagram of an apparatus for realizing eyeball three-dimensional line-of-sight tracking provided by an embodiment of the present invention.

Detailed ways

The core of the present invention is to provide a method, device, device and computer-readable storage medium for realizing eyeball three-dimensional line of sight tracking, which can obtain the three-dimensional line of sight vector of the eyeball through two-dimensional face images, and has a wide range of application scenarios.

In order to enable those skilled in the art to better understand the solution of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. Apparently, the described embodiments are only some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Please refer to Fig. 1, Fig. 1 is the flow chart of the first kind of specific embodiment of the method for realizing eyeball three-dimensional line of sight tracking provided by the present invention; Concrete operation steps are as follows:

Step S101: Input the face image to be detected into the pre-built head pose detection network to obtain the head pose in the face image;

The human face image to be detected is input to the pre-built head posture detection network, and before the head posture in the human face image is obtained, multiple pieces of human face images with head posture and three-dimensional labels of eye sights are first collected , constructing a face image data set; and constructing an initial head pose detection network and an initial eye movement detection network; using the face image data set to perform the initial head pose detection network and the initial eye movement detection network respectively Training to obtain the trained head posture detection network and the eye movement detection network.

In order to make the initial head posture detection network and the initial eye movement detection network have better generalization ability, the face image data set collected in this embodiment needs to have the following characteristics: a. has a wide distribution, Cover all head postures and eye movements as much as possible. At the same time, the data image should also include different light intensities, and even reflective interference from glasses. B. The face image data set has three-dimensional labels of head pose and eye gaze. c. The face images in the face image data set are preferably general RGB images, rather than relying on specific camera equipment.

In order to make the face image data set have a wider distribution, this embodiment uses a 3×4 camera array to represent different head postures through different camera angles of view. However, the area array camera array can only represent the difference of the head posture in the two directions of (y,p). Therefore, in order to obtain the difference of the head posture in the r direction, the collected face images are rotated clockwise and counterclockwise to obtain Indicates the change of the side swing movement of the head. For each corresponding head pose, the position of the camera array and the angle of image rotation correspond to a head pose label (y ^GT , p ^GT , r ^GT ).

In order to obtain richer eye movements, while collecting the face image data set, let the data provider's eyes track and watch a moving point on the display screen. The moving point on the display screen contains random letters, and the data provider needs to identify the letters to Make sure that the data provider’s eyeball is watching the moving point of the screen, so as to ensure the accuracy of the data label, in order to obtain different eyeball movements, corresponding to the position of each eyeball tracking, and record the eyeball vector label at this time (φ ^GT , θ ^GT ). While collecting the face image data set, record the head posture in each face image and the corresponding three-dimensional vector label of the eyeball sight.

In this embodiment, when collecting the face image data set, only the RGB image of the face needs to be collected without relying on other special equipment. The application cost is reduced, and because the head is free and unconstrained, it has better convenience.

Before constructing the initial head posture detection network, the preliminary eyeball movement detection network and the initial 3D gaze vector detection network, the geometric analysis and coordinate system used in this embodiment are described first. In this embodiment, two coordinate systems are used, the head coordinate system (X _h , Y _h , Z _h ) and the camera coordinate system (X _c , Y _c , Z _c ), and g is the line of sight vector. In order to further simplify the expression of the head posture, the embodiment of the present invention adopts a three-dimensional spherical rotation angle representation (y, p, r), where y represents the yaw angle (rotation angle along the Y _h axis), and p represents the tilt angle (the rotation angle along the X _h axis), and r represents the yaw angle (the rotation angle along the Z _h axis). The eyeball movement is represented by a two-dimensional spherical coordinate system (θ, φ), where θ and φ represent the horizontal and vertical angles between the line of sight vector and the head coordinate system, respectively.

Using eyeball movements to describe the line of sight vector in the head coordinate system is as follows:

g _h ＝[-cos(φ)sin(θ),sin(φ),-cos(φ)cos(θ)] ^T

The camera coordinate system (X _c , Y _c , Z _c ) is defined as the center of the camera as the origin, the depth direction of the camera as the Z _c axis, and the two directions of the plane perpendicular to the depth direction as the X _c and Y _c axes respectively. Since the 3D line-of-sight vector finally output by the network is expressed in the camera coordinate system, the embodiment of the present invention defines g _c as the 3-D line-of-sight vector in the camera coordinate system. According to geometric knowledge, g _c depends on g _h , and g _h is It is defined under the head coordinate system, so the overall mapping relationship of the embodiment of the present invention can be obtained:

Step S102: Input the human face image into the pre-built eye movement detection network to obtain the eye movement of the human face image;

Step S103: Input the head pose and the eye movement into a pre-built 3D line of sight vector detection network to obtain a 3D line of sight direction vector of the eye in the face image.

The head pose and the eye movement are input to a pre-built three-dimensional line of sight vector detection network to obtain the three-dimensional line of sight vector of the eyeball in the face image.

In order to reuse existing data sets, the network in this embodiment adopts an end-to-end structure. First, the initial head pose detection network and the eye movement detection network are respectively established, and then the structure detection results of the two parts of the network are input into A fully connected network to obtain the final 3D line of sight vector, the network is divided into two branches, the upper branch is used to detect the head posture, the lower part is used to detect eyeball movements, and then through the geometrically constrained line of sight conversion layer, the camera coordinate system is obtained Sight 3D direction vector.

Based on the above-mentioned embodiments, in this embodiment, in order to reuse the collected face image data sets, this embodiment adopts an end-to-end structure, and first establishes a head posture detection network and an eye movement detection network respectively, and then The structure detection results of the two parts of the network are input into a fully connected network to obtain the final 3D line of sight vector. The network is divided into two branches, the upper branch is used to detect the head posture, and the lower part is used to detect eye movements, and then through geometric constraints The line-of-sight transformation layer of the camera coordinate system obtains the line-of-sight three-dimensional direction vector. Please refer to Fig. 2, Fig. 2 is the flow chart of the second specific embodiment of the method for realizing eyeball three-dimensional line of sight tracking provided by the present invention; the specific operation steps are as follows:

Step S201: using an area array camera array to collect multiple face images of data providers, and recording the three-dimensional vector labels of the head posture and eye movements in each face image to obtain the first subset of face images;

Step S202: Rotate the face images in the first subset of face images clockwise and counterclockwise, respectively, to obtain the second subset of face images;

Step S203: merging the first subset of face images and the second subset of face images to obtain the face image data set;

Step S204: Using the face image data set to train the pre-built initial head pose detection network and initial eyeball movement detection network respectively, to obtain the target head pose detection network and the target eyeball detection network;

The basic network structure of the initial head pose detection network adopts the structure of Alex Net, which is simplified and modified accordingly. The number of layers of the network remains unchanged, but the number of channels in each layer is appropriately reduced, and at the same time, the local response normalization is changed to batch normalization, and the activation function uses PReLU. The network structure of the initial head pose detection network is as follows: C(3,1,6)-BN-PReLU-P(2,2)-C(3,1,16)-BN-PReLU-P(2, 2)-C(3,1,24)-BN-PReLU-C(3,1,24)-PReLU(3,1,16)-BN-PReLU-P(2,2)-FC(256)- FC(128)-PReLU-FC(3)

Among them, C(k,s,c) indicates that the convolution kernel size is k, the convolution step size is s, and the convolution layer with the number of channels is c, and P(k,s) indicates that the kernel size is k, and the step size is the maximum pooling layer of s, BN means batch normalization, PReLU means activation function, FC(n) means fully connected layer, and the number of neurons is n.

The input of the eyeball movement detection network is the eye area intercepted by the original picture of the face image, which is divided into two parts, the left eye and the right eye. Since the two parts of the network are completely symmetrical, the following will describe its part in detail, and adjust the eyeball image block to a consistent size of 36x36, and then through a convolutional neural network and a fully connected network, the initial eye movement detection network structure is as follows: C(11,2,96)-BN-PReLU-P(2,2)-C(5, 1,256)-BN-PReLU-P(2,2)-C(3,1,384)-BN-PReLU-P(2,2)-C(1,1,64)-BN-PReLU-P(2,2 )-FC(128)-FC(2).

Step S205: Use the target head pose detection network and the target eyeball movement detection network to detect each face in the face image data set, and obtain the head pose and eyeballs of each face image action;

Step S206: Using the head posture and eyeball movements of each face image in the face image data set to input to the pre-built initial 3D line of sight vector detection network for training, to obtain the target 3D line of sight vector detection network;

The initial three-dimensional line of sight vector detection network is obtained from (y, p, r) obtained by the target head posture detection network and (θ, φ) obtained by the target eye movement detection network as the initial three-dimensional line of sight vector detection network The input of the initial three-dimensional line of sight vector detection network is a two-layer fully connected network, the number of neurons in the first layer of the network is 128, and the number of neurons in the last layer is 3, corresponding to the three-dimensional line of sight vector.

和所述初步眼球动作检测网络损失函数

之和。When the head pose detection network and the initial eye movement detection network are trained, the loss function Loss ₁ =Loss _h +Loss _e is the loss function of the preliminary head pose detection network

and the preliminary eye movement detection network loss function

Sum.

之和。When using the pre-established initial three-dimensional line of sight vector detection network for training, the current loss function Loss ₂ = Loss ₁ + Loss _g = Loss _h + Loss _e + Loss _g is the loss function Loss ₁ and the initial three-dimensional line of sight vector detection Network loss function

Sum.

Loss _h ＝||hh ^GT || ₂ , h＝{y,p,r}

Loss _e ＝||ee ^GT || ₂ , e＝{φ,θ}

Loss _g ＝||g _c -g _c ^GT || ₂ ,g _c ＝{x,y,z}

Step S207: Input the face image to be detected into the target head pose detection network to obtain the head pose in the face image to be detected;

Step S208: Input the face image to be detected into the target eye movement detection network to obtain the eye movement of the face image to be detected;

Step S209: Input the head posture of the face image to be detected and the eyeball movement of the face image to be detected into the target 3D line of sight vector detection network to obtain the eyeball in the face image to be detected The 3D view direction vector of .

In the prior art, only the two-dimensional labeling of the eyeball center position is carried out in the eyeball recognition, and finally only the two-dimensional information of the eyeball can be obtained, so the application is limited, and the method provided in this embodiment is also based on the deep neural network, but this implementation The example not only processes the movement information of the eyeball, but also predicts the head posture, and at the same time predicts the three-dimensional line of sight vector of the eyeball, so that it has higher-level information and better application value. In this embodiment, the network training adopts end-to-end step-by-step training. In the first step of training, the existing head posture data set and eye movement data set can be fully utilized, thereby greatly increasing the training data. set, so that the deep network in this embodiment has better generalization ability.

Please refer to FIG. 3. FIG. 3 is a structural block diagram of a device for realizing eyeball three-dimensional line-of-sight tracking provided by an embodiment of the present invention; the specific device may include:

The head pose detection module 100 is used to input the human face image to be detected to the pre-built head pose detection network to obtain the head pose in the human face image;

The eye movement detection module 200 is configured to input the human face image into a pre-built eye movement detection network to obtain the eye movement of the human face image;

The 3D line of sight detection module 300 is configured to input the head pose and the eye movement into a pre-built 3D line of sight vector detection network to obtain a 3D line of sight direction vector of the eyeballs in the face image.

The device for realizing eyeball three-dimensional line of sight tracking in this embodiment is used to implement the aforementioned method for realizing eyeball three-dimensional line of sight tracking. Therefore, the specific implementation of the device for realizing eyeball three-dimensional line of sight tracking can be seen in the implementation of the method for realizing eyeball three-dimensional line of sight tracking in the above text. In the example part, for example, the head posture detection module 100, the eyeball movement detection module 200, and the three-dimensional line of sight detection module 300 are respectively used to implement steps S101, S102 and S103 in the method for realizing eyeball three-dimensional line of sight tracking. Therefore, the specific implementation method Reference may be made to the descriptions of the corresponding partial embodiments, and details are not repeated here.

The specific embodiment of the present invention also provides a device for realizing eyeball three-dimensional line of sight tracking, including: a memory for storing computer programs; a processor for implementing the above-mentioned method for realizing eyeball three-dimensional line of sight tracking when executing the computer program A step of.

A specific embodiment of the present invention also provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the above-mentioned method for realizing three-dimensional sight tracking of the eyeball is realized. step.

Each embodiment in this specification is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same or similar parts of each embodiment can be referred to each other. As for the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and for the related information, please refer to the description of the method part.

Professionals can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software or a combination of the two. In order to clearly illustrate the possible For interchangeability, in the above description, the composition and steps of each example have been generally described according to their functions. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present invention.

The steps of the methods or algorithms described in connection with the embodiments disclosed herein may be directly implemented by hardware, software modules executed by a processor, or a combination of both. Software modules can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other Any other known storage medium.

The method, device, equipment and computer-readable storage medium for realizing eyeball three-dimensional line-of-sight tracking provided by the present invention are described above in detail. In this paper, specific examples are used to illustrate the principle and implementation of the present invention, and the descriptions of the above embodiments are only used to help understand the method and core idea of the present invention. It should be pointed out that for those skilled in the art, without departing from the principle of the present invention, some improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims (8)

1. A method for realizing eyeball three-dimensional line of sight tracking, characterized in that, comprising: Input the face image to be detected to the pre-built head pose detection network to obtain the head pose in the face image; The human face image is input to a pre-built eye movement detection network to obtain the eye movement of the human face image; Inputting the head posture and the eyeball movement into a pre-built three-dimensional line of sight vector detection network to obtain the three-dimensional line of sight direction vector of the eyeball in the face image; Said inputting the human face image to be detected into the pre-built head posture detection network, before obtaining the head posture in the human face image includes: collecting multiple human faces with three-dimensional labels of head posture and eyeball sight image, constructing a face image data set, wherein the face image is an RGB image; constructing an initial head posture detection network and an initial eyeball movement detection network; The detection network and the initial eye movement detection network are trained to obtain the trained head posture detection network and the eye movement detection network; The network structure of the initial head posture detection network is: C(3,1,6)-BN-PReLU-P(2,2)-C(3,1,16)-BN-PReLU-P(2,2)-C(3,1,24)-BN -PReLU-C(3,1,24)-PReLU(3,1,16)-BN-PReLU-P(2,2)-FC(256)-FC(128)-PReLU-FC(3); The initial eye movement detection network structure is: C(11,2,96)-BN-PReLU-P(2,2)-C(5,1,256)-BN-PReLU-P(2,2)-C(3,1,384)-BN-PReLU-P (2,2)-C(1,1,64)-BN-PReLU-P(2,2)-FC(128)-FC(2); Among them, C(k,s,c) means that the convolution kernel size is k, the convolution step size is s, and the number of channels is c convolutional layer, and P(k,s) means that the kernel size is k, and the step size is s The maximum pooling layer of BN means batch normalization, PReLU means activation function, FC(n) means fully connected layer, and the number of neurons is n; The input of the head posture and the eyeball movement to the pre-built three-dimensional line of sight vector detection network, before obtaining the three-dimensional line of sight direction vector of the eyeball in the face image, includes: Using the head posture detection network and the eye movement detection network to detect the face images in the face image data set respectively, to obtain the head posture and eye movement of each face image; Utilizing the head postures and eye movements of each of the face images to train the pre-established initial three-dimensional line of sight vector detection network, thereby obtaining the three-dimensional line of sight vector detection network that has completed the training; The initial three-dimensional line of sight vector detection network is a two-layer fully connected network, the number of neurons in the first layer of the network is 128, and the number of neurons in the last layer is 3, corresponding to the three-dimensional line of sight vector. 2. method as claimed in claim 1, is characterized in that, described collection has the face image of the three-dimensional label of head pose and eyeball line of sight a plurality of pieces, constructs face image data set and comprises: Using each camera in the area array camera array to separately collect the face images of the data provider to obtain the first subset of face images; Each row of cameras in the area array camera array collects multiple face images, representing different head postures of the data provider in the y direction; The multiple face images collected by each row of cameras in the area array camera array represent different head postures of the data provider in the p direction; Rotating the face images collected by the area camera array clockwise and counterclockwise, respectively, to obtain a second subset of face images representing different head postures of the data provider in the r direction; Combining the first subset of face images and the second subset of face images to obtain the face image data set. 3. The method according to claim 2, wherein said utilizing each camera in the area camera array to collect the face image of the data provider respectively comprises: When collecting the face image of the data provider, record the moving point on the display screen where the data provider's eyes are facing up, so as to determine the three-dimensional vector label of the data provider's eye sight, and record each face image at the same time head pose in . 4. method as claimed in claim 1, is characterized in that, described utilizing described human face image data set to respectively carry out training to described initial head gesture detection network and described initial eyeball movement detection network comprising: Using the face image data set to train the head pose detection network and the initial eye movement detection network; Wherein, the loss function Loss ₁ =Loss _h +Loss _e is the sum of the loss function Loss _h of the preliminary head pose detection network and the loss function Loss _e of the preliminary eye movement detection network. 5. The method according to claim 4, wherein the head pose and the eyeball movement are input to a pre-built three-dimensional line of sight vector detection network to obtain the three-dimensional line of sight of the eyeball in the face image Include before the direction vector: The current loss function Loss ₂ =Loss ₁ +Loss _g =Loss _h +Loss _e +Loss _g is the sum of the loss function Loss ₁ and the loss function Loss _g of the initial 3D view vector detection network. 6. A device for realizing eyeball three-dimensional line-of-sight tracking, characterized in that it comprises: The head attitude detection module is used to input the human face image to be detected to the pre-built head attitude detection network to obtain the head attitude in the human face image; the human face image to be detected is input to the pre-built The constructed head pose detection network, before obtaining the head pose in the face image, includes: collecting multiple face images with three-dimensional labels of head pose and eyeball sight, and constructing a face image data set, wherein the The face image is an RGB image; construct an initial head posture detection network and an initial eye movement detection network; utilize the face image data set to train the initial head posture detection network and the initial eye movement detection network respectively , obtain the described head pose detection network and the described eyeball motion detection network that have completed the training; The network structure of the initial head pose detection network is: C(3,1,6)-BN-PReLU-P(2, 2)-C(3,1,16)-BN-PReLU-P(2,2)-C(3,1,24)-BN-PReLU-C(3,1,24)-PReLU(3,1 ,16)-BN-PReLU-P(2,2)-FC(256)-FC(128)-PReLU-FC(3); the initial eye movement detection network structure is: C(11,2,96) -BN-PreLU-P(2,2)-C(5,1,256)-BN-PReLU-P(2,2)-C(3,1,384)-BN-PReLU-P(2,2)-C( 1,1,64)-BN-PReLU-P(2,2)-FC(128)-FC(2); among them, C(k,s,c) indicates that the convolution kernel size is k, and the convolution step size is the convolutional layer with s and the number of channels c, P(k,s) represents the maximum pooling layer with a kernel size of k and a step size of s, BN represents batch normalization, PReLU represents the activation function, FC(n ) represents a fully connected layer, and the number of neurons is n; An eye movement detection module, configured to input the human face image into a pre-built eye movement detection network to obtain the eye movement of the human face image; The three-dimensional line of sight detection module is used to input the head posture and the eyeball movement to the pre-built three-dimensional line of sight vector detection network to obtain the three-dimensional line of sight direction vector of the eyeball in the face image; the head The posture and the eyeball movement are input to the pre-built three-dimensional line of sight vector detection network, and before obtaining the three-dimensional line of sight direction vector of the eyeball in the face image includes: using the head posture detection network and the eyeball movement detection network to respectively The human face images in the human face image data set are detected to obtain the head posture and eyeball movements of each human face image; the pre-established initial three-dimensional line of sight is determined by using the head posture and eyeball movements of each human face image The vector detection network is trained to obtain the three-dimensional line of sight vector detection network that completes the training; the initial three-dimensional line of sight vector detection network is a two-layer fully connected network, the number of neurons in the first layer of the network is 128, and the number of neurons in the last layer is 3, corresponding to the three-dimensional line of sight vector. 7. A device for realizing eyeball three-dimensional line-of-sight tracking, characterized in that it comprises: memory for storing computer programs; The processor is configured to implement the steps of a method for realizing eyeball three-dimensional line-of-sight tracking according to any one of claims 1 to 5 when executing the computer program. 8. A computer-readable storage medium, characterized in that, a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the computer program according to any one of claims 1 to 5 is implemented. Steps of a method for realizing eyeball three-dimensional line of sight tracking.

Images