KR102420924B1 - 3D gaze estimation method and apparatus using multi-stream CNNs - Google Patents

Abstract

A deep learning-based 3D gaze prediction method and apparatus are disclosed. The deep learning-based 3D gaze prediction method includes the steps of extracting a right eye patch and a left eye patch by normalizing after detecting each eye region from a visible light input image; extracting a right eye feature vector and a left eye feature vector by applying the right eye patch and the left eye patch to a right eye feature extraction module and a left eye feature extraction module; and estimating the viewing angle by applying the right eye feature vector, the left eye feature vector, and the head pose to a learned viewing angle estimation model.

Description

Deep learning-based 3D gaze estimation method and apparatus {3D gaze estimation method and apparatus using multi-stream CNNs}

The present invention relates to a deep learning-based 3D gaze prediction method and an apparatus therefor.

In terms of visual and cognitive processing, eye movement and gaze estimation are very important. In particular, eye movement has been extensively studied for the study of human gaze concentration, emotion analysis, and behavioral disorder identification.

Gaze estimation has been studied in the field of computer vision because it has a wide range of applications, such as human-computer interaction, psychology, disability research, navigation, driver behavior detection, robotic surgery, and marketing research.

Previous models and feature-based methods for gaze prediction have limitations depending on lighting conditions, camera calibration methods, and individual head pose changes. Computer vision researchers have recently explored appearance-based methods for estimating a person's gaze in an uncontrolled environment, typically using convolutional neural networks (CNNs) due to the availability of large gaze data sets.

Although the deep learning approach has achieved remarkable success in estimating human gaze in the natural environment, the current approach achieves only about 3.6 degrees, making it difficult to apply to real-time applications.

An object of the present invention is to provide a deep learning-based 3D gaze prediction method and apparatus capable of accurately predicting an observer's gaze only with a webcam image.

In addition, an object of the present invention is to provide a deep learning-based 3D gaze prediction method and apparatus that trains a model using only various natural face images, does not have a long execution time, and has excellent performance.

According to one aspect of the present invention, there is provided a deep learning-based 3D gaze prediction method capable of accurately predicting the gaze of an observer only with a webcam image.

According to an embodiment of the present invention, there is provided a method comprising: extracting a right eye patch and a left eye patch by normalizing each eye region from a visible light input image; extracting a right eye feature vector and a left eye feature vector by applying the right eye patch and the left eye patch to a right eye feature extraction module and a left eye feature extraction module; and estimating the viewing angle by applying the right eye feature vector, the left eye feature vector, and the head pose to a learned gaze angle estimation model.

The gaze angle estimation model includes three fully connected layers and a linear regression module, wherein the right eye feature vector, the left eye feature vector, and the head pose are learned after combining through the three fully connected layers. Applied to the module, the viewing angle can be estimated.

The linear regression module may be trained such that a loss function calculated using the predicted viewing angle and the actual viewing angle is minimized.

The viewing angles are yaw and pitch.

The right eye patch and the left eye patch may be normalized patch images cut to a predetermined size from the center p with a head roll removed corresponding to the reference point x.

According to another aspect of the present invention, there is provided an apparatus capable of accurately predicting an observer's gaze only from a webcam image.

According to an embodiment of the present invention, there is provided a pre-processing unit for extracting a right eye patch and a left eye patch by normalizing each eye region from a visible light input image; a feature extraction unit including a right eye feature extraction module for analyzing the right eye patch to extract a right eye feature vector, and a left eye feature extraction module for extracting a left eye feature vector by analyzing the left eye patch; and a viewing angle estimator for estimating the viewing angle through linear regression analysis by applying the right eye feature vector, the left eye feature vector, and the head pose to a model.

The model includes three fully connected layers and a linear regression module, wherein the right eye feature vector, the left eye feature vector, and the head pose are combined through the three fully connected layers and then applied to the learned linear regression module Thus, the viewing angle can be estimated.

The preprocessor detects a face region from the input image, detects both eye regions from the detected face region, respectively, and removes the head roll corresponding to the reference point x at the center p. The right eye patch and the left eye patch may be extracted by cutting to a predetermined size and normalizing it.

By providing a deep learning-based 3D gaze prediction method and apparatus according to an embodiment of the present invention, there is an advantage in that the gaze of an observer can be accurately predicted using a visible light image.

In addition, the present invention has an advantage in that it is possible to predict a gaze with excellent performance without a long execution time by learning a model using only various visible light face images without using an infrared image.

1 is a flowchart illustrating a deep learning-based 3D gaze prediction method according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating extraction of a left eye patch and a right eye patch according to an embodiment of the present invention; FIG.
3 is a diagram illustrating a detailed structure of a feature extraction module according to an embodiment of the present invention.
4 is a diagram illustrating a detailed configuration of a viewing angle estimator according to an embodiment of the present invention.
5 is a diagram illustrating example images from an MPIIGaze data set and an EYEDIAP data set according to an embodiment of the present invention;
6 is a diagram illustrating a method for augmenting a data set according to an embodiment of the present invention;
7 is a diagram illustrating a configuration of a 3D gaze prediction apparatus according to an embodiment of the present invention.
8 is a view comparing an estimated viewing angle and an actual viewing angle according to an embodiment of the present invention.

As used herein, the singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as “consisting of” or “comprising” should not be construed as necessarily including all of the various components or various steps described in the specification, some of which components or some steps are It should be construed that it may not include, or may further include additional components or steps. In addition, terms such as "...unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software. .

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart illustrating a deep learning-based 3D gaze prediction method according to an embodiment of the present invention, and FIG. 2 is a diagram illustrating extraction of a left eye patch and a right eye patch according to an embodiment of the present invention, FIG. 3 is a diagram showing a detailed structure of a feature extraction module according to an embodiment of the present invention, FIG. 4 is a diagram showing a detailed configuration of a viewing angle estimator according to an embodiment of the present invention, and FIG. 5 is a diagram showing the present invention is a diagram illustrating an example image from an MPIIGaze data set and an EYEDIAP data set according to an embodiment of the present invention, and FIG. 6 is a diagram illustrating a method for augmenting a data set according to an embodiment of the present invention.

In step 110 , the gaze prediction apparatus 100 analyzes the input image and extracts the normalized left eye patch and right eye patch, respectively.

A visible light image, which is an input image, has a different resolution depending on lighting conditions and camera conditions. Accordingly, according to an embodiment of the present invention, after detecting the left eye region and the right eye region from the input image, normalizing the detection and then extracting the left eye patch and the right eye patch, respectively.

The gaze prediction apparatus 100 may detect a face region from an input image that is a visible light image, and may detect both eye regions from the detected face region. Next, the gaze prediction apparatus 100 normalizes the detected both eye regions and then extracts both eye patches.

For example, the gaze prediction apparatus 100 may extract normalized both eye patches from which a head roll is removed from the input image, respectively.

According to an embodiment of the present invention, the input image may be a visible light image captured by a webcam.

The gaze prediction apparatus 100 is a normalized bilateral eye patch from which a head roll is removed in extracting both eye patches to accurately predict a gaze by overcoming an appearance variation regardless of a camera parameter in an input image. can be extracted. This will be described in more detail.

The input image will be referred to as I. Given an input image I and a reference point x, the goal is to calculate a conversion matrix M (M). Here, the conversion matrix M may be calculated using Equation (1).

로부터 기준점(x)를 보도록 정의될 수 있다. 여기서, 스케일 행렬(S)는 수학식 2와 같이 나타낼 수 있다. With a rotation matrix R, the head coordinate system and the x-axis of the camera are parallel. Therefore, the scale matrix S is the distance at which the virtual camera is fixed.

It can be defined to look at the reference point (x) from Here, the scale matrix S can be expressed as in Equation (2).

는 기준점으로부터의 이동(translation)을 나타내는 행렬이다. where diag() represents a diagonal matrix,

is a matrix representing translation from the reference point.

The gaze prediction apparatus 100 may normalize the input image I through perspective warping by using a transformation matrix W (transformation matrix). This is expressed as Equation 3 as shown in Equation 3.

는 정규화된 카메라의 투영 행렬을 나타내고,

는 실제 카메라 행렬을 나타낸다. here,

denotes the projection matrix of the normalized camera,

represents the actual camera matrix.

As a result, the gaze prediction apparatus 100 detects both eye regions in the visible light input image, and has a predetermined size (W x H) at the center (p) in a state in which the head roll with respect to the reference point (x) is removed. Each of the eye patches can be extracted with a cropped patch having .

2 shows an example of both eye patch extraction. The gaze prediction apparatus 100 detects a face region from the webcam image, detects both eye regions from the detected face region, and then a normalized eye patch I _L from which a head roll is removed with respect to the reference point x. , I _R ) can be extracted, respectively.

In operation 115 , the gaze prediction apparatus 100 extracts feature vectors from the normalized left eye patch I _L and the right eye patch I _R , respectively.

The normalized left-eye patch and right-eye patch may be input to respective feature extraction units, and feature vectors may be respectively extracted by each feature extraction unit. For convenience, hereinafter, the left eye feature vector and the right eye feature vector are referred to as the left eye feature vector.

The left eye patch and right eye patch are respectively input to the left eye feature extraction module and the right eye feature extraction module. Here, the left-eye feature extraction module and the right-eye feature extraction module do not share weights with each other. That is, it is natural that the left-eye feature extraction module and the right-eye feature extraction module can be learned to have different weights, and can be independently learned/operated.

The detailed configuration of the left eye feature extraction module and the right eye feature extraction module is the same. The structures of the left eye feature extraction module and the right eye feature extraction module are as shown in FIG. 3 .

The feature extraction module includes 5 convolutional layers for feature extraction, and the spatial weight matrix, the output value of the spatial attention module, is sigmoided with the result value (active map) passed through the 5 convolutional layers to activate the final weight Each map may be generated as a feature vector.

The spatial attention module consists of three convolutional layers with filter size of 1 x 1 and a refined linear unit (ReLU). Accordingly, the feature extraction module may generate a final weighted activation map using a result value (activation map) that has passed through five convolutional layers and a spatial weight matrix that is a result value applied with a spatial attention module. This is expressed as Equation 4 as shown in Equation 4.

Here, W denotes a spatial weight matrix that is the output result of the spatial attention module, and U denotes an activation map that has passed through five convolutional layers.

The weighted activation map goes through the final maximum pooling layer and can be reduced in dimension. The dimension (size) of the feature vector can be reduced by passing through the maximum pooling layer through multiplication of each element of the spatial weight matrix and the final weighted activation map.

In operation 120 , the gaze prediction apparatus 100 estimates the gaze angle by applying the left eye feature vector, the right eye feature vector, and the head pose to the gaze angle estimation model.

Here, the estimated viewing angle may be a yaw and a pitch. Here, yaw and pitch may be yaw and pitch according to pupil movement.

As a result, according to an embodiment of the present invention, the 3D gaze prediction apparatus 100 may estimate a 3D gaze angle by detecting a pupil movement using a 2D visible light input image.

The viewing angle estimation model will be described in more detail with reference to FIG. 4 .

The gaze angle estimation model includes three fully connected layers and a linear regression module.

In this case, a batch-normalization layer (BN) layer and a ReLU layer may be respectively positioned at the rear end of the fully connected layer.

According to an embodiment of the present invention, the sizes of the three fully connected layers included in the viewing angle estimation model may be different from each other. That is, the sizes of the three fully connected layers may be 512, 256, and 2, respectively. The size of the fully connected layer is not necessarily limited thereto, and it is natural that the number and size of the fully connected layer may be different according to an implementation method.

Before combining the left and right eye feature vectors, the dropout layer (p=0.5) was connected to a fully connected layer of 512 size, and then the BN layer and the ReLU layer were connected.

Thereafter, a fully connected layer of size 256 is added, the fully connected layer of size 256 is connected to the BN layer and the ReLU layer, and a fully connected layer of size 2 may be added. In addition, it can be seen that the performance is improved when the BN layer is used in connection with the fully connected layer as in an embodiment of the present invention.

A head pose may be input to the final layer.

A head pose vector may be input through the rearmost layer.

Due to this, the linear regression module may calculate the viewing angle yaw and pitch using the left eye feature vector, the right eye feature vector, and the head pose.

It is assumed that the linear regression model is pre-trained using the training data set. The linear regression model can calculate the loss function by estimating the Euclidean distance between the predicted viewing angle and the actual viewing angle. In this case, the linear regression model may be trained such that the loss function is minimized.

For example, the loss function may be defined as in Equation 5.

는 i번째 영상의 예측된 시선각을 나타내며,

는 i번째 영상의 실제 시선각을 나타낸다. where N represents the total number of images,

represents the predicted viewing angle of the i-th image,

denotes the actual viewing angle of the i-th image.

According to an embodiment of the present invention, the gaze angle estimation model may be trained in advance using the MIPGaze and EYEDIAP data sets, which are used as global standards for gaze tracking. 5 shows an example of images included in the MIPGaze and EYEDIAP data sets. Although images are included in the MIPGaze and EYEDIAP data sets, respectively, when the model is trained using only these images, the accuracy may be relatively low. Therefore, in an embodiment of the present invention, in addition to the MIPGaze and EYEDIAP data sets, Gaussian blur, gamma transform, and noise are added to the image of the corresponding data set to increase the data set and use it for learning (see FIG. 6 ).

By adding gamma-corrected images to the images included in the data set, the model can be trained to robustly adapt to different lighting conditions (eg, dark lighting, bright lighting).

In addition, in order to more strongly train the model in the blur condition of the camera, using OpenCV Gaussian blur, a corrected image with a kernel size of 7 x 7, 3 x 3, etc. was generated to enable additional training. Also, a model can be trained using an image in which various noises such as Gaussian salt pepper are added to the eye patch.

In one embodiment of the present invention, in generating the training data set as described above, the data is augmented through Gaussian blur, gamma correction, noise addition, etc. to enable learning about various lighting conditions and camera conditions to increase the accuracy of the model. There are advantages that can be

7 is a diagram illustrating a configuration of a 3D gaze prediction apparatus according to an embodiment of the present invention, and FIG. 8 is a diagram comparing an estimated gaze angle and an actual gaze angle according to an embodiment of the present invention.

Referring to FIG. 7 , the 3D gaze prediction apparatus 100 according to an embodiment of the present invention includes a preprocessor 710 , a feature extraction unit 715 , a gaze angle estimation unit 720 , a learning unit 725 , and a memory. 730 and a processor 735 .

The preprocessor 710 detects a face region in the visible light image (input image), detects and normalizes both eye regions (left eye region and right eye region) from the detected face region to extract the left eye patch and the right eye patch, respectively. is a means for

Since this is the same as that described with reference to FIG. 1 , the overlapping description will be omitted.

The feature extraction unit 715 includes a left eye feature extraction module and a right eye feature extraction module. The left-eye feature extraction module is a means for extracting a left-eye feature vector using a left-eye patch. Also, the right eye feature extraction module is a means for extracting a right eye feature vector using a right eye patch.

Detailed configurations of the left-eye feature extraction module and the right-eye feature extraction module included in the feature extraction unit 715 are as shown in FIG. 3 . Since this is the same as that described above, the overlapping description will be omitted.

The viewing angle estimator 720 includes a linear regression module. Three fully connected layers are positioned in front of the linear regression module, and the left eye feature vector and the right eye feature vector may be combined through the three fully connected layers. Also, a head pose may be input to the final layer (layer).

Accordingly, the linear regression module receives a vector value including the left eye feature vector, the right eye feature vector, and the head pose, and the linear regression module can estimate the 3D viewing angle using the left eye feature vector, the right eye feature vector, and the head pose. . The 3D viewing angle may be yaw and pitch.

The learning unit 725 is a means for learning the viewing angle estimator 720 using the training data set.

The learning unit 725 may learn the linear regression module such that the loss function is minimized. Since the loss function is the same as that described above, a further description thereof will be omitted.

As already described above, the training data set uses the MIPGaze and EYEDIAP data sets, but can be used for learning by adding data through Gaussian blur, gamma transformation, and noise addition using MIPGaze and EYEDIAP.

The memory 730 is a means for storing various instructions (program codes) necessary to perform a deep learning-based 3D gaze prediction method according to an embodiment of the present invention.

The processor 735 includes internal components (eg, a preprocessor 710 , a feature extractor 715 , and a gaze angle estimator 720 ) of the 3D gaze prediction apparatus 100 according to an embodiment of the present invention. , the memory 730, etc.).

8 illustrates a result of predicting a 3D viewing angle. In FIG. 8 , green indicates a predicted viewing angle, and red indicates an actual viewing angle. As shown in FIG. 8 , it can be seen that the 3D viewing angle is accurately predicted from the visible light image.

The apparatus and method according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the computer readable medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - Includes magneto-optical media and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Up to now, the present invention has been looked at focusing on the embodiments thereof. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention may be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

100: gaze prediction device
710: preprocessor
715: feature extraction unit
720: gaze angle estimation unit
725: Learning Department
730: memory
735: Processor

Claims (10)

extracting a right eye patch and a left eye patch by normalizing each eye region from the visible light input image;
extracting a right eye feature vector and a left eye feature vector by applying the right eye patch and the left eye patch to a right eye feature extraction module and a left eye feature extraction module; and
estimating the viewing angle by applying the right eye feature vector, the left eye feature vector, and the head pose to a learned viewing angle estimation model,
The right-eye feature extraction module and the left-eye feature extraction module sigmoidally compute the result of passing five convolution layers for the right-eye patch and the left-eye patch and the spatial weight matrix that is the result of applying the spatial attention module. Generate each weighted activation map as a feature vector,
The right eye patch and the left eye patch are normalized patch images cut to a certain size from the center (p) with the head roll removed corresponding to the reference point (x),
The gaze angle is a gaze prediction method, characterized in that the yaw (yaw) and the pitch (pitch).
The method of claim 1,
The viewing angle estimation model is
Includes 3 fully connected layers and a linear regression module,
The gaze prediction method, characterized in that the right eye feature vector, the left eye feature vector, and the head pose are combined through the three fully connected layers and then applied to the learned linear regression module to estimate the viewing angle.
3. The method of claim 2,
The linear regression module is a gaze prediction method, characterized in that it is learned to minimize the loss function calculated using the predicted gaze angle and the actual gaze angle.
delete delete A computer-readable recording medium product on which a program code for performing the method according to any one of claims 1 to 3 is recorded.
a preprocessor that detects and normalizes each eye region from the visible light input image to extract a right eye patch and a left eye patch, respectively;
a feature extraction unit including a right eye feature extraction module for analyzing the right eye patch to extract a right eye feature vector, and a left eye feature extraction module for extracting a left eye feature vector by analyzing the left eye patch; and
Comprising a viewing angle estimator for estimating the viewing angle through linear regression analysis by applying the right eye feature vector, the left eye feature vector, and the head pose to a model,
The right-eye feature extraction module and the left-eye feature extraction module sigmoidally compute the result of passing five convolution layers for the right-eye patch and the left-eye patch and the spatial weight matrix that is the result of applying the spatial attention module. Generate each weighted activation map as a feature vector,
The preprocessor detects a face region from the input image, detects both eye regions from the detected face region, respectively, and removes the head roll corresponding to the reference point (x) to a certain size from the center (p). The right eye patch and the left eye patch are extracted by cutting and normalizing, respectively,
The gaze angle is a gaze prediction apparatus, characterized in that the yaw (yaw) and the pitch (pitch).
8. The method of claim 7,
The model is
Includes 3 fully connected layers and a linear regression module,
The gaze prediction apparatus, characterized in that the right eye feature vector, the left eye feature vector, and the head pose are combined through the three fully connected layers and then applied to the learned linear regression module to estimate the gaze angle.


delete delete

Images