KR20190100982A - Deep learning based gaze detection system and method for automobile drivers - Google Patents

Abstract

The present invention relates to a technology of tracking a gaze of a vehicle driver. More specifically, provided are an apparatus of tracking a gaze of a vehicle driver, which classifies a gaze area of a vehicle driver from an image obtained from a camera, and a method thereof. According to one embodiment of the present invention, based on deep learning, the gaze tracking accuracy of the vehicle driver can be improved by considering head movements and eye movements of the driver without requiring an initial user calibration step.

Description

DEEP LEARNING BASED GAZE DETECTION SYSTEM AND METHOD FOR AUTOMOBILE DRIVERS

The present invention relates to a vehicle driver gaze tracking technology, and more particularly, to a vehicle driver gaze tracking apparatus and method for classifying a gaze area of a vehicle driver in an image acquired from a camera.

The existing eye tracking system has been mainly used in the indoor desktop monitor environment, and the eye tracking is performed by moving only the eyes without moving the head as much as possible. However, in a vehicle environment, the driver moves his head to stare at both side mirrors and room mirrors. However, since the gaze tracking method in the indoor room is not a gaze tracking method based on the head movement, the gaze tracking accuracy decreases. In addition, the gaze tracking research in the existing vehicle performed a gaze tracking method based on head movements to solve this problem, but in this case, gaze tracking accuracy decreases when a driver gazes at an object without moving his head. In order to solve this problem, the method using the center of the corneal reflected light generated in the cornea due to the pupil center and the illumination has been performed.However, in a vehicle environment, due to various external light changes, the reflection of the glasses surface, and the optical blur caused by the driver's movement. There was a problem in that the center of the pupil and the center of the corneal reflected light were not accurately detected.

Background of the present invention is disclosed in Republic of Korea Patent No. 10-1395819.

An object of the present invention is to provide an apparatus and method for tracking a vehicle driver's eye gaze based on deep learning considering a driver's head movement and eyeball movement without requiring an initial user calibration step.

According to an aspect of the present invention, a deep learning based driver gaze tracking device is provided.

According to an exemplary embodiment of the present disclosure, a deep learning based driver gaze tracking device may include a driver image input unit that receives a driver image, track face feature identification points from the input driver image, and based on the detected face feature identification point. Feature region detector for extracting image, left eye region image and right eye region image, Deep learning feature extractor for outputting deep learning feature set of each region image using face region image, left eye region image and right eye region image And a driver gaze tracking unit classifying the driver's gaze area by using the output feature set and the stored gaze area feature sets.

According to another aspect of the present invention, a deep learning based driver gaze tracking method and a computer program for executing the same are provided.

A deep learning based driver gaze tracking method and a computer program for executing the same according to an embodiment of the present disclosure may include receiving a driver face image and detecting a face feature identification point by tracking a face feature identification point in the driver face image. Extracting the face region image, the left eye region image and the right hand region image based on the facial feature identification point, and each of the three detected images, such as the face region image, the left eye region image, and the right hand region image The method may include generating a feature set including feature values by performing learning-based convolutional neural networks (CNNs) and classifying a driver's gaze area using the feature set.

According to an embodiment of the present disclosure, the gaze tracking accuracy of the vehicle driver may be improved based on the deep learning in consideration of the head movement and the eye movement of the driver without requiring an initial user calibration step.

According to an embodiment of the present invention, when converting from autonomous driving to driver driving in an autonomous vehicle, the driver's gaze tracking is accurately performed to determine the drowsy driving, distraction, and stare of the driver, which helps in converting the driving. This can provide accurate driver status information.

According to one embodiment of the present invention, the accuracy is higher than the method using the pupil center and the corneal reflection light center, and even if the driver's head is excessively rotated, the driver's gaze can be accurately tracked.

1 to 9 are views for explaining a deep gaze-based driver gaze tracking device according to an embodiment of the present invention.
10 to 13 are diagrams for describing a driver gaze tracking method based on deep learning according to an embodiment of the present invention.

As the inventive concept allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. In describing the present invention, when it is determined that the detailed description of the related known technology may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted. Also, the singular forms used in the specification and claims are to be interpreted as generally meaning "one or more", unless stated otherwise.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, and in the following description with reference to the accompanying drawings, the same or corresponding components are given the same reference numerals and redundant description thereof will be omitted. Shall be.

1 to 9 are diagrams for describing an apparatus for tracking a driver's gaze based on deep learning according to an exemplary embodiment.

Referring to FIG. 1, a deep learning based driver gaze tracking device 100 may include a driver image input unit 110, a feature region detector 120, a deep learning feature extractor 130, and a driver gaze tracker 140. do.

The driver image input unit 110 receives an image of a driver.

Referring to FIG. 2, the driver image input unit 110 is photographed by a near-infrared light (for example, a near-infrared LED of 850 nm) and a near-infrared camera (for example, a USB camera having a NIR band pass filter) located near the instrument panel of the vehicle. Can input the image of the driver. The driver image input unit 110 may also input a visible ray driver image photographed through the visible ray camera.

The feature region detector 120 tracks face feature identification points in the input driver image and extracts a face region image, a left eye region image, and a right eye region image based on the detected face feature identification points. The feature region detector 120 may detect 68 face feature identification points using, for example, a Dlib face feature tractor. Here, facial feature identification points may be used to define and represent protruding areas such as eyes, eyebrows, nose, mouth, chin line, and the like.

Referring to FIG. 3, the feature region detector 120 extracts two-dimensional coordinates of 68 face feature identification points. The feature region detector 120 detects a face region (face feature identification points 1 to 68), a left eye region (face feature identification points 36 to 41), and a right eye region (face feature) based on the two-dimensional coordinates of the extracted face feature identification points. Identification points 42 to 47), and extracts a face region image, a left eye region image, and a right eye region image. The feature area detector 120 may further use a haar cascade detector, a HOG and linear SVM-based detector, or a deep learning algorithm to define a face region, a left eye region, or a right eye region in the driver's image. have.

The deep learning feature extractor 130 outputs a feature set of each region image by using the face region image, the left eye region image, and the right eye region image.

4, the deep learning feature extractor 130 includes a deep learning unit 132, a feature value extractor 134, a feature value normalizer 136, and a feature set generator 138.

The deep learning unit 132 learns face gaze tracking using a CNN algorithm. The deep learning unit 132 machine learns the VGG face network using 26 million face images obtained from 2622 people included in the VGG face 16 model. The deep learning unit 132 uses the training data of its own database (DDGC-DB1) in addition to the VGG face 16 model whose accuracy is evaluated by the Labeled Faces in the Wild (LFW) database and the YouTube Faces (YTF) database. Do this.

As shown in FIG. 5, the deep learning unit 132 has built its own database DDGC-DB1 for the driver's eye tracking system in a vehicle environment. Here, the self database (DDGC-DB1) has obtained test information at various time zones, mornings, lunches and evenings of the day in order to understand the influence of various kinds of external light on driver gaze detection. The deep learning unit 132 uses a face region image, a left hand region image, and a right hand region image of 224 × 224 pixels in its database (DDGC-DB1) for CNN learning and verification. The deep learning unit 132 extracts a set of 4096 feature values from the face region image, the left eye region image, and the right eye region image in response to the second fully connected layer at the final level through CNN learning, and then minimum distance. Based on the above, the gaze area of 17 areas is determined. Here, the seventeen gaze regions may be designated as five or more gaze regions of the driver through the gaze region experiment of the driver, referring to FIG. 6.

7 and 8, the deep learning unit 132 may use a CNN structure including 13 convolutional layers, 5 pulling layers, and 3 fully connected layers. The deep learning unit 132 may include, for example, 64 filters having a size of 3 × 3 for the input of the first convolutional layer of 224 × 224 × 3 (width, height, and size of the channel). The feature map of is obtained. For example, the deep learning unit 132 may output 112 × 112 × 64 through the maximum pooling layer after the obtained 224 × 224 × 64 feature map passes through the ReLU layer of the first convolutional layer. The pooling layer may be kernel size 2 × 2, stride 2 × 2. The deep learning unit 132 may have, for example, a kernel size of 3 × 3, a padding number of 1 × 1 and a stride number of 1 × 1 used in 13 convolution layers, and only 64, 128, 256, and the number of filters. It can be changed to 512, where each ReLU layer is located after the convolutional layer so that each maximum pooling layer can be used after ReLU-1_2, ReLU-2_2, ReLU-3_3, ReLU-4_3, and ReLU-5_3. Here, each maximum pooling layer may include a kernel size 2 × 2, a stride number 2 × 2, and a padding number 0 × 0. For example, the deep learning unit 132 has a 224 × 224 × 64 ReLU-1_2 layer connected to a 112 × 112 × 64 Pool-1 layer and a 112 × 112 × 128 ReLU-2_2 layer 56 × 56. Connected to a Pool-2 layer of × 128 and a ReLU-3_3 layer of 56 × 56 × 256 connected to a Pool-3 layer of 28 × 28 × 246 and a 14 × 14 × ReLU-4_3 layer of 28 × 28 × 512. Connected to a Pool-4 layer of 512 and a ReLU-5_3 layer of 14 × 14 × 512 to a Pool-5 layer of 7 × 7 × 512, with the input image 13 convolutional layers, 13 ReLU layers and 5 pooling layers Finally, the 7 × 7 × 512 feature map can be finally output. Thereafter, the deep learning unit 132 may output feature maps of 4096 × 1, 4096 × 1, and 17 × 1, respectively, through three additional fully connected layers. Here, Equation (1) below is used for the Softmax function in the third fully connected layer.

(1)

(One)

Where r is an array of output neurons

The deep learning unit 132 may use a data increment and loss method to solve the low recognition accuracy in the CNN-based recognition system due to data overfitting, and between the first and second fully connected layers for data loss. Can employ data loss of 50% probability to break the link at random. The deep learning unit 132 extracts three separate feature sets (three 4096 feature sets) from the face region image, the left eye region image, and the right eye region image, and extracts them by min-max scaling. Normalize feature values. The deep learning unit 132 stores normalized feature values obtained from the face region image, the left eye region image, and the right eye region image for the 17 gaze regions.

The feature value extractor 134 converts a face region image, a left eye region image, and a right eye region image, inputs them to deep learning-based convolutional neural networks (CNNs), and extracts feature values of each image. Here, the face region image, the left eye region image, and the right eye region image are converted to a 224 × 224 pixel image including 50 pixels of free space using bi-linear interpolation and used as input values. Can be. The feature value extractor 134 extracts three separate feature values (4096 feature values) from the face region image, the left eye region image, and the right eye region image.

The feature set generator 136 normalizes feature values extracted from the face region image, the left eye region image, and the right eye region image, and generates a feature set including three normalized feature values. In this case, normalization may use a min-max scaling method.

Referring back to FIG. 1, the driver gaze tracking unit 140 classifies the driver gaze area by using the Euclidean distance calculated between the output feature set and the stored feature sets of the 17 gaze areas.

Referring to FIG. 9, the driver gaze tracking unit 140 may include a gaze area distance calculator 142 and a driver gaze classifier 144.

The gaze area distance calculating unit 142 inputs a face area image, a left eye area image, and a right eye area image to a deep learning based CNN algorithm, and outputs 17 gaze areas previously stored using a feature set and learning data. Calculate the Euclidean distance between the stored feature sets.

The driver gaze classification unit 144 combines the three Euclidean distances calculated by a score level fusion method, and the gaze area having the minimum combined scores for the 17 gaze areas as the driver gaze area. Decide Here, the score level fusion is compared with the weighted sum method of the following formula (2) or the weighted product method of the formula (3), and the final weight value is determined by the training data.

(2)

(2)

Where WS is the weighted sum, wi is the weight, m is 3, and di represents the Euclidean distance.

(3)

(3)

Where WP is the weighted product, wi is the weight, m is 3, and di represents the Euclidean distance.

10 to 13 are diagrams for describing a driver's gaze tracking method based on deep learning according to an exemplary embodiment.

Referring to FIG. 10, in operation S1010, the driver gaze tracking apparatus 100 inputs a driver face image. The driver gaze tracking device 100 may use a compact gaze tracking device incorporating an infrared camera.

In step S1020 the driver gaze tracking device 100 The facial feature identification point is detected by tracking the facial feature identification point in the acquired driver's face image.

In step S1030 the driver gaze tracking device 100 The face region image, the left eye region image, and the right hand region image are extracted based on the facial feature identification point.

In step S1040 the driver gaze tracking device 100 Deep learning based CNN is performed on each of three detected images such as a face region image, a left eye region image, and a right hand region image to generate a feature set including feature values.

Referring to FIG. 11, in step S1110, the driver gaze tracking apparatus 100 learns a face gaze tracking using a deep learning-based CNN.

In operation S1120, the driver gaze tracking apparatus 100 converts the face region image, the left eye region image, and the right eye region image into the CNN, and extracts feature values of each image.

In operation S1130, the driver gaze tracking apparatus 100 normalizes the extracted feature values and generates a feature set including the normalized feature values.

Referring back to FIG. 10, in step S1050, the driver gaze tracking apparatus 100 calculates a Euclidean distance of each area image between the generated feature set and the 17 gaze area feature sets previously stored.

In operation S1060, the driver gaze tracking apparatus 100 classifies the driver gaze area using the Euclidean distance of each area image. The driver gaze tracking device 100 combines the calculated eyelid region by Euclidean distance level fusion method of each area image, and the driver gaze region includes a gaze region having a minimum combined score for 17 gaze regions. Can be determined. Here, the level level fusion method may use a weighted sum or weighted product method.

Referring to FIG. 12, the driver's eye tracking method according to the present invention accurately measures a strict accuracy estimation rate (SCER) and a roughly accurate estimation rate (LCER). Here, the strong accuracy evaluation degree (SCER) is the accuracy for the corresponding point, and the approximately accurate evaluation degree (LCER) is the accuracy including the corresponding point and neighboring adjacent points.

The deep learning-based CNN algorithm according to the present invention was cross-validated. The learning data was divided into two groups, and the first group was trained as 16,310 chapters, the second group was 16,280 chapters, and the test (Test) group 1,3,256. Chapter 3, Group 2, 3262 chapters were conducted to measure the accuracy. Detailed accuracy of each region is shown in FIG. 12, and final accuracy is 92.8% SCER and 99.6% LCER for 17 gaze regions.

Referring to FIG. 13, the driver gaze tracking method according to the present invention was applied to a published Columbia gaze database CAVE-DB. Here, CAVE-DB contains 5,880 images of 56 people with varying head poses and gaze directions, with 105 gaze directions for 5 head poses, with 21 gaze directions per head pose, but 13 A gaze direction image was selected.

The driver gaze tracking method according to the present invention measured the Strictly Correct Estimation Rate (SCER) and the Loosely Correct Estimation Rate (LCER) with accuracy against the published Columbia Eye gaze database CAVE-DB. The detailed accuracy of each region is shown in FIG. 13, and the highest accuracy is 88.9% SCER and 98.8% LCER for 13 gaze regions.

The driver's gaze tracking method based on deep learning according to an exemplary embodiment of the present invention may be implemented in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. Computer-readable media may include, alone or in combination with the program instructions, data files, data structures, and the like. The program instructions recorded on the computer readable medium may be those specially designed and constructed for the present invention, or may be known and available to those skilled in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Hardware devices specially configured to store and execute program instructions such as magneto-optical media and ROM, RAM, flash memory and the like. In addition, the above-described medium may be a transmission medium such as an optical or metal wire, a waveguide, or the like including a carrier wave for transmitting a signal specifying a program command, a data structure, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

So far, the present invention has been described with reference to the embodiments. Those skilled in the art will appreciate that the present invention can be implemented in a modified form without departing from the essential features of the present invention. Therefore, the disclosed embodiments should be considered in descriptive sense only and not for purposes of limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope will be construed as being included in the present invention.

Claims (14)

In the deep learning based driver gaze tracking device,
A driver image input unit to receive a driver image;
A feature region detector for tracking face feature identification points in the input driver image and extracting a face region image, a left eye region image, and a right eye region image based on the detected face feature identification points;
A deep learning feature extractor configured to output a deep learning feature set of each area image by using a face area image, a left eye area image, and a right eye area image; And
A deep gaze-based driver gaze tracking device including a driver gaze tracking unit for classifying a gaze area of a driver using the output feature set and the stored gaze area feature sets.
The method of claim 1,
The deep learning feature extraction unit
A deep learning unit learning face gaze tracking using deep learning-based convolutional neural networks (CNN);
A feature value extracting unit configured to convert the face region image, the left eye region image, and the right eye region image, input the same to the CNN, and extract feature values of each image; And
A deep learning based driver gaze tracking device including a feature set generator for normalizing extracted feature values and generating a feature set including the normalized feature values.
The method of claim 1,
The deep learning unit
Deep learning by extracting 4096 feature values from the face region image, the left eye region image, and the right eye region image into images of 224 × 224 pixels, respectively, extracting 4096 feature values Driver's eye tracking device based.
The method of claim 3,
The deep learning unit
Use a CNN structure with 13 convolutional layers, 5 pooling layers, and 3 fully connected layers,
The kernel size used in the 13 convolution layers is 3 × 3, the number of padding is 1 × 1, the number of strides is 1 × 1, and only the number of filters is changed to 64, 128, 256 and 512, and each pooling layer is a kernel. Deep learning based driver gaze tracking device including size 2 × 2, number of strides 2 × 2 and padding 0 × 0.
The method of claim 4, wherein
The deep learning unit
The three fully connected layers are deep learning based driver gaze tracking device for outputting feature maps of 4096 × 1, 4096 × 1 and 17 × 1, respectively.
The method of claim 2,
The feature set generation unit
And a deep learning based driver gaze tracking device for normalizing feature values extracted from the face region image, the left eye region image, and the right eye region image using a min-max scaling method.
The method of claim 1,
The driver's eye tracking unit
A gaze area distance calculator configured to calculate an Euclidean distance between the stored feature sets for the 17 gaze areas stored in advance using the output feature set and the training data; And
A driver gaze classification unit for combining the calculated three Euclidean distances by a score level fusion method and determining a gaze area having a minimum combined score for 17 gaze areas as a driver gaze area; Deep learning based driver gaze tracking device.
The method of claim 7, wherein
The score level fusion method is a deep learning based driver gaze tracking device using a weighted sum or weighted product method.
In the deep learning based driver gaze tracking method,
Receiving a driver's face image;
Detecting a facial feature identification point by tracking a facial feature identification point on the driver's face image;
Extracting a face region image, a left eye region image, and a right hand region image based on the face feature identification point;
Generating a feature set including feature values by performing deep learning-based convolutional neural networks (CNNs) on three detected images such as the face region image, the left eye region image, and the right hand region image; And
And classifying the driver's gaze area using the feature set.
In the deep learning based driver gaze tracking method,
Receiving a driver's face image;
Detecting a facial feature identification point by tracking a facial feature identification point on the driver's face image;
Extracting a face region image, a left eye region image, and a right hand region image based on the face feature identification point;
Generating a feature set including feature values by performing deep learning-based convolutional neural networks (CNNs) on three detected images such as the face region image, the left eye region image, and the right hand region image; And
And classifying the driver's gaze area using the feature set.
The method of claim 9,
Generating a feature set including feature values by performing deep learning based CNN on each of the three images,
Learning face gaze tracking using a deep learning based CNN;
Converting the face region image, the left eye region image, and the right eye region image into the CNN, and extracting feature values of each image; And
And normalizing the extracted feature values, and generating a feature set including the normalized feature values.
The method of claim 9,
Categorizing the driver's gaze area using the feature set
Calculating a Euclidean distance of each region image between the feature set and the 17 pre-stored gaze region feature sets; And
A driver's gaze tracking method based on deep learning, comprising classifying a driver's gaze area using Euclidean distance of each area image.
The method of claim 11,
Classifying the driver's gaze area using the Euclidean distance of each area image
Driver based on deep learning that combines by the Euclidean distance level fusion method of each area image, and determines the gaze area with the minimum combined score for 17 gaze areas as the driver gaze area. Eye tracking method.
The method of claim 12,
The core level fusion method
Deep learning based driver gaze tracking method using weighted sum or weighted product method.
A computer program that executes the deep learning based driver gaze tracking method of any one of claims 9 to 13 and is recorded on a computer-readable recording medium.

Images