KR102286189B1 - Deblurring device and method based on deep learning for gaze estimation - Google Patents

Abstract

The present invention relates to a deblurring device and method based on deep learning for driver's gaze estimation. According to an embodiment of the present invention, a deblurring device and method based on deep learning for gaze estimation can accurately estimate a driver's gaze by deblurring an image with blur through deep learning. The deblurring device includes an input part, a feature extraction part; a deblurring part, and a deep learning part.

Description

DEBLURRING DEVICE AND METHOD BASED ON DEEP LEARNING FOR GAZE ESTIMATION

The present invention relates to an apparatus and method for restoring blur of an image, and more particularly, to restore the blur of an image acquired using deep learning to track the driver's gaze, and to accurately track the driver's gaze. It relates to a deep learning-based blur restoration apparatus and method.

As technologies related to the 4th industrial revolution are developed, various artificial intelligence-based technologies for various drivers, such as autonomous vehicles and intelligent driver reporting systems, are being introduced. In the case of an unmanned vehicle, the present invention may be used to determine a switching time between autonomous driving and direct driving by a driver. The driver's gaze information has the closest effect on driving among the driver's state information, and through driver gaze tracking, it is possible to safely determine the transition time between autonomous driving and the driver's direct driving.

In the real vehicle environment, when a driver is photographed through a camera installed in the vehicle to track the driver's gaze, the image is blurred due to various factors such as vehicle vibration and road environment. However, the existing deep learning-based driver gaze tracking method was mainly learned using images without blur. However, when there is blur in the driver's image, there is inevitably a difficulty in performing eye tracking, and performance tends to decrease. Therefore, there is a limitation that the driver's eye tracking is not continuous because the conventional driver's gaze tracking method excludes an image with blur and performs deep learning-based learning. That is, if blur occurs while tracking the driver's gaze through the camera installed in the actual driving vehicle, and a short time gap exists in the driver's gaze tracking, it is difficult to understand the driver's state due to this gap. restrictions will arise.

Background art of the present invention is disclosed in Korean Patent Registration No. 10-1628394.

The present invention provides an apparatus and method for restoring blur based on deep learning for eye tracking with high accuracy of driver eye tracking by performing blur restoration of an image using deep learning even if blur exists in an image due to various factors.

The present invention provides a deep-learning-based blur restoration apparatus and method for accurately grasping continuous driver's state information by accurately tracking the gaze information of a person through blur restoration even when an acquired image is out of focus.

According to one aspect of the present invention, a deep learning-based blur restoration apparatus for eye tracking is provided.

An input unit for acquiring an image using a plurality of near-infrared cameras according to an embodiment of the present invention, a feature extraction unit for extracting facial feature points from the image, a blur measurement unit for measuring the degree of blur in the combined image, and a threshold value In the above case, it may include a blur restoration unit for performing blur restoration and a deep learning unit for tracking the gaze of the combined image by performing deep learning.

According to another aspect of the present invention, there is provided a computer-readable recording medium in which a deep learning-based blur restoration method for eye tracking and a computer program executing the same are recorded.

A deep learning-based blur restoration method for eye tracking according to an embodiment of the present invention and a recording medium storing a computer program executing the same according to an embodiment of the present invention include: acquiring images from a plurality of cameras; detecting facial feature points in the images; Extracting the left eye, right eye and face images as facial feature points, combining the extracted images to generate a combined image, measuring a focus value of the combined image, and blurring the combined image according to the focus value It may include a step of restoring.

According to an embodiment of the present invention, an apparatus and method for restoring blur based on deep learning for eye tracking may accurately track a driver's gaze by restoring an image with blur through deep learning.

According to an embodiment of the present invention, a deep learning-based blur restoration apparatus and method for eye tracking accurately track information on a person's gaze through blur restoration even if an acquired image is out of focus to accurately grasp continuous driver's state information. can

1 to 11 are diagrams for explaining a deep learning-based blur restoration apparatus for eye tracking according to an embodiment of the present invention.
12 is a diagram for explaining a deep learning-based blur restoration method for eye tracking according to an embodiment of the present invention.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and will be described in detail through detailed description. However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. In describing the present invention, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. Also, as used herein and in the claims, the terms "a" and "a" and "a" are to be construed to mean "one or more" in general, unless stated otherwise.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. do it with

1 to 11 are diagrams for explaining a deep learning-based blur restoration apparatus for eye tracking according to an embodiment of the present invention.

Referring to FIG. 1 , the deep learning-based blur restoration apparatus 10 for eye tracking is mounted inside a vehicle and acquires side and front face images of the driver acquired to track the driver's gaze. The deep learning-based blur restoration apparatus 10 for eye tracking restores the blur when there is blur in the acquired image, so that strong eye tracking is possible. The deep learning-based blur restoration apparatus 10 for eye tracking is mounted on an ECU and performs blur restoration by acquiring an image of an in-vehicle camera using in-vehicle communication.

2, the deep learning-based blur restoration apparatus 10 for eye tracking includes an input unit 100, a feature extraction unit 200, a blur measurement unit 300, a blur restoration unit 400, and a deep learning unit ( 500).

The input unit 100 acquires an image of the driver for eye tracking. For example, the input unit 100 may acquire the driver's face image using a plurality of near-infrared cameras.

Referring to FIG. 3 , the input unit 100 acquires a driver's face image from a near-infrared camera equipped with near-infrared illumination of the front and the side attached to the vehicle.

Referring back to FIG. 2 , the feature extraction unit 200 extracts a facial feature point from the driver's face image acquired by the input unit 100 . The feature extraction unit 200 extracts facial feature points from the driver's front image acquired from the front camera and the driver's side image acquired from the side camera, respectively.

Referring to FIG. 4 , the feature extraction unit 200 detects the driver's left eye, right eye, and face based on the extracted facial feature points. The feature extraction unit 200 detects a left eye, a right eye, and a face from the front and side images, respectively.

Referring to Figure 5. The feature extraction unit 200 generates one image having three channels by combining three images detected from the driver's front image and three images detected from the driver's side image.

The blur measuring unit 300 measures the degree of blur from the combined driver's image. For example, as shown in FIG. 6 , the blur measuring unit 300 uses a 5 X 5 convolution kernel of the image. The high-frequency part is extracted and the focus value is measured.

When the measured focus value is equal to or greater than a predetermined reference value, the blur measurement unit 300 performs learning and driver gaze tracking through the deep learning unit 500 because there is no need to perform the blur restoration step.

The blur measurement unit 300 calls the blur restoration unit 400 when the focus value is less than a predetermined reference value.

The blur restoration unit 400 performs blur restoration when the focus value of the combined driver image is less than a predetermined reference value. For example, the blur restoration unit 400 restores the blur using Cycle Generative Adversarial Networks (CycleGAN). In this case, CycleGAN is a model learned through driver images that do not cause blur. GAN is configured so that Generator Network and Discriminator Network compete with each other while learning. The generator network plays a role in creating data distribution from the input image, and the discriminator network plays a role in distinguishing whether the image given as an input is a generator network image or an actual image. In the case of a general GAN, paired data is required for learning. Unlike general GAN, CycleGAN used in the present invention performs bidirectional learning using two pairs of Generator Network and Discriminator Network.

Referring to FIG. 7 , it can be confirmed that the method of restoring an image most similar to the original when blur is restored is CycleGAN (FIG. 7(c)). 7 is an exemplary image obtained by performing blur restoration using various restoration methods. FIG. 7(a) is an original image, and FIG. 7(b) is a blur image. FIG. 7(c) is an image obtained by performing blur restoration using CycleGAN on a blurred image, and FIG. 7(d) is an image obtained by performing blur restoration using DeblurGAN. FIG. 7(e) is an image obtained by performing blur restoration using Pix2Pix, FIG. 7(f) is using SRN-DeblurNet, and FIG. 7(g) is an image obtained by performing blur restoration using DeepDeblur.

The deep learning unit 500 learns a deep learning model for tracking a driver's gaze with an image having a focus value greater than or equal to a certain reference value that does not require a blur restoration step and an image that has undergone a blur restoration step because the focus value is less than a predetermined reference value.

Referring to FIG. 8 , for example, the deep learning unit 500 may track the driver's gaze region using a ResNet-152 (deep residual networks-152) model.

9 is an example in which the deep learning unit 500 calculates a total of 16 driver gaze regions including 15 regions shown in FIG. 10 and a situation in which the driver closes his/her eyes. Accuracy is measured by Strictly Correct Estimation Rate (SCER) and Loosely Correct Estimation Rate (LCER)fh. SCER is the accuracy for the corresponding point, and LCER is the accuracy including the corresponding point and the surrounding area. As a result of cross-validation by dividing into two groups, the final accuracy was SCER 90.11% and LCER 98.94%. If there is blur, high accuracy can be confirmed when deep learning is performed by performing the blur restoration step.

Referring to FIG. 11, it is the experimental result that the blur restoration step using CycleGAN used in the present invention and the result of tracking the driver's gaze using the ResNet-152 model show higher accuracy than the results and performance measured by applying other techniques. can also be checked.

12 is a diagram illustrating a deep learning-based blur restoration method for eye tracking according to an embodiment of the present invention. Each process described below is a process performed by each functional unit constituting a deep learning-based blur restoration device for eye tracking, but for a concise and clear explanation of the present invention, a deep learning-based blur restoration for eye tracking of the subject of each step for a concise and clear explanation of the present invention to be referred to as a device.

Referring to FIG. 12 , the deep learning-based blur restoration apparatus 10 for eye tracking acquires face images from a plurality of cameras in step S1201. For example, the deep learning-based blur restoration apparatus 10 for eye tracking may acquire front and side images of the driver using a front camera and a side camera.

In step S1202, the deep learning-based blur restoration apparatus 10 for eye tracking detects a facial feature point from the acquired image.

In step S1203, the deep learning-based blur restoration apparatus 10 for eye tracking extracts left eye, right eye, and face images by using facial feature points. For example, the deep learning-based blur restoration apparatus 10 for eye tracking generates a total of six images by extracting the left eye, right eye, and face images from the front image and the side image of the driver, respectively.

In step S1204, the deep learning-based blur restoration apparatus 10 for eye tracking generates one image composed of three channels by combining images extracted based on facial feature points. For example, the deep learning-based blur restoration apparatus 10 for eye tracking uses a left eye front image, a left eye side image, a right eye front image and a right eye side image, and a front face image and a side face image as channels, respectively. Creates a 3-channel single image.

In step S1205, the deep learning-based blur restoration apparatus 10 for eye tracking measures the degree of blur from a three-channel single image. The deep learning-based blur restoration apparatus 10 for eye tracking calculates a focus value of an eye region of interest in a three-channel single image. For example, the deep learning-based blur restoration apparatus 10 for eye tracking measures a focus value by extracting a high-frequency portion of a 3-channel single image using a 5X5 convolution kernel.

In step S1206, if the deep learning-based blur restoration apparatus 10 for eye tracking is less than a pre-selected threshold, it moves to step S1207 and restores the blur of a three-channel single image. For example, the deep learning-based blur restoration apparatus 10 for eye tracking performs blur restoration of a 3-channel single image using CycleGAN. The deep learning-based image blur restoration apparatus 10 for eye tracking performs driver gaze tracking using the deep learning of step S1208 with a three-channel single image restored by blurring.

Alternatively, if the deep learning-based blur restoration apparatus 10 for eye tracking in step S1206 is greater than or equal to a pre-selected threshold, it moves to step S1208 and performs a deep learning model for eye tracking with a three-channel single image.

In step S1208, the deep learning-based blur restoration apparatus 10 for eye tracking learns the image in which the blur does not exist and the image that has undergone the blur restoration step (S1207) for which the blur restoration step (S1207) is not performed as a deep learning model. . For example, the deep learning-based blur restoration apparatus 10 for eye tracking uses the ResNet-152 model, and the detailed structure is shown in FIG. 7 .

The above-described deep learning-based blur restoration method for eye tracking may be implemented as a computer-readable code on a computer-readable medium. The computer-readable recording medium may be, for example, a removable recording medium (CD, DVD, Blu-ray disk, USB storage device, removable hard disk) or a fixed recording medium (ROM, RAM, computer-equipped hard disk). can The computer program recorded in the computer-readable recording medium may be transmitted to another computing device through a network, such as the Internet, and installed in the other computing device, thereby being used in the other computing device.

In the above, even though it has been described that all components constituting the embodiment of the present invention are combined or operated as one, the present invention is not necessarily limited to this embodiment. That is, within the scope of the object of the present invention, all the components may operate by selectively combining one or more.

Although acts are shown in a specific order in the drawings, it should not be understood that the acts must be performed in the specific order or sequential order shown, or that all shown acts must be performed to obtain a desired result. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of the various components in the embodiments described above should not be construed as necessarily requiring such separation, and the described program components and systems may generally be integrated together into a single software product or packaged into multiple software products. It should be understood that there is

So far, the present invention has been looked at focusing on the embodiments thereof. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention can be implemented in modified forms without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

10: Deep learning-based blur restoration device for eye tracking
100: input unit
200: feature extraction unit
300: blur measurement unit
400: blur restoration unit
500: deep learning unit

Claims (8)

In the deep learning-based blur restoration device for eye tracking,
an input unit for acquiring front and side face images using a near-infrared camera;
A facial feature point is detected in the image, and the front left eye, front right eye, front face, side left eye, side right eye and side face image are extracted from the image using the facial feature point, The front left eye image and the side left eye image, the front right eye image and the side right eye image, and the front face image and the side face image are combined, and the combined image is used as a channel, respectively. a feature extraction unit generating a single image;
a blur measurer for measuring a focus value by extracting a high-frequency portion of an image using a convolution kernel from the three-channel single image;
a blur restoration unit for performing blur restoration using a CycleGAN model when the degree of blur is greater than or equal to a threshold value; and
A deep learning-based blur restoration apparatus for eye tracking comprising a deep learning unit that tracks the gaze of the combined image by performing deep learning.
delete delete delete In a deep learning-based blur restoration method for eye tracking,
Acquiring front and side face images using a near-infrared camera;
detecting facial feature points from the image;
extracting a front left eye, a front right eye, a front face, a side left eye, a side right eye, and a side face image from the image using the facial feature points;
3 combining the extracted front left eye image and lateral left eye image, the front right eye image and lateral right eye image, and the front face image and lateral face image, and using the combined image as a channel generating a single channel image;
measuring a focus value by extracting a high-frequency portion of the image using a convolution kernel from the three-channel single image; and
and performing blur restoration using a CycleGAN model when the degree of blur is greater than or equal to a threshold value according to the focus value.
delete delete A computer program recorded in a computer-readable recording medium executing the deep learning-based blur restoration method for eye tracking according to claim 5.

Images