KR20230159739A - Driving assisting system through tracking of eye of fork lift driver based on deep learning, method for the same, and computer-readable recording including the same - Google Patents

Abstract

The present invention includes a vision camera 110 for photographing the face area of a forklift driver, a database 120 for storing the face area image captured by the vision camera 110, and an eye image of the face area image of the database 120. A deep learning model unit 130 that learns and predicts to segment the eye object and the white of the eye object, calculates the positional relationship between the pupil object and the white of the eye object divided by the deep learning model unit 130, and determines the gaze direction of the forklift driver. The eye area segmentation prediction result by the gaze direction measurement unit 140 and the deep learning model unit 130 that measures the gaze direction and the gaze direction by the gaze direction measurement unit 140 are analyzed to determine whether the forklift is drowsy or safe. It includes a control unit 150 that determines and provides warning information according to the judgment result, and analyzes the driver's eye image using a deep learning model to predict the direction of gaze in real time to prevent safety accidents due to drowsy driving, carelessness, or inexperience. We are launching a driving assistance system through deep learning-based forklift driver eye tracking that can prevent accidents.

Description

A driving assistance system through deep learning-based forklift driver eye tracking, a computer-readable recording medium on which the method and a computer program for executing the same on a computer are recorded {DRIVING ASSISTING SYSTEM THROUGH TRACKING OF EYE OF FORK LIFT DRIVER BASED ON DEEP LEARNING, METHOD FOR THE SAME, AND COMPUTER-READABLE RECORDING INCLUDING THE SAME}

The present invention relates to a driving assistance system through deep learning-based forklift driver eye tracking, the method, and a computer-readable recording medium on which a computer program for executing the same is recorded on a computer. More specifically, the present invention relates to a computer-readable recording medium on which a computer program for executing the same on a computer is recorded. Driving assistance system through deep learning-based forklift driver gaze tracking that analyzes the driver's eye image and predicts the gaze direction in real time to prevent safety accidents due to drowsy driving, carelessness, or inexperience, the method, and execution of the method on a computer It relates to a computer-readable recording medium on which a computer program for performing the following is recorded.

As is well known, when driving and working with a conventional forklift, there is a system that emits a warning sound so that pedestrians, other workers, or other vehicles other than the forklift driver can avoid. However, even though the majority of forklifts have warning sounds, safety accidents occur due to the forklift driver's carelessness or inexperience.

In other words, the problem with the existing system is that it is a system that warns the external environment in addition to the forklift driver, rather than a warning system using the forklift driver's status information, and has the disadvantage of not being able to respond if the forklift driver is careless.

Accordingly, when driving and working on a forklift, state information such as the direction in which the forklift driver's gaze is directed is learned based on deep learning to detect whether the forklift driver is not looking forward, is driving while using a mobile device, or is drowsy. We would like to propose a system that recognizes states such as driving and takes action using the recognized states.

Korean Patent Publication No. 10-2254384 (Deep learning-based driver eye tracking device and method, 2021.05.24) Korean Patent Publication No. 10-2017766 (Deep learning-based vehicle driver eye tracking device and method, October 15, 2019)

The technical problem to be achieved by the idea of the present invention is to develop a deep learning-based forklift that can prevent safety accidents due to drowsy driving, carelessness, or inexperience by predicting the gaze direction in real time by analyzing the driver's eye image using a deep learning model. The aim is to provide a computer-readable recording medium on which a driving assistance system through driver eye tracking, the method, and a computer program for executing the same on a computer are recorded.

In order to achieve the above-described object, an embodiment of the present invention includes a vision camera for photographing the face area of a forklift driver; a database storing face area images captured by the vision camera; A deep learning model unit that learns and predicts to segment eye objects and eye white objects from the eye image of the face area image in the database; a gaze direction measurement unit that measures the gaze direction of the forklift driver by calculating a positional relationship between the pupil object and the white eye object divided by the deep learning model unit; and a control unit that analyzes the eye area segmentation prediction result by the deep learning model unit and the gaze direction by the gaze direction measurement unit to determine whether the forklift is drowsy or safe to drive, and provides warning information according to the determination result. Discloses a driving assistance system through deep learning-based forklift driver eye tracking, including.

Here, the deep learning model unit may be a semantic segmentation model.

At this time, the DeepLab V3+ model can be applied to the semantic segmentation model.

Specifically, the DeepLab V3+ model performs an atrous convolution module that extracts a low-resolution feature map from the eye image and a plurality of atrous convolutions with different rates in parallel to extract a random resolution feature map for the feature map, and then An encoder including concatenate Atrous Spatial Pyramid Pooling (ASPP) and a first convolution module for 1×1 convolution of the result of the ASPP; and a second convolution module for 1×1 convolution of the low-resolution feature map from the atrous convolution module, a first upsampling module for upsampling the feature map from the first convolution module, and A concatenate module that connects a feature map and a feature map upsampled by the first upsampling module, a third convolution module that performs 3×3 convolution of the feature map by the concatenate module, and A decoder including a second upsampling module that upsamples and restores a feature map; including a segmentation label that divides the classes of the eye object and the white of the eye object from pixels of the eye image and has different pixel values for each class. can be predicted.

Here, the ASPP may include a 1×1 convolutional layer, a 3×3 convolutional layer at a rate of 6, a 3×3 convolutional layer at a rate of 12, a 3×3 convolutional layer at a rate of 18, and an image pooling layer. You can.

In addition, the gaze direction measuring unit calculates the center coordinates of each divided pixel of the pupil object and the white eye object predicted by the deep learning model unit to obtain a gaze vector value to track the gaze direction of the forklift driver. You can.

In addition, the control unit calculates each area of the divided pixels of the pupil object and the white eye object predicted by the deep learning model unit, and the act of continuously closing the eyes or the act of closing the eyes through each calculated area. Drowsy driving can be determined by analyzing the frequency, and safe driving can be determined by comparing the gaze direction measured by the gaze direction measurement unit with the driving direction of the forklift.

Additionally, the vision camera is installed on the inner center fascia of the forklift and can capture the face area of the forklift driver with an 80-degree FoV.

Additionally, the control unit may analyze the distribution area of pixels of the head image of the forklift driver from the facial area image captured by the vision camera to determine whether the driver is drowsy due to repetitive up and down head movements.

Meanwhile, another embodiment of the present invention includes a first step of photographing the face area of a forklift driver using a vision camera; A second step of storing the facial area image captured by the vision camera in a database; A third step of predicting to segment the eye object and the white of the eye object from the eye image of the face area image in the database by a previously learned and constructed deep learning model unit; A fourth step of measuring the gaze direction of the forklift driver by calculating a positional relationship between the pupil object and the white eye object divided by the deep learning model unit by a gaze direction measurement unit; And through the control unit, the prediction result of eye area division by the deep learning model unit and the gaze direction by the gaze direction measurement unit are analyzed to determine whether the forklift is drowsy or safe to drive, and provides warning information according to the judgment result. Provides a driving assistance method through deep learning-based forklift driver eye tracking, including a fifth step of providing.

Here, the deep learning model unit may be a semantic segmentation model.

At this time, the DeepLab V3+ model can be applied to the semantic segmentation model.

Specifically, the DeepLab V3+ model includes an atrous convolution module that extracts a low-resolution feature map from the eye image, and an ear that performs multiple atrous convolutions with different rates in parallel to extract a random resolution feature map for the feature map. An encoder including concatenate Atrous Spatial Pyramid Pooling (ASPP) and a first convolution module for 1×1 convolution of the result of the ASPP; and a second convolution module for 1×1 convolution of the low-resolution feature map from the atrous convolution module, a first upsampling module for upsampling the feature map from the first convolution module, and A concatenate module that connects a feature map and a feature map upsampled by the first upsampling module, a third convolution module that performs 3×3 convolution of the feature map by the concatenate module, and A decoder including a second upsampling module that upsamples and restores a feature map; including a segmentation label that divides the classes of the eye object and the white of the eye object from pixels of the eye image and has different pixel values for each class. can be predicted.

Here, the ASPP may include a 1×1 convolutional layer, a 3×3 convolutional layer at a rate of 6, a 3×3 convolutional layer at a rate of 12, a 3×3 convolutional layer at a rate of 18, and an image pooling layer. You can.

In addition, the gaze direction measuring unit calculates the center coordinates of each divided pixel of the pupil object and the white eye object predicted by the deep learning model unit to obtain a gaze vector value to track the gaze direction of the forklift driver. You can.

In addition, in the fifth step, the control unit calculates each area of the divided pixels of the pupil object and the white eye object predicted by the deep learning model unit, and continuously controls the eye through each calculated area. Drowsy driving can be determined by analyzing the winding act or the frequency of the winding act, and safe driving can be determined by comparing the gaze direction measured by the gaze direction measurement unit with the traveling direction of the forklift.

Additionally, the vision camera is installed on the inner center fascia of the forklift and can capture the face area of the forklift driver with an 80-degree FoV.

In addition, a sixth step is performed in which the control unit analyzes the distribution area of pixels of the head image of the forklift driver from the facial area image captured by the vision camera and determines whether the drowsiness is precipitated due to repetitive up and down head movements. It can be included.

Meanwhile, another embodiment of the present invention provides a computer-readable recording medium on which a computer program for executing the driving assistance method through deep learning-based forklift driver eye tracking listed above on a computer is recorded.

According to the present invention, by applying a deep learning model that performs real-time segmentation of the pupil and the white of the eye object from the image of the face image, the gaze direction is predicted in real time by analyzing the eye image, and the anterior vagus due to drowsy driving, carelessness, or inexperience is prevented. It has the effect of determining whether driving is safe based on eye movements caused by the use of a mobile device and immediately warning the driver based on the judgment result to prevent safety accidents in advance.

Figure 1 shows a schematic configuration diagram of a driving assistance system through deep learning-based forklift driver eye tracking according to an embodiment of the present invention.
Figure 2 is a separate illustration of the deep learning model part of the driving assistance system through deep learning-based forklift driver eye tracking of Figure 1.
Figure 3 is a flowchart of a driving assistance method through deep learning-based forklift driver eye tracking according to another embodiment of the present invention.

Hereinafter, embodiments of the present invention having the above-described features will be described in more detail with reference to the attached drawings.

The driving assistance system through deep learning-based forklift driver eye tracking according to an embodiment of the present invention includes a vision camera 110 that photographs the face area of the forklift driver, and stores the face area image captured by the vision camera 110. The database 120, the deep learning model unit 130 that learns and predicts to segment the eye object and the white of the eye object from the eye image of the face area image of the database 120, and the deep learning model unit 130 divide (classify). ), a gaze direction measurement unit 140 that measures the gaze direction of the forklift driver by calculating the positional relationship between the pupil object and the white of the eye object, and an eye region segmentation prediction result and a gaze direction measurement unit by the deep learning model unit 130 ( 140), including a control unit 150 that analyzes the gaze direction to determine whether the forklift is drowsy and/or safe to drive, and provides warning information according to the judgment result, and the driver's eyes are controlled by a deep learning model. The aim is to prevent safety accidents caused by drowsy driving, carelessness, or inexperience by analyzing images and predicting gaze direction in real time.

Hereinafter, with reference to FIGS. 1 and 2, the driving assistance system through deep learning-based forklift driver eye tracking of the above-described configuration will be described in detail as follows.

First, the vision camera 110 is a camera that captures the facial area of the driver driving the forklift 10. It is installed on the inner center fascia of the forklift and can capture the front facial area of the forklift driver with an 80-degree FoV (Field of View). You can.

For example, the vision camera 110 may include a near-infrared camera and a near-infrared LED light. The near-infrared LED light has little glare, so it does not interfere with the driver's vision or affect driving, and can be used by the driver regardless of the external lighting environment day or night. For eye detection, images capable of identifying eye images can be acquired, and the driver can be continuously monitored with a wide angle FoV.

Next, the database 120 receives and stores the face area image captured by the vision camera 110 while the forklift 10 is running or working, and the deep learning model unit 130 or the gaze direction measurement unit 140 As learning data, the database 120 can be provided with various facial area images acquired by weather, season, various times of the day, morning, lunch, and evening for segmentation of the driver's eye image and tracking the driver's gaze.

Next, the deep learning model unit 130 is trained to segment the pupil object and the white of the eye object from the eye image of the face area image provided in real time from the database 120, so that the pixels of each eye image are classified into the class of the pupil area or the white of the eye area. By predicting whether the object belongs to a class, the pupils and whites of the eyes can be distinguished and recognized.

Here, the deep learning model unit 130 performs regional proposals and classifications at the same time to classify the entire image without extracting the region of interest (RoI), and also performs fast It can be a semantic segmentation model based on a 1-stage detector capable of real-time prediction by enabling real-time detection at a computation speed of 30 fps or more.

Additionally, as a semantic segmentation model, the DeepLab V3+ model using atrous separable convolution, which combines depthwise separable convolution and atrous convolution, can be applied.

Here, depthwise separable convolution, unlike existing convolution filters, can reduce the number of parameters and computation amount by processing spatial dimension and channel dimension separately, and atrous convolution maintains the same amount of parameters and computation amount as existing convolution while maintaining receptive By enlarging the field and minimizing information loss without pooling, results with better resolution can be obtained.

Specifically, referring to Figure 2, the DeepLab V3+ model uses ResNet-50 as a backbone network, ASPP (132) (Atrous Spatial Pyramid Pooling) as an encoder (130A) to extract a feature map, and a decoder ( This is a DeepLab V3 model that connects the encoder 130A and the decoder 130B by introducing a skip architecture to upsample the feature map of the encoder 130A through 130B).

That is, the encoder 130A has an atrous convolution module 131 that extracts a low-resolution feature map from the eye image, and a rate that leaves an empty space inside the filter to extract an arbitrary resolution feature map for the feature map. It will include a concatenate ASPP (Atrous Spatial Pyramid Pooling) 132 that performs a number of different atrous convolutions in parallel, and a first convolution module 133 that performs 1×1 convolution of the result of ASPP 132. You can.

Here, ASPP 132 includes a 1×1 convolutional layer, a 3×3 convolutional layer with a rate of 6, a 3×3 convolutional layer with a rate of 12, a 3×3 convolutional layer with a rate of 18, and an image pooling layer. can do.

In addition, the decoder 130B includes a second convolution module 134 that performs 1×1 convolution of the low-resolution feature map from the atrous convolution module 131, and a 4-fold upsampling of the feature map from the first convolution module 133. A first upsampling module 135, a concatenate module 136 that connects the feature map by the second convolution module 134 and the feature map upsampled by the first upsampling module 135, and a concatenate module A third convolution module 137 that convolves the feature map by 3×3 by (136), and a second upsampling module 138 that restores the feature map by upsampling 4 times the feature map by the third convolution module 137. Including, it is possible to segment the classes of the eye object and the white of the eye object from the pixels of the eye image and predict a segmentation label with different pixel values for each class.

The DeepLab V3+ model with this structure upsamples the final feature map from the encoder (130A) by 4 times, and applies 1×1 convolution to the low-resolution feature map from the middle of the encoder (130A) to reduce the channel and concatenate it. The module 136 enables concatenation of the feature map from the first convolution module 133 and the feature map from the second convolution module 134, and the third convolution module 137 and the third convolution module 137 of the decoder 130B The size of the input eye image is restored through the 2-up-sampling module 138, and the data finally divided into the classes of the eye object and the white of the eye object can be predicted.

Next, the gaze direction measurement unit 140 calculates the positional relationship between the pupil object and the white of the eye object divided by the preceding deep learning model unit 130 to measure the driver's gaze direction while driving or working.

For example, the gaze direction measurement unit 140 calculates the center coordinates of each divided pixel of the eye object and the white of the eye object predicted by the deep learning model unit 130, and changes the driver's gaze through changes in each center coordinate. It may be a deep learning model learned to track the gaze direction of the forklift driver by obtaining the facing gaze vector value, thereby learning a certain number of gaze directions or more to predict the driver's gaze direction.

Next, the control unit 150 analyzes the eye region segmentation prediction results by the deep learning model unit 130 and analyzes the gaze direction measurement (tracking) by the gaze direction measurement unit 140 to determine whether the forklift driver's drowsy driving and , determines whether the forklift is driving safely due to incorrect gaze direction unrelated to the driving direction, gaze movement due to use of mobile device, etc., and depending on the judgment result, warning information is provided as a flashing signal through LED lights or a warning exceeding a certain dB through a speaker. A signal or voice signal can be immediately provided to the forklift driver to warn him or her to drive safely.

In addition, the control unit 150 calculates each area of the divided pixels of the eye object and the white of the eye object predicted by the deep learning model unit 130, analyzes the relative change in each calculated area, and continuously monitors the eye. Drowsy driving is determined by analyzing the winding action or the frequency of the winding action, and the gaze direction measured (tracked) by the gaze direction measurement unit 140 is compared with the driving direction of the forklift (for example, the gaze direction and the forklift By comparing the driving direction (up and down, left and right, forward and backward), it is possible to determine whether driving is safe due to missing attention ahead due to carelessness or inexperience, or downward movement of gaze due to use of a mobile device.

Additionally, the control unit 150 may analyze the pixel distribution area of the forklift driver's head image from the facial area image captured by the vision camera 110 to determine whether or not the driver is drowsy due to repetitive up and down head movements. For example, the deep learning model unit 130 mentioned above is trained to segment the classes of facial objects and head objects from facial image images to predict which class the pixels of each object belong to, and determine the change in area distribution of the corresponding pixels. It can also be analyzed to determine drowsy driving, etc.

In addition, the control unit 150, in conjunction with the driving system 10 of the forklift, can make emergency braking temporarily stop while emitting a warning sound when drowsy driving is determined, and allows driving or work to be possible after the driver awakens from drowsiness. Emergency braking may be canceled directly by the driver, or emergency braking may be canceled by the control unit 150 after tracking the normal gaze direction by the deep learning model unit 130 and the gaze direction measurement unit 140.

Therefore, by applying a deep learning model that performs segmentation of the pupil and the white of the eye object in real time from the forklift driver's face image by the driving assistance system through deep learning-based forklift driver eye tracking as described above, the eye image By analyzing and predicting the direction of gaze in real time, it determines safe driving due to drowsy driving, not looking ahead due to carelessness or inexperience, and gaze movement due to use of mobile devices, and immediately warns the driver based on the judgment result to prevent safety accidents. can be prevented in advance.

Figure 3 is a flowchart of a driving assistance method through deep learning-based forklift driver eye tracking according to another embodiment of the present invention. With reference to this, the driving assistance method through deep learning-based forklift driver eye tracking is described in detail. If you do so, it is as follows.

First, in the first step (S110), the face area of the forklift driver is photographed by the vision camera 110. Here, the vision camera 110 is installed on the inner center fascia of the forklift and can capture the face area of the forklift driver with an 80-degree FoV.

Subsequently, in the second step (S120), the facial area image captured by the vision camera 110 is stored in the database 120.

Subsequently, in the third step (S130), the previously learned and constructed deep learning model unit 130 predicts to segment the eye object and the white of the eye object from the eye image of the face area image in the database 120.

Here, the deep learning model unit 130 may be a semantic segmentation model, and the DeepLab V3+ model may be applied as the semantic segmentation model.

Specifically, the DeepLab V3+ model includes an atrous convolution module 131 that extracts a low-resolution feature map from an eye image, and an ear that performs multiple atrous convolutions with different rates in parallel to extract a feature map of arbitrary resolution for the feature map. An encoder (130A), including a concatenate ASPP (Atrous Spatial Pyramid Pooling) (132), and a first convolution module (133) for 1×1 convolution of the result of ASPP (132), and an atrous convolution module ( A second convolution module 134 for 1×1 convolution of the low-resolution feature map from 131), a first upsampling module 135 for upsampling the feature map from the first convolution module 133, and a second convolution A concatenate module 136 that connects the feature map by the module 134 and the feature map upsampled by the first upsampling module 135, and a 3×3 convolution of the feature map by the concatenate module 136. Including a decoder (130B) including a third convolution module (137) and a second upsampling module (138) that upsamples and restores the feature map by the third convolution module (137), pixels of the eye image From , the classes of the eye object and the white of the eye object can be divided to predict segmentation labels with different pixel values for each class.

In addition, the ASPP 132 includes a 1×1 convolutional layer, a 3×3 convolutional layer with a rate of 6, a 3×3 convolutional layer with a rate of 12, a 3×3 convolutional layer with a rate of 18, and an image pooling layer. It can be included.

Subsequently, in the fourth step (S140), the gaze direction measurement unit 140 calculates the positional relationship between the pupil object and the white of the eye object divided by the deep learning model unit 130 to determine the gaze direction of the forklift driver. Make sure to measure it.

Here, the gaze direction measurement unit 140 calculates the center coordinates of each divided pixel of the eye object and the white of the eye object predicted by the deep learning model unit 130 to obtain a gaze vector value and determines the gaze direction of the forklift driver. You can have it tracked.

Subsequently, in the fifth step (S150), the control unit 150 analyzes the eye region segmentation prediction result by the deep learning model unit 130 and the gaze direction by the gaze direction measurement unit 140 to prevent drowsy driving of the forklift. Determine whether driving is safe and/or safe, and provide warning information based on the judgment results.

Additionally, in the fifth step (S150), the control unit 150 calculates the area of each divided pixel of the eye object and the white of the eye object predicted by the deep learning model unit 130, and calculates the area of each divided pixel of the eye object and the white of the eye object predicted by the deep learning model unit 130. Drowsy driving can be determined by continuously winding or analyzing the frequency of winding, and safe driving can be determined by comparing the gaze direction tracked by the gaze direction measurement unit 140 with the driving direction of the forklift.

Meanwhile, the control unit 150 further includes a sixth step (S160) in which the control unit 150 analyzes the distribution area of pixels of the head image from the face area image captured by the vision camera 110 and determines whether drowsiness is present due to the up and down head movement. can do.

Therefore, by applying the driving assistance method through deep learning-based forklift driver eye tracking as described above, a deep learning model that performs segmentation of the pupil and the white of the eye object in real time from the forklift driver's face image is applied to obtain the eye image. By analyzing and predicting the direction of gaze in real time, it determines safe driving due to drowsy driving, not looking ahead due to carelessness or inexperience, and gaze movement due to use of mobile devices, and immediately warns the driver based on the judgment result to prevent safety accidents. can be prevented in advance.

Another embodiment of the present invention provides a computer-readable recording medium on which a computer program for executing the driving assistance method through eye tracking of a forklift driver based on deep learning listed above is recorded on a computer.

The embodiments described in this specification and the configurations shown in the drawings are only one of the most preferred embodiments of the present invention and do not represent the entire technical idea of the present invention, so various equivalents may be substituted for them at the time of filing the present application. It should be understood that variations and variations may exist.

110: vision camera 120: database
130: Deep learning model unit 130A: Encoder
130B: Decoder 131: atrous convolution module
132: ASPP 133: 1st convolution module
134: second convolution module 135: first upsampling module
136: concatenate module 137: third convolution module
138: Second upsampling module 140: Gaze direction measurement unit
150: Control unit 10: Forklift

Claims (19)

A vision camera that captures the facial area of the forklift driver;
a database storing face area images captured by the vision camera;
A deep learning model unit that learns and predicts to segment eye objects and eye white objects from eye images of the face area image in the database;
a gaze direction measuring unit that measures the gaze direction of the forklift driver by calculating a positional relationship between the pupil object and the white eye object divided by the deep learning model unit; and
A control unit that analyzes the eye area segmentation prediction result by the deep learning model unit and the gaze direction by the gaze direction measurement unit to determine whether the forklift is drowsy or safe to drive, and provides warning information according to the judgment result. containing,
Driving assistance system through deep learning-based forklift driver eye tracking. According to claim 1,
The deep learning model unit is a semantic segmentation model,
Driving assistance system through deep learning-based forklift driver eye tracking. According to claim 2,
The semantic segmentation model is characterized by applying the DeepLab V3+ model,
Driving assistance system through deep learning-based forklift driver eye tracking. According to claim 3,
The DeepLab V3+ model is,
An atrous convolution module that extracts a low-resolution feature map from the eye image, and an Atrous Spatial Pyramid (ASPP) that performs and concatenates a number of atrous convolutions with different rates in parallel to extract a random resolution feature map for the feature map. Pooling) and an encoder including a first convolution module for 1×1 convolution of the result of the ASPP; and
A second convolution module for 1×1 convolution of the low-resolution feature map from the atrous convolution module, a first upsampling module for upsampling the feature map from the first convolution module, and features by the second convolution module A concatenate module that connects a map and a feature map upsampled by the first upsampling module, a third convolution module that performs 3×3 convolution of the feature map by the concatenate module, and features by the third convolution module. Including a decoder including a second upsampling module that upsamples and restores the map,
Characterized in predicting segmentation labels with different pixel values for each class by dividing the classes of the eye object and the white eye object from the pixels of the eye image.
Driving assistance system through deep learning-based forklift driver eye tracking. According to claim 4,
The ASPP includes a 1×1 convolutional layer, a 3×3 convolutional layer at a rate of 6, a 3×3 convolutional layer at a rate of 12, a 3×3 convolutional layer at a rate of 18, and an image pooling layer. to,
Driving assistance system through deep learning-based forklift driver eye tracking. According to claim 1,
The gaze direction measuring unit calculates the center coordinates of each divided pixel of the pupil object and the white eye object predicted by the deep learning model unit to obtain a gaze vector value to track the gaze direction of the forklift driver. Characterized by,
Driving assistance system through deep learning-based forklift driver eye tracking. According to claim 1,
The control unit,
Calculate each area of the divided pixels of the pupil object and the white eye object predicted by the deep learning model unit, and analyze the act of continuously closing the eyes or the frequency of the act of closing the eyes through each calculated area. Determine whether you are drowsy driving,
Characterized in that the gaze direction measured by the gaze direction measurement unit is compared with the driving direction of the forklift to determine whether the vehicle is driving safely.
Driving assistance system through deep learning-based forklift driver eye tracking. According to claim 1,
The vision camera is installed on the inner center fascia of the forklift and captures the face area of the forklift driver with an 80-degree FoV.
Driving assistance system through deep learning-based forklift driver eye tracking. According to claim 1,
The control unit analyzes the distribution area of pixels of the head image of the forklift driver from the facial area image captured by the vision camera and determines whether or not the drowsiness is present due to repetitive up and down head movements.
Driving assistance system through deep learning-based forklift driver eye tracking. A first step of photographing the face area of the forklift driver using a vision camera;
A second step of storing the facial area image captured by the vision camera in a database;
A third step of predicting to segment the eye object and the white of the eye object from the eye image of the face area image in the database by a previously learned and constructed deep learning model unit;
A fourth step of measuring the gaze direction of the forklift driver by calculating a positional relationship between the pupil object and the white eye object divided by the deep learning model unit by a gaze direction measurement unit; and
Through the control unit, the prediction result of eye area division by the deep learning model unit and the gaze direction by the gaze direction measurement unit are analyzed to determine whether the forklift is drowsy or safe to drive, and provides warning information according to the judgment result. 5th step; including,
Driving assistance method using deep learning-based forklift driver eye tracking. According to claim 10,
The deep learning model unit is a semantic segmentation model,
Driving assistance method using deep learning-based forklift driver eye tracking. According to claim 11,
The semantic segmentation model is characterized by applying the DeepLab V3+ model,
Driving assistance method using deep learning-based forklift driver eye tracking. According to claim 12,
The DeepLab V3+ model is,
An atrous convolution module that extracts a low-resolution feature map from the eye image, and an Atrous Spatial Pyramid (ASPP) that concatenates a number of atrous convolutions with different rates in parallel to extract an arbitrary resolution feature map for the feature map. Pooling) and an encoder including a first convolution module for 1×1 convolution of the result of the ASPP; and
A second convolution module for 1×1 convolution of the low-resolution feature map from the atrous convolution module, a first upsampling module for upsampling the feature map from the first convolution module, and features by the second convolution module A concatenate module that connects a map and a feature map upsampled by the first upsampling module, a third convolution module that performs 3×3 convolution of the feature map by the concatenate module, and features by the third convolution module. Including a decoder including a second upsampling module that upsamples and restores the map,
Characterized in predicting segmentation labels with different pixel values for each class by dividing the classes of the eye object and the white eye object from the pixels of the eye image.
Driving assistance method using deep learning-based forklift driver eye tracking. According to claim 13,
The ASPP includes a 1×1 convolutional layer, a 3×3 convolutional layer at a rate of 6, a 3×3 convolutional layer at a rate of 12, a 3×3 convolutional layer at a rate of 18, and an image pooling layer. to,
Driving assistance method using deep learning-based forklift driver eye tracking. According to claim 10,
The gaze direction measuring unit calculates the center coordinates of each divided pixel of the pupil object and the white eye object predicted by the deep learning model unit to obtain a gaze vector value to track the gaze direction of the forklift driver. Characterized by,
Driving assistance method using deep learning-based forklift driver eye tracking. According to claim 10,
In the fifth step, the control unit,
Calculate each area of the divided pixels of the pupil object and the white eye object predicted by the deep learning model unit, and analyze the act of continuously closing the eyes or the frequency of the act of closing the eyes through each calculated area. Determine whether you are drowsy driving,
Characterized in that the gaze direction measured by the gaze direction measurement unit is compared with the driving direction of the forklift to determine whether the vehicle is driving safely.
Driving assistance method using deep learning-based forklift driver eye tracking. According to claim 10,
The vision camera is installed on the inner center fascia of the forklift and captures the face area of the forklift driver with an 80-degree FoV.
Driving assistance method using deep learning-based forklift driver eye tracking. According to claim 10,
A sixth step in which the control unit analyzes the distribution area of pixels of the head image of the forklift driver from the facial area image captured by the vision camera and determines whether the head is drowsy due to repetitive up and down movement, further comprising a sixth step. Characterized by,
Driving assistance method using deep learning-based forklift driver eye tracking. A computer-readable recording medium on which a computer program for executing the driving assistance method through deep learning-based forklift driver eye tracking according to any one of claims 10 to 18 on a computer is recorded.