KR20220082173A - Multimodal deep learning based LDWS black box equipment using images and PPG data - Google Patents

Abstract

The present invention is a drowsy driving prevention black box that integrates deep learning technology and LDSW technology through images and PPG data. More specifically, it relates to a method of grafting a PPG device and a black box, and an operation method connected to deep learning by collecting image data and pulse data. To this end, the present invention increases the economic feasibility of space by locating a PPG sensor that collects the driver's pulse data on the back of the steering wheel and mixing a camera that collects the driver's image data with a black box. In addition, a drowsy driving detection confirmation procedure is added by detecting lane departure through LDWS technology. It sends all detected information to the main control unit in charge of deep learning calculations to determine drowsy driving and delivers a warning message through a speaker attached to the black box.

Description

Multimodal deep learning based LDWS black box equipment using images and PPG data}

The present invention relates to a system for preventing drowsy driving, and more specifically, data such as lane departure collected by detecting a lane through a front camera of a black box based on LDWS and data collected by an internal camera of the black box. The image and PPG data collected by the PGG sensor are used to construct a drowsiness recognition model through deep learning, and based on this, the driver's drowsiness state is accurately determined to prevent vehicle accidents caused by drowsy driving in advance. How to configure it.

According to the Korea Highway Corporation, the number one cause of highway traffic accidents over the past 10 years has been drowsy driving. In particular, job workers such as surrogate drivers and transporters who mainly drive at night are vulnerable to drowsy driving. Accordingly, many technologies for preventing drowsy driving through various forms and technologies have been invented.

Deep learning-based drowsiness using MTCNN (Multi-Task Cascaded Convolution Networks) to learn the full face of the driver's both eyes and mouth while simultaneously using only the left eye and two mouths to determine the driver's status in a faster time The cognitive model is classified into three states: normal, yawning, and drowsy.

It uses PPG data and a drowsiness recognition model to construct a multi-modal deep learning-based driver drowsiness detection system consisting of a preprocessor, a multi-modal network, and a classification network.

The preprocessor extracts only the necessary data characteristics, removes unnecessary parts, collects only the necessary data, and divides it into an image collection unit, a main control unit, and a PPG collection unit. The image collection unit receives and collects the driver's face image taken by the camera mounted in the vehicle, and the main controller uses MTCNN to extract eyes and mouth from the driver's face image. The PPG collection unit collects PPG data obtained by measuring a person's heart rate through a change in blood flow.

The multimodal network removes heterogeneity from different types of data extracted from the preprocessor and simultaneously extracts only data features, and consists of a convolutional neural network, a layer reconstruction unit, and a JR construction unit. The convolutional neural network extracts only features from the input eye and mouth images. The extracted eye and mouth features are reconstructed into one layer by the layer reconstruction unit, and the size of the layer can be arbitrarily adjusted. The JR component consists of a layer reconstructed by extracting eye and mouth features by the layer reconstruction unit, and the PPG standardized by the main control unit by fusion into one JR (Joint Representation).

The classification network receives input data reconstructed as one data with all the characteristics of eyes, mouth, and PPG from the JR component of the multi-modal network, and classifies the driver's state into normal, yawning, and drowsy states and provides the corresponding state information. to output

Since the quality of the image acquired by the front camera is directly related to the operation performance of the drowsy driving prevention system, the accuracy of the image acquired by the front camera should be high. In particular, light that is diffusely reflected by the mechanical structure of the front camera device and is incident on the camera lens may cause problems such as glare or blur with respect to the image acquired by the front camera. The phenomenon may be further exacerbated when backlight is incident on the front camera device.

In the process of obtaining lane information, the vehicle's lane recognition system cannot distinguish the lane from the shadow caused by sunlight, so the lane recognition system of a vehicle running in the sun is shining. The possibility of failing to recognize a lane increases when it is over the lane or when the lane is obscured by the shadow of a guard rail.

To prevent these problems, a concave groove is made in the black box to reduce the change in luminance through a light trap to reduce reflected light on the bottom and sidewalls of the groove, and the color image acquired through the camera is pre-processed to reduce the change in illuminance. Based on the post-processed image that excludes changes in luminance and illuminance, the distance between the lane and the vehicle is recognized and lane departure is determined.

KR 10-2013-0146934 A KR 10-2017-0096516 A KR 10-2018-0159741 A

According to a survey on drowsy driving conducted by the Transportation Safety Authority on 400 drivers, 4 out of 10 drivers experienced drowsy driving, and 19% of them had a ‘near accident’ experience that almost caused an accident. In the past 10 years, drowsiness overtook speeding as the number one cause of highway traffic accidents. Therefore, the purpose of this is to reduce traffic accidents and casualties caused by drowsy driving.

The present invention is a multi-modal deep learning-based driver drowsiness detection system made by using a deep learning-based drowsiness recognition model that determines the driver's condition using a driver's face image and a PPG that measures heart rate, and a concave groove at the bottom of the groove. It is a black box camera made by combining LDWS, which is a camera equipped with a light trap for reducing reflected light on the side wall and a camera that can obtain an image with increased accuracy by reducing the change in luminous intensity.

The present invention is by using a driver drowsiness detection system based on multi-modal deep learning using images and PG data together with LDWS. It has the effect of judging and preventing drowsy driving through complex analysis of abstract single elements (visual information, biometric data, vehicle changes).

1 is a front view of a handle to which a PPG sensor is attached;
Figure 2 is a rear structural view of the handle to which the PPG sensor is attached.
3 is a black box to which an internal camera and an external camera are attached.
Fig. 4 is a front view of Fig. 3;

Among the core technologies of the present invention, the multi-modal deep learning-based driver drowsiness detection system is largely composed of a preprocessor, a multi-modal network, and a classification network.

The pre-processing unit extracts only the characteristics of the necessary data, removes unnecessary parts, collects only the necessary data, and is divided into an image collection unit, a main control unit, and a PPG collection unit. The main controller uses MTCNN when extracting the eyes and mouth from the driver's face image (because the extraction of the location information of the eyes and mouth includes changes in the eyes and mouth when drowsiness occurs), the PPG collector uses the blood flow by changing PPG is collected by measuring a person's heart rate. PPG shows a decrease due to parasympathetic nervous system when drowsiness occurs, and it can be used together with image characteristics.

The multimodal network is a fusion of eye and mouth image data and PPG into one. Since each data extracted from the preprocessor is composed of different shapes and changes appearing differently when drowsiness occurs, the multimodal network removes heterogeneity from each extracted data and simultaneously extracts only the characteristics of the data and performs convolution It consists of a neural network, a layer reconstruction unit, and a JR construction unit. The convolutional neural network extracts only the eye and mouth features from the input eye and mouth images, and the extracted eye and mouth features are reconstructed into one layer by the layer reconstruction unit, and the size of the layer can be arbitrarily adjusted. The JR component consists of a layer reconstructed by extracting eye and mouth features by the layer reconstruction unit and a PPG standardized by the main control unit by fusion into one JR (Joint Representation).

The classification network identifies the driver's drowsiness state with the convergence data from the multimodal network, and the classification network that receives the input data reconstructed as one data with all the features of eyes, mouth, and PPG from the JR component part Outputs state information by classifying the state.

The camera for LDWS is a camera equipped with a light trap to reduce reflected light on the bottom and side walls of the groove by making a concave groove. However, by acquiring a color image through this camera and pre-processing the image to exclude changes in illuminance, it is possible to recognize the distance between the lane and the vehicle to determine whether or not to deviate from the lane.

Therefore, it is possible to determine whether drowsiness is detected by examining the lane state and the driver's state with a black box camera using LDWS and a deep learning-based sensor system.

The present invention can be installed in a vehicle (eg, an express bus, a freight car, etc.) that mainly drives long-distance on a highway where drowsy driving occurs a lot.

In addition to preventing accidents through warning messages, it is possible to quickly recognize the cause of the accident with the information stored after the accident.

In the case of a high-speed bus that provides basic Wi-Fi for guests, it can be connected to a central server and controlled.

In addition, insurance companies can provide this service to prevent accidents and handle accidents with reliable information.

(1) Handle
(2) Home: Path for moving the PPG sensor position (preset)
(3) Sensor: The part responsible for sensing is exposed to the outside
(4) Fixing device: A device that fixes the sensor so that it is fixed in the groove at the junction.
(5) Conductor: A spring-type wire passing through the inside of the handle to connect the sensor and the control unit.
(6) Black box: Black box body
(7) Speaker: A device that gives a warning message
(8) Internal camera
(9) External camera

Claims (3)

In the drowsiness prevention device using a black box and a sensor installed on the left side of the rearview mirror, an internal camera (8) that collects information from an image inside the vehicle and a face image of a driver and an exterior that collects an external image and location information of a lane Lane departure prevention LDWS black box 6 to which the camera 9 is attached; a sensor (3) installed on the vehicle handle (1) to measure the driver's pulse by contacting the driver's body and an electronic device (3) for transmitting the measured information to the black box; a driver's drowsiness warning device based on multi-modal deep learning and LDWS that detects the driver's drowsiness using the black box and electronic device and delivers a warning message to the driver through the speaker (7); The method according to claim 1, wherein the sensor is installed on the rear side of the steering wheel to measure the driver's pulse, emits an LED, detects a pulse signal, and collects PPG data; Electronic device characterized in that it can be in contact with the driver for a long time; The electronic device according to claim 1, wherein the electronic device can be preset along the groove (2) on the rear side of the steering wheel according to a different position of the hand during long-distance driving according to a driver's habit;

Images