KR102045320B1 - Method and System for Preventing Driver Drowsiness Through Interaction with Driver by Based on Driver Status Monitoring - Google Patents

Abstract

The present invention relates to a method and system for preventing driver drowsiness through interaction with a driver based on driver condition monitoring. The method according to the present invention includes obtaining driver monitoring data for monitoring a driver driving a vehicle, and obtaining driver monitoring data. Applying to a predetermined driver drowsiness prediction model to predict a driver drowsiness state, and outputting a stimulus for interacting with the driver based on the driver drowsiness state prediction result. The method may further include checking a driver's response to the stimulus, and outputting a predetermined alarm if the driver's response is inappropriate. According to the present invention, if the driver's drowsiness is predicted, it is possible to effectively prevent drowsy driving through interaction with the driver.

Description

Method and System for Preventing Driver Drowsiness Through Interaction with Driver by Based on Driver Status Monitoring}

The present invention relates to a method and system for preventing driver drowsiness, and more particularly, to a method and system for preventing driver drowsiness through interaction with a driver based on driver condition monitoring.

Recently, driver drowsiness prevention technology that analyzes the image of the driver to recognize the driver's drowsiness and outputs an alarm to the driver has been commercialized.

Korean Patent No. 1,078,466 discloses a technology for determining awakening state by analyzing eye blink patterns by photographing a subject's face, and outputting a loud sound when the subject is drowsy with a driver based on this. A technique for alerting the user is disclosed.

In addition, Korean Patent Application Publication No. 2016-0104245 discloses a technology for controlling a vehicle when the driver is drowsy to open a car window, turn on a blinking light, output a loud sound from the vehicle audio, or generate a vibration in the steering wheel or the driver's seat. Started.

However, the conventional method of applying a unilateral stimulus to the driver has not been effective in preventing driver drowsiness.

Korea Patent Registration No. 1,078,466 (Registration Date: 2011.10.25) Korean Patent Publication No. 2016-0104245 (Published: 2016.09.05)

Therefore, the technical problem to be solved by the present invention is to provide a driver drowsiness prevention method and system through interaction with the driver condition monitoring based on the driver condition can prevent drowsy driving through interaction with the driver when the driver drowsiness is predicted.

According to an aspect of the present invention, a driver drowsiness prevention method through interaction with a driver condition monitoring-based driver according to the present invention may include obtaining driver monitoring data for monitoring a driver driving a vehicle, and previewing the driver monitoring data in advance. Predicting a driver drowsiness state by applying to a predetermined driver drowsiness prediction model, and outputting a stimulus for interacting with the driver based on a result of the driver drowsiness state prediction.
Analyze the driver monitoring data to learn the degree to which the arousal level is increased according to the action taken by the driver in the drowsiness state, select an operation of increasing the arousal level in the drowsiness state as the driver's drowsiness prevention operation, and the selected driver's drowsiness prevention operation Outputs a stimulus for inducing a stimulus for interacting with the driver.

The method may further include identifying a driver's response to the stimulus and outputting a predetermined alert if the driver's response is inappropriate.

Driver drowsiness prevention system through interaction with the driver condition monitoring-based driver according to the present invention for solving the above technical problem, the vehicle terminal for obtaining driver monitoring data for monitoring the driver driving the vehicle, and the driver monitoring data Is applied to a predetermined driver drowsiness prediction model to predict a driver drowsiness state, and a service server for requesting the vehicle terminal to output a stimulus for interacting with the driver based on the driver drowsiness state prediction result.
The service server analyzes the driver monitoring data to learn the degree to which the arousal level increases according to the driver's drowsy state, and selects an operation of raising the most arousal level in the drowsy state as the driver's drowsiness prevention operation. A stimulus for inducing a selected driver drowsiness prevention operation is output through the vehicle terminal as a stimulus for interacting with the driver.

The vehicle terminal may identify a driver's response to the stimulus, and output a predetermined alarm if the driver's response is inappropriate.

The driver drowsiness prediction model may be trained using big data including a plurality of data obtained by mapping driver monitoring data, vehicle driving data, and vehicle driving environment data.

The stimulus for interacting with the driver may be output through at least one of a visual method and an auditory method.

The driver's response may be obtained through at least one of an image sensor and a voice sensor.

According to the present invention, if the driver's drowsiness is predicted, it is possible to effectively prevent drowsy driving through interaction with the driver.

1 is a block diagram of a system for preventing driver drowsiness through interaction with a driver condition monitoring-based driver according to an embodiment of the present invention.
2 is a block diagram showing the configuration of a vehicle terminal according to an embodiment of the present invention.
3 is a flowchart illustrating an operation of a driver drowsiness prevention system through interaction with a driver condition monitoring-based driver according to an embodiment of the present invention.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention.

1 is a block diagram of a driver drowsiness prevention system through interaction with a driver condition monitoring-based driver according to an embodiment of the present invention, Figure 2 is a block diagram showing the configuration of a vehicle terminal according to an embodiment of the present invention .

1 and 2, the driver drowsiness prevention system according to the present invention may include a plurality of vehicle terminals 100a, 100b, 100c,..., 100n-1, 100n and a service server 200.

The plurality of vehicle terminals 100a, 100b, 100c,..., 100n-1, 100n and the service server 200 may exchange various information and data through the communication network 10.

The communication network 10 may include a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), the Internet, 2G, 3G, 4G, LTE mobile communication network, Bluetooth, It may include Wi-Fi, Wibro, satellite communication networks, and Low Power Wide Area (LPWA) networks such as LoRa and Sigfox, and the communication method may be wired or wireless and may be any communication method. .

The vehicle terminals 100a, 100b, 100c,..., 100n-1, 100n include the image sensor 110, the voice sensor 120, the driving data collector 130, the controller 140, and the storage 150. It may include a stimulus output unit 160 and the communication unit 170.

The vehicle terminals 100a, 100b, 100c,..., 100n-1, 100n may be located in a vehicle (not shown) corresponding to each of the vehicle terminals to acquire driver monitoring data that monitors a state of a driver driving the vehicle. have.

The driver monitoring data may include driver image data photographing a driver's figure, and in some embodiments, the driver monitoring data may include driver voice data obtained by obtaining a driver's voice as an electrical signal. The driver's image data may include a driver's face and may also include an upper body of the driver.

The image sensor 110 may be implemented through a camera, and may acquire driver image data photographing the driver's image.

The voice sensor unit 120 may be implemented through a microphone, and may obtain driver voice data by converting the driver voice into an electric signal.

The vehicle terminals 100a, 100b, 100c,..., 100n-1 and 100n may acquire vehicle driving data and / or vehicle driving environment data.

The vehicle driving data may include data collected during vehicle driving, such as vehicle speed, steering angle control, brake operation, and gear operation.

The driving data collection unit 130 may obtain vehicle driving data through an on-board diagnostics (OBD) interface of the vehicle. Of course, when the vehicle terminals 100a, 100b, 100c, ..., 100n-1, 100n are provided with the GNSS receiving module, the driving data collecting unit 130 may collect the position and speed of the vehicle obtained by processing the GNSS signal. It may be.

The vehicle driving environment data is data about an environment in which the vehicle is driven, and may include, for example, a vehicle indoor temperature, a vehicle outdoor temperature, weather in a vehicle driving region, road geometric information of a vehicle driving road, and the like. In addition, the vehicle driving environment data may include road traffic information such as a traffic volume and a road traffic speed of the road on which the vehicle operates.

The driving data collection unit 130 may collect the vehicle indoor temperature and the vehicle outdoor temperature through a temperature sensor mounted inside or outside the vehicle. In addition, the driving data collector 130 may collect rainfall or illuminance around the vehicle in operation through a rain sensor or an illumination sensor mounted on the vehicle.

The storage unit 150 may store various information, data, and programs related to the operations of the vehicle terminals 100a, 100b, 100c,..., 100n-1, 100n. The storage unit 150 may permanently store or temporarily store driver monitoring data, vehicle driving data, and vehicle driving environment data collected from the vehicle.

The weather in the vehicular region may include temperature, rainfall, snowfall, wind speed, humidity, insolation and sunshine, and can be obtained from weather data provided by a weather server (not shown) operated by the Meteorological Administration or a private meteorological service provider. It may be.

Road geometry information may include road end slope, road crossing slope, road curvature, and the like. Road geometric information may be stored in advance in the service server 200.

Road traffic information may be provided from a traffic information providing server (not shown).

When the vehicle terminal 100a, 100b, 100c, ..., 100n-1, 100n is requested to interact with the driver for preventing driver drowsiness from the service server 200, the vehicle terminals 100a, 100b, 100c, ..., 100n-1, 100n output a stimulus for interacting with the driver. The reaction can be confirmed.

The stimulus output unit 160 may output a stimulus for interacting with the driver in an auditory or visual manner. For example, the stimulus output unit 160 may output a voice message inducing the driver to perform a specific action or answer. Of course, the stimulus may be output in a visual way such as text or picture for the hearing impaired. To this end, the stimulus output unit 160 may be implemented as a speaker or a display module.

The vehicle terminals 100a, 100b, 100c,..., 100n-1, 100n may output a predetermined alarm when the driver's response is inappropriate. The alert may be made by various methods such as visual, auditory, tactile and the like.

The controller 140 controls the overall operation of the vehicle terminals 100a, 100b, 100c,..., 100n-1, 100n. When the stimulus output for interaction with the driver is requested from the service server 200, the controller 140 may control the stimulus output unit 160 to output the stimulus for interaction to the driver. In addition, the controller 140 checks a response to the driver's interaction obtained through the image sensor 110 or the voice sensor 120.

How to interact with the driver to prevent drowsiness can be made in a variety of ways. For example, the controller 140 may transmit a message requesting the driver to perform a specific motion, such as 'shake the head from side to side', 'open and close the mouth', or the like. 160). The controller 140 may check whether the driver performs a corresponding operation for a predetermined time through the image sensor 110. When the driver performs the corresponding operation for a predetermined time, the controller 140 may determine that the driver response is appropriate. On the other hand, if the driver does not perform the operation or other operation within a predetermined time, the controller 140 may determine that the driver response is inappropriate.

Meanwhile, the controller 140 sends a message requesting the driver to say a specific phrase, such as 'tell me a name' or 'tell me XXX' through the stimulus output unit 160 as an audible stimulus method or a visual stimulus method. You can print In addition, the controller 140 may check whether the driver speaks the corresponding phrase for a predetermined time through the voice sensor unit 120. When the driver speaks a predetermined phrase for a predetermined time, the controller 140 may determine that the driver response is appropriate. On the other hand, if the driver does not speak the phrase or the other phrase within a predetermined time, the controller 140 may determine that the driver's response is inappropriate.

The controller 140 may output a question requesting an answer according to the current driver's situation through the stimulus output unit 160 using an auditory stimulus method or a visual stimulus method. For example, "Where are you going?", "Is it day or night now?" For example, the driver may ask questions that may vary depending on the situation. Such a question may be prepared in advance in the controller 140 or may be provided to the vehicle terminal 100a from the service server 200. In addition, the controller 140 determines that the driver has a proper answer to the question for a predetermined time (for example, 'going home (home is set as a navigation destination) or if the destination is determined to be home as a result of driver driving pattern analysis). ) "," It's daytime (if it's daytime, it's a good answer) ", you can determine that the driver's response is appropriate. Of course, the driver's answer may be received from the vehicle terminal 100a, and the service server 200 may determine whether the response is appropriate and feed back the result to the vehicle terminal 100a.

The controller 140 may output an alarm through the stimulus output unit 160 if the reaction to the driver's interaction is inappropriate. Of course, the controller 140 may output an alarm through a speaker, a horn, and a warning light provided in the vehicle.

The controller 140 may be linked with an electronic control unit (ECU) provided in the vehicle to open a vehicle window or play a specific content in the entertainment apparatus provided in the vehicle.

The communication unit 170 provides a communication interface for allowing the vehicle terminals 100a, 100b, 100c,..., 100n-1, 100n to exchange various information and data with the service server 200 through the communication network 10.

The service server 200 uses a big data including driver monitoring data collected from a plurality of vehicle terminals 100a, 100b, 100c,..., 100n-1, 100n and a plurality of data mapped to vehicle driving data. Learn a drowsiness prediction model. According to an exemplary embodiment, the driver drowsiness prediction model may be more precisely trained using big data including a plurality of data mapped not only to the driver monitoring data and the vehicle driving data but also to the vehicle driving environment data.

Various methods of analyzing the driver's face image to determine the driver's drowsiness are already known. In addition, various methods are already known in connection with determining vehicle drowsiness by analyzing vehicle driving data such as a vehicle driving speed control operation or a steering angle control operation. In addition, various methods are already known for predicting a driver's drowsiness according to road geometry, weather conditions, and road traffic conditions. Therefore, a detailed description of the method for learning the driver drowsiness prediction model is omitted.

When the service server 200 receives driver monitoring data for monitoring a driver's current state of a vehicle from the vehicle terminals 100a, 100b, 100c,..., 100n-1, 100n, the service server 200 applies the driver monitoring data to a predetermined driver drowsiness prediction model. The driver's drowsiness can be predicted.

The driver's drowsy state prediction may include not only the current driver's drowsiness but also a prediction of a driver's drowsiness as time passes.

The service server 200 may request the vehicle terminals 100a, 100b, 100c,..., 100n-1, 100n to output a stimulus for interacting with the driver based on the driver drowsiness prediction result for the specific driver. For example, when the degree of drowsiness of the driver of the vehicle in which the vehicle terminal 100a is located is greater than or equal to a predetermined criterion, or when the vehicle driving data or vehicle driving environment data currently acquired for the vehicle is reflected, the driver may be in a drowsy state in advance. If the driver drowsiness prediction result indicates that the drowsiness driving possibility is equal to or more than a predetermined criterion, for example, when it is greater than or equal to a predetermined criterion, the service server 200 may request an interaction for preventing driver drowsiness from the vehicle terminal 100a. Here, the driving environment data of the vehicle may use weather data, road geometry, road traffic conditions, etc. corresponding to the vehicle location provided from the vehicle terminal 100a.

Meanwhile, by reinforcing the artificial intelligence factor, the service server 200 may learn about an effective method that the driver takes to prevent drowsiness based on the driver monitoring data. For example, the service server 200 analyzes the driver monitoring data to determine the degree of the driver's drowsiness (awakening level is increased) according to a specific action (eg, opening a car window or listening to a specific music) that the driver takes in a drowsy state. ), You can choose the action that will most effectively prevent driver drowsiness. The service server 200 selects a stimulus for inducing an effective drowsiness prevention operation learned for the driver, for example, 'Do you want to listen to xx music?', 'Do you want to open the window for a while?' And output through the vehicle terminal 100a. You can. When the driver answers a positive question to the question, the vehicle terminal 100a may play the music through an entertainment device (not shown) provided in the vehicle, or may open the window of the vehicle.

3 is a flowchart illustrating an operation of a driver drowsiness prevention system through interaction with a driver condition monitoring-based driver according to an embodiment of the present invention.

1 to 3, first, the service server 200 maps a plurality of driver monitoring data and vehicle driving data collected from a plurality of vehicle terminals 100a, 100b, 100c,..., 100n-1, 100n. According to an embodiment, the driver drowsiness prediction model is trained using big data constructed by mapping vehicle driving environment data (S310).

Thereafter, the vehicle terminal 100a may acquire driver monitoring data for monitoring a driver currently driving the vehicle and transmit the driver monitoring data in real time to the service server 200 (S320). In operation S320, the vehicle terminal 100a may acquire vehicle driving data and transmit the vehicle driving data to the service server 200 in real time together with the driver monitoring data. According to an exemplary embodiment, vehicle driving environment data and vehicle location information obtained from a vehicle may also be transmitted to the service server 200.

Next, the service server 200 may predict driver drowsiness state by applying driver monitoring data transmitted in real time from the vehicle terminal 100a to the driver drowsiness prediction model learned in step S310 (S330). When the vehicle location information is transmitted from the vehicle terminal 100a, the service server 200 also applies the road geometry, weather data, road traffic information, etc. of the road on which the vehicle operates using the vehicle location information to the driver drowsiness prediction model. Predict driver drowsiness.

Next, when the service server 200 determines that the driver's drowsy state interaction is necessary as a result of the driver's drowsy state prediction, the service server 200 requests the vehicle terminal 100a to interact with the driver's drowsiness prevention. It may be (S340-Y).

Then, the vehicle terminal 100a may output a stimulus for interacting with the driver (S350).

Thereafter, the vehicle terminal 100a may check whether the driver's response to the stimulus output in step S350 is inappropriate (S360). If there is no driver's response to the stimulus in step S360, it may be determined to be inappropriate. In addition, it outputs a stimulus for instructing the driver to take a specific action, it may be determined that the driver is inappropriate even if the driver takes a different action or there is no action. In addition, the stimulus is outputted to the driver to induce an answer to the driver, it can be determined that it is inappropriate even if the question is inappropriately answered.

If the driver's response is inappropriate (S360-Y), the vehicle terminal 100a may output an alarm (S370).

On the other hand, if the driver's response is appropriate (S360-Y), the steps S320 to S360 may be repeated until the driver finishes driving the vehicle or stops the operation.

Embodiments of the invention include a computer readable medium containing program instructions for performing various computer-implemented operations. This medium records a program for executing the method described above. The media may include, alone or in combination with the program instructions, data files, data structures, and the like. Examples of such media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CDs and DVDs, floppy disks and program commands such as magnetic-optical media, ROM, RAM, flash memory, and the like. Hardware devices configured to store and perform such operations. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

Although the preferred embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concepts of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

Claims (8)

Obtaining driver monitoring data monitoring a driver driving a vehicle;
Predicting driver drowsiness by applying the driver monitoring data to a predetermined driver drowsiness prediction model, and
Outputting a stimulus for interacting with the driver based on the driver drowsiness prediction result;
Including,
Analyze the driver monitoring data to learn the degree to which the arousal level is increased according to the action taken by the driver in the drowsiness state, select an operation of increasing the arousal level in the drowsiness state as the driver's drowsiness prevention operation, and the selected driver's drowsiness prevention operation Driver condition monitoring based on the driver condition monitoring to output a stimulus for inducing the driver as a stimulus for interaction with the driver. In claim 1,
Confirming the driver's response to the stimulus, and
Outputting a predetermined alarm if the driver's response is inappropriate;
Driver drowsiness prevention method through interaction with the driver based on driver condition monitoring further comprising. In claim 2,
The driver drowsiness prediction model,
A driver drowsiness prevention method through interaction with a driver condition monitoring-based driver trained using big data including a plurality of data mapping driver monitoring data, vehicle driving data, and vehicle driving environment data. In claim 3,
The stimulus for interacting with the driver is output through at least one of a visual method and an audio method,
And the driver's response is acquired through at least one of an image sensor and a voice sensor. A vehicle terminal for obtaining driver monitoring data for monitoring a driver driving a vehicle, and
The service server requesting the vehicle terminal to apply the driver monitoring data to a predetermined driver drowsiness prediction model to predict the driver drowsiness and output a stimulus for interacting with the driver based on the driver drowsiness prediction result.
Including,
The service server,
Analyze the driver monitoring data to learn the degree to which the arousal level is increased according to the action taken by the driver in the drowsiness state, select an operation of increasing the arousal level in the drowsiness state as the driver's drowsiness prevention operation, and the selected driver's drowsiness prevention operation The driver's drowsiness prevention system through interaction with a driver's condition monitoring-based driver that outputs a stimulus for inducing a stimulus for interacting with the driver through the vehicle terminal. In claim 5,
The vehicle terminal,
The driver's drowsiness prevention system by checking the driver's response to the stimulus, and outputting a predetermined alarm if the driver's response is inappropriate. In claim 6,
The driver drowsiness prediction model,
Driver drowsiness prevention system through interaction with a driver condition monitoring-based driver learned using big data including a plurality of data mapping driver monitoring data, vehicle driving data and vehicle driving environment data. In claim 7,
The stimulus for interacting with the driver is output through at least one of a visual method and an audio method,
The driver's drowsiness prevention system through interaction with the driver's condition monitoring-based driver is obtained through at least one of the image sensor and the audio sensor.

Images