KR102520188B1 - Vehicle device for determining a driver's condition using artificial intelligence and control method thereof - Google Patents

Abstract

Disclosed is a vehicle apparatus. The vehicle apparatus comprises: a camera; a memory which stores at least one instruction; and a processor which executes at least one command to determine whether a driver is drowsy based on a video filmed by the camera and, when it is determined that the driver is drowsy, to provide a feedback corresponding to a drowsy status. The processor may determine whether the driver is drowsy based on driver's brain wave data and driver's eye data related to the driver's captured video. An objective of the present invention is to provide the vehicle apparatus which determines whether the driver is drowsy and provides the corresponding feedback, and a control method thereof.

Description

Vehicle device for determining a driver's condition using artificial intelligence and control method thereof}

The present disclosure relates to a vehicle device and a control method thereof, and more particularly, to a vehicle device and a control method for determining drowsy driving of a driver.

According to a survey by the Korea Road Traffic Authority, 6 to 7 out of 10 deaths in highway traffic accidents were confirmed to be due to drowsy driving and neglect of looking ahead.

Therefore, in order to prevent the driver's drowsiness, a technology and system capable of determining the driver's drowsiness are required.

Most currently developed driver drowsiness determination systems acquire driver's face data through a camera and determine the current driver's drowsiness state using the eyelid height ratio detected from the driver's face data. However, the technology has a problem in that the accuracy of judgment according to the angle of the driver's face is rapidly reduced.

The present disclosure has been made in accordance with the above-described needs, and an object of the present disclosure is to provide a vehicle apparatus and a control method for determining whether a driver is drowsy driving based on brain wave data and eyeball data of the driver and providing feedback corresponding thereto. is in providing

In order to achieve the above object, a vehicle device according to an embodiment of the present disclosure includes a camera, a memory for storing at least one command, and determining whether a driver is in a drowsy state based on a driver's captured image obtained through the camera. and a processor configured to provide feedback corresponding to the drowsy state when it is determined that the driver is in the drowsy state, wherein the processor determines the driver's brain wave data and the driver's eyeball data related to the driver's captured image. It is possible to determine whether the driver is in a drowsy state.

In addition, it further includes an brain wave measurement unit, processing the driver's brain wave data obtained through the brain wave measurement unit to obtain PSD (Power Spectral Density) brain wave data, and based on the driver's eye data obtained from the driver's photographic image Eye blink data is acquired, and if it is determined that the driver has closed his or her eyes for a predetermined period of time or more based on the eye blink data, it is determined that the driver is in a drowsy state, and the driver has not closed their eyes for a predetermined period of time or longer. Otherwise, if the PSD brain wave data satisfies a specific condition, it may be determined that the driver is in a drowsy state.

The memory may store the learned artificial neural network model, and the processor may determine whether the driver is in a drowsy state by inputting the driver's captured image acquired through the camera to the artificial neural network model. Here, the artificial neural network model may be a model learned by using a plurality of driver images for learning and brain wave data corresponding to each of the plurality of driver images for learning as input data, and using information on drowsiness as output data.

In addition, the processor maps a drowsy state to an image for which brain wave data satisfies a specific condition among driver images for learning in which the time during which eyes are closed in a predetermined time interval is greater than or equal to a threshold time, and maps a non-drowsy state to the remaining images. Thus, the artificial neural network model can be trained.

In addition, the processor determines whether the driver is in a drowsy state by inputting the driver's image and additional information to the artificial neural network model, and the additional information may include at least one of driving environment information and driver profile information. .

In addition, the processor may obtain the eye blink data from the eyeball data by calculating horizontal and vertical aspect ratios of the eyeball using a distance threshold method (DTM) technique.

In addition, the above specific condition is a condition that satisfies (α _H + α _L ) / θ > 1 and α _H > α _L , α _H > δ, θ > δ and α _H > β, and α _H is High-alpha wave, α _L may be a low-alpha wave, δ may be a delta wave, β may be a beta wave, and θ may be a theta wave.

Meanwhile, a method for controlling a vehicle device according to an embodiment includes determining whether a driver is in a drowsy state based on a driver photographed image obtained through a camera, and responding to the drowsy state if it is determined that the driver is in a drowsy state. In the step of determining whether the driver is in a drowsy state, it may be determined whether the driver is in a drowsy state based on the driver's brain wave data and the driver's eye data related to the driver's captured image.

In addition, the step of determining whether the driver is in a drowsy state may include processing the driver's EEG data to obtain PSD (Power Spectral Density) EEG data, and eye blink data based on the driver's eye data obtained from the driver's photographic image. is obtained, and if it is determined based on the eye blink data that the driver has closed his or her eyes for more than a predetermined time, it is determined that the driver is in a drowsy state, and if the driver has not closed their eyes for more than a predetermined time, the If the PSD brain wave data satisfies a specific condition, it may be determined that the driver is in a drowsy state.

In the step of determining whether the driver is in a drowsy state, the captured image of the driver is input to a learned artificial neural network model to determine whether the driver is in a drowsy state, and the artificial neural network model includes a plurality of driver images for learning and the plurality of It may be a model learned by using brain wave data corresponding to each driver image for learning as input data and information on whether or not a drowsy state is output data.

According to the various embodiments described above, it is possible to accurately determine the driver's condition and provide a customized stimulus or/and warning corresponding thereto. In addition, it is possible to determine whether the driver is drowsy or not from other facial features as well as the driver's eyes. Accordingly, it is possible to determine whether the driver is drowsy regardless of the driver's various postures and face angles. Accordingly, the driver can be protected from danger.

1 is a block diagram showing the configuration of a vehicle device according to an embodiment of the present disclosure.
2 is a diagram for explaining a method for obtaining eye blink data according to an exemplary embodiment.
3, 4, 5A and 5B are diagrams for explaining an EEG data analysis method according to an embodiment.
6A and 6B are diagrams for explaining a learning method of an artificial neural network model according to an embodiment.
7 is a diagram for explaining data for learning according to an exemplary embodiment.
8 is a diagram for explaining the operation of a learned artificial neural network model according to an embodiment.
9 is a diagram for explaining effects of the present disclosure according to an embodiment.
10 is a diagram illustrating a detailed configuration of a vehicle device according to an exemplary embodiment.
11 is a flowchart illustrating a vehicle control method according to an exemplary embodiment.
12 is a flowchart illustrating a vehicle control method according to another embodiment.

Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

Terms used in this specification will be briefly described, and the present disclosure will be described in detail.

The terms used in the embodiments of the present disclosure have been selected from general terms that are currently widely used as much as possible while considering the functions in the present disclosure, but they may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technologies, and the like. . In addition, in a specific case, there is also a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the disclosure. Therefore, terms used in the present disclosure should be defined based on the meaning of the term and the general content of the present disclosure, not simply the name of the term.

In this specification, expressions such as “has,” “can have,” “includes,” or “can include” indicate the existence of a corresponding feature (eg, numerical value, function, operation, or component such as a part). , which does not preclude the existence of additional features.

The expression at least one of A and/or B should be understood to denote either "A" or "B" or "A and B".

Expressions such as "first," "second," "first," or "second," as used herein, may modify various components regardless of order and/or importance, and may refer to one component It is used only to distinguish it from other components and does not limit the corresponding components.

A component (e.g., a first component) is "(operatively or communicatively) coupled with/to" another component (e.g., a second component); When referred to as "connected to", it should be understood that an element may be directly connected to another element, or may be connected through another element (eg, a third element).

The expression “configured to (or configured to)” as used in this disclosure means, depending on the situation, for example, “suitable for,” “having the capacity to.” ," "designed to," "adapted to," "made to," or "capable of." The term "configured (or set) to" may not necessarily mean only "specifically designed to" hardware.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "comprise" or "consist of" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other It should be understood that the presence or addition of features, numbers, steps, operations, components, parts, or combinations thereof is not precluded.

In the present disclosure, a “module” or “unit” performs at least one function or operation, and may be implemented in hardware or software or a combination of hardware and software. In addition, a plurality of "modules" or a plurality of "units" are integrated into at least one module and implemented by at least one processor (not shown), except for "modules" or "units" that need to be implemented with specific hardware. It can be.

Hereinafter, an embodiment of the present disclosure will be described in more detail with reference to the accompanying drawings.

1 is a block diagram showing the configuration of a vehicle device according to an embodiment of the present disclosure.

According to FIG. 1 , a vehicle device 100 includes a camera 110 , a memory 120 and a processor 130 .

The camera 110 may be turned on according to a preset event to take a picture. The camera 110 may convert a captured image into an electrical signal and generate image data based on the converted signal. For example, an object may be converted into an electrical image signal through a charge coupled device (CCD), and the image signal thus converted may be amplified and converted into a digital signal and then signal processed. For example, the camera 110 may be implemented as a general camera, a stereo camera, or a depth camera.

According to an example, the camera 110 may be disposed at a location within the vehicle device 100 capable of capturing the driver's face and obtain an image of the driver's face.

The memory 120 may store data necessary for various embodiments of the present disclosure. The memory 120 may be implemented in the form of a memory embedded in the vehicle device 100 or in the form of a memory capable of communicating with (or detachable from) the vehicle device 100 according to a data storage purpose. For example, data for driving the vehicle device 100 is stored in a memory embedded in the vehicle device 100, and data for an extended function of the vehicle device 100 is communicable with the vehicle device 100. can be stored in memory. Meanwhile, in the case of memory embedded in the vehicle device 100, volatile memory (eg, DRAM (dynamic RAM), SRAM (static RAM), SDRAM (synchronous dynamic RAM), etc.), non-volatile memory ( Examples: OTPROM (one time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, flash memory (such as NAND flash or NOR flash, etc.) ), a hard drive, or a solid state drive (SSD). In addition, in the case of a memory capable of communicating with the vehicle device 100, a memory card (eg, a compact flash (CF)) , SD (secure digital), Micro-SD (micro secure digital), Mini-SD (mini secure digital), xD (extreme digital), MMC (multi-media card), etc.), external memory that can be connected to the USB port (eg For example, a USB memory) may be implemented in the form of the like.

According to an example, the memory 120 may store at least one instruction or a computer program including instructions for controlling the vehicle device 100 .

According to another example, the memory 120 may store various data, programs, or applications for driving/controlling the vehicle device 100 . The vehicle device 100 ′ may store a control program for controlling the vehicle device 100 and the processor 130, an application initially provided by a manufacturer or downloaded from the outside, databases, or related data. For example, a memory ( 120) may store information such as EEG data conditions, eye data conditions, various feedback notifications, and various guide notifications for determining the drowsiness condition according to an embodiment.

According to another example, the memory 120 may store information about an artificial neural network model including a plurality of layers. Here, storing information about the artificial neural network model means various information related to the operation of the artificial neural network model, for example, information about a plurality of layers included in the artificial neural network model, parameters used in each of the plurality of layers (eg, , filter coefficients, bias, etc.) For example, the memory 120 may store information about an artificial neural network model learned to obtain information on whether a driver is in a drowsy state according to an embodiment. However, when the processor 130 is implemented as hardware dedicated to the artificial neural network model, information about the artificial neural network model may be stored in an internal memory of the processor 130 .

However, according to another embodiment, the artificial neural network model may be stored in an external device such as a server, and the vehicle device 100 transmits the driver's captured image to the external device to obtain information on whether the driver is in a drowsy state from the external device. It is also possible to obtain.

According to an embodiment, the memory 120 may be implemented as a single memory that stores data generated in various operations according to the present disclosure. However, according to another embodiment, the memory 120 may be implemented to include a plurality of memories each storing different types of data or each storing data generated in different steps.

The processor 130 is electrically connected to the camera 110 and the memory 120 to control overall operations of the vehicle device 100 . Processor 130 may be composed of one or a plurality of processors. Specifically, the processor 130 may perform the operation of the vehicle device 100 according to various embodiments of the present disclosure by executing at least one instruction stored in a memory (not shown).

According to an embodiment, the processor 130 may include a digital signal processor (DSP), a microprocessor, a graphics processing unit (GPU), an artificial intelligence (AI) processor, and a neural processing unit (NPU) for processing digital image signals. Processing Unit), time controller (TCON), but is not limited to, central processing unit (CPU), micro controller unit (MCU), micro processing unit (MPU), controller (controller), an application processor (AP), a communication processor (CP), or one or more of an ARM processor, or may be defined by the term. In addition, the processor 140 may be implemented in the form of a system on chip (SoC) with a built-in processing algorithm, large scale integration (LSI), application specific integrated circuit (ASIC), or field programmable gate array (FPGA).

In addition, the processor 130 for executing the artificial neural network model according to an embodiment includes a general-purpose processor such as a CPU, AP, digital signal processor (DSP), and the like, a graphics-only processor such as a GPU, a vision processing unit (VPU), or an NPU. It can be implemented through a combination of the same artificial intelligence dedicated processor and software. The processor 130 may control input data to be processed according to a predefined operation rule or an artificial neural network model stored in the memory 120 . Alternatively, if the processor 130 is a dedicated processor (or artificial intelligence dedicated processor), it may be designed as a hardware structure specialized for processing a specific artificial neural network model. For example, hardware specialized for processing a specific artificial neural network model may be designed as a hardware chip such as an ASIC or FPGA. When the processor 130 is implemented as a dedicated processor, it may be implemented to include a memory for implementing an embodiment of the present disclosure or to include a memory processing function for using an external memory.

The processor 130 may determine whether the driver is in a drowsy state based on the driver's captured image obtained through the camera 110 and, if it is determined that the driver is in a drowsy state, provide feedback corresponding to the drowsy state. Here, the driver's captured image may be an image related to information about the driver's brain wave data and the driver's eye data.

According to an embodiment, the processor 130 may determine whether the driver is in a drowsy state based on the driver's brain wave data obtained through the brain wave measurer 140 and the driver's eye data obtained from the driver's photographic image.

Specifically, the processor 130 may process the driver's EEG data to obtain Power Spectral Density (PSD) EEG data, and may obtain eye blink data based on the driver's eyeball data obtained from the driver's photographed image. Subsequently, the processor 130 determines that the driver is in a drowsy state when it is determined based on the eye blink data that the driver has closed his/her eyes for a predetermined period of time or longer, and if the driver has not closed his or her eyes for a predetermined period of time or longer, the PSD brain wave data When satisfies a specific condition, it may be determined that the driver is in a drowsy state.

In this case, the brain wave measurement unit 140 may detect the driver's brain waves and transmit them to the processor 130 . According to an example, the EEG measurement unit 140 may include an EEG measurement sensor, an amplification unit, and an AD conversion unit. Here, the brain wave measurement sensor may generate a real-time brain wave signal of the driver. The amplification unit may amplify the EEG signal and filter the 60Hz alternating current that is normally performed when measuring the EEG. The AD conversion unit may convert the amplified and filtered EEG signal from an analog form to a digital form. In this case, the converted EEG signal may be transmitted to the processor 130 through a predetermined interface provided in the EEG measuring unit 140 .

According to an example, the brain wave measurement unit 140 can be implemented in various forms wearable on the driver's head. For example, it may be implemented as a band or a headset type that is detachable from the driver's head. In addition, the brain wave measurement unit 140 may include a plurality of electrode units. At least one electrode unit may be disposed to detect EEG generated in the driver's temporal lobe and occipital lobe, and at least one other electrode unit may be disposed to detect EEG generated in the frontal lobe and occipital lobe of the driver's brain. it is not going to be

According to an embodiment, the processor 130 may obtain power spectral density (PSD) brain wave data by processing the driver's brain wave data obtained through the brain wave measurement unit 140 . Here, PSD (Power Spectral Density) is a function representing random vibration as an independent variable frequency, and may represent the intensity of energy according to frequency. For example, it may be expressed as an energy level (Watt/Hz) per unit frequency, but is not limited thereto. In the PSD data, it is possible to determine which frequency waveforms are more frequent in a waveform according to time.

Also, the processor 130 may obtain eye blink data based on eye data of the driver through the camera 110 .

According to an example, the processor 130 may acquire eye data by detecting a face area from an image captured by the camera 110 and detecting an eye area from the face area. As a face area detection method, various conventional methods may be used. Specifically, a direct recognition method and a method using statistics may be used. In the direct recognition method, rules are created using physical characteristics such as outline, skin color, size of components, and distance between components of a face image, and comparison, inspection, and measurement are performed according to the rules. The method using statistics may detect a face region according to a pre-learned algorithm. In other words, it is a method of comparing and analyzing the unique characteristics of the input face with a large amount of prepared database (shapes of faces and other objects) by converting them into data. In particular, a facial area can be detected according to a pre-learned algorithm, and methods such as Multi Layer Perceptron (MLP) and Support Vector Machine (SVM) can be used. Alternatively, an eye image is identified from a photographed image through a face modeling technique. In this case, the face modeling technology is an analysis process of converting a face image into digital information for processing and transmission, and one of an Active Shape Modeling (ASM) technique and an Active Appearance Modeling (AAM) technique may be used.

When eyeball data, i.e. eyeball data, is obtained from a photographed image, the processor 130 may obtain eye blink data from the eyeball data by calculating horizontal and vertical aspect ratios of the eyeball using a Distance Threshold Method (DTM) technique. A detailed description thereof will be described later with reference to the drawings.

According to an embodiment, the processor 130 may determine that the driver is in a drowsy state when it is determined that the driver has closed his or her eyes for a predetermined time or longer based on the eye blink data. Here, the preset time may be predetermined. For example, the preset time may be 2 seconds, but is not limited thereto. For example, the preset time may be determined based on the driver's driving condition. For example, the processor 130 may determine a critical time for determining that the driver is in a drowsy state based on a driving state history in which the driver's driving state is recorded.

In addition, the processor 130 may determine that the driver is in a drowsy state if the PSD brain wave data satisfies a specific condition when the driver does not close his or her eyes for a predetermined period of time or longer. In this case, the specific conditions may be conditions satisfying (α _H + α _L )/θ > 1 and α _H > α _L , α _H > δ , θ > δ , and α _H > β . Here, α _H may be a high-alpha wave, α _L may be a low-alpha wave, δ may be a delta wave, β may be a beta wave, and θ may be a theta wave. A detailed description thereof will be described later with reference to the drawings.

According to another embodiment, the processor 130 may determine whether the driver is drowsy by using the learned artificial neural network model. According to an example, the artificial neural network model may include a recurrent neural network (RNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), and a bidirectional recurrent deep neural network (BRDNN). ) or deep Q-networks (Deep Q-Networks), etc., but is not limited thereto.

Specifically, the processor 130 inputs the driver's captured image obtained through the camera 110 to the artificial neural network model to determine whether the driver is in a drowsy state, and if it is determined that the driver is in a drowsy state, the processor 130 provides feedback corresponding to the drowsy state. can provide According to an example, the driver's photographed image may be an image including the driver's whole body, face, and other environments.

Here, the artificial neural network model may be a model learned using a plurality of driver images for learning and brain wave data corresponding to each of the plurality of driver images for learning as input data and information on drowsiness as output data.

Next, if it is determined that the driver is drowsy, the processor 130 may provide feedback corresponding to the drowsy state. According to an example, if it is determined that the driver is drowsy, the processor 130 may provide a warning alarm or guide to the driver or switch to an autonomous driving mode. According to an example, it is possible to perform shoulder parking, emergency stop, and the like by switching to an autonomous driving mode.

For example, if it is determined that the driver's condition is not recovered after providing a warning alarm, the processor 130 may switch to the autonomous driving mode.

As another example, the processor 130 may provide only a warning alarm according to the level of the driver's condition or immediately switch to the autonomous driving mode simultaneously with the warning alarm.

As another example, the processor 130 may control the driving speed of the vehicle simultaneously with a warning alarm when the speed of the vehicle is greater than or equal to a threshold value. In this case, the processor 130 may control the traveling speed of the vehicle using an electronic control unit (ECU).

Meanwhile, the warning alarm may be provided in various forms such as a sound alarm, a haptic alarm, and a visual alarm (for example, strong lighting). For example, a sound alarm may be provided by outputting a human voice or a preset alarm sound through a speaker (a speaker of a car audio system, an AV system, a navigation device, or a telematics terminal). For example, a haptic alarm may be provided through a vibration device installed on a driver's seat or steering wheel. For example, a visual alarm may be provided by turning on an LED light installed inside the vehicle.

2 is a diagram for explaining a method for obtaining eye blink data according to an exemplary embodiment.

2 shows a method of acquiring eye blink data by calculating the horizontal and vertical aspect ratios of the eyeball using a distance threshold method (DTM) technique.

As shown in FIG. 2 , data representing eye blinking may be obtained based on the distance between each point of the eyeball region.

For example, as illustrated, corresponding points among a plurality of points p1, p2, p3, p4, p5, and p6 of the eyeball area are matched, and the horizontal and vertical aspect ratios of the eyeball are determined based on the distance between the matched points. can be calculated Equation 1 is a formula representing a method of calculating the horizontal and vertical aspect ratios of the eyeball according to an example.

Eye blink data may be obtained as shown in FIG. 2 based on the horizontal and vertical aspect ratios of the eyeball obtained based on Equation 1. In addition, it is possible to obtain an average value and a threshold value in a closed-eye state from eye-blink data.

3, 4, 5A and 5B are diagrams for explaining an EEG data analysis method according to an embodiment.

In general, human brain waves generate delta waves during deep sleep, theta waves during the process leading to sleep, that is, during drowsiness, and during rest or work without stress. When it can be processed, alpha brain waves are generated, and when excited or tense, beta waves are generated. Accordingly, it is possible to determine the driver's drowsiness state based on the fact that human brain waves generate brain waves representing different frequencies and different waveforms for each specific situation.

For example, as shown in FIG. 3 , the characteristic of closing the eyes at the moment when the driver becomes drowsy is detected, and a change occurs in the wave fluctuation of the brain wave. In this case, the EEG data may be analyzed using a Discrete Fouier Transform (DFT) technique.

For example, as shown in FIG. 4 , when the driver becomes drowsy, it can be confirmed that the brain waves of a specific frequency band are activated in the brain waves near the occipital lobes (regions O1 and O2). In addition, by distinguishing drowsiness in a normal state from drowsiness in a driving state, driving drowsiness characteristics may be extracted according to the PSD brain wave frequency band classification criteria as shown in Table 1 below. In addition, driver eye blink data may be extracted as an auxiliary index for determining driver drowsiness.

PSD EEG classification Frequency (Hz) Delta(δ) 0.5 to 4 Theta(θ) 4 to 8 Low-alpha (α _L ) 8 to 10 High-alpha (α _H ) 10 to 13 Beta (β) 13 to 30

5A shows EEG characteristics according to drowsiness in a normal state, and FIG. 5B shows EEG characteristics according to drowsiness in a driving state.

Referring to FIGS. 5A and 5B , it can be confirmed that alpha waves are activated in both the normal state and the driving state in a drowsy environment. However, it can be confirmed that changes in the characteristics of the high-alpha wave and the low-alpha wave activated due to the specificity of the driving environment are different.

For example, as shown in FIG. 5A, both low alpha waves and high alpha waves are activated in the general drowsy state, but as shown in FIG. can In addition, it can be confirmed that activation characteristics of other EEG data other than alpha waves change according to the drowsiness environment. For example, as shown in FIG. 5A , beta waves are deactivated during general drowsiness, but as shown in FIG. 5B , beta waves tend to be activated due to driving work in drowsiness under driving conditions. Accordingly, it is possible to determine drowsiness in a special environment of driving through relational analysis between EEG data.

Based on the analysis, PSD EEG sleepiness conditions can be defined as shown in Table 2 below.

PSD EEG sleepiness conditions (α _H + α _L )/ θ > 1 and α _H > α _L α _H > δ θ > δ α _H > β

Here, α _H may be a high-alpha wave, α _L may be a low-alpha wave, δ may be a delta wave, β may be a beta wave, and θ may be a theta wave.

Meanwhile, according to another example, not only PSD brain wave data and eye blink data, but also additional information may be used to determine whether or not a drowsy state exists. Here, the additional information may be various information such as driving environment information (eg, weather information, temperature information, humidity information, etc.), driver profile information (gender, age, etc.).

6A and 6B are diagrams for explaining a learning method of an artificial neural network model according to an embodiment.

According to an embodiment, the artificial neural network model 10 may be learned based on a pair of input training data and output training data or may be learned based on the input training data. Here, learning of the artificial neural network model means that a basic artificial neural network model (eg, an artificial neural network model including random parameters) is learned using a plurality of training data by a learning algorithm, so that desired characteristics (or, This means that a predefined action rule or an artificial neural network model set to perform the purpose) is created. Such learning may be performed through the vehicle device 100, but is not limited thereto and may be performed through a separate server and/or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the above examples. However, this is an example of supervised learning, and it goes without saying that an artificial neural network model can be trained based on unsupervised learning in which an artificial neural network model is trained by inputting only input data without using output data.

For example, an artificial neural network model may be composed of an input layer, a hidden layer, an out layer, and an activation function (f).

Here, the activation function (f) can be implemented as a linear, sigmoid, or hyperbolic tangent function (tanh), a Rectified Linear Unit (ReLU) function, or the like. For example, activation functions of the hidden layer and the out layer may be implemented as sigmoid and ReLU, respectively, but are not limited thereto.

According to an example, training data may be randomly mixed and used for learning an artificial neural network model. However, some of the training data may be used to verify the artificial neural network model. For example, 80% of the training data may be used for training and the remaining 20% for verification.

According to an example, training of an artificial neural network model may be performed in an external device such as a server. However, it is also possible that the learning of the artificial neural network model is performed in the vehicle device 100 itself. Hereinafter, for convenience of description, an embodiment in which the processor 130 of the vehicle device 100 performs learning of an artificial neural network model will be assumed and described.

According to an embodiment, the processor 130 maps a drowsiness state for each driver image for learning whose brain wave data satisfies a specific condition, and maps a non-drowsy state for each driver image for training whose brain wave data does not satisfy a specific condition. In this way, the artificial neural network model can be trained. Here, mapping may mean a pair of input training data (driver image for learning) and output training data (drowsy state or not) as shown in FIG. 6A, but as shown in FIG. 6B, labeled input training data (driver for training) It may also mean video-drowsy state).

According to an example, the processor 130 maps a drowsiness state for an image whose brain wave data satisfies a specific condition among driver images for training in which the time of closing the eyes is equal to or longer than a threshold time in a preset time interval, and maps the drowsiness state to the remaining images. By mapping the state, an artificial neural network model can be trained. Here, the preset time interval may be 5 seconds and the threshold time may be 3 seconds, but is not limited thereto. For example, the processor 130 slides a 5-second-sized image at 1-second intervals and classifies the driver's image for training based on whether eyes are closed for 3 seconds or more, but the numerical value is not necessarily limited thereto.

Here, the specific condition is that PSD (Power Spectral Density) EEG data satisfies (α _H + α _L )/ θ> 1 and α _H > α _L , α _H > δ, θ > δ and α _H > β can be Here, α _H may be a high-alpha wave, α _L may be a low-alpha wave, δ may be a delta wave, β may be a beta wave, and θ may be a theta wave.

Meanwhile, a method of acquiring EEG data and eye blink data used for learning is the same as/similar to the above-described method of obtaining EEG data and eye blink data. Accordingly, a detailed description of the corresponding configuration will be omitted. However, although the processor 130 provided in the vehicle device 100 has been described as the subject in the above-described embodiment and the following embodiment, learning of an artificial neural network model and/or acquisition of training data and related information used for learning is Of course, it can be performed in an external device.

According to an example, the processor 130 may label the driver's image for training based on at least one of PSD brain wave data and eye blink data.

For example, an artificial neural network model may be trained by labeling an image whose brain wave data satisfies a specific condition among driver images for training as a drowsy state and labeling remaining images as a non-drowsy state. Alternatively, among training driver images in which the time of closing the eyes in a predetermined time interval is greater than or equal to the critical time, an artificial neural network model is performed by labeling an image whose brain wave data satisfies a specific condition as a drowsy state and labeling the remaining images as a non-drowsy state. can be learned. Here, the preset time interval may be 5 seconds, and the threshold time may be 2 seconds or 3 seconds, but is not limited thereto.

7 is a diagram for explaining data for learning according to an exemplary embodiment.

According to an example, a driving simulation environment may be established, and biometric data and image data of a driver generated while driving in the driving simulation environment may be obtained and used for learning an artificial neural network model. In the driving simulation environment, devices for acquiring image data such as IR camera, normal camera, and depth camera, devices for acquiring biometric data such as EEG, EOG, EMG, ECG, RSP, PPG, GSR, and SKT, driving platforms, etc. Simulation equipment and the like may be used. According to an example, the driver's drowsiness condition derived from biometric data and image data may be labeled at the coordinates of the boundary box from which the driver's face is extracted. The size of the labeled driver image data may be converted into a pixel image having a predetermined size (eg, 416×416) and used as training data. 7 shows an example of acquired biometric data and image data of a driver.

However, it is not limited thereto, and the learning data can be acquired through various paths (eg, data servers) as long as the driver's image data and biometric data (eg, brain wave data) can be mapped.

8 is a diagram for explaining the operation of a learned artificial neural network model according to an embodiment.

As shown in FIG. 8 , the learned artificial neural network model 10 ′ may output information on whether or not a driver is in a drowsy state when an image taken by a driver is input.

According to an example, the trained artificial neural network model 10' may output a probability value corresponding to a drowsy state. For example, the processor 130 may determine whether the driver is in a drowsy state based on a probability value corresponding to the drowsy state output from the learned artificial neural network model 10'.

In this case, the output part of the artificial neural network model 10' may be implemented to enable softmax processing. Here, softmax is a function that normalizes all input values to a value between 0 and 1 and always makes the sum of output values 1. It outputs a probability value corresponding to each class, for example, drowsy state, non-drowsy state, etc. function can be In some cases, the output part of the artificial neural network model 10' may be implemented to enable Argmax processing. Argmax is a function that selects the most probable one among multiple labels, and here, it can function to select the ratio with the highest probability value for each class probability value. That is, when each output part of the artificial neural network model 10' is Argmax-processed, only state information (eg, drowsy state) having the highest probability value can be output.

Meanwhile, according to another embodiment, not only the driver's photographed image but also additional information may be used to determine the driver's drowsy state.

According to an example, additional information may be input to the artificial neural network model 10' together with the driver's photographed image, and the corresponding information may be used to output information on whether or not the driver is in a drowsy state. Here, the additional information may be various information such as driving environment information (eg, weather information, temperature information, humidity information, etc.), driver profile information (gender, age, etc.). However, when such additional information is applied, it goes without saying that the additional information should be used for learning of the artificial neural network model 10'.

9 is a diagram for explaining effects of the present disclosure according to an embodiment.

The artificial neural network model according to an embodiment of the present disclosure may be learned by inputting not only features of the driver's eyes and mouth but also the entire region of the driver's face as learning data. Accordingly, it is possible to determine whether the driver is drowsy regardless of the driver's posture and face angle. For example, as shown in FIG. 9 , it is possible to determine whether or not the driver is drowsy at various angles of the driver's face. In addition, a case of drowsy driving without closing the driver's eyes may be determined.

10 is a diagram illustrating a detailed configuration of a vehicle device according to an exemplary embodiment.

10, the vehicle device 100' includes a camera 110, a memory 120, a processor 130, an EEG measurement unit 140, a speaker 150, a user interface 160, and a communication interface 170. and a display 180 . Of the components shown in FIG. 10 , detailed descriptions of components overlapping those of FIG. 1 will be omitted.

The speaker 150 may be a component that outputs not only various types of audio data processed by the processor 130 but also various notification sounds or voice messages. According to one example, the processor 130 may control the speaker 150 to output a warning notification according to various embodiments of the present disclosure.

The communication interface 160 is a component for communicating with various external devices and may include a wireless communication module, for example, a Wi-Fi module, a Bluetooth module, and the like. However, it is not limited thereto, and the communication interface 160 may be used in addition to the above-described communication method such as zigbee, 3rd generation (3G), 3rd generation partnership project (3GPP), long term evolution (LTE), LTE-A (LTE Advanced), 4G (4th Generation), 5G (5th Generation), etc., various wireless communication standards, infrared communication (IrDA, Infrared Data Association) technology, etc. may perform communication. In addition, various wired communication interfaces (eg, USB terminals) may be included.

The user interface 170 is a component for receiving various user commands, and can be implemented as a button, a touch pad, a wheel, or the like according to an implementation example of the vehicle device 100'.

The display 180 may be implemented as a display including a self-light emitting element or a display including a non-light emitting element and a backlight. For example, LCD (Liquid Crystal Display), OLED (Organic Light Emitting Diodes) display, LED (Light Emitting Diodes), micro LED (micro LED), Mini LED, PDP (Plasma Display Panel), QD (Quantum dot) display , QLED (Quantum dot light-emitting diodes), etc. can be implemented in various types of displays. The display 130 may also include a driving circuit, a backlight unit, and the like that may be implemented in the form of an a-si TFT, a low temperature poly silicon (LTPS) TFT, or an organic TFT (OTFT). Meanwhile, the display 180 is implemented as a touch screen combined with a touch sensor, a flexible display, a rollable display, a 3D display, a display in which a plurality of display modules are physically connected, and the like. It can be. Also, since the display 180 has a built-in touch screen, a program may be executed using a finger or a pen (eg, a stylus pen).

Meanwhile, the vehicle device 100' may further include a microphone (not shown). The microphone is a component for receiving a user's voice or other sounds and converting them into audio data. For example, a user voice command related to various embodiments of the present disclosure may be received through a microphone (not shown).

Meanwhile, the vehicle device 100' may further include a microphone (not shown). The microphone is a component for receiving a user's voice or other sounds and converting them into audio data. For example, a user voice command related to various embodiments of the present disclosure may be received through a microphone (not shown).

11 is a flowchart illustrating a vehicle control method according to an exemplary embodiment.

According to the vehicle control method shown in FIG. 11 , power spectral density (PSD) brain wave data is obtained by processing the driver's brain wave data (S1110). In addition, eye blink data is acquired based on the eyeball data (S1120). However, the order of S1110 and S1120 is not limited thereto, and may be performed simultaneously or in a different order.

Then, based on the eye blink data, it is determined whether the driver has closed his/her eyes for a predetermined period of time or longer (S1130).

If it is determined that the driver has closed his/her eyes for a predetermined time or longer (S1130:Y), it is determined that the driver is in a drowsy state (S1140).

If it is determined that the driver has not closed his/her eyes for a predetermined time or more (S1130:N), it is determined whether the PSD brain wave data satisfies a specific condition (S750). Here, in this case, the specific conditions may be conditions satisfying (α _H + α _L )/θ> 1 and α _H > α _L , α _H > δ, θ > δ, and α _H > β. Here, α _H may be a high-alpha wave, α _L may be a low-alpha wave, δ may be a delta wave, β may be a beta wave, and θ may be a theta wave.

If it is determined that the PSD brain wave data satisfies a specific condition (S1150:Y), it is determined that the driver is in a drowsy state (S1140).

Thereafter, feedback corresponding to the drowsy state is provided (S1150).

In this case, in step S1120, eye blink data may be obtained from eye data by calculating horizontal and vertical aspect ratios of the eyeball using a distance threshold method (DTM) technique.

In addition, in step S1150, a warning alarm may be provided to the driver or the autonomous driving mode may be switched. Alternatively, in step S1150, it is also possible to provide a warning alarm to the driver and switch to an autonomous driving mode.

12 is a flowchart illustrating a vehicle control method according to another embodiment.

According to the control method of the vehicle device shown in FIG. 12 , first, it is determined whether the driver is in a drowsy state by inputting the driver's captured image acquired through the camera to the learned artificial neural network model (S1210).

Subsequently, when it is determined that the driver is drowsy based on the determination result of S1210, feedback corresponding to the drowsy state is provided (S1220).

Here, the artificial neural network model may be a model learned by taking as input data a plurality of driver images for learning and brain wave data corresponding to each of the plurality of driver images for learning, and using information on drowsiness as output data.

In addition, the control method maps the drowsiness state for each driver image for learning whose brain wave data satisfies a specific condition, and maps the non-drowsy state for each driver image for training whose brain wave data does not satisfy a specific condition, and then maps the artificial neural network model. It may further include the step of learning. Here, the specific condition is that PSD (Power Spectral Density) EEG data satisfies (α _H + α _L )/ θ> 1 and α _H > α _L , α _H > δ, θ > δ and α _H > β , α _H may be a high-alpha wave, α _L may be a low-alpha wave, δ may be a delta wave, β may be a beta wave, and θ may be a theta wave.

In addition, the control method maps the drowsiness state for driver images for learning in which the EEG data satisfies a specific condition among the driver images for learning in which the time of closing the eyes is equal to or longer than the threshold time in a preset time interval, and the drowsiness state is mapped for the remaining driver images for learning. The method may further include training an artificial neural network model by mapping a drowsy state.

In addition, in step S1210, it is possible to determine whether the driver is in a drowsy state by inputting the driver's image and additional information to the artificial neural network model. Here, the additional information may include at least one of driving environment information and driver profile information.

Meanwhile, in some cases, an EOG sensor for measuring the driver's eye conduction (EOG) and a PPG sensor for measuring the driver's photoplethysmogram (PPG) may be used as auxiliary indicators to determine drowsy driving.

In addition, in the above-described embodiment, only the case of identifying the drowsy state has been described, but in some cases, the driver's drowsy state as well as various states, such as inattention and tension, are based on the PSD brain wave data and/or eye blink data. , fatigue, or stress conditions, etc. may be detected and corresponding customized feedback may be provided based on the detection.

According to the various embodiments described above, it is possible to accurately determine the driver's condition and provide a customized stimulus or/and warning corresponding thereto. In addition, it is possible to determine whether the driver is drowsy or not from other facial features as well as the driver's eyes. Accordingly, it is possible to determine whether the driver is drowsy regardless of the driver's various postures and face angles. In addition, it is possible to determine whether the driver is drowsy even when he is dozing off without closing his eyes. Accordingly, the driver can be protected from danger.

Meanwhile, the methods according to various embodiments of the present disclosure described above may be implemented in the form of an application that can be installed in an existing vehicle device. Alternatively, the above-described methods according to various embodiments of the present disclosure may be performed using a deep learning-based artificial neural network (or deep artificial neural network), that is, a learning network model.

In addition, the above-described methods according to various embodiments of the present disclosure may be implemented only by upgrading software or hardware of an existing vehicle device.

In addition, various embodiments of the present disclosure described above may be performed through an embedded server included in the vehicle device or an external server of the vehicle device.

Meanwhile, according to an example of the present disclosure, the various embodiments described above may be implemented as software including instructions stored in a machine-readable storage medium (eg, a computer). can A device is a device capable of calling a stored command from a storage medium and operating according to the called command, and may include an electronic device (eg, the electronic device A) according to the disclosed embodiments. When a command is executed by a processor, the processor may directly or use other components under the control of the processor to perform a function corresponding to the command. An instruction may include code generated or executed by a compiler or interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' only means that the storage medium does not contain a signal and is tangible, but does not distinguish whether data is stored semi-permanently or temporarily in the storage medium.

Also, according to an embodiment of the present disclosure, the method according to the various embodiments described above may be included in a computer program product and provided. Computer program products may be traded between sellers and buyers as commodities. The computer program product may be distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)) or online through an application store (eg Play Store™). In the case of online distribution, at least part of the computer program product may be temporarily stored or temporarily created in a storage medium such as a manufacturer's server, an application store server, or a relay server's memory.

In addition, each of the components (eg, modules or programs) according to various embodiments described above may be composed of a single object or a plurality of entities, and some sub-components among the aforementioned sub-components may be omitted, or other sub-components may be used. Components may be further included in various embodiments. Alternatively or additionally, some components (eg, modules or programs) may be integrated into one entity and perform the same or similar functions performed by each corresponding component prior to integration. According to various embodiments, operations performed by modules, programs, or other components may be executed sequentially, in parallel, repetitively, or heuristically, or at least some operations may be executed in a different order, may be omitted, or other operations may be added. can

Although the preferred embodiments of the present disclosure have been shown and described above, the present disclosure is not limited to the specific embodiments described above, and is common in the technical field belonging to the present disclosure without departing from the gist of the present disclosure claimed in the claims. Of course, various modifications and implementations are possible by those with knowledge of, and these modifications should not be individually understood from the technical spirit or perspective of the present disclosure.

100: vehicle device 110: camera
120: memory 130: processor

Claims (10)

In the vehicle device,
camera;
It includes a plurality of electrode units, at least one electrode unit is disposed to detect EEG generated in the driver's temporal lobe and occipital lobe, and the other at least one electrode unit detects EEG generated in the frontal lobe and occipital lobe of the driver's brain. EEG measuring unit arranged to do so;
a memory storing at least one instruction; and
By executing the at least one command,
determining whether the driver is in a drowsy state based on the driver's captured image obtained through the camera;
a processor providing feedback corresponding to the drowsy state when it is determined that the driver is in the drowsy state; Including,
the processor,
By analyzing EEG data using a Discrete Fourier Transform (DFT) technique in consideration of the fact that the characteristic of closing the eyes is detected at the moment when the driver becomes drowsy and the wave fluctuation change of the EEG occurs, determining whether the driver is in the drowsy state based on the brain wave data of the driver and the eye data of the driver;
When the driver becomes drowsy, it is confirmed that an EEG of a specific frequency band is activated in an EEG near the occipital lobe through the plurality of electrode units;
The vehicle device, wherein the eye blink data is obtained from the eyeball data by calculating horizontal and vertical aspect ratios of the eyeball using a distance threshold method (DTM) technique. According to claim 1,
Processing the driver's brain wave data obtained through the brain wave measuring unit to obtain PSD (Power Spectral Density) brain wave data - where, based on the PSD brain wave data, it is possible to determine which frequency of the waveform is more in the waveform over time,
Obtaining the eye blink data based on the eyeball data of the driver obtained from the driver photographed image;
When it is determined that the driver has closed his/her eyes for a predetermined time or more based on the eye blink data, it is determined that the driver is in a drowsy state;
If the PSD brain wave data satisfies a specific condition when the driver does not close his or her eyes for a predetermined period of time or longer, it is determined that the driver is in the drowsy state. According to claim 1,
the memory,
Stores the trained artificial neural network model,
the processor,
The driver's captured image obtained through the camera is input to the artificial neural network model to determine whether the driver is in a drowsy state;
The artificial neural network model,
A vehicle device that is a model learned using a plurality of driver images for learning and brain wave data corresponding to each of the plurality of driver images for learning as input data and information on whether or not a drowsy state is used as output data. According to claim 3,
the processor,
Among the driver images for learning in which the time of closing the eyes in a preset time interval is longer than the critical time, the artificial neural network model is calculated by mapping the drowsy state to an image whose brain wave data satisfies a specific condition, and mapping the non-drowsy state to the remaining images. Teaching, vehicle device. According to claim 3,
the processor,
determining whether the driver is in a drowsy state by inputting the driver's image and additional information to the artificial neural network model;
The additional information above is
A vehicle device comprising at least one of driving environment information or driver profile information. delete According to claim 2 or 4,
The above specific conditions are
(α _H + α _L ) / θ > 1 and α _H > α _L , α _H > δ, θ > δ and α _H > β are satisfied,
α _H is a High-alpha wave, α _L is a Low-alpha wave, δ is a delta wave, β is a beta wave, and θ is a theta wave. In the control method of the vehicle device,
detecting brain waves generated in the temporal lobe and occipital lobe of the driver's brain, and detecting brain waves generated in the frontal lobe and the occipital lobe of the driver's brain, respectively;
determining whether the driver is in a drowsy state based on a photographed image of the driver obtained through a camera; and
providing feedback corresponding to the drowsy state when it is determined that the driver is in the drowsy state; Including,
The step of determining whether the driver is in the drowsy state,
By analyzing EEG data using a Discrete Fourier Transform (DFT) technique in consideration of the fact that the characteristic of closing the eyes is detected at the moment when the driver becomes drowsy and the wave fluctuation change of the EEG occurs, determining whether the driver is in the drowsy state based on the brain wave data of the driver and the eye data of the driver;
When the driver becomes drowsy, it is confirmed that an EEG of a specific frequency band is activated in the EEG near the occipital lobe;
A control method of obtaining the eye blink data from the eyeball data by calculating horizontal and vertical aspect ratios of the eyeball using a distance threshold method (DTM) technique. According to claim 8,
The step of determining whether the driver is in the drowsy state,
Processing the driver's EEG data to obtain PSD (Power Spectral Density) EEG data - where, based on the PSD EEG data, it is possible to determine which frequency of the waveform is more in the waveform over time,
Obtaining the eye blink data based on the eyeball data of the driver obtained from the driver photographed image;
When it is determined based on the eye blink data that the driver has closed his/her eyes for a predetermined time or longer, it is determined that the driver is in the drowsy state;
and determining that the driver is in the drowsy state if the PSD brain wave data satisfies a specific condition when the driver does not close his or her eyes for a predetermined period of time or longer. According to claim 8,
The step of determining whether the driver is drowsy,
Inputting the driver's photographed image to the trained artificial neural network model to determine whether the driver is in a drowsy state;
The artificial neural network model,
A control method, wherein the model is learned using a plurality of driver images for learning and brain wave data corresponding to each of the plurality of driver images for learning as input data and information on whether or not a drowsy state is used as output data.

Images