KR20200075556A - System for recognizing drowsiness of driver - Google Patents

Abstract

According to one embodiment of the present invention, provided is a drowsiness recognition system of a vehicle driver, which comprises: a learning step, including a step of photographing an eye image of a driver and storing the eye image in a central processing device, a step of extracting first features vector of the stored eye image of the driver, and a step of storing the extracted first features in a neuromorphic semiconductor module; after the learning step, a step of receiving the eye image of the driver through the central processing device and extracting a second feature vector; mapping the first feature vector and the second feature vector through a neurocell operation of the neuromorphic semiconductor module; deciding any one of the first feature vectors matching the second feature vector and determining the drowsiness recognition; and outputting the drowsiness to the driver if the drowsiness of the driver is determined when determining the drowsiness recognition.

Description

System for recognizing drowsiness of driver

One embodiment of the present invention relates to a drowsiness recognition system of a vehicle driver.

In general, a car provides mobility convenience and time efficiency to a person, but due to driver's carelessness, it can cause a great damage to a driver as well as to a nearby person. Therefore, recently, when a vehicle senses driver's drowsiness by an equipped device, the vehicle warns to prevent an accident due to driver's carelessness.

However, a system capable of informing a driver other than the light person through the drowsiness recognition of a conventional driver includes various algorithms for recognizing actual drowsiness with information that senses the shape of the driver's eyes, and high specifications for increasing the recognition rate of the corresponding image In the case of the central processing unit and recently introduced deep learning method, there are practical difficulties in installing and using it inside the vehicle due to the physical devices, networks, and complicated computational operations required for high-speed computation, and to realize accurate reliability. As high specification and high power are required, there is a problem that can greatly affect the driving system of the vehicle itself.

KR 100778059 B1

An object according to an embodiment of the present invention is to learn a vehicle driver's eye image while driving, extract a feature vector for it, store it in a neuromorphic semiconductor module, and then input the feature vector of the eye image through neurocell computation. It is to provide a vehicle driver drowsiness recognition system that can realize effective driver drowsiness recognition through a driver's eye image input in real time by mapping quickly.

In addition, reliability of the drowsiness recognition of the vehicle driver can be improved more effectively by mapping the vehicle driver's eye image input during driving and the feature vector of the pre-stored eye image, and combining the vehicle driving information monitored in real time.

In addition, the feature vector of the vehicle driver's eye image input during driving and the feature vector of the pre-stored driver's eye image are mapped through neurocell computation in the neuromorphic semiconductor module to effectively increase the processing speed and to achieve a low-power and low-spec system structure. Even a stable driver's drowsiness recognition system can be effectively driven.

The drowsiness recognition system of a vehicle driver according to an embodiment of the present invention includes photographing a driver's eye image and storing it in a central processing unit, extracting a first feature vector of the stored driver's eye image, and the extracted Storing a first feature vector in a neuromorphic semiconductor module; a learning step comprising, after the learning step, receiving a driver's eye image through a central processing unit and extracting a second feature vector; Mapping the feature vector and the second feature vector through neurocell computation of the neuromorphic semiconductor module, determining one of the first feature vectors matching the second feature vector, and then determining whether drowsiness is recognized or not. And determining the driver's drowsiness when determining whether or not the drowsiness is recognized, and outputting it to the driver.

Here, the step of mapping the first feature vector and the second feature vector through neurocell operation of the neuromorphic semiconductor module may include calculating a distance value between the second feature vector and the first feature vector, The method may further include repeatedly performing the step of extracting and storing the first feature vector at the closest distance.

Further, after determining any one of the first feature vectors matching the second feature vector, determining whether drowsiness is recognized may include: a plurality of first feature vectors matching the second feature vectors continuously input sequentially; Based on the basis, it may further include the step of determining whether or not the sleepiness is recognized based on the data of the period and the number of times the blindness per unit time.

Further, after determining any one of the first feature vectors matching the second feature vector, determining whether drowsiness is recognized may include: a plurality of first feature vectors matching the second feature vectors continuously input sequentially; The method may further include determining whether sleepiness is recognized by matching the data of the number of times of blinding per unit time and the number of times of blinding, driving time information of the vehicle input from the central processing unit, and lane departure information of the vehicle.

In addition, after determining one of the first feature vectors matching the second feature vector, in determining whether to recognize drowsiness, a plurality of first features corresponding to the sequentially input second feature vector values According to the vector, if it is determined that the number of times per second is less than 1 time and the period of the eyes closed is 500 ms or less, and if it is determined that the driver is drowsy when determining whether the drowsiness is recognized, the step of outputting it to the driver is a warning step. And outputting a message or an alarm.

In addition, after determining one of the first feature vectors matching the second feature vector, in determining whether to recognize drowsiness, a plurality of first feature vectors corresponding to the sequentially input second feature vectors According to the case, when the number of times of blindness per second is less than 1 time and the period of blindness is less than or equal to 300 ms, and the driving time information of the vehicle is determined to be 2 hours or more, when it is determined whether the drowsiness is recognized, the driver is determined to be drowsy , The step of outputting this to the driver may include the step of outputting a message or alarm of the warning step.

In addition, after determining one of the first feature vectors matching the second feature vector, in determining whether to recognize drowsiness, a plurality of first features corresponding to the sequentially input second feature vector values Depending on the vector, the number of times per second is less than 1 times, the period of the eyes is less than 300 ms, the driving time information of the vehicle is 1 hour or less, and the lane departure information of the vehicle is 2 times or less, or sequentially input. According to a plurality of first feature vectors corresponding to the second feature vector, the number of times per second is less than 2 times, the period of eyes is less than 300 ms, the driving time information of the vehicle is 2 hours or more, and the lane of the vehicle When it is determined that the departure information is two or less times, when it is determined that the driver is drowsy when determining whether the drowsiness is recognized, the step of outputting it to the driver may further include outputting a message or alarm of the caution step. have.

In addition, the learning step, the neuromorphic semiconductor module includes a plurality of neuromorphic semiconductor modules stacked, and the step of storing the extracted first feature vector in the neuromorphic semiconductor module is the first feature The method may further include subdividing the first feature vector into a plurality of first detailed feature vectors so as to correspond to the stacked plurality of neuromorphic semiconductor modules.

Further, after determining one of the first feature vectors matching the second feature vector, determining whether or not the drowsiness recognition is performed includes: the first feature vector and the second feature for determining whether the drowsiness is recognized; The method may further include storing a second feature vector extracted by mapping a vector through a neurocell operation of the neuromorphic semiconductor module as a corresponding first feature vector of the learning step.

Features and advantages of the present invention will become more apparent from the following detailed description based on the accompanying drawings.

Prior to this, the terms or words used in the specification and claims should not be interpreted in a conventional and lexical sense, and the inventor may properly define the concept of terms in order to best describe his or her invention. Based on the principle of being able to be interpreted as meanings and concepts consistent with the technical idea of the present invention.

According to an embodiment of the present invention, a feature vector of a specific driver's eye image of a vehicle is stored in a neuromorphic semiconductor module through a learning process of extracting a feature vector of the vehicle driver's eye image, and the vehicle is input after learning. By quickly mapping the feature vector of the eye image in the neuromorphic semiconductor module through neurocell computation, it is possible to improve the processing speed and obtain stable output results even in a low-power, low-spec environment.

In addition, by extracting and mapping the feature vector of the vehicle driver's eye image and the driving information of the vehicle input in real time, it is possible to recognize the actual vehicle driver's drowsiness and output a warning to the vehicle driver's drowsiness. have.

In addition, the feature vector obtained from the vehicle driver's eye image can be further subdivided and stored through the stacked structure of the neuromorphic semiconductor module, thereby effectively improving the reliability of the mapping result with the feature vector of the input vehicle driver's eye image. It has the effect.

In addition, as the design freedom of the device environment or conditions according to the real-time processing speed of the mapping stage of the feature vector of the image for determination of drowsiness increases, the compatibility of the system connection with the existing equipment is increased, as well as the technical compatibility of vehicle application. There is also an effect that can improve the driving reliability.

1 is a flow chart for an example of a drowsiness recognition system of a vehicle driver according to an embodiment of the present invention and
2 is a flow chart according to another example of a drowsiness recognition system of a vehicle driver according to an embodiment of the present invention.

The objects, specific advantages and novel features of the present invention will become more apparent from the following detailed description and preferred embodiments that are associated with the accompanying drawings. It should be noted that in this specification, when adding reference numerals to components of each drawing, the same components have the same number as possible, even if they are displayed on different drawings. In addition, the terms "one side", "other side", "first", "second", etc. are used to distinguish one component from another component, and the component is limited by the terms no.

Hereinafter, in describing one embodiment of the present invention, detailed descriptions of related well-known technologies that may unnecessarily obscure the subject matter of the present invention will be omitted.

1 is a flow chart for an example of a drowsiness recognition system of a vehicle driver according to an embodiment of the present invention.

Drowsiness recognition system of a vehicle driver according to an embodiment of the present invention comprises the steps of photographing and storing a driver's eye image in a central processing unit (S10), and extracting a first feature vector of the stored driver's eye image ( S20) and a learning step comprising the step of storing the extracted first feature vector in a neuromorphic semiconductor module (S30) and a second feature after receiving the driver's eye image through the central processing unit after the learning step Extracting a vector (S40), mapping the first feature vector and the second feature vector through neurocell operation of the neuromorphic semiconductor module (S50), the agent matching the second feature vector After determining any one of the feature vectors, determining whether or not the drowsiness is recognized (S60), and when determining whether the driver is drowsiness when determining whether the drowsiness is recognized, outputting it to the driver (S70).

As shown in FIG. 1, the drowsiness recognition system of a vehicle driver according to an embodiment of the present invention is a step of first capturing an image of the driver's eye and storing it in the central processing unit.

First, a learning step for obtaining an eye image of a specific person of a vehicle driver is performed. In-vehicle navigation The vehicle's driver's eye image is acquired and stored while driving through a photographing device such as a normally equipped camera (S10). The scope of learning in the learning phase is not particularly limited, but the learning phase can be terminated by acquiring an image that can extract the characteristic image of the driver's eyes, such as mileage or driving time, and the vehicle that can be progressed later. It goes without saying that the image storage of the learning stage can be cyclically performed simultaneously in the data processing process for the new input image detected in the mapping stage for driver's drowsiness recognition.

When the eye image of the vehicle driver is photographed and stored in this way, the first feature vector for the eye image is extracted (S20). The extraction of the first feature vector for the eye image is the basic data for the mapping process with the feature vector of the subsequently input image by closing or opening the eye for each image or giving features for various types of eyes. The value is stored in the neuromorphic semiconductor module. Here, the extraction of the first feature vector for the eye image means that the neurocell in the neuromorphic semiconductor module is converted into vectorization capable of operation. After performing the pre-processing step of performing such vectorization in the central processing unit, the pre-processed first feature vector is stored in the neurocell for the neurocell calculation of the neuromorphic semiconductor module (S30). Since the input eye image is pre-processed in the central processing unit, there is no need for a separate network device or the use of wireless data, and thus there is an advantage that sufficient performance can be performed even in the low-power mode of the low-spec central processing unit.

Next, the step (S30) of storing the first feature vector pre-processed by the central processing unit in the neurocell of the neuromorphic semiconductor module. Subsequently, after the learning process is completed, it is possible to obtain an image of a vehicle driver's eye and to calculate a neurocell for mapping with the acquired second feature vector.

After the learning process is finished, a second feature vector is extracted by extracting a feature of the newly input vehicle driver's eye image (S40). This process can also be performed in the same way as the operation of the central processing unit in the learning process, and extracts the features of the vehicle driver's eye image input for drowsiness recognition to form a second feature vector.

Next, in step S50, the pre-stored first feature vector and the newly input second feature vector in the neuromorphic semiconductor module are mapped in parallel through a neurocell operation. This step is not a separate algorithm or a process that requires high power and high specification such as conventional deep learning, but the first feature vector is pre-stored by quickly mapping the first feature vector and the second feature vector for the feature of the image. Among them, the first feature vector with high matching is selected. At this time, the mapping operation is performed by calculating a distance value between the input second feature vector and the previously stored first feature vector to extract the first feature vector having the closest distance.

Next, after determining one of the first feature vectors matching the second feature vector, it is determined whether the vehicle driver is drowsy (S60). As the driver's eye images are captured and stored in real time while driving the vehicle, the second feature vector and the first feature on the neuromorphic semiconductor module are input as the second feature vectors extracting the features of each image are sequentially input in time order. The real-time mapping of the vector is done quickly, and the mapping results are sequentially derived. At this time, as the matched first feature vectors are sequentially stored, the number of times the eyes are closed per unit time (millisecond; ms) through these first feature vectors, and the cycle of closing the eyes (ie, the time difference of blinking) Can be extracted. Through the extracted data, a step-by-step display of whether or not the driver of a vehicle is drowsy can be used as an objective judgment criterion according to the following practical simulation results.

Lastly, if it is determined that the driver is drowsiness when determining whether the drowsiness is recognized, it is a step of outputting it to the driver (S70).

By effectively combining the features of the driver's eye images respectively recognized on the pre-stored first feature vector, each data result value is obtained. According to an embodiment of the present invention, when the number of times of closing the eyes per second is less than 1 through the various simulations of the corresponding system and the period of closing the eyes is determined to be 500 ms (millisecond) or less, the possibility of drowsiness of the vehicle driver is high. The output message is output in a form that can be recognized by the vehicle driver on the output message or output warning sound or other display.

2 is a flow chart according to another example of a drowsiness recognition system of a vehicle driver according to an embodiment of the present invention.

The drowsiness recognition system of a vehicle driver according to an embodiment of the present invention according to FIG. 2 considers whether the vehicle driver is aware of drowsiness recognition by the central processing device in the vehicle or the lane departure information of the vehicle in the above description By doing so, it is possible to further increase the accuracy and reliability of the vehicle driver's drowsiness recognition judgment.

According to Figure 2, the step of storing the driver's eye image in the central processing unit (S100), extracting the first feature vector of the stored driver's eye image (S200) and the extracted first feature vector A learning step comprising the step of storing in a neuromorphic semiconductor module (S300), after the learning step, receiving a driver's eye image through a central processing unit and extracting a second feature vector (S400), the second Mapping a feature vector and the second feature vector through neurocell operation of the neuromorphic semiconductor module (S500), after determining one of the first feature vectors matching the second feature vector, drowsiness It includes a step of determining whether to recognize (S600) and, if it is determined as the driver's drowsiness when determining whether the drowsiness is recognized, outputting it to the driver (S800).

After determining any one of the first feature vectors matching the second feature vector, in determining whether to recognize drowsiness, a plurality of first feature vector values matching the second continuously input second feature vector values The method may further include determining whether drowsiness is recognized by additionally matching the driving time information of the vehicle and the lane departure information of the vehicle input from the central processing unit in addition to the data of the period of closing and closing the number of eyes per unit time (S700). have.

Specifically, another example of the drowsiness recognition system of a vehicle driver according to an embodiment of the present invention is information of a vehicle state input in real time from a central processing unit in a vehicle, that is, driving time of a vehicle, lane departure due to vehicle driving By adding an additional calculation process to the result derived by the mapping process of the first feature vector and the second feature vector performed in the neuromorphic semiconductor module, it is possible to discriminate and output whether the vehicle driver is drowsy or more reliable. It is.

That is, according to a plurality of first feature vector values corresponding to the sequentially input second feature vector values, the number of times of blinding per second is 1 or less, and the period of blinding is 300 ms or less, and driving time information of the vehicle If it is determined that it is 2 hours or more (S700), an alarm of the'warning stage' or other output of the vehicle driver's recognizable form (S800) according to whether the vehicle driver is drowsiness may be emitted in the output stage.

In addition, although it is lower risk than the'warning stage', it is possible to output a'attention stage' for attention to drowsiness of the vehicle driver. That is, according to a plurality of first feature vector values corresponding to the sequentially input second feature vector values, the number of times of blinding per second is 1 or less, and the period of blinding is 300 ms or less, and driving time information of the vehicle Is 1 hour or less and the lane departure information of the vehicle is 2 times or less, or according to a plurality of first feature vector values corresponding to the sequentially input second feature vector values, the number of times the eyes are closed per second is 2 times or less When the eye is judged to have a period of 300 ms or less, the driving time information of the vehicle is 2 hours or more, and the vehicle's lane departure information is 2 times or less, the'caution level' that is lower than the warning level is given to the vehicle driver. By outputting, the risk of drowsiness can be effectively prevented.

The drowsiness recognition system of a vehicle driver according to an embodiment of the present invention stacks a plurality of neuromorphic semiconductor modules in order to perform mapping through extraction of more detailed images with various drivers for calculating neurocells of a neuromorphic semiconductor module This is made possible by implementing a parallel structure of neurocell computation by implementing it as a structure.

That is, by subdividing the first feature vector into a plurality of first detailed feature vectors so as to correspond to the stacked plurality of neuromorphic semiconductor modules and re-storing, the first feature vector By storing the subdivided feature vector and a larger amount of the first feature vector, it is possible to effectively improve the reliability of the vehicle driver's drowsiness determination according to the mapping process with the second feature vector according to the actually input eye image. .

In addition, the second feature vector for the eye image feature input to determine whether the vehicle driver is drowsy is circulated and stored together in the pre-processing process of the corresponding first feature vector by the central processing unit, after the learning step ends, This system is used by storing the corresponding first feature vector or more subdivided or expanded similar images of the learning stage in the central processing unit by feeding back the input image of the result of the mapping process for drowsy judgment of the vehicle driver. In the process, the accuracy and reliability of system information can be doubled.

The present invention has been described in detail through specific embodiments, but this is for specifically describing the present invention, and the drowsiness recognition system of a vehicle driver according to the present invention is not limited to this, and within the technical spirit of the present invention It will be apparent that modification or improvement is possible by a person having ordinary knowledge. All simple modifications and variations of the present invention belong to the scope of the present invention, and the specific protection scope of the present invention will become apparent by the appended claims.

Claims (9)

Photographing the driver's eye image and storing it in the central processing unit;
Extracting a first feature vector of the stored driver's eye image; And
A step of storing the extracted first feature vector in a neuromorphic semiconductor module;
After the learning step, receiving a driver's eye image through a central processing unit and extracting a second feature vector;
Mapping the first feature vector and the second feature vector through neurocell operation of the neuromorphic semiconductor module;
Determining whether one of the first feature vectors matches the second feature vector, and then determining whether drowsiness is recognized; And
When determining whether the drowsiness is recognized, if it is determined as the driver's drowsiness, outputting it to the driver, the vehicle driver's drowsiness recognition system. The method according to claim 1,
The mapping of the first feature vector and the second feature vector through neurocell operation of the neuromorphic semiconductor module may include:
And calculating a distance value between the second feature vector and the first feature vector, and repeatedly performing the step of extracting and storing the first feature vector of the closest distance. The method according to claim 2,
After determining any one of the first feature vectors matching the second feature vector, determining whether drowsiness is recognized,
A drowsiness recognition system for a vehicle driver further comprising determining whether drowsiness is recognized based on data of the number of closed eyes per unit time and the number of closed eyes based on a plurality of first feature vectors matching the second feature vectors continuously input in sequence. . The method according to claim 3,
After determining any one of the first feature vectors matching the second feature vector, determining whether drowsiness is recognized,
Based on a plurality of first feature vectors matched with the sequentially input second feature vectors, the data of the number of closed periods and the number of closed eyes per unit time, the driving time information of the vehicle input from the central processing unit and the lane departure information of the vehicle Further comprising the step of determining whether the sleepiness recognition by matching the drowsiness recognition system of the vehicle driver. The method according to claim 3,
After determining one of the first feature vectors matching the second feature vector, in determining whether to recognize drowsiness,
According to a plurality of first feature vectors corresponding to the sequentially input second feature vectors, if the number of times per second is less than 1 times and the period of eyes closed is determined to be 500 ms or less,
When determining whether the drowsiness is recognized, when it is determined that the driver is drowsiness, outputting it to the driver includes outputting a warning message or an alarm of the vehicle driver. The method according to claim 4,
After determining one of the first feature vectors matching the second feature vector, in determining whether to recognize drowsiness, according to a plurality of first feature vectors corresponding to the sequentially input second feature vectors , If the number of times per second is less than 1 times, the period of eyes is less than 300 ms, and the driving time information of the vehicle is determined to be 2 hours or more,
When determining whether the drowsiness is recognized, when it is determined that the driver is drowsiness, outputting it to the driver includes outputting a warning message or an alarm of the vehicle driver. The method according to claim 4,
After determining any one of the first feature vectors matching the second feature vector, in determining whether to recognize drowsiness, a plurality of first feature vector values corresponding to the sequentially input second feature vectors are determined. Accordingly, the number of times of eye per second is less than or equal to 1 time, and the period of eye closing is 300 ms or less, the driving time information of the vehicle is 1 hour or less, and the lane departure information of the vehicle is 2 times or less,
According to a plurality of first feature vectors corresponding to the sequentially inputted second feature vectors, the number of times per second is less than or equal to 2 times, the period of eyes is less than 300 ms, and the driving time information of the vehicle is 2 hours or more. And when it is determined that the lane departure information of the vehicle is 2 times or less,
If it is determined that the driver is drowsiness when determining whether the drowsiness is recognized, the step of outputting it to the driver is:
Drowsiness recognition system of a vehicle driver further comprising the step of outputting a message or alarm of the caution level. The method according to claim 1,
The learning step,
The neuromorphic semiconductor module includes a plurality of neuromorphic semiconductor modules stacked,
In the step of storing the extracted first feature vector in a neuromorphic semiconductor module, the first feature vector may include a plurality of first features such that the first feature vector corresponds to the stacked plurality of neuromorphic semiconductor modules. Drowsiness recognition system of a vehicle driver further comprising the step of subdividing into a detailed feature vector and re-storing. The method according to claim 1,
After determining any one of the first feature vectors matching the second feature vector, determining whether drowsiness is recognized,
The second feature vector extracted by mapping the first feature vector and the second feature vector through neurocell operation of the neuromorphic semiconductor module to determine whether the drowsiness is recognized corresponds to the learning step Drowsiness recognition system of a vehicle driver further comprising the step of storing as a first feature vector.

Images