KR102596957B1 - Multi sensor fusion-based driver monitoring device and method - Google Patents

Abstract

The present invention relates to a driver monitoring device and method based on multi-sensor fusion. The driver monitoring device based on multi-sensor fusion according to the present invention includes an information collection unit that collects front image information and biometric information of the driver; a preprocessing unit that preprocesses the front image information and biometric information; a shape vector generator that extracts each feature vector based on the front image information and biometric information preprocessed in the preprocessor and combines the feature vectors to generate a shape vector; a determination unit that determines the driver's emotions and actions based on the shape vector; and an output unit that outputs the driver's emotions and actions. It monitors and outputs the driver's emotions and actions so that the efficiency of the autonomous vehicle can be increased.

Description

Multi-sensor fusion-based driver monitoring device and method {MULTI SENSOR FUSION-BASED DRIVER MONITORING DEVICE AND METHOD}

The present invention relates to a driver monitoring device and method based on multi-sensor fusion, and more specifically, to a driver monitoring device and method based on multi-sensor fusion that determines the driver's emotions and activities through the driver's front image and biological signals.

Driver Monitoring Systems (DMS), conventionally known as driver attention monitors, play a very important role in autonomous vehicles.

The driver monitoring system acquires images of the driver through hardware devices such as cameras and biometric recognition devices installed inside the vehicle, and determines the driver's actions such as facial recognition, fatigue detection, forward looking detection, and gesture detection. Based on this, the driver's emotions are judged.

The driver's emotions or behavior are used as information that is very helpful in driving safely in autonomous vehicles and improving the efficiency of autonomous vehicle driving, and the driver monitoring system processes the driver's emotions or behavior in real time to efficiently implement autonomous driving. Technology that can be performed without delay is required.

However, technology that performs the driver's emotions or actions without information processing is not free from errors that occur in the process of judging the driver's actions or emotions, and errors occur frequently in the autonomous driving system implemented in real time. do.

Therefore, there is a need for research and development on technology that is free from classification and implementation errors and determines the driver's emotions or behavior in real time.

(Republic of Korea) Registered Patent Publication No. 10-1241841

The present invention was created to solve the above problems, and the purpose of the present invention is to provide a driver monitoring device and method based on multi-sensor fusion that reduces classification errors and simultaneously predicts the driver's emotions and behavior.

In order to achieve the above object, a multi-sensor fusion-based driver monitoring device according to an embodiment of the present invention includes an information collection unit that collects front image information and biometric information of the driver; a preprocessing unit that preprocesses the front image information and biometric information; a shape vector generator that extracts each feature vector based on the front image information and biometric information preprocessed in the preprocessor and combines the feature vectors to generate a shape vector; a determination unit that determines the driver's emotions and actions based on the shape vector; and an output unit that outputs the driver's emotions and actions.

Here, the information collection unit includes an image collection unit that collects front image information of the driver; and a biometric collection unit that collects the driver's biometric information, including a heart rate collection unit that collects the driver's heart rate information and a voice collection unit that collects the driver's voice information.

Accordingly, the preprocessor sets the driver's area of interest in the front image information, and converts the front image information set as the area of interest into foreground extraction information, front feature point information, and front heat map information, which are present in the heart rate information. Heart rate filtering can be performed to remove noise signals present in the voice information, and voice filtering can be performed to remove noise signals present in the voice information.

Accordingly, the feature vector generator removes noise signals from the foreground extraction information, the front feature point information, the front heatmap information, and the biometric information converted from the front image information using a feature vector extractor that performs vector combining. Based on the heart rate information and the voice information, each feature vector may be extracted and each feature vector may be combined to generate one shape vector.

A multi-sensor fusion-based driver monitoring method according to another embodiment of the present invention is performed using a multi-sensor fusion-based driver monitoring device, and includes an information collection step of collecting front image information and biometric information of the driver; A preprocessing step of preprocessing the front image information and biometric information; A shape vector generation step of extracting each feature vector based on the preprocessed front image information and biometric information and combining the feature vectors to generate a shape vector; a determination step of determining the driver's emotions and actions based on the shape vector; and an output step of outputting the driver's emotions and actions.

Here, the information collection step includes collecting front image information of the driver; and collecting the biometric information including the driver's heart rate information and the driver's voice information.

Accordingly, the preprocessing step includes setting the driver's area of interest in the front image information; Converting the front image information set as the region of interest into foreground extraction information, front feature point information, and front heat map information; and performing filtering to remove noise signals present in the heart rate information and the voice information included in the biometric information.

Accordingly, the shape vector generation step uses a feature vector extractor that performs vector combining to extract the foreground extraction information, the front feature information, the front heat map information, and the heart rate information and the voice included in the biometric information. extracting each feature vector based on information; and combining the respective feature vectors to generate one shape vector.

According to one aspect of the present invention described above, by providing a driver monitoring device and method based on multi-sensor fusion, it is possible to solve the problems of combining errors that occur in the process of combining driver information measured by different devices and to recognize the driver's emotions and behavior. Recognition error problems that occur can be resolved.

In addition, by providing a driver monitoring device and method based on multi-sensor fusion, driver convenience and efficiency of autonomous driving can be increased.

Figure 1 is a block diagram of a multi-sensor fusion-based driver monitoring device according to an embodiment of the present invention.
FIG. 2 is a block diagram of the information collection unit of FIG. 1.
FIG. 3 is a diagram of the multi-sensor fusion-based driver monitoring device of FIG. 1.
FIG. 4 is a diagram illustrating preprocessing for converting front image information into foreground extraction information in the preprocessor of FIG. 1 .
FIG. 5 is a diagram illustrating preprocessing for converting front image information into front feature information in the pre-processing unit of FIG. 1.
Figure 6 is a flow diagram of a driver monitoring method based on multi-sensor fusion according to another embodiment of the present invention.
FIG. 7 is a flow diagram of the information collection step of the multi-sensor fusion-based driver monitoring method of FIG. 6.
Figure 8 is a flow diagram of the preprocessing step of the multi-sensor fusion-based driver monitoring method of Figure 6.

The detailed description of the present invention described below refers to the accompanying drawings, which show by way of example specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the present invention are different from one another but are not necessarily mutually exclusive. For example, specific shapes, structures and characteristics described herein may be implemented in one embodiment without departing from the spirit and scope of the invention. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the detailed description that follows is not intended to be taken in a limiting sense, and the scope of the invention is limited only by the appended claims, together with all equivalents to what those claims assert, if properly described. Similar reference numbers in the drawings refer to identical or similar functions across various aspects.

The terminology used herein is for descriptive purposes only and is not intended to constitute the invention. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context. As used in the specification, “comprises” and “comprising” do not exclude the presence or addition of one or more other elements in addition to the mentioned elements. Throughout the specification, like reference numerals refer to like elements, and “and/or” includes each and every combination of one or more of the referenced elements, although “first”, “second”, etc. may refer to various elements. It is used to describe the components, but of course, these components are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, the first components mentioned below Of course, the component may be a second component within the technical spirit of the present invention.

Unless otherwise defined, all terms used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly specifically defined.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

First, a multi-sensor fusion-based driver monitoring device according to an embodiment of the present invention will be described through FIGS. 1 to 5.

Figure 1 is a block diagram of a multi-sensor fusion-based driver monitoring device 100 according to an embodiment of the present invention.

Referring to FIG. 1, the multi-sensor fusion-based driver monitoring device 100 according to an embodiment of the present invention includes an information collection unit 110, a preprocessing unit 130, a shape vector generation unit 150, and a determination unit 170. ) and an output unit 190.

The multi-sensor fusion-based driver monitoring device 100 according to an embodiment of the present invention may be a part of a module of the vehicle control system or a separate terminal to be implemented as a vehicle option function. In addition, the information collection unit 110, preprocessing unit 130, shape vector generation unit 150, determination unit 170, and output unit 190 may be formed as an integrated module or may be composed of one or more modules. there is. However, on the contrary, each component may be comprised of a separate module.

Additionally, the multi-sensor fusion-based driver monitoring device 100 may be mobile or fixed. For example, the multi-sensor fusion-based driver monitoring device 100 may be in the form of a server or engine, and may include a device, apparatus, terminal, user equipment (UE), It may be called by other terms such as MS (mobile station), wireless device, or handheld device.

In addition, the multi-sensor fusion-based driver monitoring device 100 can run or produce various software based on an operating system (OS), that is, a system. Here, the operating system is a system program that allows software to use the hardware of the device, and includes mobile computer operating systems such as Android OS, Windows Mobile OS, Bada OS, Symbian OS, and Blackbeard OS, as well as Windows series, Linux series, Unix series, MAC, It can include all computer operating systems such as AIX and HP-UX.

Figure 2 is a block diagram of the information collection unit 110 included in the multi-sensor fusion-based driver monitoring device 100 according to an embodiment of the present invention.

First, the information collection unit 110 collects the driver's front image information and biometric information.

At this time, the information collection unit 110 may collect the driver's face image captured by any one of the camera, vision sensor, and motion recognition sensor provided in the vehicle as the driver's front image information. To this end, the information collection unit 110 may include an image collection unit 111 that collects front image information of the driver.

Additionally, the information collection unit 110 may further include a biometric collection unit 113 that collects the user's biometric information. At this time, the biometric collection unit 113 may include a heart rate collection unit 11a that collects the user's heart rate information and a voice collection unit 113b that collects the user's voice information.

The heart rate collection unit 113a included in the biometric collection unit 113 is measured by a heart rate measuring means, either a smart watch worn by the driver or a heart rate measuring device capable of measuring the driver's heart rate. The driver's heart rate is collected as heart rate information, and the voice collection unit 113b can collect the driver's voice measured by any one voice recognition means among the microphone, voice input/output device, and voice recognition sensor provided in the vehicle as voice information. there is. Here, the smart watch that measures the driver's heart rate is preferably used as a watch-type device that can measure the driver's heart rate, such as the Galaxy Watch series or Apple Watch series that exist in the smart watch market, but is not limited to this.

Figure 3 is a diagram of a driver monitoring device based on multi-sensor fusion according to an embodiment of the present invention.

Referring to FIG. 3, when the information collection unit 110 collects the driver's front image information and biometric information, the preprocessing unit 130 preprocesses the collected driver's front image information and biometric information.

First, the pre-processing unit 130 may set a region of interest (ROI) in the front image information collected by the information collection unit 110.

Accordingly, the pre-processing unit 130 may select some areas included in the front image information according to the front image information collected by the information collection unit 110 and set the selected areas as areas of interest.

The pre-processing unit 130 is provided as a single module and simultaneously performs pre-processing of the driver's front image information and biometric information, or is provided as a plurality of pre-processing modules and simultaneously performs pre-processing of the front image information and biometric information in each pre-processing module. It can be converted into different information. Accordingly, in the specification of the present invention, for ease of explanation, the preprocessing unit 130 included in the multi-sensor fusion-based driver monitoring device 100 according to an embodiment of the present invention is described as being provided as a plurality of preprocessing modules. , but is not limited to this.

Accordingly, although not shown in FIG. 3, the preprocessing unit 130 may include a first preprocessing module for preprocessing front image information and a second preprocessing module for preprocessing the driver's biometric information.

First, the process in which the first pre-processing module pre-processes the driver's front image information collected by the information collection unit 110 and in which the area of interest is set will be described with reference to FIGS. 4 and 5.

FIG. 4 is a diagram illustrating preprocessing for converting front image information into foreground extraction information in the preprocessor 130 of FIG. 1 .

Referring to FIG. 4, the first pre-processing module included in the pre-processing unit 130 may pre-process the driver's front image information collected in the information collection unit 110 and set as a region of interest and convert it into foreground extraction information.

In this regard, the first preprocessing module may generate a data set based on the driver's front image information in which the area of interest is set.

At this time, the first preprocessing module may set the area set as the area of interest in the generated data set as the driver's face area.

Additionally, the first preprocessing module may set a region that is not set as a region of interest in the generated data set as a foreground region of the front image information.

Accordingly, the first preprocessing module may extract the foreground area included in the data set and not set as the area of interest using a foreground extraction algorithm. Here, the first pre-processing module can extract the foreground from the driver's front image information using the GrabCut algorithm as a foreground extraction algorithm, and for this purpose, the first pre-processing module may be provided with the GrabCut algorithm. Here, the Grabcut algorithm is used as a foreground extraction algorithm to easily explain the present invention, but any known algorithm or foreground extraction method may be used to extract or delete the foreground from an image, so the present invention is not limited thereto.

Through this, the first preprocessing module can convert the front image information into foreground extraction information containing only the driver's front image face based on the data set from which the foreground area is extracted. Additionally, in order to reduce the complexity of calculations performed in the deep learning model described later, the first preprocessing module may further convert foreground extraction information containing only the driver's front image face into gray foreground extraction information.

Meanwhile, FIG. 5 is a diagram for explaining pre-processing for converting front image information into front feature point information in the pre-processing unit of FIG. 1.

Referring to FIG. 5, the first preprocessing module extracts the driver's feature points from the foreground extraction information obtained by converting the driver's front image information for which the area of interest is collected by the information collection unit 110 and further converts it into front feature point information. You can.

Accordingly, the first preprocessing module extracts the foreground using the Grabcut algorithm from the front image information with the region of interest set, and uses facial image threshing (FIT) on the foreground extraction information converted to gray to create the face area. You can adjust the size of the foreground extraction information by deleting the excluded area. Here, the facial image thrashing technique is used to correct missing emotional or behavioral data, remove inappropriate data, merge large-scale data, resize and crop images, and combine the emotional and behavioral video sequences input with the facial image thrashing technique. It can be converted to an output image with image editing and separation applied. This face image thrashing technique is performed from a face image thrashing machine, which includes a data receiver, a Multi-Task Cascaded Convolutional Network (MTCNN), an image resizer, and a pre-trained Xception algorithm model. May include a separator. In this way, the facial image thrashing machine converts emotional and behavioral video sequences into images with a data receiver, identifies human faces in the converted images using MTCNN, and reduces the size of the images with an image resizer. . Therefore, finally, the data separator can be used to separate emotional and behavioral images into appropriately labeled directories.

Subsequently, the first preprocessing module can identify feature points on the driver's face using a feature point tracking algorithm in foreground extraction information in which only the face area exists. This feature point tracking algorithm may use the SURF (Speeded UP Robust Features) algorithm. Accordingly, in this specification, the known SURF algorithm is used to easily explain the present invention, but the feature points of the face present in the image are detected or Known algorithms or facial feature point methods may be used to extract the information, but are not limited thereto. In addition, the first preprocessing module may further include a device or module capable of performing a facial image thrashing technique and a SURF algorithm to extract the driver's facial feature points from the foreground extraction information and convert them into front feature point information.

Accordingly, the first preprocessing module may extract pixel values of the region of interest from the resized foreground extraction information and combine them with the identified feature points to further convert them into front feature point information. This front feature point information can be used for learning or classifying a deep learning model that extracts the driver's emotions and behavior.

Meanwhile, the first preprocessing module may further convert the front image information of the driver with the area of interest set into front heat map information. Accordingly, the first preprocessing module can check the object in the area set as the area of interest and track the location of the object. Here, when the driver's expression changes, the object may be specified as a body organ of the face used for expression change, such as eyes, eyebrows, and mouth.

This first preprocessing module can determine the movement of the object according to the change in pixel value in the image, and can check whether the object is grounded in the area of interest according to the pixel position where the pixel value changes is in contact with the pixel present in the area of interest. there is.

Additionally, the first preprocessing module can check the pixel value of the object that changes according to the movement of the object and track the location of the object according to the position where the pixel value changes.

In this case, the first preprocessing module may selectively further track objects that are larger than a certain size among objects that touch the region of interest. Through this, the first preprocessing module can further track other body organs, such as fingers, hands, and arms, other than the body organs present on the driver's face.

Accordingly, the first preprocessing module tracks objects grounded in the area of interest, that is, the eyes, eyebrows, mouth, etc. of the driver's face, and the driver's fingers and hands, and confirms the coordinate value of the pixel present at the location of the tracked object, By accumulating the coordinate values of pixels, a heat map of the image information in front of the driver can be created. Through this, the first preprocessing module can generate a heatmap for the driver's face based on the front image information in which the region of interest is set and further convert it into front heatmap information.

Accordingly, the first preprocessing module can convert the driver's front image information for which the area of interest is set into foreground extraction information, front feature point information, and front heat map information.

Meanwhile, the second preprocessing module included in the preprocessing unit 130 may perform filtering to remove noise signals from the biometric information collected by the information collection unit 110. Accordingly, the second preprocessing module may perform filtering to remove the sleep signal from the driver's heart rate information collected by the heart rate collection unit 113a included in the information collection unit 110. Here, the heart rate information is a one-dimensional analog signal that measures the action current generated in the myocardium according to the heartbeat using the standard 12 induction method, and the measured signal may include changes in the amplitude and size of the heart rate measured in real time.

Accordingly, the second preprocessing module may remove baseline fluctuation noise from the heart rate signal included in the heart rate information. Baseline fluctuation is a low-frequency component of less than 1Hz caused by human breathing, etc., and can be removed using a band-pass filter. Accordingly, the second preprocessing module can remove noise signals included in heart rate information.

Meanwhile, the second preprocessing module may perform filtering to remove noise signals from the driver's voice information collected by the voice collection unit 113b. Here, the voice signal included in the voice information is a signal generated by a person vibrating the vocal cords, and the inter-frequency correlation between adjacent signals is very high, making it possible to distinguish surrounding background noise.

Accordingly, the second preprocessing module may determine the background noise by setting a signal with an inter-frequency correlation below the threshold in the driver's voice information as background noise. Here, the background noise may be an audio signal other than the driver's voice, such as a vehicle's exhaust sound or a vehicle's clunk sound.

Accordingly, the second preprocessing module can separate and delete the voice signal determined to be background noise from the voice information and delete the noise signal included in the voice information.

In this way, the pre-processing unit 130 can remove noise signals by filtering the heart rate information and voice information collected by the information collection unit 110 through the second pre-processing module.

Meanwhile, the shape vector generator 150 extracts each feature vector based on the front image information and biometric information preprocessed in the preprocessor 130 and combines the feature vectors to generate a shape vector.

Here, the shape vector generator 150 uses a feature vector extractor that performs vector combining to remove noise signals from the foreground extraction information, front feature point information, front heatmap information, and biometric information converted from the driver's front image information. Each feature vector can be extracted based on heart rate information and voice information.

Additionally, the shape vector generator 150 may extract each feature vector using a feature vector extractor and combine the extracted feature vectors to generate one shape vector.

The shape vector generator 150 may extract the pixel value, front feature point, and front heatmap of the face image included in the foreground extraction information, front feature point information, and front heatmap information converted in the first preprocessing module.

At this time, the shape vector generator 150 may extract a facial feature vector based on the pixel value, front feature point, and front heat map of the face image using a feature vector extractor. Here, the feature vector extractor can extract the facial feature vector using a known wavelet transform technique.

Meanwhile, the shape vector generator 150 may detect a waveform from the heart rate signal included in the heart rate information in order to extract a characteristic vector from the filtered heart rate information. Here, the heart rate signal is a signal consisting of a waveform including P wave, Q wave, R wave, S wave, and T wave in one cycle, and can be detected by various known technologies. Additionally, the heart rate signal may be a heart rate signal having one cycle or multiple cycles.

Accordingly, in the case of a heart rate signal with one cycle, five waveforms including the P wave, Q wave, R wave, S wave, and T wave can be detected, and in the case of a heart rate signal with multiple cycles, one waveform can be detected depending on the number of cycles. A double of the waveform included in the ECG signal of the cycle can be detected.

Accordingly, the shape vector generator 150 can extract specific coordinates from each detected waveform. Here, the specific coordinates are those specified through five waveforms: PP, the peak point of the P wave, QP, the peak point of the Q wave, RP, the peak point of the R wave, SP, the peak point of the S wave, and the peak of the T wave. Includes branch TP.

In addition, the shape vector generator 150 sets the point with the highest amplitude value for the P-wave, R-wave, and T-wave as the peak point, and sets the point with the lowest amplitude value for the Q-wave and S-wave as the peak point. You can.

Through this, the shape vector generator 150 can extract a feature vector based on two or more specific coordinates included in a heart rate signal having one cycle or multiple cycles. Here, the characteristic vector can be calculated using the time corresponding to the x-axis and the amplitude corresponding to the y-axis of specific coordinates.

In this regard, when the shape vector generator 150 extracts a feature vector using two or more specific coordinates from a heart rate signal consisting of one cycle, the distance between PP, the peak point of the P wave, and RP, the peak point of the R wave or slope, the distance or slope between RP, the peak point of the R wave, and TP, the peak point of the T wave, the distance or slope between SP, the peak point of the S wave, and TP, the peak point of the T wave, and PP, the peak point of the P wave. The distance or slope between SP, the peak point of the S wave, the distance or slope between PP, the peak point of the P wave, and TP, the peak point of the T wave, the distance between PP, the peak point of the P wave, and QP, the peak point of the Q wave. or slope, the distance or slope between QP, the peak point of the Q wave, and RP, the peak point of the R wave; the distance or slope between QP, the peak point of the Q wave, and SP, the peak point of the S wave; and QP, the peak point of the Q wave. The distance or slope between TP, the peak point of the T wave, and the distance or slope between RP, the peak point of the R wave, and SP, the peak point of the S wave, can be used to calculate the distance or slope between a total of 10 peak points.

In addition, when the distance and slope between the peak points are not calculated using the peak points of all waveforms, the shape vector generator 150 generates the RP and S waves, which are the peak points of the R wave, which clearly show the external characteristics of the heart rate signal. The distance and slope can be calculated using only SP, which is the peak point of , and TP, which is the peak point of the T wave.

Accordingly, the shape vector generator 150 may calculate the distance between peak points using known methods such as Manhattan Distance, Euclidean Distance, and Minkowski Distance.

Additionally, the shape vector generator 150 may calculate the slope between peak points using a known slope formula between two coordinate points.

Through this, the shape vector generator 150 can calculate specific coordinates of two or more waveforms among the five waveforms included in the heart rate signal and extract them as characteristic vectors.

In addition, when the heart rate signal has a plurality of cycles, the shape vector generator 150 specifies each waveform extracted from the heart rate signal corresponding to the current cycle (n) and the heart rate signal corresponding to the previous cycle (n-1). A feature vector can be extracted by selecting one or more coordinates from each cycle.

In this regard, when the heart rate signal includes a plurality of cycles and uses specific coordinates of two or more waveforms, the shape vector generator 150 generates a P wave (P) corresponding to the heart rate signal of the previous cycle (n-1). Between PP(n-1), the peak point of (n-1)), and RP(n), the peak point of the R wave (R(n)) corresponding to the heart rate signal of the current cycle (n) (R(n)) -The distance or slope of P(n-1)), the peak point of the P wave (P(n-1)) corresponding to the heart rate signal of the previous cycle (n-1), and PP(n-1) of the current cycle ( The distance or slope between TP(n) (T(n)-P(n-1)), which is the peak point of the T wave (T(n)) corresponding to the heart rate signal of n), can be calculated and extracted as a feature vector. You can. In this way, the shape vector generator 150 can extract a heart rate characteristic vector for the heart rate information from the filtered heart rate information.

Meanwhile, the shape vector generator 150 may calculate feature values for the voice signal included in the voice information using the MFCC (Mel Frequency Cepstral Cofficent) method in order to extract feature vectors from the filtered voice information. . This MFCC method is a method of detecting effective spectrum-based feature values using the nonlinear frequency characteristics of the human ear. Accordingly, the shape vector generator 150 can calculate the feature value of the voice signal not only using the MFCC method, but by various known methods, and is not limited to this.

Next, the shape vector generator 150 may extract a feature vector for the voice signal based on a specific value of the voice signal calculated using the MFCC method. In this regard, the shape vector generator 150 may detect a feature value, that is, a feature sequence, for the voice signal using the MFCC method.

Additionally, the shape vector generator 150 may calculate the i-vector using the combined factor analysis method using Baum-Welch statistics for the feature values of the speech signal. Here, Baum-Welch statistics is a known technique, and description thereof will be omitted in this specification.

Accordingly, the shape vector generator 150 can extract the i-vector calculated through the feature values of the voice signal as a voice feature vector for voice information.

Meanwhile, the shape vector generator 150 further uses a feature vector extractor that performs vector combining to extract a facial feature vector based on foreground extraction information, front feature point information, and front heat map information and a heart rate feature vector extracted from a heart rate signal. Voice feature vectors extracted from voice signals can be combined.

At this time, the shape vector generator 150 may generate one shape vector by using each extracted feature vector as an input and outputting the feature vectors in a combined form.

Meanwhile, the determination unit 170 determines the driver's emotions and actions based on the shape vector generated by the shape vector generator 150. At this time, the determination unit 170 may determine the driver's emotions and behavior by inputting the shape vector into a previously learned deep learning model. Accordingly, the driver's emotions are judged as one of normal, surprise, sadness, happiness, fear, disgust, anger, and boredom, and the driver's behavior is normal driving, yawning, blinking, checking the mirror, smoking, and using a cell phone. It can be judged by one of the following actions: use and checking the driver's field of vision.

Here, deep learning is defined as a collection of machine learning algorithms that attempt a high level of abstraction through a combination of several non-linear transformation techniques, and is a field of machine learning that teaches computers how to think in a large scale.

Accordingly, the deep learning models used to extract driver's emotions and behavior in the present invention include deep neural networks (Deep Neural networks) model, CNN (Convolution Neural Networks) model, DBN (Deep Believe networks) model, and Deep Residual Neural Network (DBN) model. One of the Learning for image Recognition (ResNet) models can be used. Accordingly, the present invention uses a 34-layer residual neural network model that modifies the structure of a known residual neural network model, but is not limited to this.

Accordingly, the 34-layer residual neural network model is an image classification algorithm model, and a shortcut is added to prevent accuracy from decreasing when using deep layers of the algorithm model. This residual neural network model is most preferably used with a structure of 34 layers, so it is referred to as a 34-layer residual neural network model, but it can be used with a structure of 18, 34, 50, 101, and 152 layers. It can be composed of a structure with a larger number of floors.

Subsequently, the 34-layer residual neural network model has 64 output channels and a 7x7 convolutional layer including a 3x3 max pooling layer, and a batch normalization layer is provided after each convolutional layer, which is formed with the same number of output channels. It may be composed of four modules provided with a plurality of residual blocks. At this time, the first module can use the same number of channels as the input channel number.

Through this, compared to the Inception architecture model, the 34-layer residual neural network model has the number of channels of the first residual block of the subsequent module doubled compared to the previous module (here, the first module), and its height and width are reduced by half, resulting in the Inception architecture. Optimization is simple because it can be modified more simply than the model, and when the depth of the network increases by providing a structure with a larger number of layers, optimization is simpler and higher accuracy can be achieved.

Therefore, the decision unit 170 can solve the problem of recognition errors occurring in recognition of the driver's emotions and actions by using a residual neural network model that provides higher accuracy as the depth of the network increases, and the decision unit 170 collects Based on the characteristic vectors of the images and signals, the driver's emotions and behavior can be judged with higher accuracy.

Finally, the output unit 190 outputs the driver's emotions and actions determined by the determination unit 170. In this regard, the autonomous vehicle can control vehicle functions based on the driver's emotions and actions output from the output unit 190. For example, the determination unit 170 determines the driver's emotion as bored and the driver's behavior as yawning using a deep learning model, and accordingly, the output unit 190 determines the driver's emotion and behavior as bored. and yawns. At this time, the autonomous vehicle determines that the driver is tired based on the boredom and yawning output from the output unit 190 and opens the vehicle window or plays an exciting song to change the driver's condition. You can control it. For another example, the determination unit 170 uses a deep learning model to determine the driver's emotion as anger and determines the driver's behavior as mobile phone use, and accordingly, the output unit outputs anger and mobile phone use. do. At this time, based on the anger output by the output unit 190 and the use of the mobile phone, the autonomous vehicle determines that a situation in which a traffic accident may occur while the driver is using the mobile phone has occurred and controls the speed of the vehicle or generates a warning sound. This allows the driver to control vehicle functions so that he or she can concentrate on driving.

In this way, the multi-sensor fusion-based driver monitoring device 100 according to an embodiment of the present invention can monitor and output the driver's emotions and behavior so that the efficiency of the autonomous vehicle can be increased through the above-described configuration.

Figure 6 is a flow diagram of a driver monitoring method based on multi-sensor fusion according to another embodiment of the present invention.

The multi-sensor fusion-based driver monitoring method according to another embodiment of the present invention is a method performed through the multi-sensor fusion-based driver monitoring device 100 of FIG. 1, and is performed by the multi-sensor fusion-based driver monitoring device 100. It may be carried out in a configuration that is substantially the same as the configuration. Therefore, the same components as those of the multi-sensor fusion-based driver monitoring device 100 of FIG. 1 will be assigned the same reference numerals, and repeated descriptions will be omitted.

Referring to FIG. 6, the multi-sensor fusion-based monitoring method according to another embodiment of the present invention includes an information collection step 610 of collecting the driver's front image information and biometric information, and a preprocessing step of preprocessing the front image information and biometric information. (630), a shape vector generation step (650) of extracting each feature vector based on the preprocessed front image information and biometric information and combining each feature vector to generate a shape vector, the driver based on the shape vector It includes a judgment step 670 for determining the driver's emotions and actions and an output step 690 for outputting the driver's emotions and actions.

FIG. 7 is a flow diagram of the information collection step of the multi-sensor fusion-based driver monitoring method of FIG. 6.

Referring to FIG. 7, the information collection step 610 may include a step 611 of collecting the driver's front image information and a step 613 of collecting biometric information including the driver's heart rate information and the driver's voice information. You can. At this time, the information collection step 610 may simultaneously perform the step 611 of collecting the driver's front image information and the step 613 of collecting the driver's biometric information.

Meanwhile, FIG. 8 is a flow diagram of the preprocessing step of the multi-sensor fusion-based driver monitoring method of FIG. 6.

Referring to FIG. 8, the preprocessing step 630 is a step of setting the driver's area of interest in the front image information (631), and converting the front image information set as the area of interest into foreground extraction information, front feature point information, and front heat map information. It may include a step 633 of performing filtering to remove noise signals present in heart rate information and voice information included in biometric information (635).

Accordingly, the step 633 of converting the foreground extraction information, the front feature point information, and the front heatmap information includes the step of converting the front image information into the foreground extraction information (633a), and the step of converting the front image information into the front feature point information (633b). ), including a step 633c of converting front image information into front heat map information, and each step may be performed simultaneously, or, although not shown in FIG. 8, a step 633a of converting into foreground extraction information, front feature point The step of converting to information (633b) and the step of converting to front heatmap information (633c) may be performed in this order.

Meanwhile, the preprocessing step 630 may simultaneously perform the step 631 of setting the driver's area of interest and the step 635 of performing filtering.

In this regard, the step of performing filtering (635) includes performing filtering to remove noise signals from heart rate information (635a) and performing filtering to remove noise signals from voice signals (635b), , each step may be performed simultaneously, or, although not shown in FIG. 8, may be performed in the order of step 635a of performing filtering on the heart rate information and step 635b of performing filtering on the voice signal.

Through this, the shape vector generation step 650 uses a feature vector extractor that performs vector combining based on foreground extraction information, front feature information, front heat map information, and heart rate information and voice information included in biometric information, respectively. It may include extracting feature vectors and combining each feature vector to create one shape vector.

As such, the multi-sensor fusion-based driver monitoring method according to another embodiment of the present invention can monitor and output the driver's emotions and behavior so that the efficiency of the autonomous vehicle can be increased through the above-described steps.

Although the above has been described with reference to embodiments, those skilled in the art will understand that various modifications and changes can be made to the present invention without departing from the spirit and scope of the present invention as set forth in the following patent claims. You will be able to.

100: Multi-sensor fusion-based driver monitoring device
110: Information collection department
111: Video collection unit
113: biometric collection unit, 113a: heart rate collection unit, 113b: voice collection unit
130: preprocessing unit
150: Shape vector generation unit
170: Judgment unit
190: output unit

Claims (8)

An information collection unit that collects the driver's front image information and biometric information;
a preprocessing unit that preprocesses the front image information and biometric information;
a shape vector generator that extracts each feature vector based on the front image information and biometric information preprocessed in the preprocessor and combines the feature vectors to generate a shape vector;
a determination unit that determines the driver's emotions and actions based on the shape vector; and
Characterized in that it includes an output unit that outputs the driver's emotions and actions,
The shape vector generator,
Extracting a feature vector of heart rate information included in the biometric information using a feature vector extractor that performs vector combining,
Extracting the feature vector of the heart rate information includes:
Detecting a waveform from the heart rate signal included in the heart rate information and extracting specific coordinates from each detected waveform,
When the heart rate signal has a plurality of cycles, a feature vector is extracted by selecting one or more specific coordinates of each waveform extracted from the heart rate signal corresponding to the current cycle and the heart rate signal corresponding to the previous cycle in each cycle,
When the heart rate signal includes a plurality of cycles and uses specific coordinates of two or more waveforms, the distance or slope between the peak point of the heart rate signal corresponding to the previous cycle and the peak point of the heart rate signal corresponding to the current cycle is calculated. A driver monitoring device based on multi-sensor fusion that extracts characteristic vectors.
According to paragraph 1,
The information collection department,
an image collection unit that collects front image information of the driver; and
A biometric collection unit that collects the driver's biometric information, including a heart rate collection unit that collects the driver's heart rate information and a voice collection unit that collects the driver's voice information; a multi-sensor fusion-based driver comprising a. Monitoring device.
According to paragraph 2,
The preprocessor,
The driver's area of interest is set in the front image information, and the front image information set as the area of interest is converted into foreground extraction information, front feature point information, and front heat map information, and the heart rate information included in the biometric information, A driver monitoring device based on multi-sensor fusion, characterized in that it performs filtering to remove noise signals present in voice information.
According to paragraph 3,
The shape vector generator,
The foreground extraction information, the front feature point information and front heat map information converted from the front image information using the feature vector extractor that performs the vector combination, and the heart rate information and the voice information from which noise signals are removed from the biometric information. A driver monitoring device based on multi-sensor fusion, characterized in that extracting each feature vector based on and combining each feature vector to generate one shape vector.
In a driver monitoring method performed through a multi-sensor fusion-based driver monitoring device,
An information collection step of collecting the driver's front image information and biometric information;
A preprocessing step of preprocessing the front image information and biometric information;
A shape vector generation step of extracting each feature vector based on the preprocessed front image information and biometric information and combining the feature vectors to generate a shape vector;
a determination step of determining the driver's emotions and actions based on the shape vector; and
Characterized in that it includes an output step of outputting the driver's emotions and actions,
The shape vector generation step is,
Extracting the feature vector of heart rate information included in the biometric information using a feature vector extractor that performs vector combining,
Extracting the feature vector of the heart rate information includes:
Detecting a waveform from the heart rate signal included in the heart rate information and extracting specific coordinates from each detected waveform,
When the heart rate signal has a plurality of cycles, a characteristic vector is extracted by selecting one or more specific coordinates of each waveform extracted from the heart rate signal corresponding to the current cycle and the heart rate signal corresponding to the previous cycle in each cycle,
When the heart rate signal includes a plurality of cycles and uses specific coordinates of two or more waveforms, the distance or slope between the peak point of the heart rate signal corresponding to the previous cycle and the peak point of the heart rate signal corresponding to the current cycle is calculated and A multi-sensor fusion-based driver monitoring method that extracts feature vectors.
According to clause 5,
The information collection step is,
collecting front image information of the driver; and
A multi-sensor fusion-based driver monitoring method comprising: collecting the biometric information including the driver's heart rate information and the driver's voice information.
According to clause 6,
The preprocessing step is,
setting a region of interest for the driver in the front image information;
Converting the front image information set as the region of interest into foreground extraction information, front feature point information, and front heat map information; and
A driver monitoring method based on multi-sensor fusion, comprising: performing filtering to remove noise signals present in the heart rate information and the voice information included in the biometric information.
In clause 7,
The shape vector generation step is,
Using a feature vector extractor that performs the vector combination, each feature vector is generated based on the heart rate information and the voice information included in the foreground extraction information, the front feature point information, the front heat map information, and the biometric information. Extracting step; and
A driver monitoring method based on multi-sensor fusion, comprising combining the respective feature vectors to generate one shape vector.