KR102429822B1 - Apparatus and Method for recognizing a driver’s emotional state - Google Patents

Abstract

The present invention relates to an apparatus and method for recognizing a driver's emotional state. An apparatus for recognizing a driver's emotional state according to an embodiment of the present invention includes a data collection unit including a microphone for acquiring a voice of a target driver and a camera for photographing the face of the target driver, and a first preset first from the face image taken by the camera A first feature vector is extracted according to an algorithm, a second feature vector is extracted from the voice data obtained by the microphone according to a preset second algorithm, and the first feature vector and the second feature vector are integrated to form a multi-modal Using the emotional feature extraction unit for generating data (Multimodal data), the emotional state data storage unit for storing the frequency of a plurality of pre-classified emotional states for each driver, and the frequency of the multi-modal data and the plurality of pre-classified emotional states Thus, it may include an emotional state determination unit for determining the emotional state of the target driver.

Description

Apparatus and Method for recognizing a driver's emotional state

The present invention relates to an apparatus and method for recognizing a driver's emotional state, and to a technology for improving driver safety while driving by determining an individual driver's individual emotional state and providing a convenient function accordingly.

Since the mica era in the 1980s, the number of registered automobiles has increased rapidly. For example, from 15.4 million units in 2005 to 20.12 million units in 2014, it is gradually increasing every year, and thus the risk of traffic accidents is also increasing. Accordingly, automobile companies are providing safety technologies using various sensors to provide convenience to drivers.

Representative technologies include a lane departure warning system and a forward collision warning system. The lane departure warning system uses a camera to recognize the lane ahead and warns the driver when the vehicle deviates from the lane. It is a system that generates a warning in case of danger of collision.

Such a system not only does not work properly in bad weather or environments such as a broken lane or a winding road, but also has a problem that the warning timing is not optimal because the driver's status information is not taken into account.

Recently, after recognizing the driver's concentration and emotional state information, technologies for improving the driver's safety by using such information have been introduced.

This technology utilizes contact and non-contact sensors to recognize the driver's gaze, bio-signals, head movements, and emotional changes while driving to understand the driver's condition, thereby improving driving safety and providing convenience functions and provide an alert.

This technology makes it difficult to track or track changes in the driver's emotions while driving when there are several drivers for a vehicle, and in the case of general concentration or emotional models, personal information is not included when learning. There is a problem in that the recognition rate for the classification of the emotional state is lowered.

Accordingly, there is a demand for the development of an apparatus and method for more accurately recognizing a driver's emotional state.

The present invention has been devised to solve the problems described above, and an object of the present invention is to provide a user with a system for recognizing a driver's emotional state through a personalized emotion recognition method.

On the other hand, the technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned are clearly to those of ordinary skill in the art to which the present invention belongs from the description below. can be understood

An apparatus for recognizing a driver's emotional state related to an example of the present invention for realizing the above-described problem includes a data collection unit including a microphone for acquiring a voice of a target driver and a camera for photographing the face of the target driver, A first feature vector is extracted from the face image according to a preset first algorithm, and a second feature vector is extracted from the voice data obtained by the microphone according to a preset second algorithm, and the first feature vector and the second An emotional feature extraction unit for generating multimodal data by integrating feature vectors, an emotional state data storage unit for storing the frequency of a plurality of pre-classified emotional states for each driver, and the multi-modal data and the pre-classified plurality It may include an emotional state determination unit for determining the emotional state of the target driver by using the frequency of the emotional state.

In addition, the emotional state determination unit derives the degree of each of the plurality of emotional states from the multi-modal data, and reflects the frequency of each of the plurality of emotional states of the target driver in each of the plurality of emotional state degrees to the plurality of emotional states. It is possible to calculate an output value for each emotional state of , and determine the emotional state having the largest output value among the plurality of output values as the emotional state of the target driver.

In addition, the apparatus may further include a face data storage unit configured to store a plurality of driver face data, and an identification unit configured to recognize the target driver using the first feature vector and a plurality of driver face data stored in the driver face data storage unit. have.

Also, the identification unit may detect a face region of the target driver from the face image and extract the first feature vector from the detected face region of the target driver.

Also, when the target driver is a driver not stored in the data storage unit, the identification unit may store the face region image of the target driver in the face data storage unit.

In addition, the emotion feature extracting unit may reduce the number of dimensions of the first feature vector and the second feature vector according to a preset third algorithm, and then integrate the first feature vector and the second feature vector.

Also, the emotional state determining unit may calculate an output value for each of the plurality of emotional states by using the following equation.

[Equation 1]

S = H + H x ω

S is an output value for the emotional state, H is the degree of the emotional state, and ω is a probability density value according to the frequency of the emotional state of the target driver.

In addition, when the output value of the emotional state is out of a preset output value range, it may further include a risk determination and action unit for outputting a warning sound.

An embodiment of the present invention may provide a user with a system for recognizing a driver's emotional state through a personalized emotion recognition method.

On the other hand, the effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the description below. will be able

The following drawings attached to the present specification illustrate a preferred embodiment of the present invention, and serve to further understand the technical spirit of the present invention together with the detailed description of the present invention, so the present invention is limited to the matters described in such drawings It should not be construed as being limited.
1 is a block diagram of an apparatus for recognizing a driver's emotional state according to an embodiment of the present invention.
2 is a diagram illustrating a process of generating multi-modal data from a face image and a voice signal according to an embodiment of the present invention.
3 is a diagram illustrating classification of a plurality of emotional states according to an embodiment of the present invention.
4 is a histogram of a driver's emotional state probability density according to an embodiment of the present invention.
5 is a flowchart illustrating a method for recognizing a driver's emotional state according to an embodiment of the present invention.
6 is a block diagram illustrating a computing system for executing a method for recognizing a driver's emotional state according to an embodiment of the present invention.

Hereinafter, some embodiments of the present invention will be described in detail with reference to exemplary drawings. In adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in describing the embodiment of the present invention, if it is determined that a detailed description of a related known configuration or function interferes with the understanding of the embodiment of the present invention, the detailed description thereof will be omitted.

In describing the components of the embodiment of the present invention, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the elements from other elements, and the essence, order, or order of the elements are not limited by the terms. In addition, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

1 is a block diagram of an apparatus for recognizing a driver's emotional state according to an embodiment of the present invention.

Referring to FIG. 1 , the apparatus for recognizing a driver's emotional state includes a data collection unit 100 , a face data storage unit 200 , an identity verification unit 300 , an emotional feature extraction unit 400 , and an emotional state determination unit 500 . , it may include an emotional state data storage unit 600 and a risk determination and action unit 700 , and the like.

However, since the components shown in FIG. 1 are not essential, an apparatus for recognizing a driver's emotional state having more or fewer components may be implemented.

The data collection unit 100 is configured to acquire the voice and face images of the target driver, and may include a microphone 110 and a camera 120 .

That is, the target driver's voice may be acquired using the microphone 110 , and the target driver's face may be photographed using the camera 120 to acquire a face image.

The face data storage unit 200 is configured to store a plurality of driver face data.

The plurality of driver face data is used for the identification unit 300 to be described later to recognize the target driver. In addition, the plurality of driver face data may be a plurality of driver face images or a feature vector extracted from a plurality of driver face images according to a preset first algorithm.

In addition, new driver's face data may be stored in the face data storage unit 200 by the identification unit 300 to be described later, which will be described in detail below.

The identification unit 300 is configured to recognize the target driver from the face image captured by the camera 120 .

The identification unit 300 extracts a first feature vector from the face image according to a preset first algorithm to recognize the target driver.

And the identification unit 300 may use an algorithm such as LBP (Local Binary Patterns), Gabor as the first algorithm.

The identification unit 300 compares the extracted first feature vector with a plurality of driver face data stored in the face data storage unit 200 to recognize the target driver. In this case, when a plurality of driver face data stored in the face data storage unit 200 is a feature vector, a target driver may be recognized using a classifier such as a support vector machine (SVM). In addition, when a plurality of driver face data stored in the face data storage unit 200 is a driver's face image, a target driver may be recognized after a feature vector is extracted from the plurality of driver's face images according to the first algorithm.

Meanwhile, according to an embodiment of the present invention, the identification unit 300 performs a function of detecting a face region from a face image captured by the camera 120 before extracting the first feature vector and detects the detected face region. A first feature vector can be extracted from . This is because, when the camera 120 captures the face of the target driver, objects other than the face of the target driver may be included in the captured image.

In this case, a commonly used Adaboost-based algorithm may be used as an algorithm for detecting the face region, but is not limited thereto.

In addition, when the target driver to be recognized according to an embodiment of the present invention is a driver who is not stored in the face data storage unit 200 , the identification unit 300 determines that the target driver is an unregistered driver, and the face data storage unit In 200 , the face image of the target driver and the first feature vector extracted from the face image may be additionally stored.

The emotional feature extraction unit 400 is configured to generate multimodal data including voice data and face image data.

The emotional feature extraction unit 400 may extract the first feature vector according to a preset first algorithm like the identity verification unit 300 according to an embodiment of the present invention.

And the emotional feature extraction unit 400 can use an algorithm such as LBP (Local Binary Patterns), Gabor, etc. as the first algorithm, like the identity verification unit 300, before extracting the first feature vector, the camera 120 may further perform a function of detecting a face region from the captured face image and extract a first feature vector from the detected face region.

The microphone 110 extracts a second feature vector from the acquired driver's voice data according to a preset second algorithm. In this case, as the second algorithm, MFCC (Mel-frequency cepstral coefficients) or the like may be used.

The emotional feature extraction unit 400 generates multi-modal data by integrating the first feature vector and the second feature vector. In this case, the emotional feature extraction unit 400 may use a deep learning method DBN (Deep Belief Network) to integrate the first feature vector and the second feature vector.

The emotional feature extraction unit 400 reduces the number of dimensions of the first feature vector and the second feature vector by using an algorithm such as PCA (Principle Component Analysis) before generating multi-modal data according to an embodiment of the present invention. can also be integrated.

In order to schematically describe a process of generating such multi-modal data, it will be described with reference to FIG. 2 .

2 is a diagram illustrating a process of generating multi-modal data from a face image and a voice signal according to an embodiment of the present invention. Referring to FIG. 2 , the upper left picture is a facial image of a target driver acquired by the camera 120 , and the lower left picture is the voice signal of the target driver acquired by the microphone 110 .

The above-described identity verification unit 300 or emotional feature extracting unit 400 extracts a first feature vector from the face image, and the emotional feature extracting unit 400 extracts a second feature vector from the voice signal.

A figure showing the first feature vector and the second feature vector is shown in the lower right corner of FIG. 2 .

As an additional step, the number of dimensions of the extracted first and second feature vectors may be reduced by the emotional feature extraction unit 400, and the first and second feature vectors with reduced dimensions are shown in the right middle of FIG. 2 . is shown in

Thereafter, the emotional feature extraction unit 400 may generate multi-modal data by integrating the first feature vector and the second feature vector, which is shown in the upper right corner of FIG. 2 .

Referring back to FIG. 1 , the emotional state determining unit 500 is configured to determine the emotional state of the target driver using multi-modal data.

The emotional state determination unit 500 derives the degree of each of a plurality of pre-classified emotional states in order to determine the emotional state of the target driver.

3 is a diagram illustrating classification of a plurality of emotional states according to an embodiment of the present invention.

The emotional state determination unit 500, for example, as shown in FIG. 3 , 'awakened-discomfort', 'awakened', 'awakened-pleasant', 'dissatisfied', 'neutral', 'pleasant', 'relaxed- The degree of 'discomfort', 'relaxation', and 'relaxation-pleasant' can be derived, and in this case, a classifier such as HMM (Hidden Markov Model) can be used.

The emotional state data storage unit 600 stores the frequency of a plurality of pre-classified emotional states for each driver. The frequency may be stored as the number of occurrences of the emotional state, or may be stored as a probability density.

The emotional state determination unit 500 may determine the emotional state having the highest degree among the derived results as the emotional state of the target driver, but the emotional state data storage unit 600 ) stored data.

That is, the emotional state determining unit 500 may calculate an output value for each of the plurality of emotional states by reflecting the frequency of each of the plurality of emotional states in the degree of each emotional state derived for determining the customized emotional state for each driver. In addition, the emotional state having the largest output value among the plurality of calculated output values may be determined as the emotional state of the target driver.

The emotional state determination unit 500 may use the following equation to calculate an output value reflecting the frequency of each of the plurality of emotional states.

[Equation 1]

S = H + H x ω

Where S is an output value reflecting the frequency in the degree of emotional state, H is the degree of emotional state, and ω is a probability density value according to the frequency of the emotional state of the target driver.

Hereinafter, an embodiment in which the emotional state determining unit 500 determines the emotional state of the target driver using the above equation will be described with reference to FIG. 4 .

4 is a histogram of the probability density of an emotional state of a target driver according to an embodiment of the present invention.

First, the degree of arousal-displeasure derived by the emotional state determination unit 500 is 4, the degree of arousal is 4, the degree of arousal-pleasure is 5, the degree of displeasure is 6, the degree of neutrality is 5, and the degree of pleasure is 11, It is assumed that the degree of relaxation-displeasure is 4, the degree of relaxation is 2, and the degree of relaxation-pleasant is 10.

If the emotional state of the target driver is simply judged by the degree of emotional state, the ‘pleasant’ with the greatest degree will be judged as the emotional state of the target driver. However, if the probability density histogram of FIG. 4 is used to determine the customized emotional state for each driver, the result is different.

If the output value is calculated using the above equation, the output value of arousal-discomfort is about 4.44, the output value of arousal is about 4.72, the output value of arousal-pleasure is 6, the output value of displeasure is 6.3, the output value of neutral is about 5.2, and the output value of pleasure is about 5.2. The output value is about 11.66, the output value of relaxation-discomfort is about 4.36, the output value of relaxation is about 2.24, and the output value of relaxation-pleasant is about 13.1.

Of these, the emotional state with the highest output value is 'relaxation-pleasant', so the emotional state of the target driver derived by reflecting the frequency of emotional states of the target driver is 'relaxed-pleasant', which is different from the result simply considering the degree of emotional state.

As described above, by using the emotional state data storage unit 600 for each driver, it is possible to improve the individual emotional state recognition performance.

According to an embodiment of the present invention, the apparatus for recognizing the driver's emotional state may further include a risk determination and action unit 700 .

The risk determination and action unit 700 is configured to determine a risk according to the emotional state of the target driver and take measures such as an alarm accordingly.

The risk determination and action unit 700 may determine that the target driver is in a dangerous state if the degree or output value of the emotional state determined by the emotional state determination unit 500 to determine the risk is out of the reference range.

For example, if the emotional state of the target driver appears as arousal-discomfort while driving and the output value exceeds a reference value, the state of the target driver is determined as dangerous.

When it is determined that the target driver is in a dangerous state as described above, the risk determination and action unit 700 provides a function such as an alert.

For example, it is possible to inform the target driver of danger through the output of a warning sound, etc., and to provide a function for stably changing the emotional state of the target driver through operation of a massage chair, operation of a heated seat, and playback of music.

By further including the risk determination and action unit 700 , the safety of driving can be improved.

Hereinafter, a method for recognizing a driver's emotional state based on the above-described configurations will be described with reference to FIG. 5 .

5 is a flowchart illustrating a method for recognizing a driver's emotional state according to an embodiment of the present invention.

The data collection unit acquires the voice and face images of the target driver (S100).

The microphone of the above-described configuration acquires the voice of the target driver, and the camera acquires the face image of the target driver.

The identification unit extracts the first feature vector from the face image (S200).

The identification unit extracts the first feature vector from the face image according to a preset first algorithm, and as described above, an algorithm such as LBP (Local Binary Patterns) or Gabor may be used.

Also, before extracting the first feature vector, the face region may be detected from the face image and the first feature vector may be extracted from the detected face region.

The identification unit recognizes the target driver from the first feature vector (S300).

This is a step of recognizing a target driver by comparing a plurality of pre-stored driver face data with the first feature vector.

In step S300 , a classifier such as a support vector machine (SVM) may be used to recognize the driver.

The emotional feature extraction unit extracts a second feature vector from the voice (S400).

The second feature vector is extracted from the voice data obtained by the microphone according to a preset second algorithm, and an algorithm such as MFCC (Mel-frequency cepstral coefficients) may be used.

The emotion feature extraction unit generates multi-modal data by integrating the first feature vector and the second feature vector (S500).

As a step of generating multi-modal data by integrating the previously extracted first and second feature vectors, a deep learning method DBN (Deep Belief Network) may be used.

In addition, before integrating the first feature vector and the second feature vector, the number of dimensions of the first feature vector and the second feature vector may be reduced and then combined.

The emotional state determination unit determines the emotional state of the target driver by using the multi-modal data (S600).

In step S600 , the degree of each of a plurality of pre-classified emotional states may be derived for determining the emotional state customized to the driver, and an output value may be calculated by reflecting the pre-stored frequency of each emotional state of the target driver.

As described above, the frequency of each emotional state of the driver may be stored as a probability density as shown in FIG. 4 , and the following equation may be used to reflect the frequency in the degree of each emotional state.

[Equation 2]

S = H + H x ω

S is an output value reflecting the frequency of the emotional state, H is the emotional state, and ω is a probability density value according to the frequency of the driver's emotional state.

S, which is an output value for each emotional state, will be derived using the above equation, and the emotional state having the largest value among the plurality of S may be determined as the emotional state of the target driver.

Meanwhile, according to an embodiment of the present invention, the step of generating an alarm according to the emotional state of the target driver may be further performed.

If the output value calculated to determine the emotional state of the target driver in step S600 deviates from the preset reference range, it may be determined that the target driver's state is dangerous, and accordingly, an alarm may be generated by outputting a warning sound or the like.

Accordingly, it is possible to improve the safety of the target driver's driving operation.

6 is a block diagram illustrating a computing system for executing a method for recognizing a driver's emotional state according to an embodiment of the present invention.

Referring to FIG. 6 , the computing system 1000 includes at least one processor 1100 , a memory 1300 , a user interface input device 1400 , a user interface output device 1500 , and storage connected through a bus 1200 . 1600 , and a network interface 1700 .

The processor 1100 may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in the memory 1300 and/or the storage 1600 . The memory 1300 and the storage 1600 may include various types of volatile or nonvolatile storage media. For example, the memory 1300 may include read only memory (ROM) and random access memory (RAM).

Accordingly, the steps of a method or algorithm described in connection with the embodiments disclosed herein may be directly implemented in hardware, a software module executed by the processor 1100 , or a combination of the two. A software module resides in a storage medium (ie, memory 1300 and/or storage 1600 ) such as RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM. You may. An exemplary storage medium is coupled to the processor 1100 , the processor 1100 capable of reading information from, and writing information to, the storage medium. Alternatively, the storage medium may be integrated with the processor 1100 . The processor and storage medium may reside within an application specific integrated circuit (ASIC). The ASIC may reside within the user terminal. Alternatively, the processor and storage medium may reside as separate components within the user terminal.

The apparatus and method for recognizing the driver's emotional state described above are not limited to the configuration and method of the above-described embodiments, but all or part of each of the embodiments may be modified so that various modifications may be made. They may be selectively combined and configured.

100: data collection unit
110: microphone
120 : camera
200: face data storage unit
300: identification unit
400: emotional feature extraction unit
500: emotional state determination unit
600: emotional state data storage unit
700: Risk judgment and action unit
1000: computing system
1100: Processor
1200: system bus
1300: memory
1310: ROM
1320: RAM
1400: user interface

Claims (10)

a data collection unit including a microphone for acquiring the target driver's voice and a camera for photographing the target driver's face;
A first feature vector is extracted from the face image taken by the camera according to a first preset algorithm, and a second feature vector is extracted from the voice data obtained by the microphone according to a preset second algorithm, and the first feature an emotion feature extraction unit for generating multimodal data by integrating the vector and the second feature vector;
an emotional state data storage unit for storing the frequency of a plurality of pre-classified emotional states for each driver; and
An emotional state determination unit that determines the emotional state of the target driver by using the multi-modal data and the frequency of the plurality of pre-classified emotional states;
The emotional state determination unit
The degree of each of the plurality of emotional states is derived from the multi-modal data, and a value obtained by multiplying the degree of each of the plurality of emotional states by a probability density value according to the frequency of each of the plurality of emotional states of the target driver is reflected to reflect the A system for recognizing an emotional state of a driver, comprising calculating an output value for each of a plurality of emotional states, and determining the emotional state having the largest output value among the plurality of output values as the emotional state of the target driver. delete The method according to claim 1,
a face data storage unit configured to store a plurality of driver face data; and
The driver's emotional state recognition system according to claim 1, further comprising an identification unit for recognizing the target driver by using the first feature vector and the plurality of driver's face data. 4. The method of claim 3,
The identification unit
and detecting the face region of the target driver from the face image, and extracting the first feature vector from the detected face region of the target driver. 5. The method according to claim 4,
The identification unit
When the target driver is a driver not stored in the data storage unit, the facial region image of the target driver is stored in the face data storage unit. The method according to claim 1,
The emotional feature extraction unit
The system for recognizing a driver's emotional state, characterized in that after reducing the number of dimensions of the first feature vector and the second feature vector according to a third preset algorithm, the first feature vector and the second feature vector are integrated. delete The method according to claim 1,
The driver's emotional state recognition system, characterized in that it further comprises a risk determination and action unit for outputting a warning sound when the output value for the emotional state is out of a preset output value range. extracting a first feature vector from a face image photographed by a camera photographing the target driver's face according to a preset first algorithm;
extracting a second feature vector from the voice data acquired by the microphone acquiring the target driver's voice according to a preset second algorithm;
generating multimodal data by integrating the first feature vector and the second feature vector; and
determining the emotional state of the target driver using the multi-modal data and the frequency of a plurality of emotional states pre-classified for each driver
including,
The judging step
The degree of each of the plurality of emotional states is derived from the multi-modal data, and a value obtained by multiplying the degree of each of the plurality of emotional states by a probability density value according to the frequency of each of the plurality of emotional states of the target driver is reflected to reflect the A method for recognizing an emotional state of a driver, comprising calculating an output value for each of a plurality of emotional states, and determining the emotional state having the largest output value among the plurality of output values as the emotional state of the target driver. delete

Images