WO2015152536A1

WO2015152536A1 - Method and device for detecting sleepiness using moving image-based physiological signal

Info

Publication number: WO2015152536A1
Application number: PCT/KR2015/002558
Authority: WO
Inventors: 최진관; 황성택
Original assignee: 주식회사 바이브라시스템
Priority date: 2014-03-31
Filing date: 2015-03-17
Publication date: 2015-10-08
Also published as: KR101536348B1

Abstract

Disclosed are a method and a device for reliably and accurately measuring a psychophysiological parameter in a subject. The measurement method comprises the steps of: acquiring a moving image from a psychophysiological test subject via a moving image; measuring a vibration parameter of the test subject from the moving image; generating a biological signal image on the basis of the vibration parameter; and generating a psychophysiological reaction parameter of the test subject by processing the biological signal image.

Description

Drowsiness detection method and device using video-based physiological signal

The present invention relates to a method and apparatus for predicting or detecting a driver's drowsiness and taking safety measures according to the result to promote the safety of the driver and the occupant. Specifically, the present invention relates to obtaining a physiological signal using a video obtained from a living body. The present invention relates to a method of predicting or detecting a driver's drowsiness and an apparatus applying the same.

In all traffic accidents in Korea, drowsiness driving due to fatigue accounts for 25 ~ 30% of all traffic accidents, rather than accidents caused by violation of major laws (center line violations, signal violations, speeding accidents, etc.). The mortality rate is high. The EU will implement restrictions on driver fatigue to reduce fatigue and drowsiness in commercial vehicle drivers, leading leading car manufacturers to develop competing similar technologies and carry on their vehicles. The existing technology has a low recognition rate of drowsiness when wearing sunglasses, glasses, masks, hats, etc., so it is problematic to secure reliability. There is no case.

As a conventional technique, a technology of determining a sleepiness by detecting a driver's eye state through a camera, imaging a driver's face using a CCD camera, and determining a sleepiness by detecting a change in the eye area from the image signal thus captured Technology to take driver's face image using camera, detect eye line from the face image and analyze minute movement of eye line to determine awakening state such as drowsiness and low attention, frequency of eye blink, pulse frequency of eyelid Technology to determine drowsiness based on physiological measurement parameters such as facial muscle loosening degree, eyes blink from driver's face image, eye opening degree, eyes closed time, etc. to determine drowsiness Technology to compare head orientation, exact position of eyes and nose from facial images captured by the camera There exists a technique etc. which determine a sleepiness degree by this.

As a technique of using EEG, the conventional EEG during driver's driving is memorized as reference EEG, after detecting EEG in real driving, this EEG is classified according to frequency, and the frequency of occurrence of each EEG classified in this way is calculated It is a technique of determining whether or not drowsiness by comparing with the reference EEG stored in advance.

One conventional patent document provides a method of installing a lamp for transmitting infrared rays toward a driver's face for night photographing, and imaging a driver's face image through a face-oriented camera to detect a drowsiness from the driver's face. By edge-processing the image from the acquired face image, the driver's face left and right end positions and face upper and lower end positions are detected, and the amount of movement thereof is calculated to determine whether the driver is asleep. At this time, the position of the driver's eyes or face or the movement state of the driver changes according to the driving state of the vehicle, so that the movement amount is compensated based on the driving state of the vehicle.

More accurate and faster methods for predicting and detecting driver drowsiness early are desirable and therefore require further study.

Biological signal acquisition method according to the present invention:

Photographing the driver while driving to obtain a plurality of continuous image information;

Analyzing the image information to extract the vibration parameter of the subject;

Generating psychophysiological parameters based on the vibration parameters;

Extracting the subject's physiological signal from the parameter; And

Predicting or detecting a drowsiness of the driver from the physiological signal.

According to an embodiment of the present disclosure, the method may further include generating and visualizing a corresponding image of the physiological signal of the subject using the physiological signal.

Physiological signal detection device according to the present invention:

An image acquisition unit photographing the driver to obtain a continuous image;

A light detector for continuously imaging the driver's image, an A / D converter for converting the captured image into image data, and analyzing the converted continuous image data to measure a vibration parameter, and to measure the measured vibration parameter. There is provided a driver drowsiness prediction or detection apparatus incorporating a camera, comprising: a processor configured to generate a biosignal image based on the display; and a display unit configured to display the generated biosignal image.

According to the present invention as described above, it is possible to predict or detect a drowsiness state of a driver driving a vehicle by using a camera that can reliably and accurately measure the psychophysiological parameters of the subject.

The present invention can be mounted on the vehicle to protect the driver and passengers from accidents caused by drowsiness.

1 is a flowchart illustrating an example of a method of obtaining a biosignal to detect a drowsiness state of a driver.

FIG. 2A is a schematic block diagram functionally categorizing an apparatus implementing the method shown in FIG.

Figure 2b schematically illustrates the configuration and flow of a driver drowsiness method according to the present invention and a system applying the same.

3A is a block diagram of an electronic device implementing the method illustrated in FIG. 1.

FIG. 3B is a block diagram showing correlations between components in the electronic device shown in FIG. 3.

4 illustrates the concept of matching the EEG signal and vibration parameters in the driver drowsiness prediction method according to the present invention, and FIG. 5 is a diagram illustrating a specific flow.

6 is a flowchart of a method for obtaining a biosignal of a driver in the driver drowsiness prediction method according to the present invention.

FIG. 7A illustrates the emission of bioenergy (aura) around a subject (driver) human body image formed by the amplitude component of the vibration image.

7B shows bioenergetics radiated around the actual image of the human body.

8A and 8B show the bio-image radiation according to the state of the subject, and FIG. 8A shows a stable state and FIG. 8B shows an unstable stress state.

Fig. 9A is a distribution graph of frequency components (biosignal images) of a human body vibration image in a stable state.

9B is a distribution graph of the frequency components (biosignal images) of the human body vibration image under stress.

10 is a flowchart illustrating an algorithm extraction method in an R & D stage in implementing the driver drowsiness prediction method according to the present invention.

11 is a flowchart of a driver's drowsiness prediction method using the extracted algorithm.

FIG. 12 shows a change of a parameter related to sleepiness among vibration parameters in a driver drowsiness prediction system according to the present invention.

Figure 13 illustrates a user interface appearing on a display in a system that actually performs the drowsiness prediction method according to the present invention.

14 illustrates a relationship between a camera photographing a driver and LED lighting.

Fig. 15 is a view for explaining an angle of view relationship with a driver when a camera is installed in a room mirror of a vehicle.

16 illustrates an example of a photographing mechanism for minimizing the influence of the vibration of the camera during photographing.

Figure 17 shows the result of the prediction of the present invention and the comparison of the drowsy state wearing the wireless EEG headset.

Hereinafter, a method of predicting or detecting a drowsiness of a driver according to the present invention and an apparatus applying the same will be described with reference to the accompanying drawings.

First, the concept of the present invention will be described below with reference to FIG.

The present invention obtains a real-time video from the driver as shown in FIG. 1 (11), processes (analyzes) the video to extract or generate vibration parameters (parameters), and uses the vibration parameters. The physiological signal is extracted (13). The driver's sleep state is evaluated (predicted or detected) 14 using the physiological signal. In the present invention, it is possible to take safety measures to awaken the driver from the situation notification and drowsiness by sound or display according to the detection result of the drowsiness state.

This method may be implemented by a device having a structure as shown in FIG. 2A. Fig. 2A is a functional block diagram of the drowsy state detection device. Referring to FIG. 2, the drowsiness prediction or detection detecting device according to the present invention includes a camera 21 for photographing a driver, an image processor 22 for analyzing an image obtained from the camera 21, and an image processor 22. The analysis unit 23 uses the signal to extract vibration parameters (parameters), generates a physiological signal, and processes and analyzes the physiological signal to evaluate an operator's drowsiness, wherein the application unit 24 includes: For example, the controller may be connected to a safety guard 25 including a display for generating a character or an image for arousing a driver and displaying the same or an audio device for reproducing sound or music. The image processing unit 22, the determination unit 23, and the application unit 24 may be implemented by an application program based on a central processing unit (CPU) or an application processor (AP). It can be implemented as a device or built-in to a system in an automobile.

In the present invention, the signal component corresponding to the EEG signal is extracted from the above parameters, thereby predicting or detecting the drowsiness of the driver. The signal corresponding to the EEG signal is extracted from the video, which may have properties similar to the EEG signal. For more accurate drowsiness prediction, by matching the EEG signal acquired from the real driver with the signal component extracted from the video, the vibration parameter component appearing in synchronization with the EEG signal in the drowsy state is analyzed and extracted. Or the corresponding component signal of the EEG signal for determination. The process of acquiring video from the driver, extracting vibration parameters, and extracting EEG-corresponding signals from the parameters will be described later in detail, and a system applying this method may be embedded in a vehicle, which is shown in FIG. 2B. It may have a structure as shown.

As shown in FIG. 2B, a black box for generating (A) an image signal (information) using a camera from a driver who is driving and storing the driving record including the image information and a built-in analysis built in a vehicle analyzing the same Send to system (B). In the analysis system, the extraction of vibration parameters and the like, and the analysis of the drowsiness state using the same are performed, and according to the result, the sound for converting the driver to the awake state by transmitting to various systems operating in conjunction with the vehicle system, Drive a device such as vibration (C). Separately, the vibration parameters are extracted from the black box by a terminal device such as a smartphone to analyze the driver's physiology, psychological state, for example, drowsiness, fatigue, emotional state, etc. Through the visual or auditory perception, the driver's physiological and emotional state can be controlled through various contents as well as prevention of drowsiness (E).

In such a system, the camera can be mounted on the driver's front contrast board, or on the other hand in a room mirror. The camera may have a structure for removing vibrations transmitted from the vehicle, which will be described later in detail.

Prior to the description of the method and apparatus for predicting or detecting a driver's drowsiness according to the present invention, the relationship between the vibration of each part of the living body and the psychophysiological parameters will be described.

In small particle physics, there is no clear boundary between the wave and particulate properties of matter, and the photon energy (ε) is known to be linked to the frequency of photon energy (ν) through Planck's constant (ε = hν). It is hypothesized that the energy emitted from each part of the organism is proportional to the vibration frequency of that part in the space. As a result, in order to record the energy coming from a living thing, it is necessary to record the vibrations (in or between spaces) that occur at various parts of the living thing. This can be done by using a contactless television system with sufficient resolution and fast processing power. In addition, the frequency components of the biosignal images obtained (ie, the vibration (position shift, wave) frequency at each site) have the most information on the bioenergy, or psychophysiological characteristics, of the observed organisms. Analysis of the obtained biosignal image may be performed by a person or mathematically by processing at least one of the obtained digital biosignal image and its components by a program. In order to prepare and analyze algorithms for mathematical processing, it is good to make a biosignal image which is convenient for visual analysis such as pseudo-color image of monitor screen.

In other words, the frequency component of the biosignal image to be obtained allows to continuously and clearly specify the levels of psychophysiological and emotional states of the human body and to distinguish the changes in the human state when various stimuli occur in humans. do. As it turns out, an image showing the human body's bioenergy field represented by an aura located around the human body can be used to evaluate the psychophysiological state of the human body faster and more accurately than other methods.

The term aura refers to an integral characteristic of the psychophysiological state of the human body. These auras appear around the human body and have specific relationships with the bioenergy components of the human body. The image of the human aura provides a lot of information when studying the psychophysiological parameters of the human body, and the following factors are considered. Human emotional state can literally change every second. The average person can not stay in one emotional state for a long time.

Every thought and action or reaction to a situation leads to a momentary change in the emotional state (images of each biosignal). Therefore, it is important to find the optimal correlation between the number of information on the obtained biosignal image (above all, the resolution of the camera) and the rapid processing of the system.

Adds an amplitude modulating of the aura size, and modulates the subject's maximum vibrational frequency by color modulating the frequency of the positional change or average value of the amplitude in a specific region of the human body, Changes can be recorded at a glance and instantaneously. Frequent fluctuations in the brain are known to play a key role in learning, memory, and solving a variety of tasks. Experiments have shown that the most intensive part of vibration in the human body is the brain, and in most cases auras (frequency components of vibration images) may be present only around the human head, much larger than the auras around the body. . Changes that occur in the human body are represented by the collapse of the aura or the appearance of asymmetry in color and form. This is clearly seen in the obtained biosignal image.

There is one point in that the elements of the biosignal image are associated with the elements of the actual image in topology. Experimental results show that the emotional state of the human body that contains the most information is transmitted at the maximum oscillation frequency, and the average level of the frequency or the background level of adjacent points can be shrouded or cover up the true changes that occur when visually accepting the biosignal image. It may be.

Therefore, it is shown that the geometrical correlation of the elements of the biosignal image to the elements of the real image is less effective than the frequency components of the vibration image represented by the aura located around the real image. When the elements of the biosignal image are topologically related to the elements of the actual image, the elements with the maximum vibration frequency are not visible in the entire background when the image is subjected to color-frequency adjustment. In order to mathematically analyze the biosignal image in various forms, the biosignal image to be obtained must be visually controlled in advance. The proposed image of the frequency component of the biosignal image, in the form of an aura, is consistent with the physical concept of bioenergy radiation and enables visual control and analysis of the device-generated image.

Unlike frequency components, the use of amplitude components is more effective in topological relationships. First of all, it is possible to evaluate the quality of the biosignal image obtained by using the amplitude component of the biosignal image, which is topologically connected to the vibration point, and to determine the exact parameters (variables) for adjusting the system.

First, the measurement of the vibration image parameter will be described in detail.

Acquiring information about the level of aggression of a creature consists of constructing a frequency distribution histogram and measuring the head vibration image parameters of the creature.

Aggregation of Aggression Levels (Ag) consists of:

Maximum frequency of frequency distribution density in Fm- histogram

"I" frequency aggregation of frequency distribution density obtained over N frame time in Fi- histogram

Fin-vibration image processing frequency

Aggregate including interframe differences beyond the threshold in n-N frames

The biological head vibration image parameter is measured to obtain information about the stress level of the creature. The stress level St is calculated by the following Equation 2.

-Total amplitude of thermal vibration image "I" in the left part of the subject

-Total amplitude of thermal vibration image "I" in the right part of the subject

-

From

Maximum value of liver

-"I" thermal vibration image maximum frequency in the left part of the object

Maximum frequency of "I" thermal vibration image on the right side of the subject

-

From

Maximum value of liver

n-the number of columns

To obtain information about the level of anxiety in the organism, the biological head vibration image parameters are measured. Anxiety level (Tn) is measured by the following Eq.

Pi (f)-Vibration Image Frequency Distribution Power Spectrum

fmax-maximum frequency of the vibration image frequency distribution spectrum

To obtain information about the level of compatibility of living things with other organisms, determine histograms of vibration frequency distributions for all individual organisms, construct each histogram, obtain common frequency distributions, and obtain the general law of distribution and the previously obtained co-distribution area. Make the same and find the difference between the general law of distribution and the frequency histogram. The compatibility level (C) is counted as the following <Equation 4>.

K-Acquired Frequency Histogram Generalized Correlation Coefficient

y '– normal distribution density

In determining verbal or nonverbal falsehoods, vibration image parameters of the head of the organism are measured to obtain information about the integrated level of change in the psychophysiological state.

The integrated level of change (L) in the psychophysiological state used in the false decision is summed up by the formula:

Pi-parameter to change the higher set threshold

Pc-Vibration image parameter measured when determining false level

K-Pi semantic correlation coefficient to be measured

n-the number of parameters to be measured

m-the number of parameters changed

As is well known, cybernetics and information theory examines the applicability of operational methods and techniques to organisms and living systems. Modern concepts of cognitive biology are usually related to the concepts and definitions of signal information and transfer theory, and enable the psychophysiological information of mathematical parameters established in information theory. The author's long study and observation of the study of human head micromovement with the help of statistical parameters used in information theory shows that there is a statistically reliable dependence between the state of human psychophysiology and the head micromovement information statistics parameter. Could know.

The present inventors can present their own interpretations of these phenomena and vestibular emotion reflections. First, we will define the interrelationship between psychophysiological energy coordination (metabolism). All typical emotional states can be characterized by a correlation between specific energy consumption and individual physiologically necessary energy and emotional energy. At this time, the physiological energy is formed to realize the physiological process, and the emotional energy is formed as a result of the conscious or unconscious process. For example, the attack state, if it is the same attack condition, should be expressed differently in various people, and natural adjustment process such as age, gender and education level should be considered. However, from a physiological point of view, these differences should not have a fundamental meaning in the relative amounts and locations of energy release within the body. All this leads to visible emotional signs, such as flushing of the face, frequent sighs, rapid heartbeats, and certain micromovements. The main reason for the external manifestation of the emotional state is due to the additional release of energy in the body organs which changes the correlation between physiological and emotional energy. It should be emphasized that the author took into account the body-chemical energy of natural physical processes, which are well known at the level of modern technology development. The progression of physiological processes, and the interruption and triggering processes for human thought and movement processes.

The main challenge of the vestibular system is, above all, to maintain a dynamic equivalence or equality. However, the study proved that the equilibrium state of the semi-closed system is possible only if the dynamics, chemistry, energy, and other systems (systems) forming the subject are equal. Unbalance in any one of these systems results in the destruction of the equilibrium state of the adjacent system, i.e. the breakdown of mechanical balance results in the breakdown of energy balance.

The human head in a vertical, semi-balanced state can be seen as an overly sensitive mechanical indicator of all the energy processes in the body. From a biomechanical point of view, maintaining the vertical balance and equilibrium of the heads far above the center of gravity requires tremendous continuous effort and reduction of the neck-head bone muscles. Moreover, this movement is realized reflexively under vestibular system. All meaningful phenomena in the organs lead to changes in the ongoing physiological process. This is similar to other physiological process changes traditionally used for psychophysiological analysis, such as galvanic skin response (GSR), arterial pressure, and heart rate. In addition, the parameters of head movement vary with the amount of energy expression and the location of energy expression. The spatial three-dimensional trajectory of head movements is very complicated because the shape of the head resembles a sphere. Also, the movement trajectory of each point can vary significantly in the movement of hundreds of neck muscles. Statistical analysis of informative motion parameters enables reliable quantitative parameter differentiation of head movements. In other words, it is possible to measure and confirm the emotional state through the measurement of energy and vestibular response. The laws of mechanics appear to be consistent, and behavior is always reactionary to maintain equality. Energy measurements in the body organs that naturally target a wide variety of people will result in consistent corresponding changes in head movement parameters through vestibular activity.

The overall emotional classification according to the informational / statistical parameters of the presented head movements confirms all emotional states. Currently there is no single, holistic approach to measuring emotional state, so it can be used for initial measurements in terms of other psychophysiological methods or independent experimental evaluations. Modern psychology mainly uses qualitative criteria in the evaluation of emotional state, which essentially makes it impossible to measure quantitatively, and the objective evaluation of human state is difficult. However, the suggested method allows us to measure all emotional states. Given that a change in head movement parameters is functionally related to a change in energy exchange, naturally, head movement parameters are a general characteristic psychophysiological state of man. The accuracy of agreement of the proposed formulas for counting emotional states according to existing assessment criteria is low compared to the emotional state assessment method through head micromovement. This is because there is no overall standard for assessing emotional state at the current level of technology. The proposed method is characteristic in that an integrated approach is possible for all emotion measurements. All previous methods were also used to assess various emotional states. Adopting the proposed concept for measuring emotional state allows the inclusion of psychology in precision science and enables the same emotional measurement.

Acquisition of the signal regarding the head movement of the subject is performed by comparing images by the camera. In terms of spatial and temporal distribution information statistics parameters, the movement speed of the head of the creature is measured as the average frequency of marker movement, determined in units of 10 seconds, which yields the maximum frequency of TV camera work. These characteristics can well reflect human emotional anxiety and can characterize anxiety levels.

When the vibration image simultaneously represents the spatial and temporal distribution of the target motion energy, the number of factors having the same vibration frequency for a specific time is aggregated to obtain a frequency histogram. Histograms therefore exclude information about the spatial distribution of vibration frequencies. This apparent loss of spatial information actually increases the motion information, because in terms of physiological energy, it is not very important in which part of the head the movement is performed unlike the fine movement of the face. The configuration of the frequency histogram is determined according to the following.

As is well known, a new formula has been proposed that takes into account two main factors that differ from the existing contradictory approaches that determine the level of aggression. Two main factors are the average vibration frequency, or human head motion and parameters, and the mean square deviation, which best show the characteristic spread of the vibration. This aggressive person has a high spreading frequency in movement of high frequency of hair fine movement and various points of the head part. Other official correlation coefficients show aggressive correlation coefficients for values from 0 to 1.

Fm- Maximum frequency of frequency distribution density histogram

Number of i-frequency aggregates in frequency distribution density histograms obtained per fi-50 frame hour

Fin-vibration image processing frequency

Aggregate number of interframe differences above the threshold in n-50 frames

This equation allows everyone to determine their level of aggression, with naturally lower aggression levels approaching zero. People with high aggression are close to one. The use of a vibrating imaging system's security system to identify potentially dangerous ones uses a threshold of 0.75 to identify aggressive ones.

The next step is to statistically identify meaningful vibration image information parameters that determine vibration image acquisition and subsequent aggression levels. This determines, among other things, vibration symmetry parameters for amplitude and frequency vibration images.

Unlike the well-known and contradictory approaches that determine the level of aggression, a new formula has been proposed that takes into account the motion amplitude and frequency symmetry for individual columns that scan the human head. As such, the person with the highest level of aggression exhibits maximum symmetry in vibration and fine movement for processing amplitude and frequency vibration images for 20 seconds. At the same time it shows low levels of stress and anxiety.

-Total amplitude of thermal vibration image "I" in the left part of the subject

-

From

Maximum value of liver

-"I" thermal vibration image maximum frequency in the left part of the object

-

From

Maximum value of liver

n – the number of columns occupied by the target

Similar to the information statistics parameters presented previously, the formula presented allows us to measure the stress level (St) from 0 to 1, and above all, the minimum stress level corresponds to the minimum measurement, In people with stress levels close to one.

The following is a statistical analysis of meaningful vibration image information parameters that determine vibration image acquisition and subsequent anxiety levels. This relates, among other things, to the fast activity signal frequency spectrum construction of amplitude and frequency vibration images.

Unlike the well-known and contradictory existing approaches that determine the level of anxiety, a new formula has been proposed that takes into account the fact that high anxiety increases high frequency spectral density rather than low frequency spectral density.

Tn-anxiety level

Pi (f)-Vibration Image Frequency Spread Power Spectrum

fmax-vibration image frequency spread spectrum maximum frequency

The presented formula, similar to the previously presented information statistics parameters, allows us to measure the level of anxiety from 0 to 1. In addition, the minimum level of anxiety meets the minimum measure, and those with high levels of anxiety have stress levels close to one. The fast signal frequency spread spectrum of the vibration image appears for the operator's or system operator's control.

Another example is to find statistically meaningful informative parameters of the vibration image that determine the vibration image acquisition and then the level of compatibility between the people. Best of all, this consists of a vibration image histogram configuration at each individual frequency.

In contrast to the well-known and contradictory existing approaches that determine the level of compatibility, a new formula that considers the possibility of compatibility is characterized by a close proximity to the total vibration frequency histograms of both sides of the distribution normal law. Presented.

K -normalization correlation coefficient of the first histogram

Similar to the parameters presented previously, the proposed formula measures the level of compatibility from 0 to 1. In addition, the minimum measure corresponds to the minimum compatibility (compatibility), and the high level of compatibility measure on both sides appears close to one.

Next is to find a statistically meaningful information parameter of the vibration image that determines the vibration image acquisition and human false level. First of all, it is related to the acquisition of temporary dependence of the maximum amount of vibration image parameters with the least correlation between them.

A new formula has been proposed for the detection of lies that differs from the well-known existing psychophysiological approaches. False in this formula is determined by the change in vibration image parameter measurements compared to the reporting time. The formula presented allows us to determine whether we are verbal and nonverbal. Basically, the linguistic false decision in the time dimension utilizes the time to the start of the test subject's answer. In the case of non-verbal false analysis, it is made by comparing the parameters with one time period and another time period.

In the case of a false decision, the integrated level of change (L) of the psychophysiological state utilized is calculated by the formula:

Pi-change parameter for a higher set threshold

Pc-Vibration image parameter changes when determining false level

K-Measured Pi Semantic Correlation Coefficient

n-number of measurement parameters (may vary from the number of visual parameters)

m- number of change parameters

Similar to the parameters presented previously, the formula presented allows us to measure false levels from 0 to 1. In addition, the minimum level of false matches the minimum measure, while the highest level of false has a value close to one.

This does not mean, however, that the present invention is utilized solely for the measurement of emotional and psychophysiological states of humans presented above. For reference, there are over 200 classifications of human state characteristics according to various classification systems. Above all, the present invention allows us to describe all human conditions through the head micromovement parameters and / or the head vibration image parameters. In psychology, it is an unclear principle to translate the traditional concept of motion into reflex micromovement of the human head using reliable statistical parameters. However, based on the proposed approach, it is possible to determine the human state similarly to the technical information system and to utilize the information parameters to characterize the human state. For example, it is possible to determine the level of human information and thermodynamic entropy according to the formula.

The head fine motion frequency distribution histogram is constructed as the basis for the informational entropy calculation. The informational entropy (H) calculation is based on the following formula.

As a basis for the thermodynamic entropy calculation, a histogram of the frequency distribution of the micro-movement of the head is constructed. The thermodynamic entropy (S) calculation follows the following formula.

These individual information statistics parameters are applied to better identify any emotional state in humans. For example, experiments have shown that there is a high correlation between informational entropy on false levels, and thermodynamic entropy is a state of anxiety in humans. It was found that there is a big connection with.

Based on body and thermodynamic parameters, we were able to more fully characterize and determine human behavior, energy and charismatic aspects. For example, using a vibration image 7.1 version system, human energy (E) was able to be based on the difference between the mean square error and the frequency peak, based on a frequency histogram representing the highest frequency of the vibration image recording.

Quantitative analysis of the reflexive micromovement of the head makes it possible to measure the psychophysiological state of humans more objectively and scientifically and to solve many problems in medicine, psychology, psychology, and everyday life. Through independent experiments with the system developed and the quantitative assessment of the psychological status of passengers at the airport according to aggression, stress, anxiety and potential risk levels, the present invention is positive (more than 90%) to the expert's professional evaluation. I could tell the truth. This confirms the practical feasibility of the present invention.

Hereinafter, a car built-in drowsiness prediction or detection system for performing the method of the present invention will be described with reference to the accompanying drawings.

In the device or system shown in Figs. 2A and 2B, which are functionally blocked, the camera 21 installed in a dashboard or a room mirror in an automobile is used to digitize an imaging device such as a CCD or CMOS and an analog signal therefrom. / D converter, and the image processor 22 includes an encoder for generating a video of a specific format.

As described above, the signal analyzer 23 measures the vibration parameter by the method as described above using the image, and generates or extracts psychophysiological information (signal or physiological signal) therefrom. Here, the vibration parameters include vibration frequency, amplitude, and phase according to the change of position of each part of the subject. The psychophysiological information includes psychological / emotional / emotional states such as a stable state, an excited state, and a stress state, which predict or detect drowsiness of the driver.

Meanwhile, the physiological signal application unit 24 may include a physiological signal processing algorithm for evaluating the mental and emotional state of the subject using the physiological signal and a display for displaying the result. For example, the physiological signal processing algorithm may classify the subject's states into nine emotional states according to, for example, James Russell's two-dimensional emotional model. The display provided in the application unit 24 displays the final result in the form of a text or an image as described above. In particular, the application unit 24 may include various safety measures systems for transitioning from the drowsy state of the driver to the awake state. This may include a horn-like sound, or a music or video device that can wake you up. In addition, it may include a vibration device that can be installed in the driver's seat or the like to physically stimulate the driver.

Such a device may be based on various types of systems, not only general OS for PC but also portable OS such as Windows mobile, Android, iOS, Symbian, BlackBerry, Bada.

FIG. 3 shows a driver drowsiness prediction or detection device 30 that forms part of a system implementing the device or system shown in FIGS. 2A and 2B.

Referring to FIG. 3, the driver's drowsiness prediction or detection device 30 includes an A / D converter that digitizes an analog image signal of an image photographing unit having an imaging device, that is, the camera 31 and the camera 31. 32), a processor 34 for performing image analysis and parameter extraction and physiological signal generation, drowsiness prediction or detection1, and a display unit 35 for displaying the result. In addition, the apparatus includes an input device for inputting information from the outside, for example, a key input unit 36 such as a keypad and the like, and a storage unit 33 including a memory used in the above-described image signal processing.

In order to predict and detect driver's drowsiness using a video according to the present invention as described above, as shown in FIG. 4, the driver's brain waves acquired by an actual EEG device (EEG equipment) in the actual EEG measurement process (I) and In addition, it is preferable to extract an algorithm of a specific parameter of psychophysiology for the drowsiness state by matching the video information acquired through the video shooting process II. Specifically, referring to FIG. 5, EEG equipment 40 measures brain waves 41, and measures and analyzes brain waves in a drowsy state through measured brain waves (42) to identify brain waves in a drowsy state (43). The vibration parameters are extracted from the moving image 45 obtained through the imaging device 44 (46), and the vibration parameters synchronized with the brain waves in the drowsy state are matched (47). The parameters correlated to the sleepiness obtained through the matching are applied to the sleepiness prediction and detection program (48). According to the experiments of the present invention, in the EEG, the delta component of C4 / F4, the alpha component of F3, and theta component of F3 are related to drowsiness and match three vibration image parameters ( F3, P17, P8F) were extracted.

6 is a flowchart illustrating a process of acquiring a biological signal through a camera based on a video according to the present invention.

Referring to FIG. 6, first, an image of a driver is acquired by a camera 31 and converted into an analog electric signal (S10 and S20). The electrical signal obtained from the driver's image is an analog signal and is therefore converted into digital image data by the A / D converter 32 (S30).

In a next step, the processor 34 calculates a vibration parameter by analyzing the change over time of each image data (S40). The vibration parameter includes at least one of a vibration frequency, an amplitude, and a phase according to a change in position of each part of the subject. That is, the processor 34 analyzes the position change of each part of the driver to calculate the vibration frequency of each part, the magnitude of the position change (the magnitude of the vibration), the phase, and the like. Subsequently, the processor 34 may analyze the difference between the images using a vibration image analysis program, measure a position change with respect to the center of gravity, or calculate (calculate) a vibration parameter (parameter) using a Fourier transform.

The vibration parameter calculation is described in more detail as follows. The processor 34 grasps the movement or vibration of the contour according to the driver's movement from a plurality of consecutive images and separates the contour into two equal parts (left and right). Then, determine the point indicated by the maximum vibration frequency in two parts of the row divided in half. This frequency determines the color of the corresponding horizontal row of the biosignal image. The average amplitude of the positional variation in each of the two sections of the row divided in half located in the separate contour section determines the size (length) of the biosignal image. Vibration images obtained at each point have certain positive and static characteristics, but integrated biosignal images are associated with psychophysiological parameters of the human body. This is the case in which the portable device with the camera is fixed to a non-moving support and is not affected by vibration from the outside.

Since the camera is installed in a vehicle, vibration or movement of the camera occurs, so it is preferable to go through the process of removing (filtering) the vibration component. Such filtering may include some or all of the various noise components that act as noise as well as vibration of the auto.

To this end, after A / D conversion of the target image converted into an electrical signal, it is necessary to perform a process of receiving and removing vibration transmitted from the hand as noise before calculating the vibration parameter (S40).

The processor 34 generates a biosignal image based on the calculated vibration parameter (S50). The biosignal image may include an amplitude component and a frequency component. Hereinafter, the amplitude component is referred to as "internal biosignal image" and the frequency component is referred to as "external biosignal image". The concept of this term definition will be understood in the description of FIG. 5 below.

Finally, the processor 34 obtains psychophysiological information of the subject 1 from the calculated vibration parameter (S60). That is, the processor 34 may know the psychological state of the object 20 by analyzing the vibration parameter, and in particular, may predict or detect the drowsiness of the driver by the algorithm as described above.

FIG. 7A illustrates the emission of an aura of bioenergetic energy around an image of a human body formed from the amplitude component of a vibrating image.

As described above, the internal biosignal image expresses the magnitude of the change in position of each part in color. Through this it is possible to visualize the magnitude of the position change of each part of the subject (1). The external biosignal image appears around the internal biosignal image and modulates the average peak vibration frequency into color.

7B illustrates that a biosignal image, which is bioenergy, is radiated around an actual image of the human body. In FIG. 5B, the internal biosignal image is not represented and only the biosignal image is displayed around the actual image.

8A and 8B show biosignal images in a stable state and an unstable state, respectively. FIG. 8A shows a biosignal image of a subject in a stable or finished state and FIG. 8B in a stress state.

Referring to FIG. 8A, the biosignal image is sufficiently symmetrical in shape and color, and the color of the biosignal image is about halfway between the selected color scale (overall color-green). The bio signal image shows that the driver is in a stable state.

On the other hand, referring to FIG. 8B, the aura contains a lot of red components in the biosignal image. Thus, it can be seen that the driver in this state is in an unstable state. When a person is stimulated, for example, exposed to a violent scene through the screen, the subject becomes stressed or aggressive and the color of the biosignal image changes to a reddish color.

FIG. 9A is a distribution graph of frequency components (biological signal images) of a human body vibration image in a stable state, and FIG. 9B is a distribution graph of frequency components (biological signal images) of a human body vibration image in a stress state.

The graph shown in FIG. 9A shows a typical frequency distribution of a person in normal working condition. The results show that the majority of people in a calm state generally have a distribution of distributions similar to a single-mode distribution rule. When the subject is subjected to a certain negative effect such as watching a violence scene on the screen, the state of the subject changes as shown in FIG. 8B. If in fear, stress and aggressive conditions, the mean (medium) value of the frequency distribution (M) shifts towards increasing. In a stable and relaxed state, the mean (middle) value of the frequency distribution value (M) is shifted toward decreasing. The frequency axis (X) can be expressed not only in relative units, but also in real units or time (㎐ or sec.). The distance between the display values is determined by the actual parameters of the camera's rapid processing and the software's settings (time to accumulate images and the number of images in the processing sequence).

10 is an operation flowchart of a drowsiness prediction program according to the present invention, which is an experimental flowchart of a process of extracting a drowsiness prediction algorithm by matching EEG and vibration parameters in a Research & Development (R & D) step.

The experiment is started, and first, the EEG sensor is worn by the driver or the subject to be measured (S10a), and the camera for observing the subject's head or the upper body including the same is prepared (S10b). In this state, while inducing drowsiness to the driver, while recording a video (S10d) and at the same time measures the EEG (S10c). From the video, the vibration parameter is extracted by the method as described above, and the driver's drowsiness is measured (S10f). Then, by comparing and analyzing the EEG component and vibration parameters (S10f), and extracts the vibration parameters related to sleepiness (S10h). When the vibration parameter is extracted, an algorithm for predicting drowsiness is prepared or extracted using the extracted parameter (S10i). The drowsiness prediction algorithm created or extracted here is applied to the actual drowsiness prediction apparatus.

11 illustrates a process of predicting a driver's drowsiness using the above-described drowsiness prediction algorithm and transmitting and stimulating a stimulus when the drowsiness is predicted.

Referring to Fig. 11, the sleepiness prediction algorithm is driven in a state in which a system according to the present invention is mounted on a real vehicle (S11a) (S11b). The driver's drowsiness is predicted in real time based on the vibration parameter extracted from the moving image obtained from the driver using this drowsiness prediction algorithm (S11c). As a result of the prediction of drowsiness, it is determined whether it is a mild attention state (S11d) or a warning state (S11e), where the level of the attention state or warning state is compared with a threshold value (S11f), and if the threshold value is less than or equal to the above-described process, the infinite loop Circulation is carried out, and if it is above a threshold, an appropriate stimulus is generated (S11g). Appropriate stimulation is to allow the driver to escape from the drowsiness, and may include anything that can stimulate at least one of the five senses of the human being, such as vibration, sound, and aroma.

12 shows the change in the parameters related to sleepiness among the vibration parameters. In Figure 12, an arbitrary number 20 represents the level of the variable, where 20 is a threshold indicating the start of sleepiness. As shown in Fig. 12, the driver initially maintained the awakening state of 20 or more, and after a certain time, the drowsiness parameter began to fall below 20, that is, the drowsiness precursor appeared, and about 14 seconds had elapsed. At the time of complete drowsiness.

Equation 14 below mathematically expresses an algorithm for predicting sleepiness.

Ca – the number of pixels inside the face

Ii – the intensity of pixels needed for detection inside the face

Drowsiness variable 1 (SDP 1) is a value between 0 and 1, representing a numerical value according to the sum of the values of Ii (intensity for pixels). As the drowsiness increases, the number decreases

As the awakening state increases, the number is close to 1. The sleepiness variable 1 (SDP1) threshold for predicting sleepiness uses 0.2.

K: Rate of change constant for sleepiness variable 2 (SDP2)

(Standard deviation of drowsiness variable 2 / mean value of drowsiness variable 2)

Wm: Size of external biometric image on the left

Wn: Size of external biometric image on the right

Cm: Color control size of left external bioimage

Cn: Color control size of the external biological image on the right

N: time

The change rate constant K for drowsiness variable 2 (SDP) is a change rate constant obtained by dividing the standard deviation of the drowsiness variable 2 by the mean value of the drowsiness variable 2 (Mean of SDP 2).

The above two expressions evaluate the drowsiness precursor by And operation.

That is, using the rate of change constant (K) of drowsiness variable 1 (SDP1) and drowsiness variable 2

When the threshold 20 of the drowsiness variable 1 (SDP 1) and the threshold value 0.05 of the change rate constant (K) of the drowsiness variable 2 are satisfied at the same time, it is determined as the drowsiness state prediction.

In FIG. 13, (a) is a menu screen for determining performance of the sleepiness prediction method, (b) is a sleepiness prediction initialization screen, and (c) shows a photographing screen of the driver in a state where the sleepiness prediction method is performed. As shown in (c), masking is applied around the periphery so that only the driver part appears in the video. In this state, when the driver's drowsiness is predicted, a warning sound can be heard along with the screen as shown in (d).

As shown, the LED light source irradiates light to the driver's face, and the camera receives the light reflected from the driver's face. At this time, the vibration or movement of the driver may perform frequency filtering with a synchronization structure based on a difference between perspective and reflection of the LED light.

Fig. 15 is a view for explaining an angle of view relationship with a driver when a camera is installed in a room mirror of a vehicle. Considering the internal structure of the vehicle, the position of the camera is optimal for the room mirror, and therefore, it is preferable to mount the camera and the lighting device on the room mirror itself. The camera installed in the room mirror needs to be positioned so that the driver's head-neck can be photographed. The photographing angle of the camera is about 90 at the top, bottom, left and right, and the size of the image is preferably 30% or more of the image frame.

The LED light source emits light to the driver's head-face portion in the form of a pulse, and the camera captures an image only at the pulse time in synchronization with the pulse. The vibration of the driver's head is analogous and continuous, but the vibration of the body is monotonous. Since the vibration of the human head is less than 10 Hz, it is possible to apply multi-sided filtering with the pulse of the LED light source being about 1 micro second. On the other hand, the vibration still present in the image can be attenuated by applying a software filter.

On the other hand, the vibration of the camera can be extremely suppressed by applying an optical image stabilizer (OIS) structure that has a built-in gyro sensor.

As a result of applying the present invention as described above, as a result of detecting the change value of the drowsiness variable in the physiological precursor state before the start of the drowsiness, it was possible to detect the signal of the physiological change that can predict the drowsiness state in the 11-12 second period. In addition, the driver's drowsiness could be predicted with an average detection rate of 90% 11-11 seconds before the drowsiness began.

Figure 17 shows the result of the prediction of the present invention and the comparison of the drowsy state wearing the wireless EEG headset. As shown, it can be seen that a collision (drowsiness occurrence) occurred after about 12 from the time point when the drowsiness prediction value was the highest (drowsiness precursor time point). According to the experiment, the eyelid closure time was predicted as a drowsiness precursor about 11.6 seconds before the occurrence of drowsiness according to the present invention, and the drowsiness precursor was confirmed 11-12 seconds before the drowsiness prediction result using the wireless brain wave on the US ABM.

In the driver drowsiness prediction system according to the present invention, a camera mounted in an interior of a vehicle, for example, a room mirror, may store information obtained by a program for photographing a driver and predicting and detecting a drowsiness state with an IVN (In-Vehicle Network) of the vehicle. It is possible to be connected. According to the present invention, various arousal systems may be applied according to the predicted drowsiness state, and for example, sound, image, or vibration may be applied. In other words, the software contained in the main body detects the driver's drowsiness and enters the vehicle driving information from the IVN through a program that executes an algorithm according to the present invention with the real-time image information of the driver obtained from the camera in the room mirror while driving the vehicle. In addition, it can be interlocked to take measures to prevent safety accidents caused by drowsy driving by generating alarm sound, fastening seat belt and steering wheel vibration, which causes awakening state according to vehicle driving state. According to the present invention, it is possible to detect not only the drowsiness state of the driver, but also the fatigue state, the stress state, the depressed state, etc. by using the bio-signal image analysis algorithm based on the video, so that the load generated during driving by using the driving conditions can be detected. It can be combined with measurement technology.

The present invention is a technology that has not existed until now, since it has been proved to have a very high reliability since a drowsiness prediction detection program was developed through mapping of EEG and video of the driver in real time and mapping with psychophysiological response parameters. . It is a supplementary service that detects drowsiness status in real time by taking a video of the driver and detects the driver's mental function state and emotional state (driver's fatigue, stress, concentration and depression level) and provides the driver with information about it. Very scalable. The present invention provides convenience without limiting the driver because the driver can detect the drowsiness even when the driver wears sunglasses, glasses, a mask, a hat, or the like.

To date, various exemplary embodiments of the present disclosure have been described and illustrated in the accompanying drawings. However, it should be understood that these embodiments are only part of the various embodiments. Various other modifications may occur to those skilled in the art.

Claims

Obtaining a video from a driver using a camera;
Measuring a vibration parameter of a subject from the video, and removing the natural frequency transmitted from the driver in the vibration parameter;
Generating a biosignal image based on the vibration parameter;
Processing the biosignal image to generate psychophysiological response parameters of a subject; And
Extracting a parameter related to sleepiness among the parameters and predicting sleepiness of the driver using the same; Drowsiness detection method using a video-based physiological signal comprising a.
The method of claim 1,
The vibration parameter includes at least one of a vibration frequency, an amplitude, and a phase according to a change in position of each part of the driver,
The measurement value of the amplitude is a drowsiness detection method using a video-based physiological signal, characterized in that represented by the coordinate value of the position change of each part.
The method of claim 2,
Drowsiness detection method using a video-based physiological signal characterized in that to apply the optical image stabilizer (OIS) technology capable of image stabilization to the camera.
The method of claim 1,
And matching the parameters with the EEG signals obtained from the driver and extracting a parameter matching the EEG signal correlated with the drowsiness. The drowsiness detection method using a video-based physiological signal further comprising.
The method of claim 2,
And matching the parameters with the EEG signals obtained from the driver and extracting a parameter matching the EEG signal correlated with the drowsiness. The drowsiness detection method using a video-based physiological signal further comprising.
The method of claim 3,
And matching the parameters with the EEG signals obtained from the driver and extracting a parameter matching the EEG signal correlated with the drowsiness. The drowsiness detection method using a video-based physiological signal further comprising.
In the drowsiness detection device which performs the method of Claim 1,
A camera obtaining a video signal from the driver;
A processor configured to analyze the image data to measure the vibration parameter and extract the vibration parameter;
An analyzer for predicting drowsiness of the driver using the vibration parameter;
Drowsiness detection apparatus using a video-based physiological signal comprising a; application unit for awakening the driver in accordance with the results from the analysis unit.
The method of claim 7, wherein
The vibration parameter includes at least one of a vibration frequency, an amplitude, and a phase according to a change in position of each part of the driver,
The measurement value of the amplitude is a drowsiness detection apparatus using a video-based physiological signal, characterized in that represented by the coordinate value of the change in position of each part.
The method of claim 7, wherein
Drowsiness detection device using a video-based physiological signal characterized in that to apply the optical image stabilizer (OIS) technology that can be shake correction to the camera.
The method of claim 9,
Drowsiness detection apparatus using a video-based physiological signal characterized in that for extracting a parameter matching the EEG signal correlated with drowsiness by matching the parameter and the EEG signal obtained from the driver.
The method of claim 7, wherein
Drowsiness detection apparatus using a video-based physiological signal characterized in that for extracting a parameter matching the EEG signal correlated with drowsiness by matching the parameter and the EEG signal obtained from the driver.
The method of claim 8,
Drowsiness detection apparatus using a video-based physiological signal characterized in that for extracting a parameter matching the EEG signal correlated with drowsiness by matching the parameter and the EEG signal obtained from the driver.
The method of claim 7, wherein
The application unit drowsiness detection apparatus using a video-based physiological signal, characterized in that at least one of the stimulation device of the sound, image, vibration according to the result.
The method of claim 12,
The vibration stimulation device is a drowsiness detection device using a video-based physiological signal, characterized in that provided on at least one side of the steering wheel, driver's seat of the vehicle
The method of claim 7, wherein
The application unit drowsiness detection apparatus using a video-based physiological signal further comprises a display for displaying the result.