CN103340637B - Driver alertness monitoring system and method for intelligent eye movement and EEG Fusion - Google Patents

Driver alertness monitoring system and method for intelligent eye movement and EEG Fusion Download PDF

Info

Publication number
CN103340637B
CN103340637B CN 201310225220 CN201310225220A CN103340637B CN 103340637 B CN103340637 B CN 103340637B CN 201310225220 CN201310225220 CN 201310225220 CN 201310225220 A CN201310225220 A CN 201310225220A CN 103340637 B CN103340637 B CN 103340637B
Authority
CN
Grant status
Grant
Patent type
Prior art keywords
alertness
eeg
eye
module
driver
Prior art date
Application number
CN 201310225220
Other languages
Chinese (zh)
Other versions
CN103340637A (en )
Inventor
孙杳如
李洁
曹磊
朱华平
徐一菲
Original Assignee
同济大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Grant date

Links

Abstract

本发明提供一种基于眼动与脑电融合的驾驶员警觉度智能监控系统及方法,该系统包括:实时采集驾驶员的眼动信息的眼动信号采集模块;与眼动信号采集模块通信相连,从眼动信息中分析和提取眼动警觉度特征的眼动信号处理模块;实时采集驾驶员的脑电信号的脑电信号采集模块;与脑电信号采集模块通信相连,从脑电信号中分析和提取脑电警觉度特征的脑电信号处理模块;分别与眼动信号处理模块和脑电信号处理模块通信相连,对驾驶员当前的眼动警觉度特征和脑电警觉度特征融合后形成的融合警觉度特征进行警觉度状态评定的警觉度监测模块。 The present invention provides a smart driver alertness monitoring system and method EEG and eye movement based on the fusion, the system comprising: a real-time acquisition of the driver's eye movement eye movement information signal acquisition module; a communication module connected to the signal acquisition and eye , eye movement analysis and extraction alertness eye feature from the eye movement signal processing module information; real-time acquisition of the EEG signal acquisition module driver's brain; connected EEG acquisition module in communication with the electrical signal from the brain extraction and analysis of EEG features alertness EEG processing module; respectively, after the signal processing module, and the communication processing module is connected with the EEG eye, the current eye movement of the driver alertness and EEG features feature fusion formed alertness fusion characteristics of alertness alertness state evaluation alertness monitoring module. 本发明通过融合脑电信息与眼动信息监测驾驶员的警觉度状态,判断是否是非安全驾驶,具有较好的实时性与较高的可信度。 Movable state of alertness of the present invention by fusing the driver information monitoring EEG and eye information, determines whether a non-secure driver, has better real-time and high reliability.

Description

基于眼动与脑电融合的驾驶员警觉度智能监控系统及方法 Driver alertness monitoring system and method for intelligent eye movement and EEG Fusion

技术领域 FIELD

[0001] 本发明属于生物信号感知与处理技术领域,涉及一种驾驶员警觉度智能监控系统及方法,尤其涉及一种基于眼动与脑电融合的驾驶员警觉度智能监控系统及方法。 [0001] The present invention belongs to the biological signal sensing and processing technologies, relates to system and method for intelligent monitoring driver alertness, particularly to a driver alertness monitoring system and method for intelligent eye movement and EEG fusion.

背景技术 Background technique

[0002] 警觉度指人集中精力执行一项操作任务时所表现出的灵敏程度,包括对疲劳和瞌睡状态的度量。 [0002] refers to the sensitivity level of alertness when a person performs a focus operation exhibited tasks, including a measure of the state of fatigue and drowsiness. 一些特殊的工作,如空中管制中心的管制员、飞行员和高速公路上的长途客车驾驶员等都需要保持很高的警觉度。 Some special work, such as air traffic control center controllers, pilots coach driver on the highway and so the need to maintain a high degree of vigilance. 对人的警觉度的精确估计以及实时监测是人机交互系统研究中的一项非常重要的课题。 Accurate estimation and real-time monitoring of people's awareness of human-computer interaction is the systematic study of a very important issue. 尤其是司机长时间驾驶后将产生疲劳和警觉度下降, 通过警觉度分析可以有效判断司机的疲劳程度,防止交通事故的发生。 Especially after a long drive driver fatigue and alertness decreased alertness through effective analysis can determine the extent of driver fatigue, prevent traffic accidents.

[0003] 在以往研究警觉度和睡眠程度中,眼电信号是被广泛识别的信号。 [0003] In previous studies of sleep and alertness degree, the electrical signal is a signal eye is widely recognized. 因为在不同睡眠阶段眼睛转动的频率不同,所以可以根据眼电信号来判断睡眠的阶段。 Because different at different sleep stages eye rotation frequency, it is possible to determine sleep stages according to an electrical signal eye. 现有的一种基于眼电信号的警觉度检测系统(申请号为201110066235. 7)包括信号采集系统、信号处理系统和反馈系统;信号采集系统采集眼电模拟信号并进行放大、滤波和数模转换处理后输出特征数据至信号处理系统;信号处理系统对输入的眼电信号进行特征提取并估计出警觉度状态后输出至反馈系统;反馈系统在满足警告条件时发出报警。 Existing detection systems based on eye alertness electrical signal (Application No. 201110066235.7) comprises a signal acquisition system, the feedback system and a signal processing system; EOG signal acquisition system analog signal and amplifies, filters and digital-analog after the conversion process to the signal output characteristic data processing system; electrical signal processing system of the eye, and the input feature extraction after estimating the output feedback system to the state of alertness; feedback system alarm in alarm condition is satisfied. 该基于眼电信号的警觉度检测系统能够提供比眼部视频更全面更准确的信息;结合慢速眼动、快速眼动、眨眼等多种从EOG(electrooculography,眼电图)中提取的特征,并且使用了支持实时的线性动力系统去噪方法,可及时准确地反映使用者的疲劳状态,并对超过一定程度的疲劳产生报警。 The electrical signal based alertness eye detection system can provide more complete and accurate than the eye of video information; binding slow eye movements, REM, and other extracted from the blink EOG (electrooculography, EOG) feature and using the real-time denoising linear dynamic system, can accurately reflect the fatigue state of the user in a timely manner, over a certain level and fatigue to generate an alarm.

[0004] 此外,由于脑电信号的变化通常先于人的面部表情变化和肢体动作而变化,能更及时更准确地反映人的警觉度状态,随着脑电分析技术的日益进步,脑电信号也越来越多地担当警觉度分析的重任。 [0004] In addition, changes in EEG usually precede changes in people's facial expressions and body movements change due to, more timely and more accurately reflect the state of alertness of the people, with the increasing advances in EEG analysis techniques, EEG signals are also increasingly take a leading role alertness analysis.

[0005] 现有技术中有一种用于驾车司机警觉度测定的脑电信号识别检测方法(申请号为201010548388.0),该方法通过将脑电信号的测试段频域序列分频段计算最大相关度和特征值之间的最小冗余度后,采用基于高斯核的支持向量机方法通过对每段时间所处的警觉状态进行分类,实现轻度瞌睡识别。 [0005] One prior art method of driving an electric signal identifying the driver alertness detection assay for the brain (Application No. 201010548388.0), by the method of EEG frequency bands test section calculates the maximum correlation sub-domain sequence and after the minimum redundancy between the characteristic values, are classified using a Gaussian kernel support vector machine based on the alert state by the time at which each segment to achieve recognition mild drowsiness. 该方法通过脑电信号识别人脑进入瞌睡之前的轻度瞌睡状态,来预测并防止警觉度的进一步下降。 The method of signal recognition by the human brain to enter doze state before mild drowsiness, to predict and prevent further decreased alertness. 但该方法仅采用了S段、Θ段、α段、β段、 γ段5个频段,这些频段会受到环境和人为因素的影响,产生偏差。 However, this method uses only the S segment, segment [Theta], [alpha] segment, segment beta], gamma] segment 5 bands, these bands can be influenced by environmental and human factors, vary.

发明内容 SUMMARY

[0006] 鉴于以上所述现有技术的缺点,本发明的目的在于提供一种基于眼动与脑电融合的驾驶员警觉度智能监控系统及方法,用于解决现有技术中对驾驶员的警觉度状态评测不准确、易受外界环境和人为因素影响产生偏差的问题。 [0006] In view of the foregoing disadvantages of the prior art, an object of the present invention is to provide a driver alertness monitoring system and method for intelligent eye fused with EEG based on the prior art for solving the driver alertness state evaluation is inaccurate, susceptible to external environmental factors and human impact of bias problem.

[0007] 为实现上述目的及其他相关目的,本发明提供一种基于眼动与脑电融合的驾驶员警觉度智能监控系统及方法。 [0007] To achieve the above objects and other related objects, the present invention provides an eye based intelligent driver alertness monitoring system and method of the EEG fusion.

[0008] -种基于眼动与脑电融合的驾驶员警觉度智能监控系统,该监控系统包括:用以实时采集驾驶员的眼动信息的眼动信号采集模块;与所述眼动信号采集模块通信相连,且用以从所述眼动信息中分析和提取眼动警觉度特征的眼动信号处理模块;用以实时采集驾驶员的脑电信号的脑电信号采集模块;与所述脑电信号采集模块通信相连,且用以从所述脑电信号中分析和提取脑电警觉度特征的脑电信号处理模块;所述脑电警觉度特征包括δ、θ、α、β四个频带特征和(θ+α)/β、α/β、(θ+αν(α+β)、θ/β四个频带能量比值特征;分别与所述眼动信号处理模块和脑电信号处理模块通信相连,且用以对驾驶员当前的眼动警觉度特征和脑电警觉度特征融合后形成的融合警觉度特征进行警觉度状态评定的警觉度监测模块。 [0008] - integration with the kind of REM EEG driver alertness monitoring system based on intelligent, the monitoring system comprising: a real-time acquisition of the driver's eye to eye movement information signal acquisition module; the signal acquisition eye a communication module connected to and information from the eye movement analysis and eye alertness extracted eye feature the signal processing module; for real-time acquisition of the driver's EEG EEG acquisition module; the brain signal acquisition modules connected to the communication, and to analyze and extract characteristic EEG alertness EEG processing module from the EEG signal; alertness feature comprises the EEG δ, θ, α, β four frequency bands and wherein (θ + α) / β, α / β, (θ + αν (α + β), θ / β ratio wherein four energy bands, respectively; with the eye signal processing module, and a communication processing module electrical brain fusion characteristics of alertness alertness state evaluation alertness monitoring module is connected is formed, and to the current driver's eye movement and EEG features alertness alertness wherein after fusion.

[0009]优选地,所述警觉度监测模块包括:导联确定模块,与所述脑电信号处理模块相连,建立与警觉度相关的普遍脑功能导联区;导联选择模块,与所述导联确定模块相连,利用fisherscore算法对普遍脑功能导联区进行个人的导联分析,获得适于个人的脑功能导联区域,进而获得个人的脑电导联分量;多种警觉度状态评定模型,与所述导联选择模块相连,利用脑电导联分量进行频谱能量分析,对所述S、θ、α、β四个频带特征和(θ+α)/ β、α/β、(θ+αν(α+β)、θ/β四个频带能量比值特征进行多种警觉度状态下的脑电警觉度特征的模式分类,获得警觉度脑功能区的定位关联;多窗口分布式实时监控模块,与所述多种警觉度状态评定模型和所述眼动信号处理模块分别相连,对所述导联分量、四个频带特征和四个频带能量比值特征、警觉度脑功能区的定位关 [0009] Preferably, the alertness monitoring module comprising: a lead determination module, coupled to the processing module EEG, established alertness related to brain function generally lead region; lead selection module, and the determining module connected to the lead, lead to widespread brain function analysis individual zone using leads fisherscore algorithm, brain function is obtained suitable for personal lead region, and thus to obtain the individual components of the brain electrical lead; more alertness status evaluation model , with the lead selection module is connected by electrical lead components cerebral energy spectrum analysis, the S, θ, α, β and features four bands (θ + α) / β, α / β, (θ + αν (α + β), θ / β ratio of four energy bands characteristic EEG pattern for alertness under various features alertness status classification, correlation alertness obtain positioning of brain areas; distributed real-time multi-window monitoring module , the plurality of evaluation models and the alertness status of the signal processing module is connected to each eye, the leads of the component, wherein four bands and four frequency bands wherein the ratio of energy, alertness off positioning of brain areas 联、警觉度监控相关参数以及眼动警觉度特征进行多角度实时监控。 Joint, as well as parameters related to alertness monitoring alertness eye multi-angle feature real-time monitoring.

[0010] 优选地,所述眼动信号处理模块包括:人脸检测模块,与所述眼动信号采集模块相连,对所述眼动信号采集设备采集到的图像进行人脸检测,获得脸部区域特征;闭眼时间监测模块,与所述人脸检测模块相连,对脸部区域特征使用三维十字模型进行建模,在三维十字模型覆盖的区域进行眼部特征提取,并利用眼睛睁开和闭合时颜色特征的差异进行闭眼时间的监测;头部偏转角度监测模块,与所述人脸检测模块相连,利用光流法追踪相邻帧的输入图像中脸部区域特征的变化,将变化后的三维模型映射到二维图像中,再利用姿态估计的方式从二维图像中还原出三维模型的旋转平移参数,从而获得头部的旋转平移矩阵, 进而获得头部偏转的角度。 [0010] Preferably, the signal processing module of the eye comprising: a face detection module, said signal acquisition module is connected to the eye, the eye image signal collecting devices to perform face detection, face obtained region feature; closed eye time monitoring module, connected to the face detection module, a three-dimensional model to cross the face region feature modeling, feature extraction for the eye in the region of the three-dimensional model of the cross covered eye opening and using when the difference in color characteristics close to monitor closed eye time; head deflection angle monitoring module, connected to the face detection module, a face track changes feature region adjacent frames in the input image by using optical flow, will vary after the three-dimensional model is mapped to the two-dimensional image, and then use the pose estimation method to restore the two-dimensional image three-dimensional model of the rotation translation parameters to obtain the translation matrix rotating head, thereby obtaining deflection angle of the head.

[0011] 优选地,所述三维十字模型包括水平部分和垂直部分,水平部分与垂直部分的交点为坐标原点,水平部分用水平曲线h(x)表示,垂直部分用垂直曲线v(y)表示,三维十字模型为: [0011] Preferably, the cross-dimensional model comprises a horizontal portion and a vertical portion, a horizontal portion and the vertical portion of the intersection point as the coordinate origin, the horizontal portion of the curve represented by the horizontal h (x), represented by the vertical portion of the vertical profile v (y) three-dimensional cross model:

Figure CN103340637BD00061

[0014] 优选地,所述基于眼动与脑电融合的驾驶员警觉度智能监控系统还包括一与所述警觉度监测模块相连,且在所述警觉度状态评定的结果为异常时发出报警的报警模块。 [0014] Preferably, based on a result of the driver alertness monitoring system of intelligent eye and further includes an EEG fused alertness connected to the monitoring module, and assessing the state of alertness abnormality alarm the alarm module.

[0015] 优选地,所述基于眼动与脑电融合的驾驶员警觉度智能监控系统还包括一与所述警觉度监测模块相连,且用以实时显示驾驶员眼动警觉度特征、脑电警觉度特征以及融合警觉度特征的显示模块。 [0015] Preferably, the smart driver alertness monitoring system of EEG and eye movement based on the fusion and further comprising a module connected to the monitoring of alertness, and for real-time display of driver eye movements characterized in alertness, EEG and a display module wherein alertness alertness fusion characteristics.

[0016]-种基于眼动与脑电融合的驾驶员警觉度智能监控方法,包括以下步骤: [0016] - a driver alertness intelligent species eye with EEG monitoring method based on fusion, comprising the steps of:

[0017]实时采集驾驶员的眼动信息,从所述眼动信息中分析和提取眼动警觉度特征; [0017] real-time collection of information of the driver eye movement, eye movement analysis and extraction alertness from the eye feature information;

[0018]实时采集驾驶员的脑电信号,从所述脑电信号中分析和提取脑电警觉度特征; 所述脑电警觉度特征包括S、θ、α、β四个频带特征和(θ+α)/β、α/β、(θ+α)/ (α+β)、θ/β四个频带能量比值特征; [0018] The real-time acquisition of EEG driver's alertness EEG analysis and extraction features from the EEG signal; alertness of the EEG feature comprises S, θ, α, β, and four band characteristics ([theta] + α) / β, α / β, (θ + α) / (α + β), θ / β ratio wherein four energy bands;

[0019]对驾驶员当前的眼动警觉度特征和脑电警觉度特征融合后形成的融合警觉度特征进行警觉度状态评定。 Fusion characteristics of alertness alertness state evaluation after the formation of [0019] the current driver's eye movement and EEG features alertness alertness feature fusion.

[0020] 对融合警觉度特征进行警觉度状态评定的具体过程包括: [0020] The specific process features fusion alertness alertness state evaluation comprises:

[0021] 建立与警觉度相关的普遍脑功能导联区; [0021] associated with the establishment of general alertness of brain function leads zone;

[0022] 利用fisherscore算法对普遍脑功能导联区进行个人的导联分析,获得适于个人的脑功能导联区域,进而获得个人的脑电导联分量; [0022] fisherscore algorithm using common lead brain function analysis zone individual leads, brain function is obtained suitable for personal lead region, thereby obtaining the individual components of the brain electrical lead;

[0023]利用脑电导联分量进行频谱能量分析,对所述δ、θ、α、β四个频带特征和(θ+α)/β、α/β、(θ+αν(α+β)、θ/β四个频带能量比值特征进行多种警觉度状态任务下的脑电警觉度特征的模式分类,获得警觉度脑功能区的定位关联; [0023] The use of brain electrical lead component spectrum energy analysis, the δ, θ, α, β and features four bands (θ + α) / β, α / β, (θ + αν (α + β), θ / β four band energy ratio mode features characteristic EEG alertness alertness status under various classification tasks, obtain location related brain areas of alertness;

[0024]对所述导联分量、四个频带特征和四个频带能量比值特征、警觉度脑功能区的定位关联以及警觉度监控相关参数进行多角度实时监控。 [0024] the lead component, wherein four bands and four frequency band energy ratio wherein the positioning brain areas associated with alertness and alertness monitoring parameters related to multi-angle real-time monitoring.

[0025]从所述眼动信息中分析和提取眼动警觉度特征的具体过程包括: [0025] The specific process from the eye to analyze and extract information alertness eye feature comprises:

[0026]对采集到的图像进行人脸检测,获得脸部区域特征; [0026] The collected image for face detection, face region feature is obtained;

[0027]对脸部区域特征使用三维十字模型进行建模,在三维十字模型覆盖的区域进行眼部特征提取,并利用眼睛睁开和闭合时颜色特征的差异进行闭眼时间的监测;所述三维十字模型包括水平部分和垂直部分,水平部分与垂直部分的交点为坐标原点,水平部分用水平曲线h(x)表示,垂直部分用垂直曲线v(y)表示,三维十字模型为: [0027] The use of cross-dimensional features of the face region model modeling, feature extraction for the eye in the region of the three-dimensional model of the cross covered, and using the difference in color and eye opening of closed monitoring features of closed eye time; the cross-dimensional model comprises a horizontal portion and a vertical portion, a horizontal portion and a vertical portion of the intersection as the coordinate origin, the horizontal portion of the curve represented by the horizontal h (x), represented by the vertical portion of the vertical profile v (y), the cross-dimensional model:

Figure CN103340637BD00071

[0030]追踪相邻帧的输入图像中脸部区域特征的变化,利用姿态估计的方式获得头部的旋转平移矩阵,进而获得头部偏转的角度。 [0030] Change Tracking feature region adjacent the face of the input image frames, using a pose estimation matrix obtained in a manner translating rotation of the head, and thus obtain a deflection angle of the head.

[0031]优选地,所述智能监控方法还包括:在所述警觉度状态评定结果为异常时发出报警;实时显示驾驶员眼动警觉度特征、脑电警觉度特征以及融合警觉度特征。 [0031] Preferably, the intelligent monitoring method further comprising: Rating state of alertness in the abnormal alarm; real-time display of driver eye movements characterized in alertness, alertness EEG features and characteristics fusion alertness.

[0032]如上所述,本发明所述的基于眼动与脑电融合的驾驶员警觉度智能监控系统及方法,具有以下有益效果: [0032] As described above, according to the present invention, a driver alertness monitoring system and method for intelligent eye movement based on the EEG fusion, has the following advantages:

[0033]本发明通过融合人体内在的脑电信息与外在的眼动信息,两者相互补充,不易受外界环境和人为因素影响,快速监测驾驶员的警觉度状态,判断是否是非安全驾驶状态,延时性不超过1秒,具有较好的实时性与较高的可信度。 [0033] The present invention by fusing human EEG internal information and external eye movement information, the two complement each other, not easily affected by external environmental and human factors, rapid monitoring of the state of alertness of the driver to determine whether the non-state drive safely delay of no more than one second, with better real-time performance and high reliability.

附图说明 BRIEF DESCRIPTION

[0034]图1为本发明所述的基于眼动与脑电融合的驾驶员警觉度智能监控系统的结构示意图。 [0034] FIG. 1 is a schematic structural diagram of the present invention is based on EEG and eye fusion driver alertness intelligent monitoring system.

[0035]图2为本发明所述的眼动信号处理模块的结构示意图。 [0035] FIG. 2 is a schematic view of the eye of the signal processing module of the present invention.

[0036]图3为本发明所述的警觉度监测模块的结构示意图。 [0036] FIG. 3 is a schematic structural diagram of the present invention, the alertness monitoring module.

[0037]图4为本发明的基于眼动与脑电融合的驾驶员警觉度智能监控方法的流程示意图。 [0037] FIG. 4 is a schematic view of the present invention is based on the eye of the driver alertness monitoring method and intelligent EEG fusion process.

[0038] 元件标号说明 [0038] DESCRIPTION OF REFERENCE NUMERALS element

[0039] 100 驾驶员警觉度智能监控系统 [0039] 100 driver alertness intelligent monitoring system

[0040] 110 眼动信号采集模块 [0040] The motion signal acquisition module 110

[0041] 120 眼动信号处理模块 [0041] The motion signal processing module 120

[0042] 121 人脸检测模块 [0042] The face detecting module 121

[0043] 122 闭眼时间监测模块 [0043] The monitoring module 122 closed eye time

[0044] 123 头部偏转角度监测模块 [0044] The monitoring module 123 head deflection angle

[0045] 130 脑电信号采集模块 [0045] EEG acquisition module 130

[0046] 140 脑电信号处理模块 [0046] The processing module 140 EEG

[0047] 150 警觉度监测模块 [0047] The monitoring module 150 alertness

[0048] 151 导联确定模块 [0048] The determining module 151 lead

[0049] 152 导联选择模块 [0049] The selection module 152 Lead

[0050] 153 多种警觉度状态评定模型 [0050] more than 153 kinds of evaluation model state of alertness

[0051] 154 多窗口分布式实时监控模块 [0051] more than 154 windows distributed real-time monitoring module

[0052] 160 报警模块 [0052] The alarm module 160

[0053] 170 显示模块 [0053] The display module 170

具体实施方式 detailed description

[0054] 以下通过特定的具体实例说明本发明的实施方式,本领域技术人员可由本说明书所揭露的内容轻易地了解本发明的其他优点与功效。 [0054] Hereinafter, an embodiment of the present invention by certain specific examples, those skilled in the art disclosed in this specification may readily understand the content of other advantages and effects of the present invention. 本发明还可以通过另外不同的具体实施方式加以实施或应用,本说明书中的各项细节也可以基于不同观点与应用,在没有背离本发明的精神下进行各种修饰或改变。 The present invention may also be implemented or applied through other different specific embodiments, the details of the specification may be carried out in various modified or changed without departing from the spirit of the invention based on various concepts and applications.

[0055] 请参阅附图。 [0055] Refer to the accompanying drawings. 需要说明的是,本实施例中所提供的图示仅以示意方式说明本发明的基本构想,遂图式中仅显示与本发明中有关的组件而非按照实际实施时的组件数目、形状及尺寸绘制,其实际实施时各组件的型态、数量及比例可为一种随意的改变,且其组件布局型态也可能更为复杂。 Incidentally, the present embodiment illustrates a schematic manner only examples provided to illustrate the basic idea of ​​the invention, then the drawings shows only related to the present invention, the number of components in the assembly when not in accordance with the actual embodiment, the shape and drawn to scale, its actual implementation of each component type, number and proportion of changes may be as a free, and the layout of the components may also be more complex patterns.

[0056] 下面结合实施例和附图对本发明进行详细说明。 [0056] The present invention will be described in detail in conjunction with embodiment examples and figures.

[0057] 实施例 [0057] Example

[0058] 本实施例提供一种基于眼动与脑电融合的驾驶员警觉度智能监控系统,如图1所示,所述基于眼动与脑电融合的驾驶员警觉度智能监控系统100包括:眼动信号采集模块110、眼动信号处理模块120、脑电信号采集模块130、脑电信号处理模块140、警觉度监测模块150、报警模块160、显示模块170。 [0058] The present embodiment provides a monitoring system based on eye smart driver alertness and the EEG fusion shown in Figure 1, a driver alertness monitoring system intelligent eye and EEG based on 100 comprises a fusion of the : eye signal acquisition module 110, a signal processing module 120 to the eye, EEG acquisition module 130, processing module 140 EEG, alertness monitoring module 150, an alarm module 160, display module 170.

[0059] 所述眼动信号采集模块110实时采集驾驶员的眼动信息。 [0059] The signal acquisition module 110 eye real-time acquisition of the driver's eye movement information.

[0060] 所述眼动信号处理模块120与所述眼动信号采集模块110通信相连,从所述眼动信息中分析和提取眼动警觉度特征。 [0060] The signal processing module of the eye and the eye 120 is connected to communication signal acquisition module 110, analyze and extract information alertness eye features from the eye. 所述眼动信号处理模块120通过实时分析由摄像头采集的视频,提取眼动警觉度特征。 The eye signal processing module 120 through real-time analysis of video captured by the camera, extracting characteristic eye alertness.

[0061] 进一步,如图2所示,所述眼动信号处理模块120包括:人脸检测模块121,闭眼时间监测模块122,头部偏转角度监测模块123。 [0061] Further, as shown in Figure 2, the eye movement signal processing module 120 includes: face detection module 121, monitoring module 122 closed eye time, monitoring the deflection angle of the head module 123.

[0062] 所述人脸检测模块121与所述眼动信号采集模块110相连,对所述眼动信号采集设备采集到的图像进行人脸检测,获得脸部区域特征。 [0062] The face detecting module 121 and the features of the face region of the eye is connected to the signal acquisition module 110, the eye image signal collecting devices to face detection, is obtained.

[0063] 所述闭眼时间监测模块122与所述人脸检测模块121相连,对脸部区域特征使用三维十字模型进行建模,在三维十字模型覆盖的区域进行眼部特征提取,并利用眼睛睁开和闭合时颜色特征的差异进行闭眼时间的监测。 [0063] The closed eye time monitoring module 122 is connected to the face detection module 121, a three-dimensional model to cross the face region feature modeling, feature extraction for the eye in the region of the three-dimensional model of the cross covered using eye difference in color characteristics to be monitored when open and closed eye closing time. 三维十字模型因为它的简单、较少的自由度、以及合理精确地近似了人体头部的特性用于三维头部追踪,其不会被脸部表情影响,可以稳定的运行。 Cross-dimensional model because of its simplicity, fewer degrees of freedom, and reasonably accurate approximate the characteristics of the human head for three-dimensional head tracking, facial expression which is not affected, can be run stably. 本发明中,选取眼眉中心作为三维十字模型的中心,选取两个眼角,鼻尖以及前额中心作为四个顶点。 In the present invention, a selected three-dimensionally cross the center of eyebrow model, selecting two eyes, a nose and the center of the forehead four vertices. 同时,模型保留了十字区域的深度信息,以使模型能精确匹配头部。 At the same time, retaining the model region cross depth information, so that the model accurately match head. 所述三维十字模型包括水平部分和垂直部分,水平部分与垂直部分的交点为坐标原点, 水平部分用水平曲线h(x)表示,垂直部分用垂直曲线v(y)表示,三维十字模型为: And the three-dimensional model comprises a vertical cross section, the intersection of the horizontal portion of the horizontal portion and the vertical portion is a coordinate origin, the horizontal portion of the curve represented by the horizontal h (x), represented by the vertical portion of the vertical profile v (y), the cross-dimensional model:

Figure CN103340637BD00091

[0066] 所述头部偏转角度监测模块123与所述人脸检测模块121相连,追踪相邻帧的输入图像中脸部区域特征的变化,利用姿态估计的方式获得头部的旋转平移矩阵,进而获得头部偏转的角度,同时根据特定范围的灰度值的像素数量的差异性比较有效检测出眨眼状ίέτO [0066] The monitoring module 123 and a deflection angle of the head of the face detection module 121 is connected to track changes in the facial feature area adjacent to the input image frame, rotational translation matrix is ​​obtained by way of the head pose estimation, Further deflection angle of the head is obtained, and is effective in accordance with the detected blink like ίέτO difference pixel value of a specific number of gray range

[0067] 追踪头部获得头部偏转角度的具体实现过程为:将三维十字模型映射到初始头部姿态模板来近似头部,从而进行头部姿态估计。 [0067] The head tracking head deflection angle is obtained for the specific implementation process of: mapping the three-dimensional model to an initial cross head pose templates to approximate the head so as to perform head pose estimation. 在检测到人脸基础上,人脸图像的初始参考模板以及相应的姿态被计算出来,然后三维十字模型被创建出来,头部全运动将从输入的图像中利用光流法被追踪出来。 The face is detected based on an initial reference template and a face image corresponding to the posture is calculated, and then the three-dimensional model is created cross, full-motion image input from the head by using the optical flow to be tracked out. 本发明通过Lucas-Kanade方法(简称LK算法,是光流法的一种经典算法)解决目标对象和摄像机之间的相对运动问题,以下是将头部(即目标对象)看成刚性物体,然后利用如前所述的三维十字头模型和LK方法计算估计头部状态的过程。 The present invention is by Lucas-Kanade method (LK algorithm for short, is a classic method of optical flow algorithm) to solve the problem of relative movement between the target object and the camera, the following is the head (i.e., target object) as a rigid body, and as described above using a three-dimensional model of the crosshead and the LK method calculating an estimated state of the head process.

[0068] 头部姿态估计是对刚性头部运动的估计,改变三维十字模型的姿态直到模型的特征与图像数据相同。 [0068] is an estimate of the head pose estimation rigid head movement, change the posture of the three-dimensional model until cross the same features of the model with the image data. 刚性头部运动包含旋转ReR3x3和平移Ter3X1,可以用公式(1)所示的齐次坐标描述: The rigid head includes a rotary motion and a translational ReR3x3 Ter3X1, can be used in equation (1) shown in homogeneous coordinates Description:

Figure CN103340637BD00092

[0070] 每一个刚性运动也可以表示成三维的旋转ReR3x3和沿着三个轴的平移τer3X1。 [0070] Each of rigid motion may be expressed as a three-dimensional rotation along three axes and ReR3x3 translation τer3X1. 旋转ReR3x3和平移TeR3xi可以分别表示成公式(2)和(3): ReR3x3 rotation and translation TeR3xi can be expressed as Equation (2) and (3):

Figure CN103340637BD00101

[0073] 其中[COxCOyCoJ表示相对于三个轴的旋转角度,[txtytz]表示三维平移T。 [0073] wherein [COxCOyCoJ represents the rotation angle of the three axes, [txtytz] represents the three-dimensional translational T.

[0074] 把头部运动看做一个刚性运动。 [0074] The movement of the head seen as a rigid motion. 因此,每一个三维十字模型的像素在t+1时刻可以表示成: Thus, each pixel in a three-dimensional model cross time t + 1 can be expressed as:

[0075]Xt+1=M·Xt=R3x3 ·Xt+T3X1 (4) [0075] Xt + 1 = M · Xt = R3x3 · Xt + T3X1 (4)

[0076]因为用LK方法计算运动参数向量μ=[ωχ,coy,ωζ,tx,ty,tj,可以得到旋转矩阵ReR3x3和平移矩阵TeR3X1,然后就能计算出头部的全运动并估计头部姿态。 [0076] Since the motion vector of parameters calculated by the LK method μ = [ωχ, coy, ωζ, tx, ty, tj, can be obtained rotation matrix and translation matrix ReR3x3 TeR3X1, and can calculate and estimate the full movement of the head head attitude.

[0077] 追踪头部获得眨眼状态的具体实现过程为:在检测到人脸基础上,使用LK方法稳定的追踪头部运动,借助三维十字模型的水平信息来追踪眼睛,因眼睛区域位于三维十字模型的特定区域,因此,裁剪三维十字模型水平部分的两个区域,可以得到两个眼睛图片。 [0077] The head tracking state obtained blink embodied process: the face is detected based on the LK method using a stable tracking head motion, by means of horizontal cross-dimensional model information to track the eye, the eye region is located due to the three-dimensional cross specific areas of the model, and therefore, two-dimensional area portion of the horizontal cross-cut model, two eye images can be obtained. 在实时获取、监测左眼和右眼视频图像的基础上,计算图像的灰度直方图。 In real-time access, left and right video monitor based on the image, the grayscale histogram of the image is calculated. 当眼睛睁开时,灰度值低于一个特定阈值的像素的数量与眼睛闭合时的数量相比有着明显的区别。 When the eye is opened, when the number of gradation values ​​is less than the number of pixels in a particular closed eye threshold value compared to a clear difference. 根据眼睛睁开和闭合时,特定范围的灰度值的像素数量的差异性比较可以有效检测出眨眼状ίέτO The eyes are open and when closed, the number of pixels of the gradation value of a specific range of difference comparison can effectively detect the blink like ίέτO

[0078] 所述脑电信号采集模块130实时采集驾驶员的脑电信号。 [0078] The EEG acquisition module 130 of the real-time acquisition of EEG driver.

[0079]所述脑电信号处理模块140与所述脑电信号采集模块130通信相连,从所述脑电信号中分析和提取脑电警觉度特征。 [0079] The EEG signal processing module 140 in communication with the EEG acquisition module 130 is connected, analyze and extract features from the EEG alertness EEG. 所述脑电警觉度特征包括S、θ、α、β四个频带特征和(θ+α)/β、α/β、(θ+αν(α+β)、θ/β四个频带能量比值特征。所述脑电信号处理模块140通过实时分析脑电数据,提取脑电警觉度特征。利用脑电的导联分量进行频谱能量分析,对δ、θ、α和β四个频带以及(θ+α)/β、α/β、(θ+α)/(α+β)、α/β4 个能量比值共8个特征进行多种警觉度状态任务下的脑电警觉度特征的模式分类,获得警觉度脑功能区的定位关联,可以达到提高准确率和效果的目的。 EEG characteristics of the alert comprises S, θ, α, β and features four bands (θ + α) / β, α / β, (θ + αν (α + β), θ / β ratio of four energy bands feature. the EEG signal processing module 140 through real-time analysis of EEG data, extracting characteristic EEG alertness. component leads using EEG power spectrum analysis, δ, θ, α and β, and four bands ([theta] + α) / β, α / β, (θ + α) / (α + β), α / β4 an energy ratio of 8 characteristic EEG pattern for alertness under various features alertness status classification task, obtaining a position associated with alertness and brain function area, you can achieve the purpose of improving the accuracy and effectiveness.

[0080] 所述警觉度监测模块150分别与所述眼动信号处理模块120和脑电信号处理模块140通信相连,对驾驶员当前的眼动警觉度特征和脑电警觉度特征融合后形成的融合警觉度特征进行警觉度状态评定。 [0080] The alertness monitoring module 150 are connected with the eye 140 of the communication processing module 120 and the signal processing module EEG signal, the current driver's eye movement and EEG features alertness alertness characterized syncytium formation fusion characteristics of alertness alertness state evaluation. 所述警觉度监测模块150通过脑电警觉度特征给出驾驶员警觉度的判定,警觉度至少分为清醒,疲劳和睡眠三个状态。 The alertness monitoring module 150 is given by a driver alertness alertness EEG determined characteristics, into at least awake alertness, fatigue and sleep three states. 所述警觉度监测模块150利用机器视觉方法分析头部离正前方的偏转角度和眼睛闭合的信息,由此判别驾驶员的警觉状态,警觉状态可分为正常状态和分神状态。 Analysis of information in the header and the deflection angle from the front of the eye closure alertness monitoring module 150 using machine vision methods, thereby determining the state of alertness of the driver, an alert state and a normal state can be divided into distracted state. 当驾驶员头部偏离正前方的偏角过大和偏离的时间过长,或者眼睛单次闭合的时间过长,系统将其判定为分神状态。 When the time has deviated from the driver's head in front of the large and the off angle is too long, single or eyes closed for too long, it will be determined that distracted state.

[0081] 进一步,如图3所示,所述警觉度监测模块150包括:导联确定模块151,导联选择模块152,多种警觉度状态评定模型153,多窗口分布式实时监控模块154。 [0081] Further, as shown in FIG. 3, the alertness monitoring module 150 comprises: a determining module 151 lead, lead selection module 152, a variety of state of alertness evaluation model 153, distributed real-time multi-window monitoring module 154.

[0082] 所述导联确定模块151与所述脑电信号处理模块140相连,建立与警觉度相关的普遍脑功能导联区。 [0082] The lead 151 is connected to the determining module EEG processing module 140, associated with the establishment of universal brain function alertness lead region.

[0083] 所述导联选择模块152与所述导联确定模块151相连,利用fisherscore算法对普遍脑功能导联区进行个人的导联分析,获得适于个人的脑功能导联区域,进而获得个人的脑电导联分量。 [0083] The lead selection module 152 and the determining module 151 is connected to lead, lead to widespread brain function analysis individual zone using leads fisherscore algorithm, brain function is obtained suitable for personal lead region, and thus obtained individual components of brain electrical lead. 在需要产品化和实用性的双重要求下,导联的减少和选择是系统的重要问题。 In need of products and practicality of dual requirements, reduce choice and lead the system is an important issue. 本发明利用fisherscore算法对62导的实验数据进行警觉度状态的显著性差异分析,找到了与警觉度相关的普遍脑功能区,利用这一结果则可对未有警觉度先验知识的受试者进行导联的选择,以达到减少不必要的导联增强实用性的目的。 The present invention utilizes the experimental data fisherscore algorithm is turned 62 degrees state of alertness significant difference analysis, to find a general alertness related brain areas, using the results of the test can No prior knowledge of alertness then selects lead to achieve enhanced reduce unnecessary leads practical purposes. 与此同时,对已有警觉度相关数据的受试者可利用导联确定模块151和导联选择模块152进行个人的导联分析, 找到适用于个人的导联区域,提高精确度。 At the same time, it has been the subject of alertness related data available lead determination module 151 analyzes lead, and lead selection module 152 individuals, individuals find suitable lead area, to improve accuracy.

[0084] 所述多种警觉度状态评定模型153与所述导联选择模块152相连,利用脑电导联分量进行频谱能量分析,对所述S、θ、α、β四个频带特征和(θ+α)/β、α/β、 (θ+αν(α+β)、θ/β四个频带能量比值特征进行多种警觉度状态任务(如清醒-睡眠-疲劳等任务)下的脑电警觉度特征的模式分类,获得警觉度脑功能区的定位关联。所述多种警觉度状态评定模型153根据特征提取的结果,利用分类算法对警觉度状态进行客观评定。对于无先验模型情况,利用实验采集的大量样本数据进行特征的脑功能区定位,并利用线性fisher算法训练、测试得到特征提取及状态评定的参数选择,得到无先验模型的粗粒度监控范式,对于初次使用人员进行包括清醒、睡眠、疲劳等警觉度状态的准确监控和预警。所述多种警觉度状态评定模型153在无先验模型的基础上,针对具备先验警觉 [0084] Evaluation of the state of alertness plurality of model 153 and the lead selection module 152 is connected by electrical lead components cerebral energy spectrum analysis, the characteristics of the four frequency bands S, θ, α, β, and ([theta] EEG (and other tasks such as fatigue awake - - sleeping) at + α) / β, α / β, (θ + αν (α + β), θ / β ratio of four energy bands wherein various tasks state of alertness alertness feature pattern classification, obtained alertness related positioning of brain areas. alertness status of the plurality of evaluation results of feature extraction model 153, a state of alertness assessed using objective classification algorithm for the case without a prior model , experiments using a large number of sample data acquired feature localizing brain function, and using a linear fisher training algorithm, and feature extraction to obtain the test parameters of the selected state assessment, no prior model to obtain coarse-grained monitoring paradigm for initial use personnel accurately monitor and warning awake state comprises alertness, sleep, fatigue, etc. the plurality of alertness state evaluation model based on 153 without a priori model, for vigilance comprising a priori 信息的样本,利用SVM(支持向量机)算法和GMMlcuster(高斯混合聚类)算法开发出来的,多种警觉度状态评定模型能够评定包括两种及两种以上的警觉度状态,本发明采用了3种为例进行了说明,但多种警觉度状态评定模型可以评定的警觉度状态的种类则不限于3种。所述多种警觉度状态评定模型153具备先验警觉度模型的功能,对警觉度状态的评定更加精确,其利用GMMcluster分类器进行任意警觉度状态的评定,在清醒、睡眠、疲劳等状态基础上,增加更加细致的划分,为制定灵活、准确的预警策略提供了技术支持。 Information on the sample, using the SVM (Support Vector Machine) algorithm and GMMlcuster (Gaussian mixture clustering) algorithm developed, various models can be assessed alertness status assessment and include two or more kinds of the state of alertness, the present invention employs three kinds described as an example, but the state of alertness of the plurality of types of state of alertness can be assessed evaluation model is not limited to three in combination. the plurality of alertness prior state evaluation model 153 includes a model of alertness function, assess alertness state more precisely, its use GMMcluster classifier to assess any alertness state, the state of the underlying awake, sleep, fatigue, the increase is more detailed classification, provided technical support for the development of flexible and accurate early warning strategy .

[0085] 所述多窗口分布式实时监控模块154与所述多种警觉度状态评定模型153和所述眼动信号处理模块120分别相连,对所述导联分量、四个频带特征和四个频带能量比值特征、警觉度脑功能区的定位关联、警觉度监控相关参数以及眼动警觉度特征进行多角度实时监控。 [0085] The distributed real-time multi-window monitoring module 154 and the plurality of alertness state evaluation model eye 153 and the signal processing module 120 is connected respectively to said lead components, features four bands and four wherein the ratio of frequency band energy, alertness related positioning of brain areas, alertness monitoring parameters and associated eye alertness multi-angle feature real-time monitoring.

[0086] 所述导联确定模块151和所述导联选择模块152利用大量的实验对建立的警觉度监控与预警模型进行验证,建立基于主要特征的警觉度脑功能区的定位关联。 [0086] The determining module 151 and the lead of the lead selection module 152 using a large number of experiments and early warning monitoring alertness model is verified, the positioning established brain areas associated with alertness based on essential characteristics. 在此基础上, 多种警觉度状态评定模型153利用支持向量机SVM算法和高斯混合聚类GMMCluster算法对主要特征进行警觉度状态的评定,并以此为基础建立了无先验知识的多种警觉度状态评定的模型,克服了警觉度先验模型和无中间态模型的缺陷。 On this basis, a variety of models to assess the state of alertness main characteristics were assessed alertness state 153 SVM and SVM algorithm GMMCluster Gaussian mixture clustering algorithm, and to establish a variety of non-priori knowledge-based alertness status assessment model, overcomes the deficiencies of the prior model and alertness without intermediate states model. 同时所述多窗口分布式实时监控模块154提出了多模态监控方法,对导联信号、能量特征、脑功能区关联情况以及警觉度监控相关参数提出了多窗口分布式实时监控手段,对驾驶员的生理状态进行多角度的监控并制定相应的预警策略。 The distributed real-time multi-window while monitoring module 154 to monitor a multi-modal approach to lead signal, the energy characteristics, as well as brain areas associated with alertness monitoring parameters proposed distributed real-time multi-window monitoring means, the driver physiological state staff to monitor multiple perspectives and to develop appropriate early warning strategies.

[0087] 所述报警模块160与所述警觉度监测模块150相连,在所述警觉度状态评定的结果为异常时发出报警,提醒驾驶员注意。 [0087] The alarm module 160 is connected to the alertness monitoring module 150, an alarm is abnormal when the result of the evaluation of the state of alertness, to alert the driver.

[0088] 所述显示模块170与所述警觉度监测模块150相连,实时显示驾驶员眼动警觉度特征、脑电警觉度特征以及融合警觉度特征的显示模块。 [0088] The display module 170 is connected to the alertness monitoring module 150, a real-time display of driver eye movements characterized in alertness, EEG alertness and a display module wherein the fusion characteristics alertness. 该模块包括基于脑电、眼动的警觉度实时状态显示,融合的警觉度状态坐标轴显示,融合的警觉度历史记录,用户参数配置与控制等子t吴块。 The module includes the EEG, eye movement status display in real time alertness, alertness status display axes fusion, fusion alertness history, user parameters and control sub-blocks t Wu.

[0089] 本实施例还提供一种基于眼动与脑电融合的驾驶员警觉度智能监控方法,该监控方法可以由所述基于眼动与脑电融合的驾驶员警觉度智能监控系统实现,也可以由其他装置设备实现,即所述基于眼动与脑电融合的驾驶员警觉度智能监控方法的实现装置不现于本发明所述的监控系统。 [0089] The present embodiments also provide an intelligent method for monitoring driver alertness and the EEG eye fusion based on this monitoring method may be based on eye movement by the driver alertness monitoring system and EEG intelligent integration achieved, may also be realized by means of other devices, i.e., based on the eye to achieve intelligent device driver alertness and the EEG monitoring method is not fused to the existing monitoring system of the present invention. 如图4所示,该基于眼动与脑电融合的驾驶员警觉度智能监控方法包括以下步骤: As shown, the driver alertness monitoring method based on eye smart fusion EEG 4 comprises the steps of:

[0090] 实时采集驾驶员的眼动信息,从所述眼动信息中分析和提取眼动警觉度特征。 [0090] real-time collection of information of the driver eye movement, eye movement analysis and extraction alertness from the eye feature information. 进一步,从所述眼动信息中分析和提取眼动警觉度特征的具体过程包括:对采集到的图像进行人脸检测,获得脸部区域特征;对脸部区域特征使用三维十字模型进行建模,在三维十字模型覆盖的区域进行眼部特征提取,并利用眼睛睁开和闭合时颜色特征的差异进行闭眼时间的监测;所述三维十字模型包括水平部分和垂直部分,水平部分与垂直部分的交点为坐标原点,水平部分用水平曲线h(x)表示,垂直部分用垂直曲线v(y)表示,三维十字模型为: Further, information from the eye movement analysis and eye extraction process specific features of alertness comprising: the collected images for face detection, face region feature is obtained; three-dimensional model of a cross modeling features of the face region carried out in the region of the three-dimensional feature extraction eye model cross covered, and using the difference in color and eye opening of closed monitoring features of closed eye time; the three-dimensional model comprises a horizontal cross section and a vertical portion, a horizontal portion and a vertical portion the intersection of the coordinate origin, the horizontal portion of the curve represented by the horizontal h (x), represented by the vertical portion of the vertical profile v (y), the cross-dimensional model:

Figure CN103340637BD00121

[0093]追踪相邻帧的输入图像中脸部区域特征的变化,利用姿态估计的方式获得头部的旋转平移矩阵,进而获得头部偏转的角度。 [0093] Change Tracking feature region adjacent the face of the input image frames, using a pose estimation matrix obtained in a manner translating rotation of the head, and thus obtain a deflection angle of the head. 在眼动监控中,需要监测的指标包括眼睛每次闭合的时间和头部偏转的角度。 In the eye movement monitoring, the need to include indicators for monitoring the angle of each head and eye closure time of deflection. 首先对输入的图像进行预处理,预处理包括图像的灰度化,为减小光照影响对图像进行像素均衡化;第二步对输入的图像进行人脸检测,并对检测出的人脸进行特征的提取;第三步在检测出人脸之后对图像中的人脸区域进行使用三维十字模型进行建模,同时在三维十字模型所覆盖的区域进行特征点的提取,在这一步可以确定眼部的位置,并利用眼睛睁开和闭合时颜色特征上的差异进行闭眼时间的监测;第四步对下一帧的输入图像中的相应的特诊点进行追踪;最后根据前后两帧中的模型的变化进行模型的姿态估计,从而得到头部的旋转平移矩阵,从而得到头部姿态。 First, preprocessing the input image, the gradation of an image comprising a pretreatment in order to reduce the influence of light on the image pixel equalization; a second step of the inputted face image detection, and the detected human face feature extraction; the third step of the face region in the image is detected after a three-dimensional face model modeling a cross, while the extracted feature point in a region covered by the three-dimensional model of the cross, at which the eye can be determined the position of the unit, and monitoring and closed-eye time difference in color characteristics by using the closed eye is open; a fourth step of the tracking points corresponding to the input image, VIP in the next frame; the last two frames before and after the model attitude estimation model change, whereby rotation of the head of the translation matrix to obtain a head pose.

[0094]实时采集驾驶员的脑电信号,从所述脑电信号中分析和提取脑电警觉度特征; 所述脑电警觉度特征包括S、θ、α、β四个频带特征和(θ+α)/β、α/β、(θ+α)/ (α+β)、θ/β四个频带能量比值特征。 [0094] The real-time acquisition of EEG driver's alertness EEG analysis and extraction features from the EEG signal; alertness of the EEG feature comprises S, θ, α, β, and four band characteristics ([theta] + α) / β, α / β, (θ + α) / (α + β), θ / β four band energy ratio features.

[0095]对驾驶员当前的眼动警觉度特征和脑电警觉度特征融合后形成的融合警觉度特征进行警觉度状态评定。 Fusion characteristics of alertness alertness state evaluation after the formation of [0095] the current driver's eye movement and EEG features alertness alertness feature fusion. 进一步,对融合警觉度特征进行警觉度状态评定的具体过程包括: 建立与警觉度相关的普遍脑功能导联区;利用fisherscore算法对普遍脑功能导联区进行个人的导联分析,获得适于个人的脑功能导联区域,进而获得个人的脑电导联分量;利用脑电导联分量进行频谱能量分析,对所述S、θ、α、β四个频带特征和(θ+α)/β、α/ β、(θ+αν(α+β)、θ/β四个频带能量比值特征进行清醒-睡眠-疲劳任务下的脑电特征提取和多种警觉度状态的客观评定;对所述导联分量、四个频带特征和四个频带能量比值特征、警觉度脑功能区的定位关联以及警觉度监控相关参数进行多角度实时监控。 Further specific process, fusion alertness status characteristics assessed alertness comprising: establishing a general alertness related to brain function areas lead; lead universal brain function analysis individual zone using leads fisherscore algorithm, adapted to obtain personal area leads brain function, and thus obtain the individual components of the brain electrical lead; brain electrical lead components using energy spectrum analysis, the S, θ, α, β and features four bands (θ + α) / β, α / β, (θ + αν (α + β), θ / β ratio wherein four energy bands awake - sleep - EEG feature extraction and the fatigue task more objective assessment of the state of alertness; of the guide joint component, wherein four bands and four frequency band energy ratio wherein the positioning brain areas associated with alertness and alertness monitoring parameters related to multi-angle real-time monitoring.

[0096] 在所述警觉度状态评定结果为异常时发出报警。 [0096] The alarm when the abnormal state of alertness evaluation results.

[0097] 实时显示驾驶员眼动警觉度特征、脑电警觉度特征以及融合警觉度特征。 [0097] Live View alertness driver eye movements characterized in, and EEG features alertness alertness fusion characteristics.

[0098] 本发明借助脑电采集设备与摄像头同时采集驾驶员的脑电信号与眼动信息,利用SVM,高斯等多种模式分类算法对驾驶员警觉度状态进行综合评定,并通过眼动与脑电信息融合对驾驶员疲劳、瞌睡、视线长时偏移等非正常驾驶状态进行声光报警。 [0098] EEG acquisition device by means of the present invention with the camera while acquiring EEG and eye movement information of the driver, the driver's state of alertness comprehensive assessment using the SVM, and other modes of Gaussian classification algorithms, and by eye movement EEG information fusion for non-normal driving state of the driver fatigue, drowsiness, sight alignment during long sound and light alarm. 本发明是在整合利用基于脑电信号的驾驶员警觉度分类模型与算法研究的基础上,结合实时的眼动监控对驾驶员的警觉度状态进行整合分析,实现检测驾驶员瞌睡、清醒、睁眼、长时闭眼、头部偏移等状态并适时报警的系统和方法。 The present invention is based on the integration of the driver using a classification model of alertness and EEG research algorithms, combining the real-time monitoring of the eye movement of the driver's state of alertness integrate analysis is performed to detect driver drowsiness, clear, wide open eyes, long eyes closed, head offset, status and timely alarm systems and methods.

[0099] 本发明通过对脑电信号的处理与分析能实时地对驾驶员警觉度状态进行清醒,疲劳和睡眠等3个等级的分类,由于脑电相较于人体外部表象更能反映出人体疲劳的内部特征,因此通过本发明获得的警觉度状态具有较好的实时性与较高的可信度。 [0099] The present invention, by processing and analysis of EEG signal in real time to the state of alertness of the driver are classified in three levels awake, like fatigue and sleep, EEG because compared to the human body can reflect the external appearance internal fatigue characteristics, it has good reliability and real-time state of alertness higher throughput obtained by the present invention.

[0100] 本发明提供了利用驾驶员头部运动和眼动信息来判断驾驶疲劳的机器视觉判断方法,利用普通摄像头获取驾驶员头动与眼动信息,通过图像预处理、人脸检测、头部姿态恢复及眼动追踪等工作,对因疲劳导致的驾驶员头部长时间偏离正前方,或眼睛闭合时间显著增长等非安全驾驶状态进行报警。 [0100] The present invention provides a method of using a machine vision determination of the driver head movement and eye movement information to determine the fatigue of the driver, using a general camera to obtain a driver head movement and eye movement information, the image pre-processing, face detection, head pose recovery unit and eye tracking, etc., on the head of the driver due to fatigue caused by prolonged departing from front, or a significant increase in time to eye closure and other non-operating state security alarm.

[0101] 本发明基于脑电和眼动、头动信息融合的驾驶员警觉度实时监控和及时报警,通过融合人体内在的脑电信息与外在的眼动信息,两者相互补充,可快速监测清醒、睡眠、深度疲劳所导致的瞌睡、眼睛闭合超时、眼睛或头部偏离超时等状态非安全驾驶状态(行为), 延时性不超过1秒。 [0101] The present invention is based on EEG and eye movement, head movement of the driver's alertness information fusion real-time monitoring and timely warning information by fusing the inner and outer EEG information human eye, the two complement each other, can quickly monitoring awake, sleep, deep fatigue caused by drowsiness, eye closure timeout, eye or head deviate from a non-secure state timeout driving state (behavior), a delay of less than 1 second.

[0102] 本发明不仅可以用在驾驶员警觉度监控,也可以用在其他对人的警觉度要求较高的工作场合。 [0102] The present invention may be used not only in a driver alertness monitoring, may be used in other human alertness demanding workplace.

[0103] 综上所述,本发明有效克服了现有技术中的种种缺点而具高度产业利用价值。 [0103] In summary, the present invention effectively overcomes the drawbacks of the prior art and the use of highly industrial value.

[0104] 上述实施例仅例示性说明本发明的原理及其功效,而非用于限制本发明。 [0104] the above-described embodiments are only to illustrate the principle and efficacy of the present invention, the present invention is not intended to be limiting. 任何熟悉此技术的人士皆可在不违背本发明的精神及范畴下,对上述实施例进行修饰或改变。 Any person skilled in this art can be made at without departing from the spirit and scope of the present invention, the above-described embodiments can be modified or changed. 因此,举凡所属技术领域中具有通常知识者在未脱离本发明所揭示的精神与技术思想下所完成的一切等效修饰或改变,仍应由本发明的权利要求所涵盖。 Thus, one skilled in the art that whenever all having ordinary knowledge in the technical ideas and spirit of the present invention is disclosed without departing from the completed equivalent modified or altered, yet the claims shall be encompassed by the present invention.

Claims (8)

  1. 1. 一种基于眼动与脑电融合的驾驶员警觉度智能监控系统,其特征在于,所述基于眼动与脑电融合的驾驶员警觉度智能监控系统包括: 用W实时采集驾驶员的眼动信息的眼动信号采集模块; 与所述眼动信号采集模块通信相连,且用W从所述眼动信息中分析和提取眼动警觉度特征的眼动信号处理模块; 用W实时采集驾驶员的脑电信号的脑电信号采集模块; 与所述脑电信号采集模块通信相连,且用W从所述脑电信号中分析和提取脑电警觉度特征的脑电信号处理模块;所述脑电警觉度特征包括δ、Θ、α、β四个频带特征和(θ+α)/β、α/β、(θ+α)/(α+β)、θ/β四个频带能量比值特征; 分别与所述眼动信号处理模块和脑电信号处理模块通信相连,且用W对驾驶员当前的眼动警觉度特征和脑电警觉度特征融合后形成的融合警觉度特征进行警觉度状态评定的警觉度监测 An intelligent monitoring system based on the eye of the driver alertness and the EEG fusion, wherein the fusion EEG and eye movement based on the driver's alertness intelligent monitoring system comprising: a driver with real-time acquisition W eye movement eye movement information signal acquisition module; eye connected with the communication signal acquisition module, and to analyze and extract characteristic eye alertness eye signal processing module information from the eye by W; W with real-time acquisition brain EEG signal acquisition module driver; the EEG acquisition module in communication with the connected extraction and analysis of EEG and EEG features alertness processing module from the EEG signal W is used; the wherein said EEG alertness comprising δ, Θ, α, β and features four bands (θ + α) / β, α / β, (θ + α) / (α + β), θ / β four energy bands wherein ratios; respectively connected to said eye-EEG signal processing module and the communication processing module and the dynamic fusion with W alertness features formed after alertness and EEG features alertness eye feature fusion current driver's alertness state of alertness monitoring assessment 模块; 所述警觉度监测模块包括: 导联确定模块,与所述脑电信号处理模块相连,建立与警觉度相关的普遍脑功能导联区; 导联选择模块,与所述导联确定模块相连,利用fisher score算法对普遍脑功能导联区进行个人的导联分析,获得适于个人的脑功能导联区域,进而获得个人的脑电导联分量; 多种警觉度状态评定模型,与所述导联选择模块相连,利用脑电导联分量进行频谱能量分析,对所述δ、Θ、α、β四个频带特征和(θ+α)/β、α/β、(θ+α)/(α+β)、Θ/ e四个频带能量比值特征进行多种警觉度状态下的脑电警觉度特征的模式分类,获得警觉度脑功能区的定位关联; 多窗口分布式实时监控模块,与所述多种警觉度状态评定模型和所述眼动信号处理模块分别相连,对所述导联分量、四个频带特征和四个频带能量比值特征、警觉度脑功能区的定位关联 Module; the alertness monitoring module comprising: a lead determination module, coupled to the processing module EEG, established alertness related to brain function generally lead region; lead selection module, and the module determination lead connected by universal algorithm fisher score lead brain function analysis of individual regions of lead, lead brain function is obtained suitable for personal area, thereby obtaining the individual components of the brain electrical lead; more alertness model state evaluation, and the lead connected to said selection module, electrical lead components using brain energy spectrum analysis, the δ, Θ, α, β and features four bands (θ + α) / β, α / β, (θ + α) / (α + β), Θ / e ratio wherein four energy bands characteristic of alertness EEG patterns under various alertness status classification, correlation alertness obtain positioning of brain areas; distributed real-time multi-window monitoring module, the plurality of evaluation models and the state of alertness of the eye movement signal processing module, respectively, the lead component, wherein four bands and four frequency band energy ratio wherein the positioning related brain areas of alertness 、警觉度监控相关参数W及眼动警觉度特征进行多角度实时监控。 , Alertness monitoring parameters W and vigilant eye of the multi-angle feature real-time monitoring.
  2. 2. 根据权利要求1所述的基于眼动与脑电融合的驾驶员警觉度智能监控系统,其特征在于,所述眼动信号处理模块包括: 人脸检测模块,与所述眼动信号采集模块相连,对所述眼动信号采集模块采集到的图像进行人脸检测,获得脸部区域特征; 闭眼时间监测模块,与所述人脸检测模块相连,对脸部区域特征使用Ξ维十字模型进行建模,在Ξ维十字模型覆盖的区域进行眼部特征提取,并利用眼睛睁开和闭合时颜色特征的差异进行闭眼时间的监测; 头部偏转角度监测模块,与所述人脸检测模块相连,利用光流法追踪相邻帖的输入图像中脸部区域特征的变化,将变化后的Ξ维模型映射到二维图像中,再利用姿态估计的方式从二维图像中还原出Ξ维模型的旋转平移参数,从而获得头部的旋转平移矩阵,进而获得头部偏转的角度。 The intelligent monitoring system of claim 1 is based on EEG and eye movement of a driver alertness fusion claim, wherein the signal processing module of the eye comprising: a face detection module, and the eye movement signal acquisition module is connected, the image of the eye to the signal acquisition module for face detection, feature region to obtain a face; closed eye time monitoring module, connected to the face detection module, using Ξ dimensional features of the face area of ​​the cross model modeling, feature extraction for the eye in the region of the cross-dimensional model Ξ covered, and using the difference in color and eye opening of closed monitoring features of closed eye time; monitoring the deflection angle of the head module, the face detecting module is connected, wherein tracking changes in the region adjacent the face image using the signature input optical flow, the mapping Ξ dimensional model is changed to the two-dimensional image, the pose estimation reuse method to restore the two-dimensional image from Ξ dimensional model rotation and translation parameters so as to obtain rotational translation matrix head, thereby obtaining deflection angle of the head.
  3. 3. 根据权利要求2所述的基于眼动与脑电融合的驾驶员警觉度智能监控系统,其特征在于:所述Ξ维十字模型包括水平部分和垂直部分,水平部分与垂直部分的交点为坐标原点,水平部分用水平曲线h(x)表示,垂直部分用垂直曲线v(y)表示,Ξ维十字模型为: The intelligent monitoring system of claim 2 based on EEG and eye movement of a driver alertness fusion claim, wherein: said cross Ξ dimensional model and a vertical portion comprising the intersection of the horizontal portion of the horizontal portion and the vertical portion is coordinate origin, the horizontal portion of the curve represented by the horizontal h (x), represented by the vertical portion of the vertical profile v (y), Ξ cross dimension model:
    Figure CN103340637BC00031
  4. 4. 根据权利要求1所述的基于眼动与脑电融合的驾驶员警觉度智能监控系统,其特征在于:所述基于眼动与脑电融合的驾驶员警觉度智能监控系统还包括一与所述警觉度监测模块相连,且在所述警觉度状态评定的结果为异常时发出报警的报警模块。 According to claim 1, the intelligent monitoring system based on eye movement of the driver alertness and the EEG fused, wherein: based on the driver alertness monitoring system of intelligent EEG and eye movement comprises a further fusion with the alertness monitoring module is connected, and the evaluation result of the state of alertness of the alarm when abnormal alarm module.
  5. 5. 根据权利要求1所述的基于眼动与脑电融合的驾驶员警觉度智能监控系统,其特征在于:所述基于眼动与脑电融合的驾驶员警觉度智能监控系统还包括一与所述警觉度监测模块相连,且用W实时显示驾驶员眼动警觉度特征、脑电警觉度特征W及融合警觉度特征的显示模块。 According to claim 1, intelligent monitoring system based on eye movement of the driver's alertness and the EEG fused, wherein: based on the driver alertness monitoring system of intelligent EEG and eye movement comprises a further fusion with the alertness monitoring module is connected with the W and the real-time display of driver eye movements characterized in alertness, EEG alertness wherein W and fusion characteristics alertness display module.
  6. 6. -种基于眼动与脑电融合的驾驶员警觉度智能监控方法,其特征在于,包括W下步骤: 实时采集驾驶员的眼动信息,从所述眼动信息中分析和提取眼动警觉度特征; 实时采集驾驶员的脑电信号,从所述脑电信号中分析和提取脑电警觉度特征;所述脑电警觉度特征包括δ、Θ、α、β四个频带特征和(θ+α)/β、α/β、(θ+α)/(α+β)、 θ/β四个频带能量比值特征; 对驾驶员当前的眼动警觉度特征和脑电警觉度特征融合后形成的融合警觉度特征进行警觉度状态评定;对融合警觉度特征进行警觉度状态评定的具体过程包括: 建立与警觉度相关的普遍脑功能导联区; 利用fisher score算法对普遍脑功能导联区进行个人的导联分析,获得适于个人的脑功能导联区域,进而获得个人的脑电导联分量; 利用脑电导联分量进行频谱能量分析,对所述δ、Θ、α、β四个频带特 6. - Species monitoring method based on eye smart driver alertness and the EEG fusion, wherein W comprises the steps of: collecting in real time information of the driver's eye movement, eye movement analysis and extraction information from the eye characterized in alertness; real-time acquisition of the driver's EEG analysis and extraction of features from the EEG alertness EEG; alertness feature comprises the EEG δ, Θ, α, β and features four bands ( θ + α) / β, α / β, (θ + α) / (α + β), θ / β ratio wherein four energy bands; the current driver's eye movement and EEG features alertness alertness feature fusion after forming the fusion alertness features alertness state evaluation; specific process of fusion alertness features alertness state evaluation comprises: establishing associated with alertness general brain function leads region; using fisher score algorithm function derivative of general brain joint area for individual analysis leads to obtain brain function area adapted to the individual lead, thereby obtaining the individual components of the brain electrical lead; brain electrical lead components using energy spectrum analysis, the δ, Θ, α, β four bands Laid 征和(Θ + α )/ β、α/β、(θ+α)/(α+β)、θ/β四个频带能量比值特征进行多种警觉度状态任务下的脑电警觉度特征的模式分类,获得警觉度脑功能区的定位关联; 对所述导联分量、四个频带特征和四个频带能量比值特征、警觉度脑功能区的定位关联、警觉度监控相关参数W及眼动警觉度特征进行多角度实时监控。 Zheng and (Θ + α) / β, α / β, (θ + α) / (α + β), θ / β ratio of four energy bands characteristic EEG features in a plurality of alertness alertness status of a task pattern classification, obtained alertness related positioning of brain areas; the lead component, four bands and four frequency band energy ratio characteristic features associated with the positioning of brain areas alertness, alertness monitoring parameters related to eye movement and W alertness multi-angle feature real-time monitoring.
  7. 7. 根据权利要求6所述的基于眼动与脑电融合的驾驶员警觉度智能监控方法,其特征在于,从所述眼动信息中分析和提取眼动警觉度特征的具体过程包括: 对采集到的图像进行人脸检测,获得脸部区域特征; 对脸部区域特征使用Ξ维十字模型进行建模,在Ξ维十字模型覆盖的区域进行眼部特征提取,并利用眼睛睁开和闭合时颜色特征的差异进行闭眼时间的监测;所述Ξ维十字模型包括水平部分和垂直部分,水平部分与垂直部分的交点为坐标原点,水平部分用水平曲线h(x)表示,垂直部分用垂直曲线v(y)表示,Ξ维十字模型为: 々(Λ-)二(八 According to claim eye based fusion EEG intelligent driver alertness monitoring method, characterized in that, from the eye of the specific process of analyzing and extracting information alertness eye features include the 6: collected image for face detection, face region feature is obtained; Ξ using cross-dimensional model of the face region feature modeling, feature extraction for the eye in the region of the cross-dimensional model Ξ covered, using open and closed eyes difference in color characteristics when closed-eye monitoring time; Ξ said cross-dimensional model comprises a horizontal portion and a vertical portion, a horizontal portion and the vertical portion of the intersection point as the coordinate origin, the horizontal portion of the curve represented by the horizontal h (x), with the vertical portions vertical curve v (y) represents, Ξ cross dimension model: 々 (Lambda) bis (eight
    Figure CN103340637BC00041
    追踪相邻帖的输入图像中脸部区域特征的变化,利用姿态估计的方式获得头部的旋转平移矩阵,进而获得头部偏转的角度。 Tracking changes in the face region wherein adjacent posts input image matrix obtained rotational translation of the head, and thus obtain a deflection angle of the head pose estimation using the embodiment.
  8. 8.根据权利要求6所述的基于眼动与脑电融合的驾驶员警觉度智能监控方法,其特征在于,所述智能监控方法还包括: 在所述警觉度状态评定结果为异常时发出报警; 实时显示驾驶员眼动警觉度特征、脑电警觉度特征W及融合警觉度特征。 8. The method of intelligent monitoring driver alertness and the EEG eye based fusion, wherein according to claim 6, the intelligent monitoring method further comprises: alarm when the abnormal state of alertness Rating ; real-time display of driver eye movements characterized in alertness, wherein W EEG alertness alertness and fusion characteristics.
CN 201310225220 2013-06-06 2013-06-06 Driver alertness monitoring system and method for intelligent eye movement and EEG Fusion CN103340637B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN 201310225220 CN103340637B (en) 2013-06-06 2013-06-06 Driver alertness monitoring system and method for intelligent eye movement and EEG Fusion

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN 201310225220 CN103340637B (en) 2013-06-06 2013-06-06 Driver alertness monitoring system and method for intelligent eye movement and EEG Fusion

Publications (2)

Publication Number Publication Date
CN103340637A true CN103340637A (en) 2013-10-09
CN103340637B true CN103340637B (en) 2016-04-06

Family

ID=49275504

Family Applications (1)

Application Number Title Priority Date Filing Date
CN 201310225220 CN103340637B (en) 2013-06-06 2013-06-06 Driver alertness monitoring system and method for intelligent eye movement and EEG Fusion

Country Status (1)

Country Link
CN (1) CN103340637B (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103919565A (en) * 2014-05-05 2014-07-16 重庆大学 Fatigue driving electroencephalogram signal feature extraction and identification method
CN104146701A (en) * 2014-07-22 2014-11-19 南京航空航天大学 Multi-mode 3D (three-dimensional) television health detecting system
CN104464003B (en) * 2014-12-04 2017-02-22 重庆晋才富熙科技有限公司 Attentiveness attendance method
CN105893980A (en) * 2016-04-26 2016-08-24 北京科技大学 Method and system for evaluating concentration degree of attention

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102622600A (en) * 2012-02-02 2012-08-01 西南交通大学 High-speed train driver alertness detecting method based on face image and eye movement analysis
CN202665556U (en) * 2012-07-05 2013-01-16 西南交通大学 Eyeglass type truck driver fatigue parameter acquiring device

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7435227B2 (en) * 2004-09-13 2008-10-14 Biocognisafe (Bcs) Technologies Method and apparatus for generating an indication of a level of vigilance of an individual

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102622600A (en) * 2012-02-02 2012-08-01 西南交通大学 High-speed train driver alertness detecting method based on face image and eye movement analysis
CN202665556U (en) * 2012-07-05 2013-01-16 西南交通大学 Eyeglass type truck driver fatigue parameter acquiring device

Also Published As

Publication number Publication date Type
CN103340637A (en) 2013-10-09 application

Similar Documents

Publication Publication Date Title
Li et al. Learning to predict gaze in egocentric video
El Kaliouby et al. Real-time inference of complex mental states from facial expressions and head gestures
D’Orazio et al. A visual approach for driver inattention detection
US20140204193A1 (en) Driver gaze detection system
US8462996B2 (en) Method and system for measuring human response to visual stimulus based on changes in facial expression
US7043056B2 (en) Facial image processing system
Lalonde et al. Real-time eye blink detection with GPU-based SIFT tracking
Mbouna et al. Visual analysis of eye state and head pose for driver alertness monitoring
Al-Rahayfeh et al. Eye tracking and head movement detection: A state-of-art survey
Dinges et al. Optical computer recognition of facial expressions associated with stress induced by performance demands
CN102436715A (en) Detection method for fatigue driving
Lu et al. A head pose-free approach for appearance-based gaze estimation.
CN101090482A (en) Driver fatigue monitoring system and method based on image process and information mixing technology
CN102254151A (en) Driver fatigue detection method based on face video analysis
Cheng et al. Multi-spectral and multi-perspective video arrays for driver body tracking and activity analysis
CN101375796A (en) Real-time detection system of fatigue driving
Jo et al. Detecting driver drowsiness using feature-level fusion and user-specific classification
Torricelli et al. A neural-based remote eye gaze tracker under natural head motion
CN101540090A (en) Driver fatigue monitoring device based on multivariate information fusion and monitoring method thereof
CN101908140A (en) Biopsy method for use in human face identification
CN101593425A (en) Machine vision based fatigue driving monitoring method and system
Jo et al. Vision-based method for detecting driver drowsiness and distraction in driver monitoring system
CN101201695A (en) Mouse system for extracting and tracing based on ocular movement characteristic
CN101950355A (en) Method for detecting fatigue state of driver based on digital video
Sigari et al. A driver face monitoring system for fatigue and distraction detection

Legal Events

Date Code Title Description
C06 Publication
C10 Entry into substantive examination
C14 Grant of patent or utility model