WO2023036015A1 - 一种基于多维身体状态感知的疲劳检测方法及系统 - Google Patents

一种基于多维身体状态感知的疲劳检测方法及系统 Download PDF

Info

Publication number
WO2023036015A1
WO2023036015A1 PCT/CN2022/115881 CN2022115881W WO2023036015A1 WO 2023036015 A1 WO2023036015 A1 WO 2023036015A1 CN 2022115881 W CN2022115881 W CN 2022115881W WO 2023036015 A1 WO2023036015 A1 WO 2023036015A1
Authority
WO
WIPO (PCT)
Prior art keywords
fatigue
fatigue detection
eye
facial
voice
Prior art date
Application number
PCT/CN2022/115881
Other languages
English (en)
French (fr)
Inventor
朱明增
莫梓樱
覃秋勤
吕鸣
刘小兰
陈极万
韩竞
李和峰
陈名良
欧健美
温黎明
周素君
马红康
宋嗣皇
梁维
梁朝聪
Original Assignee
广西电网有限责任公司贺州供电局
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 广西电网有限责任公司贺州供电局 filed Critical 广西电网有限责任公司贺州供电局
Publication of WO2023036015A1 publication Critical patent/WO2023036015A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
    • G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
    • G10L25/66—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination for extracting parameters related to health condition

Definitions

  • the invention relates to the field of computer technology, in particular to a fatigue detection method and system based on multi-dimensional body state perception.
  • Fatigue detection lacks obvious definite features. Contact detection such as EEG and ECG measurement generally requires electrodes to be pasted on the human body, which will cause resentment and rejection by the tester to a certain extent. The influence of physical condition. The environment of the power dispatching site is complex, noise, light, etc. cause difficulties in feature extraction, which greatly reduces the accuracy of fatigue detection. The fatigue detection process is often affected by non-universal applicability factors, and it is difficult to judge the fatigue state with a single feature, which leads to a decrease in the recognition rate.
  • the purpose of the present invention is to overcome the deficiencies of the prior art.
  • the present invention provides a fatigue detection method and system based on multi-dimensional body state perception, and improves the effectiveness of the entire system by providing fusion of various fatigue detection information.
  • an embodiment of the present invention provides a fatigue detection method based on multi-dimensional body state perception, the method comprising:
  • infrared image data of facial features through infrared technology, preprocess the infrared image data, extract the face information from the infrared image data, and extract the eye feature data and mouth feature data from the face information.
  • the feature data and mouth feature data are used to detect the state of facial fatigue and obtain the result of facial fatigue detection;
  • the facial fatigue detection results and voice fatigue detection results are fused, multi-classifier fusion is selected based on information entropy, and the content of the decision table is fused to obtain the final human fatigue detection result.
  • the infrared image data collected by infrared technology to collect facial features includes:
  • the infrared light source is used to supplement the light to assist the imaging of the tester.
  • the preprocessing of the infrared image data, extracting face information from the infrared image data, and extracting eye feature data and mouth feature data from the face information include:
  • Grayscale and binarize the infrared image extract the tested person from the image background, use the Adaboost algorithm to train on non-overlapping training sets to obtain multiple weaker classifiers, and use the trained Multiple weak classifiers are combined to obtain a strong classifier, so as to locate the tester's eyes and mouth, use the Retinex algorithm to enhance the image of the positioning area and extract feature points, and use the obtained feature point information to quantify the state data of the eyes and mouth .
  • the detection of facial fatigue state by analyzing eye feature data and mouth feature data includes:
  • the mouth closure degree of the mouth feature data is quantified according to the PMOT criterion.
  • the quantification of the fatigue state of the eye feature data according to the PERCLOS rule includes:
  • the voice information is collected by the voice input module, and the voice fatigue state is detected based on the voice information, and the voice fatigue detection result obtained includes:
  • Collect the voice information of the tester mark the collected voice information and use it as the source sample; find the feature transformation to map different data sets to the same feature space, so that the marginal probability distribution and conditional probability distribution of the source sample and the target sample are infinitely close ; and then repeatedly train the migration classifier to enhance the generalization ability and detect the fatigue state of the target sample.
  • the present invention also provides a fatigue detection system based on multi-dimensional body state perception, the system comprising:
  • the facial fatigue detection module is used to collect infrared image data of facial features through infrared technology, preprocess the infrared image data, extract face information from infrared image data, and extract eye feature data and Mouth feature data, by analyzing eye feature data and mouth feature data to detect facial fatigue status and obtain facial fatigue detection results;
  • the voice fatigue detection module is used to collect sound information through the voice input module, and detect the voice fatigue state based on the voice information, and obtain the voice fatigue detection result;
  • the fusion detection module is used to fuse the facial fatigue detection results and voice fatigue detection results, perform multi-classifier fusion based on information entropy selection, and fuse the contents of the decision table to obtain the final human fatigue detection result.
  • the facial fatigue detection module performs grayscale and binarization processing on the infrared image, extracts the person under test from the image background, and uses the Adaboost algorithm to train on non-overlapping training sets to obtain multiple weaker classifications. Combine multiple weak classifiers obtained through training to obtain a strong classifier, thereby positioning the tester's eyes and mouth, using the Retinex algorithm to enhance the image of the positioning area and extract feature points, and use the obtained feature point information to quantify State data for eyes and mouth.
  • the facial fatigue detection module quantifies the fatigue state of the eye feature data according to the PERCLOS rule; quantifies the mouth closure degree of the mouth feature data according to the PMOT rule.
  • the facial fatigue detection module sets the eye height value and the eye width value, and calculates the eye closure degree based on the eye height value and the eye width value, and matches the current eye fatigue degree value based on the eye closure degree.
  • fatigue detection technology based on multi-information fusion is used to comprehensively analyze the fatigue state of staff from multiple dimensions . Fully obtain a variety of information resources and optimize their combination to obtain more effective information, thereby improving the effectiveness of the entire system. In the process of fatigue detection, the detection results are often affected by non-universal applicability factors, and it is difficult to judge the fatigue state with a single feature, which leads to a decrease in the recognition rate.
  • the present invention combines and analyzes the facial fatigue state and the speech fatigue state, obtains multiple information resources and optimizes the combination of information to obtain more effective information and improves the effectiveness of the entire system.
  • Fig. 1 is a flowchart of a fatigue detection method based on multidimensional body state perception in an embodiment of the present invention
  • Fig. 2 is the flow chart of facial fatigue detection in the embodiment of the present invention.
  • Fig. 3 is the speech fatigue detection flowchart in the embodiment of the present invention.
  • Fig. 4 is a fatigue detection system based on multi-dimensional body state perception in an embodiment of the present invention.
  • Fig. 1 shows a flow chart of a fatigue detection method based on multi-dimensional body state perception in an embodiment of the present invention, the method includes the following steps:
  • the facial fatigue state recognition is carried out first, that is, to obtain an image containing the face information of the subject through infrared technology acquisition image technology, preprocess the image, extract the face from the image, and extract the eyes and mouth from the face feature, locate it, and detect the facial fatigue state by analyzing its features.
  • the infrared image data of facial features is collected by infrared technology, the infrared image data is preprocessed, the face information is extracted from the infrared image data, and the eye feature data and mouth feature data are extracted from the face information.
  • the eye feature data and the mouth feature data are used to detect the facial fatigue state and obtain the facial fatigue detection result.
  • collecting infrared image data of facial features through infrared technology includes: using an infrared light source to supplement light on the tester to assist in imaging.
  • preprocessing the infrared image data, extracting face information from the infrared image data, and extracting eye feature data and mouth feature data from the face information includes: grayscale and binarization of the infrared image Processing, the person under test is extracted from the image background, and the Adaboost algorithm is used to train on non-overlapping training sets to obtain multiple weak classifiers, and the multiple weak classifiers obtained by training are combined to obtain a strong classifier , so as to locate the eyes and mouth of the tester, use the Retinex algorithm to enhance the image of the positioning area and extract feature points, and use the obtained feature point information to quantify the state data of the eyes and mouth.
  • detecting the facial fatigue state by analyzing the eye feature data and the mouth feature data includes: quantifying the fatigue state of the eye feature data according to the PERCLOS rule; quantifying the mouth closure degree of the mouth feature data according to the PMOT rule.
  • the quantification of the fatigue state of the eye feature data according to the PERCLOS rule includes: setting the eye height value and the eye width value, and calculating the eye closure based on the eye height value and the eye width value, and matching the current eye closure based on the eye closure. value of fatigue.
  • Fig. 2 shows the flow chart of facial fatigue detection in the embodiment of the present invention.
  • Using video images to collect information will reveal the privacy of the testee to a large extent, so the present invention intends to use infrared imaging technology to test
  • the facial information of the user is displayed in the form of an infrared image. Affected by factors such as the occlusion of the person under test and the change in illumination, there may be information loss in image acquisition, which will greatly affect the accuracy of fatigue state detection. Therefore, an infrared light source is first used to supplement the light of the tester to assist in imaging.
  • ⁇ h eye [dist(p37,p41)+dist(p38,p40)]/2
  • the eye closure d c is calculated as follows:
  • the state of the mouth will change significantly, for example, the frequency of yawning will increase significantly, and the degree of mouth closure will increase significantly after completing the action of "yawning" compared with normal speaking or static state.
  • the calculation of the mouth closure is similar to the calculation of the eye closure. The height, width and closure of the mouth are calculated separately. The calculation process is as follows:
  • Mouth closure was quantified according to PMOT criteria. Combining the PERCLOS criterion and the PMOT criterion to fuse the state information of the eyes and mouth, the classifier is trained and the analysis of the facial fatigue state is completed.
  • S102 Collect voice information through the voice input module, and detect the voice fatigue state based on the voice information, and obtain a voice fatigue detection result;
  • the voice information is collected through the voice input module, and the voice fatigue state is detected based on the voice information.
  • Obtaining the voice fatigue detection result includes: performing Mel frequency conversion on the spectrogram formed by the voice of the testee, and using the frequency change to stretch Sensitive range of perception, enhance the system’s perception of fatigue; collect the tester’s voice information, mark the collected voice information and use it as the source sample; find the feature transformation and map different data sets to the same feature space, so that the source The marginal probability distribution and the conditional probability distribution of the sample and the target sample are infinitely close; then the migration classifier is repeatedly trained to enhance the generalization ability and detect the fatigue state of the target sample.
  • Fig. 3 shows the flow chart of speech fatigue detection in the embodiment of the present invention, which performs feature extraction for samples, adopts domain adaptation and dimensionality reduction to realize iterative optimization of data mapping, and realizes the tester's speech data by obtaining test speech Feature extraction, feature transformation based on data mapping iterative optimization, and classifier training to obtain speech fatigue detection results.
  • the three most relevant features are extracted from the tester's voice, namely: spectrum, sound quality and speech rhythm.
  • the spectrogram formed by the voice of the subject is subjected to Mel frequency transformation, and the mapping relationship between linear frequency and Mel frequency can be expressed as:
  • the linear spectrum has a higher sensitivity to fatigue, and the proportion of the low-frequency part with a higher sensitivity to fatigue in the linear spectrogram is much smaller than that after the Mel frequency change, so the frequency change is used to stretch the sensitivity. Interval, enhance the system's perception of fatigue.
  • T(x) it is necessary to find the feature transformation T(x) to map different data sets to the same feature space, so that the marginal probability distribution and the conditional probability distribution of the two are infinitely close, that is, P[T(x s )] ⁇ P[T(x t )] and Q[T(x s )
  • a single feature is easily affected by various environmental and human factors, so the facial fatigue detection results and voice fatigue detection results are fused, as shown in the figure below, the decision-making conditions are fused at the decision-making level to obtain a decision table, based on information entropy selection. Classifier fusion, the content of the decision table is fused to obtain the final decision result, and the final judgment of human fatigue detection is realized.
  • FIG. 4 shows a fatigue detection system based on multi-dimensional body state perception in an embodiment of the present invention, and the system includes:
  • the facial fatigue detection module is used to collect infrared image data of facial features through infrared technology, preprocess the infrared image data, extract face information from infrared image data, and extract eye feature data and Mouth feature data, by analyzing eye feature data and mouth feature data to detect facial fatigue status and obtain facial fatigue detection results;
  • the voice fatigue detection module is used to collect sound information through the voice input module, and detect the voice fatigue state based on the voice information, and obtain the voice fatigue detection result;
  • the fusion detection module is used to fuse the facial fatigue detection results and voice fatigue detection results, perform multi-classifier fusion based on information entropy selection, and fuse the contents of the decision table to obtain the final human fatigue detection result.
  • the facial fatigue detection module performs grayscale and binarization processing on the infrared image, extracts the person under test from the image background, and uses the Adaboost algorithm to train on non-overlapping training sets to obtain multiple weaker
  • a strong classifier is obtained by combining multiple weak classifiers obtained through training to obtain a strong classifier, thereby positioning the tester's eyes and mouth, using the Retinex algorithm to enhance the image of the positioning area and extracting feature points, and using the obtained feature points Information quantifies eye and mouth state data.
  • the facial fatigue detection module quantifies the fatigue state of the eye feature data according to the PERCLOS rule; quantifies the mouth closure degree of the mouth feature data according to the PMOT rule.
  • the facial fatigue detection module sets the eye height value and the eye width value, calculates the degree of eye closure based on the eye height value and the eye width value, and matches the current eye fatigue degree value based on the eye degree of closure.
  • fatigue detection technology based on multi-information fusion is used to comprehensively analyze the fatigue state of staff from multiple dimensions . Fully obtain a variety of information resources and optimize their combination to obtain more effective information, thereby improving the effectiveness of the entire system. In the process of fatigue detection, the detection results are often affected by non-universal applicability factors, and it is difficult to judge the fatigue state with a single feature, which leads to a decrease in the recognition rate.
  • the present invention combines and analyzes the fatigue state of the face and the fatigue state of the voice, obtains multiple information resources, optimizes the combination of information and obtains more effective information, and improves the effectiveness of the entire system.

Landscapes

  • Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Epidemiology (AREA)
  • Probability & Statistics with Applications (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Computational Linguistics (AREA)
  • Signal Processing (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)

Abstract

本发明公开了一种基于多维身体状态感知的疲劳检测方法及系统，其方法包括：通过红外技术采集人脸特征的红外图像数据，对红外图像数据进行预处理，将人脸信息从红外图像数据中提取出来，并从人脸信息中提取眼睛特征数据和嘴巴特征数据，通过分析眼睛特征数据和嘴巴特征数据对面部疲劳状态进行检测，获取面部疲劳检测结果；通过语音输入模块采集声音信息，并基于声音信息进行语音疲劳状态进行检测，获取语音疲劳检测结果；将面部疲劳检测结果和语音疲劳检测结果进行融合，基于信息熵选择进行多分类器融合，将决策表内容进行融合得到最终人体疲劳检测结果。在本发明实施例通过提供多种疲劳检测信息的融合，提升了整个系统的有效性。

Description

一种基于多维身体状态感知的疲劳检测方法及系统 技术领域
本发明涉及计算机技术领域，尤其涉及一种基于多维身体状态感知的疲劳检测方法及系统。
背景技术
目前电力系统的自动化技术水平与可靠性已经大幅提高，但人在电力系统中依旧发挥着重要作用。长时间高强度的现场作业容易造成人体疲劳，导致注意力难以集中，机体活力及反应力大大下降，这不仅会影响生产作业人员的工作效率，甚至会引发生产安全事故，因此，作为主动预防事故发生的一项措施，疲劳检测具有重要的社会意义和实际价值。
疲劳检测缺少明显的确定特征，接触式检测比如脑电、心电的测量一般需要在人的身体上粘贴电极，在一定程度会造成测试人的反感排斥，操作复杂且易受被试者情绪、身体状况的影响。电力调度现场环境复杂，噪声，光照等对特征提取造成困难，极大降低疲劳检测的准确性。疲劳检测过程常常会受到非普遍适用性因素影响，单特征难以对疲劳状态进行断定，从而导致识别率降低。
发明内容
本发明的目的在于克服现有技术的不足，本发明提供了一种基于多维身体状态感知的疲劳检测方法及系统，通过提供多种疲劳检测信息的融合，提升了整个系统的有效性。
为了解决上述技术问题，本发明实施例提供了一种基于多维身体状态感知的疲劳检测方法，所述方法包括：
通过红外技术采集人脸特征的红外图像数据，对红外图像数据进行预处理，将人脸信息从红外图像数据中提取出来，并从人脸信息中提取眼睛特征数据和嘴巴特征数据，通过分析眼睛特征数据和嘴巴特征数据对面部 疲劳状态进行检测，获取面部疲劳检测结果；
通过语音输入模块采集声音信息，并基于声音信息进行语音疲劳状态进行检测，获取语音疲劳检测结果；
将面部疲劳检测结果和语音疲劳检测结果进行融合，基于信息熵选择进行多分类器融合，将决策表内容进行融合得到最终人体疲劳检测结果。
所述通过红外技术采集人脸特征的红外图像数据包括：
使用红外光源对测试者进行补光协助成像。
所述对红外图像数据进行预处理，将人脸信息从红外图像数据中提取出来，并从人脸信息中提取眼睛特征数据和嘴巴特征数据包括：
对红外图像进行灰度化以及二值化处理，将被测人员从图像背景中进行提取，采用Adaboost算法在互不重叠的训练集上进行训练得到多个较弱的分类器，将训练得到的多个弱分类器进行联合得到强分类器，从而对测试者眼睛和嘴巴进行定位，采用Retinex算法对定位区域图像进行增强并提取特征点，利用所得到的特征点信息量化眼睛和嘴的状态数据。
所述通过分析眼睛特征数据和嘴巴特征数据对面部疲劳状态进行检测包括：
依据PERCLOS规则对眼睛特征数据的疲劳状态进行量化；
依据PMOT准则对嘴巴特征数据的嘴巴闭合度进行量化。
所述依据PERCLOS规则对眼睛特征数据的疲劳状态进行量化包括：
设定眼睛高度值和眼睛宽度值，并基于眼睛高度值和眼睛宽度值进行眼睛闭合度计算，并基于眼睛闭合度匹配当前眼睛的疲劳程度值。
所述通过语音输入模块采集声音信息，并基于声音信息进行语音疲劳状态进行检测，获取语音疲劳检测结果包括：
将被测试者声音所形成的语谱图进行Mel频率变换，利用频率变化拉伸感知度敏感区间，增强系统对疲劳的感知度；
对测试者声音信息进行采集，对所采集的声音信息进行标记并作为源样本；找到特征变换将不同的数据集映射至同一特征空间，使源样本和目标样本边缘概率分布和条件概率分布无限接近；然后反复训练迁移分类器增强泛化能力，对目标样本的疲劳状态进行检测。
相应的，本发明还提供了一种基于多维身体状态感知的疲劳检测系统，所述系统包括：
面部疲劳检测模块，用于通过红外技术采集人脸特征的红外图像数据，对红外图像数据进行预处理，将人脸信息从红外图像数据中提取出来，并从人脸信息中提取眼睛特征数据和嘴巴特征数据，通过分析眼睛特征数据和嘴巴特征数据对面部疲劳状态进行检测，获取面部疲劳检测结果；
语音疲劳检测模块，用于通过语音输入模块采集声音信息，并基于声音信息进行语音疲劳状态进行检测，获取语音疲劳检测结果；
融合检测模块，用于将面部疲劳检测结果和语音疲劳检测结果进行融合，基于信息熵选择进行多分类器融合，将决策表内容进行融合得到最终人体疲劳检测结果。
所述面部疲劳检测模块对红外图像进行灰度化以及二值化处理，将被测人员从图像背景中进行提取，采用Adaboost算法在互不重叠的训练集上进行训练得到多个较弱的分类器，将训练得到的多个弱分类器进行联合得到强分类器，从而对测试者眼睛和嘴巴进行定位，采用Retinex算法对定位区域图像进行增强并提取特征点，利用所得到的特征点信息量化眼睛和嘴的状态数据。
所述面部疲劳检测模块依据PERCLOS规则对眼睛特征数据的疲劳状态进行量化；依据PMOT准则对嘴巴特征数据的嘴巴闭合度进行量化。
所述面部疲劳检测模块设定眼睛高度值和眼睛宽度值，并基于眼睛高度值和眼睛宽度值进行眼睛闭合度计算，并基于眼睛闭合度匹配当前眼睛的疲劳程度值。
在本发明实施例中基于多维身体状态感知方式，基于面部信息特征的疲劳检测以及基于生理信息特征的疲劳检测，采用基于多信息融合的疲劳检测技术，从多个维度全面分析工作人员的疲劳状态。充分获得多种信息资源并对其进行优化组合获取更多的有效信息，从而提高整个系统的有效性。在疲劳检测过程中，检测结果常常会受到非普遍适用性因素的影响，单特征难以对疲劳状态进行断定，从而导致识别率降低。本发明将面部疲劳状态和语音疲劳状态结合分析，获得多种信息资源对信息的优化组合获 取更多的有效信息，提高了整个系统的有效性。
附图说明
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见的，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其它的附图。
图1是本发明实施例中的基于多维身体状态感知的疲劳检测方法流程图；
图2是本发明实施例中的面部疲劳检测流程图；
图3是本发明实施例中的语音疲劳检测流程图；
图4是本发明实施例中的基于多维身体状态感知的疲劳检测系统。
具体实施方式
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其它实施例，都属于本发明保护的范围。
图1示出了本发明实施例中的基于多维身体状态感知的疲劳检测方法流程图，该方法包括以下步骤：
S101、获取面部疲劳检测结果；
这里先进行面部疲劳状态识别，即为通过红外技术采集图像技术得到包含被测试者面部信息的图像，对图像进行预处理，将人脸从图像中提取出来，并从人脸中提取眼睛和嘴巴特征，对其进行定位，通过分析其特征对面部疲劳状态进行检测。
即通过红外技术采集人脸特征的红外图像数据，对红外图像数据进行预处理，将人脸信息从红外图像数据中提取出来，并从人脸信息中提取眼 睛特征数据和嘴巴特征数据，通过分析眼睛特征数据和嘴巴特征数据对面部疲劳状态进行检测，获取面部疲劳检测结果。
具体的，通过红外技术采集人脸特征的红外图像数据包括：使用红外光源对测试者进行补光协助成像。
具体的，对红外图像数据进行预处理，将人脸信息从红外图像数据中提取出来，并从人脸信息中提取眼睛特征数据和嘴巴特征数据包括：对红外图像进行灰度化以及二值化处理，将被测人员从图像背景中进行提取，采用Adaboost算法在互不重叠的训练集上进行训练得到多个较弱的分类器，将训练得到的多个弱分类器进行联合得到强分类器，从而对测试者眼睛和嘴巴进行定位，采用Retinex算法对定位区域图像进行增强并提取特征点，利用所得到的特征点信息量化眼睛和嘴的状态数据。
具体的，通过分析眼睛特征数据和嘴巴特征数据对面部疲劳状态进行检测包括：依据PERCLOS规则对眼睛特征数据的疲劳状态进行量化；依据PMOT准则对嘴巴特征数据的嘴巴闭合度进行量化。
具体的，依据PERCLOS规则对眼睛特征数据的疲劳状态进行量化包括：设定眼睛高度值和眼睛宽度值，并基于眼睛高度值和眼睛宽度值进行眼睛闭合度计算，并基于眼睛闭合度匹配当前眼睛的疲劳程度值。
需要说明的是，图2示出了本发明实施例中的面部疲劳检测流程图，利用视频图像进行信息采集将很大程度泄露被测试者的隐私，故本发明拟采用红外成像技术，将测试者面部信息以红外图像形式展示。受到被测人员衣帽遮挡以及光照变化等因素影响，图像采集可能存在信息缺失现象，这将极大影响疲劳状态检测精度，故首先使用红外光源对测试者进行补光协助成像。对红外图像进行灰度化以及二值化等一系列处理，将被测人员从图像背景中进行提取，采用Adaboost算法在互不重叠的训练集上进行训练得到多个较弱的分类器，将训练得到的多个弱分类器进行联合得到强分类器，从而对测试者眼睛和嘴巴进行定位，采用Retinex算法对定位区域图像进行增强并提取特征点，利用所得到的特征点信息量化眼睛和嘴的状态。例如，人体疲劳时眼睛闭合度相对于轻松时会极大下降，并且闭眼过程持续的时间较长，故可以依据PERCLOS规则对疲劳状态进行量化。
设定右眼高度为Δh，右眼宽度为Δw，则有：
Δh eye＝[dist(p37,p41)+dist(p38,p40)]/2
Δw eye＝dist(p36,p39)
则两点之间的距离可以表示为：
Figure PCTCN2022115881-appb-000001
由于在实际工程中对眼睛长度和宽度的计算量较大，故可以将上述公式进行简化得到：
Δh eye≈[abs(y p37-y p41+y p38-y p40)]/2
Δw eye≈abs(x p36-x p39)
眼睛闭合度d c计算如下：
Figure PCTCN2022115881-appb-000002
将某时刻眼睛的疲劳程度进行量化：
Figure PCTCN2022115881-appb-000003
人体处于疲劳状态时嘴部状态会发生明显变化例如打哈欠频率显著增加，并且完成“打哈欠”这一动作之后嘴巴闭合程度相对于正常讲话或者静止状态时会明显增大。嘴巴闭合度计算类似于眼睛闭合度的计算，分别计算得到嘴巴高度和宽度以及闭合度，计算过程如下：
Figure PCTCN2022115881-appb-000004
Δw mouth≈abs(x p60-x p64)
Figure PCTCN2022115881-appb-000005
依据PMOT准则对嘴巴闭合度进行量化。结合PERCLOS准则以及PMOT准则融合眼睛和嘴巴状态信息，对分类器进行训练并完成对面部疲劳状态的分析。
S102、通过语音输入模块采集声音信息，并基于声音信息进行语音疲劳状态进行检测，获取语音疲劳检测结果；
具体的，通过语音输入模块采集声音信息，并基于声音信息进行语音疲劳状态进行检测，获取语音疲劳检测结果包括：将被测试者声音所形成的语谱图进行Mel频率变换，利用频率变化拉伸感知度敏感区间，增强系 统对疲劳的感知度；对测试者声音信息进行采集，对所采集的声音信息进行标记并作为源样本；找到特征变换将不同的数据集映射至同一特征空间，使源样本和目标样本边缘概率分布和条件概率分布无限接近；然后反复训练迁移分类器增强泛化能力，对目标样本的疲劳状态进行检测。
具体的，图3示出了本发明实施例中的语音疲劳检测流程图，其针对样本进行特征提取，采用领域适配与降维实现数据映射迭代优化，通过获取测试语音实现对测试者语音数据的特征提取，基于数据映射迭代优化进行特征变换，实现分类器训练得出语音疲劳检测结果。
人体疲劳时，声带肌肉相对松弛，语音音质也会发生显著变化，并且由于疲劳脑活力降低，反应能力也随之下降，语言的停顿呈现无规律状态，并且对音质的清晰度也会产生影响。故对测试者语音提取三个最相关特征，分别是：语谱，音质以及话语节奏。
将被测试者声音所形成的语谱图进行Mel频率变换，线性频率和Mel频率之间的映射关系可以表示为：
el(f)＝2595lg(1+f/700)
在低频部分线性频谱对疲劳的感知度较高，对疲劳的感知度较高的低频部分在线性频谱图中所占比例远小于经过Mel频率变化以后的比例，故利用频率变化拉伸感知度敏感区间，增强系统对疲劳的感知度。
对测试者声音信息进行采集，对其进行标记并作为源样本，＝{(x s1,y s1),(x s2,y s2),...(x sn,y sn)}即为收集到的数据集，x为特征向量，y是类别编号，一共有n个样本，D t＝{(x t1,y t1),(x t2,y t2),...(x tn,y tn)}为无标记的目标样本，由于两个数据集的边缘概率分布不同，同时条件概率分布有所差别，故对D s进行训练得到的分类器对D t并不适用。故需找到特征变换T(x)将不同的数据集映射至同一特征空间，使二者边缘概率分布和条件概率分布无限接近，即为P[T(x s)]≈P[T(x t)]并且Q[T(x s)|y s]≈Q[T(x t)|y t]。然后反复训练迁移分类器f[T(x)]，增强其泛化能力，对目标样本的疲劳状态进行检测。
S103、将面部疲劳检测结果和语音疲劳检测结果进行融合，基于信息熵选择进行多分类器融合，将决策表内容进行融合得到最终人体疲劳检测结果。
单一特征容易受环境和人为因素各方面影响，故将面部疲劳检测结果和语音疲劳检测结果进行融合，如下图所示，在决策层对判定条件进行融合处理得到决策表，基于信息熵选择进行多分类器融合，将决策表内容进行融合得到最终决策结果，实现对人体疲劳检测的最终判定。
相应的，图4示出了本发明实施例中的基于多维身体状态感知的疲劳检测系统，所述系统包括：
面部疲劳检测模块，用于通过红外技术采集人脸特征的红外图像数据，对红外图像数据进行预处理，将人脸信息从红外图像数据中提取出来，并从人脸信息中提取眼睛特征数据和嘴巴特征数据，通过分析眼睛特征数据和嘴巴特征数据对面部疲劳状态进行检测，获取面部疲劳检测结果；
语音疲劳检测模块，用于通过语音输入模块采集声音信息，并基于声音信息进行语音疲劳状态进行检测，获取语音疲劳检测结果；
融合检测模块，用于将面部疲劳检测结果和语音疲劳检测结果进行融合，基于信息熵选择进行多分类器融合，将决策表内容进行融合得到最终人体疲劳检测结果。
具体的，该面部疲劳检测模块对红外图像进行灰度化以及二值化处理，将被测人员从图像背景中进行提取，采用Adaboost算法在互不重叠的训练集上进行训练得到多个较弱的分类器，将训练得到的多个弱分类器进行联合得到强分类器，从而对测试者眼睛和嘴巴进行定位，采用Retinex算法对定位区域图像进行增强并提取特征点，利用所得到的特征点信息量化眼睛和嘴的状态数据。
具体的，该面部疲劳检测模块依据PERCLOS规则对眼睛特征数据的疲劳状态进行量化；依据PMOT准则对嘴巴特征数据的嘴巴闭合度进行量化。
具体的，该面部疲劳检测模块设定眼睛高度值和眼睛宽度值，并基于眼睛高度值和眼睛宽度值进行眼睛闭合度计算，并基于眼睛闭合度匹配当前眼睛的疲劳程度值。
在本发明实施例中基于多维身体状态感知方式，基于面部信息特征的疲劳检测以及基于生理信息特征的疲劳检测，采用基于多信息融合的疲劳检测技术，从多个维度全面分析工作人员的疲劳状态。充分获得多种信息 资源并对其进行优化组合获取更多的有效信息，从而提高整个系统的有效性。在疲劳检测过程中，检测结果常常会受到非普遍适用性因素的影响，单特征难以对疲劳状态进行断定，从而导致识别率降低。本发明将面部疲劳状态和语音疲劳状态结合分析，获得多种信息资源对信息的优化组合获取更多的有效信息，提高了整个系统的有效性。
以上对本发明实施例所进行了详细介绍，本文中应采用了具体个例对本发明的原理及实施方式进行了阐述，以上实施例的说明只是用于帮助理解本发明的方法及其核心思想；同时，对于本领域的一般技术人员，依据本发明的思想，在具体实施方式及应用范围上均会有改变之处，综上所述，本说明书内容不应理解为对本发明的限制。

Claims (10)

  1. 一种基于多维身体状态感知的疲劳检测方法，其特征在于，所述方法包括：
    通过红外技术采集人脸特征的红外图像数据，对红外图像数据进行预处理，将人脸信息从红外图像数据中提取出来，并从人脸信息中提取眼睛特征数据和嘴巴特征数据，通过分析眼睛特征数据和嘴巴特征数据对面部疲劳状态进行检测，获取面部疲劳检测结果；
    通过语音输入模块采集声音信息，并基于声音信息进行语音疲劳状态进行检测，获取语音疲劳检测结果；
    将面部疲劳检测结果和语音疲劳检测结果进行融合，基于信息熵选择进行多分类器融合，将决策表内容进行融合得到最终人体疲劳检测结果。
  2. 如权利要求1所述的基于多维身体状态感知的疲劳检测方法，其特征在于，所述通过红外技术采集人脸特征的红外图像数据包括：
    使用红外光源对测试者进行补光协助成像。
  3. 如权利要求1所述的基于多维身体状态感知的疲劳检测方法，其特征在于，所述对红外图像数据进行预处理，将人脸信息从红外图像数据中提取出来，并从人脸信息中提取眼睛特征数据和嘴巴特征数据包括：
    对红外图像进行灰度化以及二值化处理，将被测人员从图像背景中进行提取，采用Adaboost算法在互不重叠的训练集上进行训练得到多个较弱的分类器，将训练得到的多个弱分类器进行联合得到强分类器，从而对测试者眼睛和嘴巴进行定位，采用Retinex算法对定位区域图像进行增强并提取特征点，利用所得到的特征点信息量化眼睛和嘴的状态数据。
  4. 如权利要求3所述的基于多维身体状态感知的疲劳检测方法，其特征在于，所述通过分析眼睛特征数据和嘴巴特征数据对面部疲劳状态进行 检测包括：
    依据PERCLOS规则对眼睛特征数据的疲劳状态进行量化；
    依据PMOT准则对嘴巴特征数据的嘴巴闭合度进行量化。
  5. 如权利要求4所述的基于多维身体状态感知的疲劳检测方法，其特征在于，所述依据PERCLOS规则对眼睛特征数据的疲劳状态进行量化包括：
    设定眼睛高度值和眼睛宽度值，并基于眼睛高度值和眼睛宽度值进行眼睛闭合度计算，并基于眼睛闭合度匹配当前眼睛的疲劳程度值。
  6. 如权利要求4所述的基于多维身体状态感知的疲劳检测方法，其特征在于，所述通过语音输入模块采集声音信息，并基于声音信息进行语音疲劳状态进行检测，获取语音疲劳检测结果包括：
    将被测试者声音所形成的语谱图进行Mel频率变换，利用频率变化拉伸感知度敏感区间，增强系统对疲劳的感知度；
    对测试者声音信息进行采集，对所采集的声音信息进行标记并作为源样本；找到特征变换将不同的数据集映射至同一特征空间，使源样本和目标样本边缘概率分布和条件概率分布无限接近；然后反复训练迁移分类器增强泛化能力，对目标样本的疲劳状态进行检测。
  7. 一种基于多维身体状态感知的疲劳检测系统，其特征在于，所述系统包括：
    面部疲劳检测模块，用于通过红外技术采集人脸特征的红外图像数据，对红外图像数据进行预处理，将人脸信息从红外图像数据中提取出来，并从人脸信息中提取眼睛特征数据和嘴巴特征数据，通过分析眼睛特征数据和嘴巴特征数据对面部疲劳状态进行检测，获取面部疲劳检测结果；
    语音疲劳检测模块，用于通过语音输入模块采集声音信息，并基于声音信息进行语音疲劳状态进行检测，获取语音疲劳检测结果；
    融合检测模块，用于将面部疲劳检测结果和语音疲劳检测结果进行融 合，基于信息熵选择进行多分类器融合，将决策表内容进行融合得到最终人体疲劳检测结果。
  8. 如权利要求7所述的基于多维身体状态感知的疲劳检测系统，其特征在于，所述面部疲劳检测模块对红外图像进行灰度化以及二值化处理，将被测人员从图像背景中进行提取，采用Adaboost算法在互不重叠的训练集上进行训练得到多个较弱的分类器，将训练得到的多个弱分类器进行联合得到强分类器，从而对测试者眼睛和嘴巴进行定位，采用Retinex算法对定位区域图像进行增强并提取特征点，利用所得到的特征点信息量化眼睛和嘴的状态数据。
  9. 如权利要求8所述的基于多维身体状态感知的疲劳检测系统，其特征在于，所述面部疲劳检测模块依据PERCLOS规则对眼睛特征数据的疲劳状态进行量化；依据PMOT准则对嘴巴特征数据的嘴巴闭合度进行量化。
  10. 如权利要求9所述的基于多维身体状态感知的疲劳检测系统，其特征在于，所述面部疲劳检测模块设定眼睛高度值和眼睛宽度值，并基于眼睛高度值和眼睛宽度值进行眼睛闭合度计算，并基于眼睛闭合度匹配当前眼睛的疲劳程度值。
PCT/CN2022/115881 2021-09-07 2022-08-30 一种基于多维身体状态感知的疲劳检测方法及系统 WO2023036015A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111043068.4A CN113780150A (zh) 2021-09-07 2021-09-07 一种基于多维身体状态感知的疲劳检测方法及系统
CN202111043068.4 2021-09-07

Publications (1)

Publication Number Publication Date
WO2023036015A1 true WO2023036015A1 (zh) 2023-03-16

Family

ID=78841403

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/115881 WO2023036015A1 (zh) 2021-09-07 2022-08-30 一种基于多维身体状态感知的疲劳检测方法及系统

Country Status (2)

Country Link
CN (1) CN113780150A (zh)
WO (1) WO2023036015A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117079255A (zh) * 2023-10-17 2023-11-17 江西开放大学 一种基于人脸识别和语音交互的疲劳驾驶检测方法

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113780150A (zh) * 2021-09-07 2021-12-10 广西电网有限责任公司贺州供电局 一种基于多维身体状态感知的疲劳检测方法及系统
CN114010170B (zh) * 2021-12-24 2023-05-05 厦门理工学院 一种基于标识显示的智能监测提醒方法及装置
CN115067945A (zh) * 2022-08-22 2022-09-20 深圳市海清视讯科技有限公司 疲劳检测方法、装置、设备及存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6070098A (en) * 1997-01-11 2000-05-30 Circadian Technologies, Inc. Method of and apparatus for evaluation and mitigation of microsleep events
CN107633657A (zh) * 2016-07-18 2018-01-26 中兴通讯股份有限公司 一种疲劳驾驶的提醒方法及终端
CN113327404A (zh) * 2021-05-13 2021-08-31 安徽中科昊音智能科技有限公司 空中交通管制员岗位疲劳状态监测告警系统
CN113780150A (zh) * 2021-09-07 2021-12-10 广西电网有限责任公司贺州供电局 一种基于多维身体状态感知的疲劳检测方法及系统

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
NZ630770A (en) * 2013-10-09 2016-03-31 Resmed Sensor Technologies Ltd Fatigue monitoring and management system
CN104881956A (zh) * 2015-06-17 2015-09-02 上海大学 一种疲劳驾驶预警系统
CN107292251B (zh) * 2017-06-09 2020-08-28 湖北天业云商网络科技有限公司 一种基于人眼状态的驾驶员疲劳检测方法及系统
CN110859609B (zh) * 2019-11-26 2020-12-18 蘑菇车联信息科技有限公司 基于语音分析的多特征融合疲劳驾驶检测方法
CN110992985A (zh) * 2019-12-02 2020-04-10 中国科学院声学研究所东海研究站 识别跑步机异音的识别模型确定方法、识别方法、系统
CN111310683B (zh) * 2020-02-24 2023-02-24 合肥工业大学 一种基于驾驶员疲劳检测的人机协同控制方法及系统

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6070098A (en) * 1997-01-11 2000-05-30 Circadian Technologies, Inc. Method of and apparatus for evaluation and mitigation of microsleep events
CN107633657A (zh) * 2016-07-18 2018-01-26 中兴通讯股份有限公司 一种疲劳驾驶的提醒方法及终端
CN113327404A (zh) * 2021-05-13 2021-08-31 安徽中科昊音智能科技有限公司 空中交通管制员岗位疲劳状态监测告警系统
CN113780150A (zh) * 2021-09-07 2021-12-10 广西电网有限责任公司贺州供电局 一种基于多维身体状态感知的疲劳检测方法及系统

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117079255A (zh) * 2023-10-17 2023-11-17 江西开放大学 一种基于人脸识别和语音交互的疲劳驾驶检测方法
CN117079255B (zh) * 2023-10-17 2024-01-05 江西开放大学 一种基于人脸识别和语音交互的疲劳驾驶检测方法

Also Published As

Publication number Publication date
CN113780150A (zh) 2021-12-10

Similar Documents

Publication Publication Date Title
WO2023036015A1 (zh) 一种基于多维身体状态感知的疲劳检测方法及系统
CN109993068B (zh) 一种基于心率和面部特征的非接触式的人类情感识别方法
CN107358180B (zh) 一种人脸表情的疼痛评估方法
Osareh Automated identification of diabetic retinal exudates and the optic disc
Kumar et al. A novel multimodal fusion framework for early diagnosis and accurate classification of COVID-19 patients using X-ray images and speech signal processing techniques
CN109497990B (zh) 一种基于典型相关分析的心电信号身份识别方法及系统
CN108256579A (zh) 一种基于先验知识的多模态民族认同感量化测量方法
CN106338597A (zh) 呼吸气体测量的方法及系统
Ding et al. Deep connected attention (DCA) ResNet for robust voice pathology detection and classification
Chen et al. Hybrid facial image feature extraction and recognition for non-invasive chronic fatigue syndrome diagnosis
CN106529377A (zh) 一种基于图像的年龄估计方法、装置及系统
CN109003677A (zh) 病历数据结构化分析处理方法
Km et al. Comparison of multidimensional MFCC feature vectors for objective assessment of stuttered disfluencies
CN114565957A (zh) 基于微表情识别的意识评估方法及系统
CN107045624A (zh) 一种基于最大加权团的脑电信号预处理与分类方法
Huang et al. A contrastive embedding-based domain adaptation method for lung sound recognition in children community-acquired pneumonia
CN108920699A (zh) 一种基于N2pc的目标识别反馈系统及方法
CN110598607B (zh) 非接触式与接触式协同的实时情绪智能监测系统
Senyurek et al. A comparison of SVM and CNN-LSTM based approach for detecting smoke inhalations from respiratory signal
WO2019218571A1 (zh) 基于opencv技术的疲劳驾驶预警系统
Yagnavajjula et al. Detection of neurogenic voice disorders using the fisher vector representation of cepstral features
Kim et al. A study on user recognition using 2D ECG image based on ensemble networks for intelligent vehicles
Samsudin et al. Forehead lesion score for facial nerve paralysis evaluation
CN105893994A (zh) 一种人脸识别系统
CN111012306A (zh) 基于双神经网络的睡眠呼吸声检测方法及系统

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE