CN112668393A - Fatigue degree detection device and method based on face recognition and key point detection - Google Patents
Fatigue degree detection device and method based on face recognition and key point detection Download PDFInfo
- Publication number
- CN112668393A CN112668393A CN202011369802.1A CN202011369802A CN112668393A CN 112668393 A CN112668393 A CN 112668393A CN 202011369802 A CN202011369802 A CN 202011369802A CN 112668393 A CN112668393 A CN 112668393A
- Authority
- CN
- China
- Prior art keywords
- face
- eye
- module
- state
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 87
- 238000000034 method Methods 0.000 title claims abstract description 21
- 238000012545 processing Methods 0.000 claims abstract description 29
- 238000005286 illumination Methods 0.000 claims abstract description 10
- 230000006854 communication Effects 0.000 claims abstract description 6
- 238000004891 communication Methods 0.000 claims abstract description 6
- 239000013589 supplement Substances 0.000 claims description 15
- 230000007175 bidirectional communication Effects 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 5
- 230000008859 change Effects 0.000 claims description 5
- 238000012360 testing method Methods 0.000 claims description 5
- 239000000284 extract Substances 0.000 claims description 3
- 230000004399 eye closure Effects 0.000 claims description 3
- 210000001747 pupil Anatomy 0.000 claims description 3
- 238000005070 sampling Methods 0.000 claims description 3
- 230000001502 supplementing effect Effects 0.000 claims description 2
- 238000012423 maintenance Methods 0.000 abstract description 4
- 238000010586 diagram Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 206010070834 Sensitisation Diseases 0.000 description 2
- 238000005034 decoration Methods 0.000 description 2
- 238000009434 installation Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000008313 sensitization Effects 0.000 description 2
- 238000000060 site-specific infrared dichroism spectroscopy Methods 0.000 description 2
- 206010053615 Thermal burn Diseases 0.000 description 1
- FFGPTBGBLSHEPO-UHFFFAOYSA-N carbamazepine Chemical compound C1=CC2=CC=CC=C2N(C(=O)N)C2=CC=CC=C21 FFGPTBGBLSHEPO-UHFFFAOYSA-N 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000035790 physiological processes and functions Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000029058 respiratory gaseous exchange Effects 0.000 description 1
- 239000003381 stabilizer Substances 0.000 description 1
- 230000000087 stabilizing effect Effects 0.000 description 1
Images
Landscapes
- Traffic Control Systems (AREA)
Abstract
The invention discloses a fatigue degree detection device based on face recognition and key point detection, which comprises a face signal processing module, a face state acquisition module, a photosensitive module, a GPS module, a voice alarm module and a power management module, wherein the face signal processing module is respectively in wired connection and communication with the face state acquisition module, the photosensitive module, the GPS module and the voice module, and is powered by the power management module; the fatigue detection method comprises a face registration stage, a fatigue detection stage, a supplementary lighting adjustment stage and an updating stage, the problems that individuals are different in fatigue driving detection, results are affected by illumination conditions and the like are solved by intelligently adjusting supplementary lighting brightness, and the method based on mobile phone APP updating and operation is provided, so that the cost of equipment maintenance is reduced.
Description
Technical Field
The invention belongs to the field of computer vision, and particularly relates to a fatigue degree detection device and a fatigue degree detection method based on face recognition and key point detection.
Background
The fatigue detection methods in the market are many, and typically include the following three methods:
firstly, a physiological sensor is utilized to detect the physiological state of a driver, such as electroencephalogram, heart rate, respiration and the like. This method is the most accurate, but costly, and most drivers will be reluctant to wear physiological sensors.
Secondly, the vehicle-mounted sensor is used for detecting the state of the vehicle, such as an accelerator, a brake, steering of a steering wheel and the like, so as to further estimate the state of the driver. The method is low in cost, but lacks real-time performance and accuracy, and the driver can give an alarm when an accident is about to happen, so that the driver cannot respond timely.
And thirdly, detecting the face state of the driver, such as eyes, mouth and the like, by using computer vision, wherein the fatigue state of the driver is detected by using a PERCLOS algorithm, and the method is also used in most of fatigue detection patents.
A. Patent No. CN105354985 introduces an algorithm for detecting fatigue degree by computer vision, specifically, using Haar-Like feature to calculate pupil size, and combining with PERCLOS algorithm to detect fatigue state of driver.
B. Patent No. CN105096528 introduces a universal fatigue detection method, which comprehensively detects fatigue from factors such as the limb and expression state of the driver, the driving time, the vehicle state, and the road condition information.
The above patent mainly has the following problems in solving the technical problems:
1. patent one, although detecting by the PERCLOS algorithm, needs several important indexes used therein, and there are differences among individuals, such as the size of the eye when the eye is fully opened, the time of the eye opening to 80%, and the like. If only the PERCLOS algorithm is used for detection, the conditions of false detection and inaccurate detection are easy to generate.
2. The second patent adopts the change of the state of the driver along with the time to judge the fatigue degree, and updates the reference face once in 4 hours, the method is not influenced by the individual difference of the driver, but the condition that the driver is in the fatigue state just begins to drive cannot be ensured, the equipment installation is troublesome, and various sensors are required to be installed.
3. In the driving process, the illumination condition is complex, the picture is easy to appear too dark or too bright, the performance of the face key point algorithm is poor, and the result is inaccurate.
4. The setting and updating of the programs of the vehicle-mounted equipment mostly require professional development or operation of maintenance personnel, and the operation cost is high.
Disclosure of Invention
The invention provides a fatigue degree detection device and a detection method thereof based on face recognition and key point detection, solves the problems of individual difference, influence of results on illumination conditions and the like in fatigue driving detection by intelligently adjusting light supplement brightness, and provides a method based on mobile phone APP updating and operation, so that the cost of equipment maintenance is reduced.
The technical scheme of the invention is as follows: fatigue detection device based on face identification, key point detect, including face signal processing module, face state collection module, sensitization module, GPS module, audio alert module and power management module, face signal processing module communicates with face state collection module, sensitization module, GPS module, voice module wired connection respectively, is supplied power by power management module simultaneously.
Further, the intelligent face signal processing system further comprises an APP module, and the APP module is in wireless connection communication with the face signal processing module through BLE and WIFI protocols.
Furthermore, the human face signal processing module adopts an ESP32 series chip as a processor, and the ESP32 series chip is connected with the human face state acquisition module, the voice module and the GSP module through UART and connected with the photosensitive module through I2C;
the method comprises the following specific steps: setting IO19 as a TXD1 interface, setting IO18 as an RXD1 interface, connecting the interfaces with a human face state acquisition module, and performing bidirectional communication; setting an RXD2 interface of the IO32 to be connected with a GPS module, and unidirectionally receiving GPS data; setting an IO33 as a TXD2 interface to be connected with the voice module, and transmitting an instruction in a unidirectional mode; the IO20 is a CLK interface, and the IO21 is an SD0 interface, and is connected to the photosensitive module for bidirectional communication.
Further, the ESP32 supports two wireless connection modes: BLE and WIFI; and supports OTA functionality.
The invention also provides a fatigue detection method based on face recognition and key point detection, which comprises a face registration stage and a fatigue detection stage;
the face registration stage: firstly extracting the face characteristics of a driver, then comparing the face characteristics with a face library, if the face characteristics are not found, keeping a normal driving posture by voice prompt, starting key point detection, and recording the face information of the driver in a normal state, wherein the face information comprises face characteristic values and the size E when eyes are completely opened0Finally, writing the face information into a face library; if the face features are found, reading face information in a face library;
the fatigue degree detection stage is as follows: firstly obtaining key points in the face characteristics, and calculating the current pupil size E1If E is1/E0Is less than the first threshold value, the eye-closing state S0Adding a duration; otherwise if E1/E0Is greater than or equal to a first threshold value, eye-open state S1Adding the duration, calculating S within one minute0/(S0+S1) If the value of (b) is greater than the second threshold value and a continuous eye-closing state exists, triggering a fatigue alarm. The first threshold value is 0.2, and the second threshold value is 0.4.
Further, the face registration stage specifically comprises the following steps:
IOU＝(C∩D)/(C∪D)
c is a current frame rectangular frame, and D is a previous frame rectangular frame;
and when the intersection ratio is larger than a sixth threshold value, entering a fatigue detection stage, otherwise, entering a step 1001 to detect the face again and setting the value of the mark A as False.
Further, the fatigue detection stage comprises the following specific steps:
Further, still include the light filling and adjust the stage, the light filling is adjusted the stage: when the photoresistor detects the light intensity each time, the photoresistor can be compared with the reference light intensity, if the light intensity is too low, the photoresistor is marked as a weak light state, otherwise, the photoresistor is marked as a strong light state; and comparing the current light intensity state with the illumination state recorded by the system, and if the difference is detected within the continuous seventh threshold time, recording the current light intensity state by the system, and adjusting the camera and the light supplementing lamp according to the light intensity state. The seventh threshold time is 5 seconds.
Further, the light supplement adjusting stage specifically includes the following steps:
Further, the method also comprises an updating stage, wherein the updating stage: when a user sends an update instruction at an APP terminal, the device receives the instruction through a BLE protocol and switches to WIFI connection, if the version number of the APP terminal is greater than the version number of the device terminal (for example, if the version number of the APP terminal is 1.1.0 and the version number of the device terminal is 1.0.0, the version number of the APP terminal is greater than the version number of the device terminal and needs to be updated), connection is established, and a check update packet is received; otherwise, directly quitting the updating; finally switching back to BLE connection.
The invention has the advantages that: 1. the fatigue detection is not influenced by individual differences of drivers in a face recognition mode; 2. when the strong light state is detected, the camera in the face state acquisition module is switched to an RGB mode, and the light supplement lamp is turned off; when a low light state is detected, the camera is switched to an IR mode, and the intensity of the light supplement lamp is adjusted to be 20; the intelligent control light supplement lamp is suitable for illuminating complicated road surfaces; 3. a communication and updating mode of a mobile terminal and an embedded terminal based on BLE and WIFI is provided, and operation cost and equipment power consumption are reduced.
Drawings
FIG. 1 is a block diagram of a fatigue detection system according to an embodiment of the present invention;
FIG. 2(a) is a schematic diagram of a power module provided by an embodiment of the present invention;
FIG. 2(b) is a schematic diagram of a face signal processing module according to an embodiment of the present invention;
FIG. 2(c) is a schematic diagram of a face state acquisition module provided by an embodiment of the present invention;
FIG. 2(d) is a schematic diagram of a photosensitive module according to an embodiment of the present invention;
FIG. 2(e) is a schematic diagram of a voice alarm module provided by an embodiment of the present invention;
FIG. 2(f) is a schematic diagram of a GPS module provided by an embodiment of the present invention;
FIG. 3 is a flow chart of a face registration phase provided by an embodiment of the present invention;
FIG. 4 is a flow chart of a fatigue detection phase provided by an embodiment of the present invention;
fig. 5 is a flowchart of a light supplement adjusting stage according to an embodiment of the present invention;
fig. 6 is a flow chart of an update phase provided by an example of the present invention.
Detailed Description
The following provides a preferred embodiment of the present invention with reference to fig. 1 to 6 to describe the technical solution of the software and hardware combination of the present invention in detail.
As shown in fig. 1, the fatigue detection includes the following six hardware modules:
1 human face signal processing module
2 human face state acquisition module
3 photosensitive module
4 GPS module
5 Voice module
6 power management module
The human face signal processing module is in wired connection and communication with the human face state acquisition module, the photosensitive module, the GPS module and the voice module respectively, and is powered by the power management module. APP is communicated with the human face signal processing module through BLE and WIFI protocols in a wireless connection mode.
As shown in fig. 2, the specific hardware model and connection method used by each module:
the human face signal processing module 1 uses a chip of the Lexin company ESP32 series as a processor. Since ESP32 can set specific interface functions of the pins, such as UART, I2C, SPI, etc. In the embodiment, the human face state acquisition module, the voice module and the GSP module are connected through UART (universal asynchronous receiver/transmitter), and the photosensitive module is connected through I2C; specifically, an IO19 is set as a TXD1 interface, an IO18 is set as an RXD1 interface, and the interfaces are connected with a face state acquisition module for bidirectional communication; setting an RXD2 interface of the IO32 to be connected with a GPS module, and unidirectionally receiving GPS data; setting an IO33 as a TXD2 interface to be connected with the voice module, and transmitting an instruction in a unidirectional mode; the IO20 is a CLK interface, and the IO21 is an SD0 interface, and is connected to the photosensitive module for bidirectional communication.
Meanwhile, ESP32 supports two wireless connection modes: BLE and WIFI. WIFI transmission speed is fast, but the consumption is big, if not add radiating element, connects for a long time, causes the equipment easily to scald. BLE is not, but the transmission speed is slower; in this case, the device is connected with the mobile phone for a long time through BLE, and functions of reading and modifying configuration are realized. If the user needs to update the equipment, the WIFI mode can be automatically switched to be connected, and the BLE connection is switched back after updating is completed.
Finally, ESP32 supports OTA functionality. In this case, the steps for implementing the device update are as follows: 1. the mobile phone receives the latest update package of the server and sends an update instruction to ESP 32; 2, ESP32 is connected with mobile phone WIFI to establish TCP/IP connection; 3. reading version number V of mobile phone terminal0If V is0Greater than the version number V of the equipment terminal1If yes, sending an updating request, otherwise, quitting updating; 4. and acquiring the size Sn of the new packet, receiving the update packet, and ending the TCP/IP connection after the transmission is finished. 5. Comparing whether the size of the received update package is equal to Sn, and if so, rewriting the version number of the equipment end to V0And loading the new program. Otherwise, exiting the updating and deleting the installation package.
The human face state acquisition module 2 adopts a Tencent priority image VisionSeed module. The module supports the functions of face detection, face key point detection, face feature extraction, face comparison, face tracking and the like, and supports a binocular camera mode, wherein an RGB camera is used when the illumination is normal, and an IR camera is used when the illumination is too dark;
and the photosensitive module 3 detects the illumination intensity and feeds the illumination intensity back to the face signal processing module by adopting BH1750FVI, and controls the camera mode of the face state acquisition module.
The GPS module 4 adopts ATGM332D-5N-31, when the speed per hour of the vehicle is detected to be less than S km/h, the detection is closed, and the value of S can be defined by the user in APP and is 20 by default.
The voice alarm module 5 adopts a WT2003M02 chip and a programmable MP3 module, and is controlled by an ESP32 to complete the functions of registration reminding and voice alarm.
The power supply module 6 adopts an AMS1117 voltage stabilizer to provide 5V voltage for the face state acquisition module and the GPS module and provide 3.3V voltage for the face signal processing module and the voice alarm module; meanwhile, one BAT760-7 diode is used as a voltage stabilizing circuit, and three diodes LESD5D5.0CT1G are used as protection circuits.
As shown in fig. 3, the implementation of the face registration phase procedure 100 includes the steps of:
judging whether a face is detected for the first time (step 1001): after the device is powered on, the initialization flag a is False, when the face state acquisition module 2 detects a face, the detected maximum face is taken, if the face angle is within a normal range (| roll | <60, | pitch | <18, | yaw | <18) and the value of a is False and the definition of the face region is greater than 3.5 (here, the definition of the picture calculated by using the sobel operator is a method disclosed in the field and is not described any more), feature matching is performed on the face library (step 1002), and the value of a is True. Otherwise, the human face is detected again. Wherein: roll is the left-right inclination angle of the face, pitch is the up-down elevation angle of the face, and yaw is the left-right head-twisting angle of the face.
Feature matching face library (step 1002): the face state acquisition module 2 extracts face features according to the face picture detected in step 1001, and then performs traversal comparison on the detected face features and features in the face library. And taking the score with the maximum comparison result, and if the score is more than 80, calling the eye size in the face library (step 1003). Otherwise, the vehicle speed is determined (step 1004).
Recall eye size in face library (step 1003): and reading the eye size field corresponding to the ID in the face library according to the result ID of the matching face library (step 1002). And proceeds to face tracking (step 1007).
Vehicle speed determination (step 1004): the speed parameters of the GPS module 4 are read, and if the speed is less than 20km/h, information is collected (step 1005), otherwise the eye size reserved by the system is read (step 1006).
Information collection (step 1005): the voice alarm module 5 prompts the user that the user is sampling and keeping a normal driving posture. The face state acquisition module 2 acquires 100 frames of relative sizes of the eyes of the user.
The formula for calculating the eye size L is:
l1 — the distance from point 2 to point 6; point 2 is the eye's lowest endpoint and point 6 is the eye's highest endpoint;
l2 — the distance from point 0 to point 4; point 0 is the leftmost eye point and point 4 is the rightmost eye point;
L＝(L1)/(L2+0.01)
and the eye size value with the largest number of times is taken out and stored in a face library together with the face features. And proceeds to face tracking (step 1007).
Reading the eye size reserved by the system (step 1006): and reading the default eye size parameters in the face signal processing module 1. Face tracking is entered (step 1007).
Face tracking (step 1007): judging the intersection ratio of the current frame face rectangular frame and the previous frame face rectangular frame, wherein the calculation formula is as follows:
IOU＝(C∩D)/(C∪D)
c is the current frame rectangular frame, D is the last frame rectangular frame.
And when the intersection ratio is larger than 0.8, entering a fatigue detection stage (figure 4), otherwise, detecting the face again (step 1001) and setting the value of the mark A as False.
As shown in fig. 4, a fatigue detection phase procedure 200 is implemented, comprising the steps of:
fatigue detection initialization (step 2001): according to the face registration stage, acquiring the normal eye size E0 of the user; recording the current time stamps T0, T1; the continuous eye-closing time TE is 0; the continuous closed-eye state V0 is False; the eye closing time TC is 0; the eye-open time TP is 0; proceed to check eye size (step 2002).
Eye size detection (step 2002): the face state acquisition module 2 first performs key point detection, calculates the eye size E1 of the current frame, records the current timestamp as T2, and finally determines whether to close the eye (step 2003).
Judgment of eye closure (step 2003): the current frame eye size E1 is divided by the user's normal eye size E0, and if the ratio is less than 0.2, the eye-closed state is updated (step 2004), otherwise the eye-open state is updated (step 2005).
Update closed-eye state (step 2004): the values of the closed-eye time TC, the continuous closed-eye time TE, plus the difference between the timestamps T2 and T1. Proceed to determine whether to close the eye continuously (step 2006).
Update the eye-open state (step 2005): the value of the eye-open state TP is added to the difference between the time stamps T2 and T1, the value of the continuous eye-closing time TE is reset to 0, and it is then determined whether or not the detection has continued for one minute (step 2007).
Judging whether to close the eyes continuously (step 2006): judging the value of the continuous eye closing time TE, and setting the value of the continuous eye closing state V0 to True when TE is more than 3 seconds; otherwise, it is not processed. Finally, it is determined whether the test has continued for one minute (step 2007).
Determine if the test has continued for one minute (step 2007): the value of T2 is assigned to T1, and then whether the value of T2 minus T0 is greater than 60 seconds is judged, if so, the fatigue is judged (step 2008), otherwise, the eye size is continuously detected (step 2002).
Fatigue level determination (step 2008): the value of PERCLOS is calculated. The fatigue value PERCLOS is calculated by the formula: PERCLOS is TC/(TC + TP). If the value of PERCLOS is greater than 0.4 and the value of the continuous eye-closing state V0 is True, it indicates that the user is in a tired state, a voice alarm is made, otherwise fatigue detection is reinitialized (step 2001).
As shown in fig. 5, the implementation of the intelligent light supplement phase procedure 300 includes the steps of:
initialization (step 3001): the reserved light intensity state of the reading system is L0The continuous change flag I is 0.
Detecting the current light intensity state (step 3002): the photo sensing module 3 detects the current light intensity and determines the current light intensity status (step 3003).
Light intensity state judgment (step 3003): and judging whether the detected light intensity is greater than 50lx, if so, judging that the light intensity is in a strong light state in the L1 state, and if not, judging that the light intensity is in a weak light state in the L1 state. Proceed to determine if the ambient light has changed (step 3004).
Determine whether the ambient light changes (step 3004): judgment of L0And L1If not, continuously changing the value of the identifier I plus one; otherwise, the value of I is reset to 0. It is judged whether five detections are continued (step 3005).
Determine whether five consecutive tests are performed (step 3005): judging whether the value of the continuous change identifier I is larger than 5, if so, indicating that the ambient light changes, and switching the camera mode (step 3006); otherwise, the current light intensity is continuously detected (step 3002).
Switching a camera mode (step 3006), when a strong light state is detected, switching the camera in the human face state acquisition module 2 to an RGB mode, and turning off a light supplement lamp; when the low light state is detected, the camera is switched to the IR mode, and the intensity of the light supplement lamp is adjusted to be 20. And records the current light intensity status and enters initialization (step 3001).
As shown in FIG. 6, an update phase procedure 400 is implemented, comprising the steps of:
initialization (step 4001): the face signal processing module 1 closes BLE connection, starts WIFI, and sets AP mode, SSID and password. And the APP terminal scans WIFI according to the SSID, inputs password connection and establishes TCP/IP. A check version number is entered (step 4002).
Check version number (step 4002): the face signal processing module 1 receives the first 16 bytes and converts them into version numbers and update packet sizes (note: the format of the version number here is 1.0.0 and includes 3 int types, the update packet size is 1 int type, and 4 int types in total, so 16 bytes are received). Comparing the version sizes (for example, if the version number of the APP side is 1.1.0 and the version number of the device side is 1.0.0, it indicates that the version number of the APP side is greater than the version number of the device side and needs to be updated), and if the version number needs to be updated, continuing to download the update package (step 4003). Otherwise, the BLE connection is switched (step 4006).
Download update package (step 4003): the face signal processing module 1 receives data sent by the APP terminal through a socket.
Check update package (step 4004): after the face signal processing module 1 receives the data, the size of the received update package is read, the update package is compared with the update package obtained in the check version number (step 4002), if the update package is equal, the equipment is updated (step 4005), otherwise, the update package is downloaded again (step 4003).
Update device (step 4005): the face signal processing module 1 installs the update package, stores the new version number in the configuration file, deletes the update package, and switches the BLE connection (step 4003).
Switching BLE connection (step 4006): the face signal processing module 1 closes the WIFI connection and starts BLE broadcasting. The APP terminal scans to BLE broadcast and connects.
According to the fatigue detection device and the detection method based on face recognition and key point detection, the problems that individuals have differences in fatigue driving detection, results are affected by illumination conditions and the like are solved through intelligent adjustment of light supplement brightness, and a method based on mobile phone APP updating and operation is provided, so that the equipment maintenance cost is reduced.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (10)
1. Fatigue degree detection device based on face identification, key point detect, its characterized in that: the human face signal processing module is in wired connection communication with the human face state acquisition module, the photosensitive module, the GPS module and the voice module respectively, and is powered by the power management module.
2. The fatigue degree detection device based on face recognition and key point detection according to claim 1, characterized in that: the intelligent face signal processing system is characterized by further comprising an APP module, wherein the APP module is in wireless connection communication with the face signal processing module through BLE and WIFI protocols.
3. The fatigue degree detection device based on face recognition and key point detection according to claim 1, characterized in that: the human face signal processing module adopts an ESP32 series chip as a processor, and the ESP32 series chip is connected with the human face state acquisition module, the voice module and the GSP module through UART and connected with the photosensitive module through I2C;
the method comprises the following specific steps: setting IO19 as a TXD1 interface, setting IO18 as an RXD1 interface, connecting the interfaces with a human face state acquisition module, and performing bidirectional communication; setting an RXD2 interface of the IO32 to be connected with a GPS module, and unidirectionally receiving GPS data; setting an IO33 as a TXD2 interface to be connected with the voice module, and transmitting an instruction in a unidirectional mode; the IO20 is a CLK interface, and the IO21 is an SD0 interface, and is connected to the photosensitive module for bidirectional communication.
4. The fatigue degree detection apparatus based on face recognition and key point detection according to claim 3, wherein: the ESP32 supports two wireless connection modes: BLE and WIFI; and supports OTA functionality.
5. The fatigue degree detection method based on face recognition and key point detection is characterized by comprising the following steps: the method comprises a face registration stage and a fatigue detection stage;
the face registration stage: firstly extracting the face characteristics of a driver, then comparing the face characteristics with a face library, if the face characteristics are not found, keeping a normal driving posture by voice prompt, starting key point detection, and recording the face information of the driver in a normal state, wherein the face information comprises face characteristic values and the size E when eyes are completely opened0Finally, writing the face information into a face library; if the face features are found, reading face information in a face library;
the fatigue degree detection stage is as follows: firstly obtaining key points in the face characteristics, and calculating the current pupil size E1If E is1/E0Is less than the first threshold value, the eye-closing state S0Adding a duration; otherwise if E1/E0Is greater than or equal to a first threshold value, eye-open state S1Adding the duration, calculating S within one minute0/(S0+S1) If the value of (b) is greater than the second threshold value and a continuous eye-closing state exists, triggering a fatigue alarm.
6. The fatigue degree detection method based on face recognition and key point detection as claimed in claim 5, wherein: the face registration stage specifically comprises the following steps:
step 1001, judging whether a face is detected for the first time: after the equipment is powered on, initializing a mark A to be False, when a face state acquisition module detects a face, taking the detected maximum face, and if the face angle is within a normal range, the value of A is False and the definition of a face region is greater than a third threshold value, entering a feature matching face library in step 1002 and setting the value of A to be True; otherwise, detecting the face again;
step 1002, feature matching face library: the human face state acquisition module extracts human face features according to the human face picture detected in the step 1001 and then performs traversal comparison on the detected human face features and features in a human face library; taking the score with the maximum comparison result, and if the score is larger than a fourth threshold value, entering a step 1003 to call the size of eyes in the face library; otherwise, go to step 1004 to judge the vehicle speed;
step 1003, calling the eye size in the face library: reading an eye size field corresponding to the ID in the face library according to the result ID matched with the face library in the step 1002, and entering the step 1007 of face tracking;
step 1004, judging the vehicle speed: reading the speed parameter of the GPS module, if the speed is less than a fifth threshold value, entering a step 1005 for collecting information, otherwise entering a step 1006 for reading the size of the eye reserved by the system; in the GPS module, when the speed per hour of the vehicle is detected to be less than a fifth threshold value, the detection is closed;
step 1005, collecting information: the voice alarm module prompts a user that the user is sampling and keeping a normal driving posture; the face state acquisition module acquires the relative sizes of eyes of a plurality of frames of users; taking out the eye size value with the largest number of times, and storing the eye size value and the face features into a face library; and step 1007 of face tracking is carried out;
step 1006, reading the eye size reserved by the system: reading a default eye size parameter in the face signal processing module, and entering into step 1007 of face tracking;
step 1007, face tracking: judging the intersection ratio of the current frame face rectangular frame and the previous frame face rectangular frame, wherein the calculation formula is as follows:
IOU＝(C∩D)/(C∪D)
c is a current frame rectangular frame, and D is a previous frame rectangular frame;
and when the intersection ratio is larger than a sixth threshold value, entering a fatigue detection stage, otherwise, entering a step 1001 to detect the face again and setting the value of the mark A as False.
7. The fatigue degree detection method based on face recognition and key point detection as claimed in claim 5, wherein: the fatigue detection stage comprises the following specific steps:
step 2001, fatigue detection initialization: according to the face registration stage, acquiring the normal eye size E0 of the user; recording the current time stamps T0, T1; the continuous eye-closing time TE is 0; the continuous closed-eye state V0 is False; the eye closing time TC is 0; the eye-open time TP is 0; entering step 2002 to detect eye size;
step 2002, detecting the size of the eye: the face state acquisition module firstly detects key points, calculates the eye size E1 of the current frame, then records the current timestamp as T2, and finally enters step 2003 to judge whether the eyes are closed;
step 2003, judging whether the eyes are closed: dividing the current frame eye size E1 by the user normal eye size E0, if the ratio is smaller than a first threshold, entering step 2004 to update the eye-closing state, otherwise entering step 2005 to update the eye-opening state;
step 2004, updating the eye-closing state: the values of the closed-eye time TC, the continuous closed-eye time TE plus the difference between the timestamps T2 and T1; step 2006, judging whether continuous eye closure is performed;
step 2005, updating the eye-open state: the value of the eye-open state TP is added to the difference between the time stamps T2 and T1, the value of the continuous eye-closing time TE is reset to 0, and then step 2007 is performed to determine whether the detection has continued for one minute;
step 2006, judging whether to close the eyes continuously: judging the value of the continuous eye closing time TE, and setting the value of the continuous eye closing state V0 to True when TE is larger than a sixth threshold value; otherwise, not processing; finally, step 2007 is entered to judge whether the detection lasts for one minute;
step 2007, judging whether the detection lasts for one minute: assigning the value of T2 to T1, judging whether the value of T2 minus T0 is greater than 60 seconds, if so, entering a step 2008 to judge the fatigue degree, otherwise, entering a step 2002 to continuously detect the size of eyes;
step 2008, judging fatigue degree: firstly, calculating the PERCLOS value, wherein the calculation formula of the fatigue value PERCLOS is as follows: PERCLOS ═ TC/(TC + TP); if the value of PERCLOS is greater than the second threshold value and the value of the continuous eye-closing state V0 is True, the user is in a fatigue state, a voice alarm is carried out, otherwise, the step 2001 is carried out to reinitialize fatigue degree detection.
8. The fatigue degree detection method based on face recognition and key point detection as claimed in claim 5, wherein: still include the light filling regulation stage, the light filling regulation stage: when the photoresistor detects the light intensity each time, the photoresistor can be compared with the reference light intensity, if the light intensity is too low, the photoresistor is marked as a weak light state, otherwise, the photoresistor is marked as a strong light state; and comparing the current light intensity state with the illumination state recorded by the system, and if the difference is detected within the continuous seventh threshold time, recording the current light intensity state by the system, and adjusting the camera and the light supplementing lamp according to the light intensity state.
9. The fatigue degree detection method based on face recognition and key point detection according to claim 8, characterized in that: the light supplement adjusting stage specifically comprises the following steps:
step 3001, initialization: the reserved light intensity state of the reading system is L0Continuously changing the identifier I to 0;
step 3002, detecting the current light intensity state: the photosensitive module detects the current light intensity, and step 3003 is performed to determine the current light intensity state;
step 3003, judging the light intensity state: judging whether the detected light intensity is greater than the eighth threshold value, if so, judging whether the detected light intensity is greater than the current light intensity state L1Is in a bright light state, otherwise L1Is in a low light state; step 3004 is entered to determine if the ambient light has changed;
step 3004, determine whether the ambient light changes: judgment of L0And L1If not, continuously changing the value of the identifier I plus one; otherwise, resetting the value of I to 0; step 3005 is entered to determine whether to perform multiple consecutive detections;
step 3005, determine whether to perform multiple consecutive tests: judging whether the value of the continuous change identifier I is larger than a ninth threshold value, if so, indicating that the ambient light changes, and entering a step 3006 to switch the camera mode; otherwise, go on to step 3002 to detect the current light intensity;
step 3006, switching camera mode, wherein when a strong light state is detected, the camera in the face state acquisition module is switched to RGB mode, and the light supplement lamp is turned off; when a low-light state is detected, switching the camera to an IR mode, and adjusting the intensity of a light supplement lamp to be 15-35; and records the current light intensity status, and enters step 3001 for initialization.
10. The fatigue degree detection method based on face recognition and key point detection as claimed in claim 3, wherein: further comprising an update phase, the update phase: when a user sends an update instruction at an APP terminal, the equipment receives the instruction through a BLE protocol and switches to WIFI connection, if the version number of the APP terminal is larger than that of the equipment terminal, connection is established, and a check update packet is received; otherwise, directly quitting the updating; finally switching back to BLE connection.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011369802.1A CN112668393A (en) | 2020-11-30 | 2020-11-30 | Fatigue degree detection device and method based on face recognition and key point detection |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011369802.1A CN112668393A (en) | 2020-11-30 | 2020-11-30 | Fatigue degree detection device and method based on face recognition and key point detection |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112668393A true CN112668393A (en) | 2021-04-16 |
Family
ID=75403027
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011369802.1A Pending CN112668393A (en) | 2020-11-30 | 2020-11-30 | Fatigue degree detection device and method based on face recognition and key point detection |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112668393A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113392792A (en) * | 2021-06-27 | 2021-09-14 | 赣州德业电子科技有限公司 | Face AI image recognition system for tower crane operation |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN203733262U (en) * | 2013-12-04 | 2014-07-23 | 安徽三联交通应用技术股份有限公司 | Real-time detecting and early-warning equipment of driver attention |
| CN104240446A (en) * | 2014-09-26 | 2014-12-24 | 长春工业大学 | Fatigue driving warning system on basis of human face recognition |
| CN104269028A (en) * | 2014-10-23 | 2015-01-07 | 深圳大学 | Fatigue driving detection method and system |
| CN105069976A (en) * | 2015-07-28 | 2015-11-18 | 南京工程学院 | Integrated fatigue detection and driving record system and fatigue detection method |
| CN105096528A (en) * | 2015-08-05 | 2015-11-25 | 广州云从信息科技有限公司 | Fatigue driving detection method and system |
| CN106530623A (en) * | 2016-12-30 | 2017-03-22 | 南京理工大学 | Fatigue driving detection device and method |
| CN107563346A (en) * | 2017-09-20 | 2018-01-09 | 南京栎树交通互联科技有限公司 | One kind realizes that driver fatigue sentences method for distinguishing based on eye image processing |
| CN108573210A (en) * | 2018-03-02 | 2018-09-25 | 成都高原汽车工业有限公司 | A kind of alarming method for fatigue drive and device |
| US10147319B1 (en) * | 2017-12-22 | 2018-12-04 | Pin-Hua CHEN | Safe driving system having function of detecting heart rate variability |
-
2020
- 2020-11-30 CN CN202011369802.1A patent/CN112668393A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN203733262U (en) * | 2013-12-04 | 2014-07-23 | 安徽三联交通应用技术股份有限公司 | Real-time detecting and early-warning equipment of driver attention |
| CN104240446A (en) * | 2014-09-26 | 2014-12-24 | 长春工业大学 | Fatigue driving warning system on basis of human face recognition |
| CN104269028A (en) * | 2014-10-23 | 2015-01-07 | 深圳大学 | Fatigue driving detection method and system |
| CN105069976A (en) * | 2015-07-28 | 2015-11-18 | 南京工程学院 | Integrated fatigue detection and driving record system and fatigue detection method |
| CN105096528A (en) * | 2015-08-05 | 2015-11-25 | 广州云从信息科技有限公司 | Fatigue driving detection method and system |
| CN106530623A (en) * | 2016-12-30 | 2017-03-22 | 南京理工大学 | Fatigue driving detection device and method |
| CN107563346A (en) * | 2017-09-20 | 2018-01-09 | 南京栎树交通互联科技有限公司 | One kind realizes that driver fatigue sentences method for distinguishing based on eye image processing |
| US10147319B1 (en) * | 2017-12-22 | 2018-12-04 | Pin-Hua CHEN | Safe driving system having function of detecting heart rate variability |
| CN108573210A (en) * | 2018-03-02 | 2018-09-25 | 成都高原汽车工业有限公司 | A kind of alarming method for fatigue drive and device |
Non-Patent Citations (2)
| Title |
|---|
| 乐鑫信息科技: "ESP32：OTA升级介绍", pages 1 - 121, Retrieved from the Internet <URL:https://www.bilibili.com/video/BV155411Y7VJ/?spm_id_from=333.337.search-card.all.click&vd_source=ec0408a1b50ad127fcc86237216e5261> * |
| 黄家才，旷文腾，毛宽诚: "基于人脸关键点的疲劳驾驶检测研究", 南京工程学院学报（自然科学版）, vol. 15, no. 4, 31 December 2017 (2017-12-31), pages 8 - 13 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113392792A (en) * | 2021-06-27 | 2021-09-14 | 赣州德业电子科技有限公司 | Face AI image recognition system for tower crane operation |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101090482B (en) | Driver fatigue monitoring system and method based on image process and information mixing technology | |
| CN106293032B (en) | Portable terminal device, and control method and apparatus thereof | |
| US20070222617A1 (en) | Vision based alert system using portable device with camera | |
| CN106774889B (en) | Gesture recognition method and device of wearable device | |
| CN106953977B (en) | A kind of monitoring method and system based on mobile terminal | |
| CN109617967A (en) | One kind being based on big data Intelligent internet of things control platform | |
| CN107065224A (en) | Kopiopia recognition methods and its intelligent glasses based on big data | |
| CN102509466A (en) | Traffic signal light auxiliary recognition system based on mobile telephone and method | |
| CN107273071A (en) | Electronic installation, screen adjustment system and method | |
| CN111179880A (en) | Brightness adjusting method and device of display screen, electronic equipment and system | |
| CN112668393A (en) | Fatigue degree detection device and method based on face recognition and key point detection | |
| CN109104689B (en) | Safety warning method and terminal | |
| JP4989249B2 (en) | Eye detection device, dozing detection device, and method of eye detection device | |
| CN205405809U (en) | Driver fatigue detection alarm system based on intelligence wrist -watch | |
| CN110101179A (en) | A kind of intelligent walking stick that multi information remotely monitors and its application method | |
| CN116110351B (en) | Backlight control method, device, chip, electronic equipment and medium | |
| CN209785217U (en) | Fatigue driving prevention system based on convolutional neural network and matrix photography | |
| CN110703910B (en) | Gesture recognition method and system based on smart watch | |
| CN106774841A (en) | Intelligent glasses and its awakening method, Rouser | |
| CN113712505B (en) | Child vision health prevention and control glasses and vision training method | |
| CN113380050B (en) | Driving safety prompting method and system based on intelligent glasses and intelligent glasses | |
| CN106550086B (en) | Terminal and push information prompting method | |
| CN115134975A (en) | Headlamp comprising improved dynamic illumination | |
| CN210328089U (en) | Illumination control device | |
| KR101349702B1 (en) | Smart tv having healthcare monitoring feature and healthcare monitoring method using the same |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |