WO2019218571A1 - Fatigued driving early warning system based on opencv technology - Google Patents

Abstract

A fatigued driving early warning system based on opencv technology, comprising an infrared sensor and a camera, wherein the infrared sensor and the camera are both connected to a processor through a communication module; the infrared sensor keeps track of a certain point of a head of a driver and acquires a physiological signal of the driver for determination, so as to perform early warning and reminding; the camera is used for performing real-time tracking of the face of the driver and transmitting the collected image as input data to a processor by using a sub-frame mode, and the processor judges the collected image and determines whether the driver is in a fatigued state at the time, so as to perform early warning and reminding; and alarm warning is performed when the infrared sensor detects that a specified lower-head distance threshold is exceeded and the camera detects that an upper-and-lower eyelid distance threshold is exceeded. The accuracy of reminding about the driver's fatigued driving is greatly improved.

Description

Fatigue driving warning system based on opencv technology Technical field

The invention relates to the field of driving safety, in particular to a fatigue driving early warning system based on opencv technology.

Background technique

Today's research presents a number of techniques for image recognition, that is, computer processing, analysis, and understanding of images to identify targets and objects in various modes. In general industrial use, industrial cameras are used to take pictures, and then the software is used to further identify and process according to the grayscale difference of the pictures. The mathematical nature of image recognition belongs to the mapping problem from pattern space to category space. At present, in the development of image recognition, there are mainly three methods of identification: statistical pattern recognition, structural pattern recognition, and fuzzy pattern recognition. Image segmentation is a key technology in image processing, including threshold segmentation method, edge detection method, region extraction method, and segmentation method combined with specific theory tools.

In the field of in-vehicle driver fatigue monitoring technology, it is essentially a technique of capturing and analyzing a driver's biological behavior information during driving, such as eyes, face, heart, brain electrical activity, and the like. However, heartbeat activity and EEG monitoring are not currently applied in batches due to contact restrictions. The most frequently used fatigue detection method is the driver's driving behavior analysis, that is, by recording and analyzing the behavior characteristics of the driver turning the steering wheel and braking, to determine whether the driver is fatigued. However, this method is greatly affected by the driver's driving habits. Another major category of detection methods is the fatigue assessment of the driver's face and eye features by means of image analysis. This method is gradually being accepted and adopted by OEMs.

However, the existing image detection method has a large false alarm rate and cannot be used well.

Summary of the invention

The technical problem to be solved by the present invention is to provide a fatigue driving early warning system based on opencv technology, which greatly improves the accuracy of the driver's fatigue driving reminder.

In order to solve the above technical problem, the present invention provides a fatigue driving warning system based on the opencv technology, comprising an infrared sensor and a camera, wherein the infrared sensor and the camera are connected to the processor through a communication module;

The infrared sensor tracks a certain point of the driver's head and collects the physiological signal of the driver. When the driver experiences fatigue and then lowers the physiological signal of the head, the infrared sensor feeds back the distance signal to the processor, and the processor The inspection is based on the distance of the head, and if the specified head distance threshold is exceeded, an early warning is given;

The camera performs real-time tracking on the driver's face and transmits the captured image as input data to the processor in a framing manner, and the processor discriminates the collected image and establishes a coordinate system to drive the driver's two The eye is projected to obtain the coordinates of the two eyes, the coordinates of the upper and lower eyelids of the two eyes are quantized by coordinates, and then the difference between the upper and lower eyelids is compared with the set upper and lower eyelid distance thresholds, and the distance between the upper and lower eyelids exceeds the upper and lower eyelid distances. After the threshold, it is determined that the driver is in a state of fatigue at this time, and an early warning is given;

When the infrared sensor detects that the specified low head distance threshold is exceeded and the camera detects that the upper and lower eyelid distance thresholds are exceeded, an alarm warning is performed;

In the process of discriminating the acquired image, the integral image algorithm is used to extract the image feature value, then the weak classifier is strengthened by the feature of the adaboost classifier, and the most useful feature is retained. Finally, the adaboost classifier is modified to obtain the level. Associated with the adaboost classifier, using a cascaded adaboost classifier to discriminate the image;

The cascaded adaboost classifier includes the following preparation steps:

Step 1) Initially initializing the training data;

D ₁ represents the weight of each sample of the first iteration, w ₁₁ represents the weight of the first sample at the first iteration, and N is the total number of samples;

Step 2) performing multiple iterations;

C. Learning using a training sample set having a weight distribution D _m (m=1, 2, 3...N) to obtain a weak classifier;

G _m(x) : x→{-1, +1}

The equation represents that the weak classifier at the mth iteration classifies the sample x into -1 or 1; criterion: the error function of the weak classifier is the smallest, that is, the sum of the weights corresponding to the sample of the fault is the smallest;

Is a constant;

Where e _m is the error function value;

D. Calculate the discourse weight of the weak classifier G _m(x) , using the following formula:

The speech right a _m represents the importance of G _m(x) in the final classifier, where a _m is equal to e _m ; the formula is a classifier that increases as e _m decreases, ie, the error rate is small, in the final classifier Of great importance;

C. Update the weight distribution of the training sample set for the next iteration;

Among them, the weight of the sample that is misclassified will increase, and the weight of the correctly divided will decrease;

D _m+1 =(w _m+1,1 ,w _m+1,2 ,...w _m+1,i ,...w _m+1,N ),

In the above formula, D _m+1 is the weight of the sample used for the next iteration, w _{m+1, i} is the weight of the i-th sample at the next iteration;

y _i represents that the category corresponding to the i-th sample is 1 or -1, and G _m (x _i ) represents the classification of the sample x _i by the weak classifier. If the pair is paired, the value of y _i *G _m (x _i ) Is 1, and vice versa -1;

Z _m is a normalization factor such that the sum of the weights corresponding to all samples is 1, and the formula for calculating Z _m is as follows:

Step 3) After the iteration is completed, combine the weak classifiers; the combination formula is as follows:

A cascaded adaboost classifier is obtained using a combination formula.

Further, the difference between the upper and lower eyelids is calculated by the Euclidean distance formula.

Further, the communication module adopts a Lora module.

Further, the warning reminder and the alarm alarm use a buzzer, and the sound volume and the warning sound are generated by the volume of the buzzer and the interval of the buzzer.

Further, the warning reminder uses a warning light to emit a reminder light, and the alarm alarm indicates that a buzzer is used to sound a warning sound.

Further, the information judgment result of the camera and the infrared sensor is information fusion;

Taking both the camera and the infrared sensor as sensing devices, first determine the consistency measure p _i (t) of the sensing device at a certain observation time, that is, the observation value of the i-th sensing device and the observation values of the remaining sensing devices. Similarity

Among them, the reliability of the single sensing device itself needs to be judged. The reliability of the single sensing device itself is represented by the measured mean and variance of the sensing device. The measured mean and variance are calculated by the following formula:

By r _i (t) and

Judging the individual sensing devices, the uniform reliability measure of a single sensing device is:

Normalize the above formula:

Where q _i (t) and q _j (t) are respectively represented as two different uniform sensing measures of a single sensing device;

The ratio of the measured mean and the variance is not included in the parameter adjustment, and the n data at time t is merged according to the following formula:

Where x(t) is the final judgment result at time t, and x _i (t) is the judgment result of the i-th sensing device at time t.

Further, when judging the reliability of the single sensing device itself, it is necessary to adjust the weight of the single sensing device in all the sensing devices by analyzing the time series fluctuation factors;

The time series fluctuation is represented by [p _i (1), p _i (2), ..., p _i (t)] ^T. When the time series fluctuation is small, the sensing device performance is stable, that is, the sensing device The reliability is high, and the weight ratio in all sensing devices is increased; when the time series fluctuates greatly, the sensing device performance is unstable, that is, the reliability of the sensing device is low, and the weights in all sensing devices are The proportion is lower.

The beneficial effects of the invention:

The invention can detect the facial features generated by the driver during the driving process and the generated series of physiological signals to prevent the driver from generating an unexpected situation during the driving process; the eyes are erratic when the driver is in a fatigue state. A feature performs tracking detection on the human eye, and determines a state of the driver at this time by setting a threshold and comparing the state of the eye at different times with the set threshold;

By setting a reasonable threshold, each driver's eye size, whether wearing glasses and lighting, etc. will affect the monitoring. Based on this situation, a reasonable threshold is set for different people's different lighting conditions to judge the state of the driver at this time. The invention can effectively avoid such false alarm problems.

The invention proposes a set of face detection method based on Viola-Jones method, firstly uses the integral image algorithm to extract image feature values, and the operation method is fast, and then uses the characteristics of the adaboost classifier to strengthen the weak classifier and retain the most useful features. Therefore, the computational complexity of the detection is reduced, and finally the adaboost classifier is transformed into a cascaded adaboost classifier, which improves the accuracy of face detection.

Infrared sensing is used to detect other physiological phenomena such as bowing when the driver is fatigued, so that the infrared sensor detects the shadow and judges the driver's physiological characteristics and then alarms.

Detailed ways

The present invention will be further described in conjunction with the specific embodiments, which are to be understood by those skilled in the art.

An embodiment of the fatigue driving early warning system based on the opencv technology of the present invention includes an infrared sensor and a camera, wherein the infrared sensor and the camera are connected to the processor through a communication module;

The infrared sensor tracks a certain point of the driver's head and collects the physiological signals of the driver. When the driver experiences fatigue and then lowers the physiological signal of the head, the infrared sensor feeds back the distance signal to the processor, and the processor is based on the head. The distance is checked, and if the specified head distance threshold is exceeded, an early warning is given;

The camera tracks the driver's face in real time and transmits the captured image as input data to the processor in a framing manner. The processor discriminates the collected image and establishes a coordinate system to perform the driver's eyes. Projection to obtain the coordinates of the two eyes, coordinate the difference between the upper and lower eyelids of the two eyes by coordinates, the difference between the upper and lower eyelids is calculated by the Euclidean distance formula, and then the upper and lower eyelid difference distances are compared with the set upper and lower eyelid distance thresholds. After the distance between the upper and lower eyelids exceeds the upper and lower eyelid distance threshold, it is determined that the driver is in a state of fatigue at this time, and an early warning is given;

The above-mentioned infrared sensor and camera can be used alone or in combination. When combined, when the infrared sensor detects that the specified head distance threshold is exceeded and the camera detects the upper and lower eyelid distance thresholds, an alarm warning is issued, and an alarm warning is performed. In order to fully determine the existence of dangerous driving conditions, it needs to be corrected immediately;

In the process of discriminating the acquired image, the integral image algorithm is used to extract the image feature value, then the weak classifier is strengthened by the feature of the adaboost classifier, and the most useful feature is retained. Finally, the adaboost classifier is modified to obtain the level. Associated with the adaboost classifier, using a cascaded adaboost classifier to discriminate the image;

The cascaded adaboost classifier includes the following preparation steps:

Step 1) Initially initializing the training data;

D ₁ represents the weight of each sample of the first iteration, w ₁₁ represents the weight of the first sample at the first iteration, and N is the total number of samples;

Step 2) performing multiple iterations;

E. learning using a training sample set having a weight distribution D _m (m=1, 2, 3...N) to obtain a weak classifier;

G _m(x) : x→{-1, +1}

The equation represents that the weak classifier at the mth iteration classifies the sample x into -1 or 1; criterion: the error function of the weak classifier is the smallest, that is, the sum of the weights corresponding to the sample of the fault is the smallest;

Is a constant;

Where e _m is the error function value;

F. Calculate the discourse weight of the weak classifier G _m(x) , using the following formula:

The speech right a _m represents the importance of G _m(x) in the final classifier, where a _m is equal to e _m ; the formula is a classifier that increases as e _m decreases, ie, the error rate is small, in the final classifier Of great importance;

C. Update the weight distribution of the training sample set for the next iteration;

Among them, the weight of the sample that is misclassified will increase, and the weight of the correctly divided will decrease;

D _m+1 =(w _m+1,1 ,w _m+1,2 ,...w _m+1,i ,...w _m+1,N ),

In the above formula, D _m+1 is the weight of the sample used for the next iteration, w _{m+1, i} is the weight of the i-th sample at the next iteration;

y _i represents that the category corresponding to the i-th sample is 1 or -1, and G _m (x _i ) represents the classification of the sample x _i by the weak classifier. If the pair is paired, the value of y _i *G _m (x _i ) Is 1, and vice versa -1;

Z _m is a normalization factor such that the sum of the weights corresponding to all samples is 1, and the formula for calculating Z _m is as follows:

Step 3) After the iteration is completed, combine the weak classifiers; the combination formula is as follows:

The cascading adaboost classifier is obtained by the combination formula. The cascaded adaboost classifier can find the appropriate threshold value to carry out the recognition and judgment, and is not affected by the driver's eye size, wearing glasses and lighting, etc., and greatly improves the accuracy of the judgment.

The communication module adopts the Lora module. The Lora module has ultra-long-distance communication, ultra-low power consumption, remarkable anti-interference characteristics, reliable data transmission, high receiving sensitivity and low cost. The main function in the present invention is to transmit the data collected by the infrared sensor and the camera to the processor.

In an embodiment, the buzzer is used for both the early warning reminder and the warning alarm, and the alarm sound and the warning sound are generated by the volume of the buzzer and the sounding interval of the buzzer, and the interval may be used when the reminder sound is needed. Longer and lighter sounds, so as to avoid excessive damage to the driving environment, when the warning is required, the surface driver has been trapped and affected the driving safety, so the sound with shorter interval, rush and louder volume can be used. To achieve a good early warning effect.

In an embodiment, the warning reminder uses a warning light to emit a reminder light. Due to a single detection, a false alarm may occur. For example, if the eye has blinking disease or is uncomfortable, it may be detected, and the reminder light is not excessive. Destroy the driving atmosphere, and when the driver appears to bow and blink frequently or close the eyes, the police alarm shows the use of a buzzer to sound a warning sound, achieving a good early warning effect.

In the above-mentioned fusion judgment of the information judgment result of the camera and the infrared sensor, it is necessary to perform reliable information judgment on the information data of the camera and the infrared sensor. First, both the camera and the infrared sensor are used as the sensing device, and the sensing device is first determined. The consistency measure p _i (t) of a certain observation time, that is, the similarity between the observation value of the i-th sensing device and the observation value of the remaining sensing devices; although the value of p _i (t) is relatively large at a certain time, However, it cannot be said that the reliability of the sensor is high throughout the observation interval, and the reliability of the single sensing device itself needs to be judged. The reliability of the single sensing device itself is represented by the measured mean and variance of the sensing device, and the mean value is measured. And the variance is calculated using the following formula:

k in the above formula is converted to the expression of the computer cycle at time t;

Due to r _i (t) and

Both are important parameters of the sensing device itself, so the uniform reliability measure of a single sensing device is:

Normalize the above formula:

Where q _i (t) and q _j (t) are respectively represented as two different uniform sensing measures of a single sensing device;

The ratio of measured mean and variance does not contain parameter adjustment, which avoids the influence of subjective factors, and more truly reflects the characteristics of the sensor itself, that is, more realistically describes the importance of the sensing device in the measurement information. The n data are merged according to the following formula:

Where x(t) is the final judgment result at time t, and x _i (t) is the judgment result of the i-th sensing device at time t, and finally a more reasonable judgment result can be obtained.

When judging the reliability of a single sensing device itself, it is necessary to adjust the weight of a single sensing device in all sensing devices by analyzing time series fluctuation factors;

The time series fluctuation is represented by [p _i (1), p _i (2), ..., p _i (t)] ^T. When the time series fluctuation is small, the sensing device performance is stable, that is, the sensing device The reliability is high, and the weight ratio in all sensing devices is increased; when the time series fluctuates greatly, the sensing device performance is unstable, that is, the reliability of the sensing device is low, and the weights in all sensing devices are The proportion is reduced to achieve more accurate sensor judgment. When the weight is very small, it can be considered that the sensing device is damaged and the stability of use is improved.

The invention is a relatively complete detection system. Not only the characteristics produced by the driver's face can be detected, but also the physiological signals generated by the driver can be detected. The technology involved is relatively wide, and the error is small. It mainly comes from the delay of signal transmission and the inevitable noise. The recognition time is short and the sensitivity is relatively high.

The above embodiments are merely preferred embodiments for the purpose of fully illustrating the present invention, and the scope of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are within the scope of the present invention. The scope of the invention is defined by the claims.

Claims (7)

1. A fatigue driving early warning system based on opencv technology, comprising: an infrared sensor and a camera, wherein the infrared sensor and the camera are connected to the processor through a communication module;

   The infrared sensor tracks a certain point of the driver's head and collects the physiological signal of the driver. When the driver experiences fatigue and then lowers the physiological signal of the head, the infrared sensor feeds back the distance signal to the processor, and the processor The inspection is based on the distance of the head, and if the specified head distance threshold is exceeded, an early warning is given;

   The camera performs real-time tracking on the driver's face and transmits the captured image as input data to the processor in a framing manner, and the processor discriminates the collected image and establishes a coordinate system to drive the driver's two The eye is projected to obtain the coordinates of the two eyes, the coordinates of the upper and lower eyelids of the two eyes are quantized by coordinates, and then the difference between the upper and lower eyelids is compared with the set upper and lower eyelid distance thresholds, and the distance between the upper and lower eyelids exceeds the upper and lower eyelid distances. After the threshold, it is determined that the driver is in a state of fatigue at this time, and an early warning is given;

   When the infrared sensor detects that the specified low head distance threshold is exceeded and the camera detects that the upper and lower eyelid distance thresholds are exceeded, an alarm warning is performed;

   In the process of discriminating the acquired image, the integral image algorithm is used to extract the image feature value, then the weak classifier is strengthened by the feature of the adaboost classifier, and the most useful feature is retained. Finally, the adaboost classifier is modified to obtain the level. Associated with the adaboost classifier, using a cascaded adaboost classifier to discriminate the image;

   The cascaded adaboost classifier includes the following preparation steps:

   Step 1) Initially initializing the training data;

   

   D ₁ represents the weight of each sample of the first iteration, w ₁₁ represents the weight of the first sample at the first iteration, and N is the total number of samples;

   Step 2) performing multiple iterations;

   A. Learning using a training sample set having a weight distribution Dm (m=1, 2, 3...N) to obtain a weak classifier;

   G _m(x) : x→{-1, +1}

   The equation represents that the weak classifier at the mth iteration classifies the sample x into -1 or 1;

   Criterion: The error function of the weak classifier is the smallest, that is, the sum of the weights corresponding to the sample of the fault is the smallest;

   

   Is a constant;

   Where e _m is the error function value;

   B. Calculate the discourse weight of the weak classifier G _m(x) , using the following formula:

   

   The speech right a _m represents the importance of G _m(x) in the final classifier, where a _m is equal to e _m ; the formula is a classifier that increases as e _m decreases, ie, the error rate is small, in the final classifier Of great importance;

   C. Update the weight distribution of the training sample set for the next iteration;

   Among them, the weight of the sample that is misclassified will increase, and the weight of the correctly divided will decrease;

   D _m+1 =(w _m+1,1 ,w _m+1,2 ,...w _m+1,i ,...w _m+1,N ),

   

   In the above formula, D _m+1 is the weight of the sample used for the next iteration, w _{m+1, i} is the weight of the i-th sample at the next iteration;

   y _i represents that the category corresponding to the i-th sample is 1 or -1, and G _m (x _i ) represents the classification of the sample x _i by the weak classifier. If the pair is paired, the value of y _i *G _m (x _i ) Is 1, and vice versa -1;

   Z _m is a normalization factor such that the sum of the weights corresponding to all samples is 1, and the formula for calculating Z _m is as follows:

   

   Step 3) After the iteration is completed, combine the weak classifiers; the combination formula is as follows:

   

   A cascaded adaboost classifier is obtained using a combination formula.
2. The fatigue driving early warning system based on the opencv technology according to claim 1, wherein the upper and lower eyelids are separated by an Euclidean distance formula.
3. The fatigue driving early warning system based on opencv technology according to claim 1, wherein the communication module adopts a Lora module.
4. The fatigue driving warning system based on opencv technology according to claim 1, wherein the warning reminder and the warning alarm are displayed by using a buzzer, and the sound volume of the buzzer and the sounding interval of the buzzer are combined to sound a reminder sound. Warning sound.
5. The fatigue driving early warning system based on the opencv technology according to claim 1, wherein the warning reminder uses a warning light to emit a reminder light, and the warning alarm indicates that a buzzer is used to issue a warning sound.
6. The fatigue driving early warning system based on the opencv technology according to claim 1, wherein the information judgment result of the camera and the infrared sensor is information fusion;

   Taking both the camera and the infrared sensor as sensing devices, first determine the consistency measure p _i (t) of the sensing device at a certain observation time, that is, the observation value of the i-th sensing device and the observation values of the remaining sensing devices. Similarity

   Among them, the reliability of the single sensing device itself needs to be judged. The reliability of the single sensing device itself is represented by the measured mean and variance of the sensing device. The measured mean and variance are calculated by the following formula:

   

   

   By r _i (t) and

   

   Judging the individual sensing devices, the uniform reliability measure of a single sensing device is:

   

   Normalize the above formula:

   

   Where q _i (t) and q _j (t) are respectively represented as two different uniform sensing measures of a single sensing device;

   The ratio of the measured mean and the variance is not included in the parameter adjustment, and the n data at time t is merged according to the following formula:

   

   Where x(t) is the final judgment result at time t, and x _i (t) is the judgment result of the i-th sensing device at time t.
7. The fatigue driving early warning system based on opencv technology according to claim 6, wherein when the reliability of the single sensing device itself is judged, it is necessary to analyze all the sensing devices by time series fluctuation factors. The weights in the device are adjusted;

   The time series fluctuation is represented by [p _i (1), p _i (2), ..., p _i (t)] ^T. When the time series fluctuation is small, the sensing device performance is stable, that is, the sensing device The reliability is high, and the weight ratio in all sensing devices is increased; when the time series fluctuates greatly, the sensing device performance is unstable, that is, the reliability of the sensing device is low, and the weights in all sensing devices are The proportion is lower.