CN103366506A - Device and method for automatically monitoring telephone call behavior of driver when driving - Google Patents

Abstract

The invention relates to an automatic monitoring device and method for a driver's mobile phone behavior while driving, and belongs to the field of intelligent transportation and assisted driving. The device includes an image acquisition device, a computing device, a warning device and a wireless transmission device; firstly, the image of the driver's head and the surrounding area is obtained; secondly, the position of the driver's face and hands in the image is obtained through skin color detection; Classify the mobile phone to determine whether the driver is answering or calling the mobile phone. If the mobile phone is connected, the driver will be warned; if the warning is useless, the image of the driver’s mobile phone will be uploaded to the monitoring center of the traffic bureau through the wireless network as evidence of his violation. The invention can effectively monitor illegal behaviors of drivers, reduce the work intensity of law enforcement personnel, improve work efficiency, and thereby reduce the occurrence rate of traffic accidents.

Description

Device and method for automatic monitoring of driver's mobile phone behavior while driving

technical field

The invention relates to the fields of intelligent transportation and assisted driving.

Background technique

With the popularization of mobile phone use, the behavior of drivers dialing or answering mobile phones while driving can be seen everywhere. Because the driver often controls the steering wheel with one hand when answering and calling the mobile phone, he cannot control the vehicle well; more importantly, answering and calling the mobile phone will distract the driver's attention. Very likely to cause traffic accidents. Studies have shown that when using a mobile phone while driving, the brain's reaction speed is 30% slower than that of drunk driving, and the risk of a car accident when using a mobile phone while driving is more than 4 times higher than that of normal driving. 70% of fatal traffic accidents are caused by drivers' inattention, and mobile phones are one of the main reasons for inattention. In view of the harmfulness of using mobile phones while driving to traffic safety, there are currently about 50 countries and regions in the world that prohibit drivers from using mobile phones while driving. my country's latest "Regulations for the Implementation of the Road Traffic Safety Law" also clearly stipulates that "if a driver calls or answers a mobile phone while driving, he will be given 3 points and a fine of 100 yuan."

However, since this behavior occurs in a car traveling at high speed, it is difficult to capture outside the car. At present, there are mainly three methods for monitoring the driver's mobile phone behavior while driving. The first is law enforcement personnel on-the-spot enforcement, but law enforcement personnel generally only appear in a few areas such as intersections, so this method has great limitations. The second is the camera monitoring outside the car. This method can capture the behavior of the driver using the mobile phone while driving. Although the monitoring range has been expanded compared with the first method, it is only limited to the range that the camera can monitor, and it cannot be monitored in most areas. the behavior. In addition, this method also requires monitoring personnel to stare at the monitor all the time to judge the illegal act, which has high work intensity and low efficiency, and the efficiency is even lower at night. The third is to install a directional antenna in a fixed area to monitor the mobile phone signal from a moving vehicle, and if it is a signal in a call, start the imaging device for on-site evidence collection. Although this method can play the function of automatic monitoring, it will take pictures of all the vehicles that detect the signal of the mobile phone that is talking. At this time, it is not necessarily the driver who is calling, or someone else in the car is talking. Lots of false positives. In the latter two methods, the cameras are all installed outside the car. Sometimes the captured pictures cannot clearly see the situation inside the car due to reasons such as weather or occlusion, and there will be missed detection and false detection.

Based on the above reasons, the behavior of "calling and answering mobile phones while driving" cannot be effectively monitored, making this behavior everywhere and causing great hidden dangers to traffic safety. At present, there is no device for automatically monitoring the driver's mobile phone behavior based on the in-vehicle camera.

Contents of the invention

In order to solve the above problems, the present invention provides an automatic monitoring device and method for the driver's mobile phone behavior while driving. Collect images of the driver's head and nearby areas through the camera in the car, use image processing technology and pattern recognition methods to automatically identify the driver's behavior of answering and calling mobile phones while driving, and take pictures of violations and upload them to the monitoring of the traffic bureau Center, as evidence of violation. The invention provides a method for automatically monitoring the driver's behavior of receiving and calling mobile phones during driving. It can greatly reduce the intensity of law enforcement, improve the efficiency of law enforcement, and effectively avoid traffic accidents caused by drivers dialing or answering mobile phones while driving. Moreover, the present invention installs the camera on the front windshield in the car, which can clearly photograph the behavior of the driver without the assistance of other electronic devices and will not affect the normal driving of the driver.

An automatic monitoring device for the driver's mobile phone behavior during driving, including: an image acquisition device, a computing device, a warning device and a wireless transmission device; wherein,

The image acquisition device uses a camera fixed on the front windshield in the vehicle to acquire images of the driver's head and nearby areas, and transmits the images to the computing device;

The computing device obtains the position of the driver's face and hands in the image through skin color detection, calculates the gradient histogram features of the face and hand regions and the driver's lip shape features, and obtains the driver's state of using a mobile phone and without it. Use the characteristics of the mobile phone state, classify through the support vector machine, so as to determine whether the driver is answering and calling the mobile phone during driving, and pass the information of whether the driver is answering and calling the mobile phone to the warning device;

The warning device sends a warning to the driver after receiving the information that the driver is receiving and calling the mobile phone from the computing device, and if the driver still receives the information that the driver is receiving and calling the mobile phone from the computing device after sending the warning, then send a signal to wireless transmission device;

The wireless transmission device uploads the image of the driver calling and calling the mobile phone to the monitoring center of the traffic bureau through the wireless network, as evidence of his violation of the law.

An automatic monitoring method for a driver's mobile phone behavior while driving, the implementation steps are as follows:

Step 1: When the vehicle engine is started, the monitoring device is automatically turned on, and the device performs a self-check operation to check whether there is a human face in the captured video screen, and whether the face is in a specific area (to prevent the driver from moving or blocking the device), if the above If the conditions are met, the steering wheel is unlocked; otherwise, the steering wheel is locked and the driver cannot drive the vehicle.

Step 2: After the device starts normally, it starts to monitor the image of the head area. If it is found that the driver's gesture of answering and calling the mobile phone exceeds a certain period of time, accompanied by lip movements, it is determined that the driver is answering and calling the mobile phone. .

Step 3: The alarm device generates an alarm. If the driver still does not stop using the mobile phone 3 seconds after the alarm is issued, the camera will upload the image of the driver using the mobile phone to the monitoring center of the Traffic Bureau through the network, as evidence of his violation, and to facilitate law enforcement personnel to deal with the violation.

Step 4: The device is only turned on when the vehicle is driving, and when the vehicle stops, the device automatically stops working.

Advantages of the present invention:

1. It can monitor in real time the violation of the driver's mobile phone calls while driving.

2. It can automatically transfer violation records to the monitoring center, which reduces the workload of law enforcement personnel and improves work efficiency.

3. The device is automatic and does not require the driver's operation, which is simple and easy to use.

4. The device is mandatory and can guarantee the realization of this function. It can effectively monitor the driver's illegal behavior and reduce the illegal behavior, thereby reducing the incidence of traffic accidents.

5. The camera is installed in the car and can capture clear images without being affected by the external environment such as weather, occlusion and other factors.

Description of drawings

Fig. 1 is the flow chart of the monitoring, warning and automatic evidence collection of the driver's mobile phone behavior on the way of driving in the present invention;

Fig. 2 is the rectangular feature used for driver's face detection in the present invention.

Detailed ways

The core idea of the present invention is: when the driver is answering and calling the mobile phone, there must be an action of holding the mobile phone close to the ear, accompanied by lip movement. Through video analysis, it is judged whether these behaviors occur, so as to judge whether it is making or receiving a mobile phone. Once the illegal behavior is determined, if the warning is invalid, the image of the violation will be sent to the monitoring center immediately, and a violation record will be formed, which is convenient for law enforcement officers to collect evidence.

The automatic monitoring device of the present invention includes: an image acquisition device, a computing device, a warning device and a wireless transmission device; wherein,

The image acquisition device uses a camera fixed on the front windshield of the vehicle to acquire images of the driver's head and nearby areas;

The computing device obtains the position of the driver's face and hands in the image through skin color detection, calculates the gradient histogram features of the face and hand regions and the driver's lip shape features, and obtains the driver's state of using a mobile phone and without it. Using the characteristics of the mobile phone state, classify through the support vector machine to determine whether the driver is answering and calling the mobile phone during driving;

The warning device issues a warning to the driver after the computing device determines that the driver is answering and calling the mobile phone;

The wireless transmission device uploads the image of the driver calling and calling the mobile phone to the monitoring center of the traffic bureau through the wireless network, as evidence of his violation of the law.

In the present invention, the flow chart of monitoring, warning and automatic evidence collection based on the driver's mobile phone behavior monitoring, warning and automatic evidence collection on the way of driving based on the camera in the car, the specific implementation steps are as follows:

Step 1: When the vehicle engine is started, the monitoring device is automatically turned on, and the device performs a self-check operation to check whether there is a human face in the captured video screen, and whether the face is in a specific area (to prevent the driver from moving or blocking the device), if the above If the conditions are met, the steering wheel is unlocked; otherwise, the steering wheel is locked and the driver cannot drive the vehicle.

The face detection method based on AdaBoost is used to locate the driver's face. This method constructs a weak classifier based on rectangular features, and then uses the AdaBoost method to select a small number of key features, and weights and sums the corresponding weak classifiers to construct a strong classifier, which is used as the final classifier for the face detection. Among them, each rectangle feature is composed of 2-3 rectangles, as shown in Figure 2, its value is the sum of the pixel values in the white rectangle minus the sum of the pixel values in the black rectangle.

Each weak classifier consists of a rectangular feature, and the training process of the strong classifier is as follows:

(1) Given training data (x ₁ ,y ₁ ),...(x _n ,y _n ), where y _i =0 represents negative samples, y _i =1 represents positive samples, and n is the number of training samples .

y_i=1时

m,l分别是负样本和正样本的个数。(2) Initialize the weight value, when y _i =0

when y _i =1

m and l are the number of negative samples and positive samples respectively.

(3) Corresponding to t=1,...,T:

A． normalized weight $w_{t,i}\←\frac{w_{t,i}}{{\mathrm{\Σ}}_{j=1}^{\mathrm{no}}{w}_{t,j}}$

B． For each feature j, train a weak classifier h _j , if the error of the classifier is ε _j =∑ _i w _i |h _j (xi ₎ -y _i |

C． Choose the classifier h _t with the smallest error

其中e_i=0如果x_i被正确分类，否则e_i=1， ${\mathrm{\β}}_t=\frac{{\mathrm{\ϵ}}_t}{1-{\mathrm{\ϵ}}_t}$ D． Update weights

where e _i = 0 if _xi is correctly classified, otherwise e _i = 1, ${\mathrm{\β}}_t=\frac{{\mathrm{\ϵ}}_t}{1-{\mathrm{\ϵ}}_t}$

(4) Then the final strong classifier is

$h\left(x\right)=\left\{\begin{array}{cc}1& {\mathrm{\Σ}}_{t=1}^T{\mathrm{\α}}_t{h}_t\left(x\right)\&Greater\; Equal;\frac{1}{2}{\mathrm{\Σ}}_{t=1}^T{\mathrm{\α}}_t\\ 0& \mathrm{otherwise}\end{array}\right.,$ in ${\mathrm{\α}}_t=\log \frac{1}{{\mathrm{\β}}_t}$

After getting the strong classifier, use the strong classifier to scan the image to get the position of the driver's face. If the face is in the center of the image, the device works normally. The self-test is successful, the steering wheel is unlocked, and the driver can start the vehicle.

Step 2: After the device starts up normally, it automatically monitors the image of the head area during the driving of the vehicle. If it is found that the driver's gesture of answering and calling the mobile phone exceeds a certain period of time, accompanied by lip movements, it is determined that the driver is using the mobile phone.

Because there is a big difference between the driver's normal driving state and the state of putting his hands on his ears when using a mobile phone. Therefore, we use the method of training the SVM classifier to judge whether the driver is answering or calling the mobile phone. Since there are obvious differences in the state of the lips of the driver when he is speaking and not speaking, the method of training the SVM classifier is also used to judge whether the driver's lips are moving. Based on the above two clues, if the driver has made the gesture of answering and calling the mobile phone for more than a certain period of time, and there is lip movement during the period, it is determined that the driver is answering and calling the mobile phone.

1) Extract the HOG features of the driver's head area

The features used to train the classifier are all HOG features. HOG is histogram of oriented gradient (histogram of gradient direction), which is a feature descriptor used for target detection. It calculates the number of directional gradients that appear locally in the image. The method divides the entire image into small connected regions (called cells), each cell generates a histogram of oriented gradients, and the combination of these histograms can represent the descriptor of the detected object. To improve accuracy, it is possible to normalize all cells within a larger region (called a block) in the image. Better lighting/shading invariance is done through the normalization process.

The HOG feature extraction method is as follows:

(1) Convert the image to a grayscale image.

(2) Divide the image into small cells (eg: 2*2).

(3) Calculate the gradient of each pixel in each cell.

Gradient size: $R(x,\mathrm{the\; y})=\sqrt{{(I(x+1,\mathrm{the\; y})-I(x-1,\mathrm{the\; y}))}^2+{(I(x,\mathrm{the\; y}+1)-I(x,\mathrm{the\; y}-1))}^2}$

Gradient direction: Ang(x,y)=arctan((I(x+1,y)-I(x-1,y))/(I(x,y+1)-I(x,y-1) ))

(4) Normalize all cells in each block. Normalize with L2-norm:

L2-norm: $v\&Right\; Arrow;v/\sqrt{{\left|\right|v\left|\right|}_2^2+{\mathrm{\ϵ}}^2}$

(5) Generate HOG features. Calculate the HOG features in all blocks in the detection window and concatenate them into the final feature vector for classification.

2) Use SVM to classify HOG features

After extracting the HOG feature of the driver's head area, it is used as the input of the SVM classifier for classification to judge the driver's head posture.

SVM stands for Support Vector Machine (Support Vector Machine), which is a supervised learning method that shows many unique advantages in solving small sample, nonlinear and high-dimensional pattern recognition. Based on the structural risk minimization theory, SVM constructs the optimal segmentation hyperplane in the feature space, so that the classifier is globally optimized, and the expected risk in the entire sample space meets a certain upper bound with a certain probability.

Assume that the mathematical form of the hyperplane is: w·xb=0. where x is a point on the hyperplane and w is a vector perpendicular to the hyperplane. (x ⁽ⁱ⁾ , y ⁽ⁱ⁾ ) are training samples. x ⁽ⁱ⁾ is the feature input of the training sample; y ⁽ⁱ⁾ is its corresponding label, where y ⁽ⁱ⁾ ∈ {1,-1}. Then in the case of linear separability, the optimal hyperplane can be obtained by solving the following quadratic optimization problem:

$\mathrm{<span\; class="notranslate">\; min</span>}\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}$

Satisfy the constraints: y ⁽ⁱ⁾ (w ^T x ⁽ⁱ⁾ +b)≥1,i=1,2,...,m

3) Judging the state of the driver's lips

The process of judging the state of the driver's lips is as follows:

1. Detect the position of the face through the Adaboost learning algorithm

2. Coarse positioning of the lip position. According to prior knowledge, lips are located in the lower half of the human face.

3. Determine the shape and position of the lips. Train the skin color and lip color, determine their color range respectively, use Fisher linear classifier to distinguish the skin color and lip color, and segment the mouth area.

4. Eliminate noise. By first corroding and then expanding the mouth area segmented in the previous step, the small connected areas are eliminated to achieve the purpose of noise removal. The area after noise removal is the mouth area.

5. The contour of the mouth is determined. The mouth area detected in the previous step is horizontally and vertically projected to obtain the mouth contour.

6. Feature extraction. Calculate the maximum width W _max , the maximum height H _max , and the height H _m between the upper lip and the lower lip of the mouth region according to the mouth contour. These values are obviously different when people are speaking and not speaking, so they are taken as features. F=(W _amx ,H _max ,H _m ).

7. Train the SVM classifier to classify the lip features to determine the state of the driver's mouth. 4) Judging whether the driver is answering or calling the mobile phone

When using SVM to classify the HOG features of the driver's head area, it is judged that it belongs to the gesture of answering and calling the mobile phone, and at the same time it is judged that the state of the driver's lips is speaking, and when the duration of this situation exceeds a given threshold, it is judged that the driving The staff is answering the phone.

Step 3: The alarm device sends out an alarm. If the driver still does not stop using the mobile phone after 3 seconds of the alarm, the wireless transmission device uploads the image of the driver using the mobile phone to the monitoring center of law enforcement officers through the network as evidence of his violation. And record its illegal behavior, time and other content, so that law enforcement officers can deal with the illegal behavior.

Step 4: The device is only turned on when the vehicle is driving, and when the vehicle stops, the device automatically stops working.

Claims (2)

1. The utility model provides a driver connects automatic monitoring device who makes a call to cell-phone action on the way that drives a vehicle which characterized in that: the automatic monitoring device comprises an image acquisition device, a computing device, a warning device and a wireless transmission device; wherein,

the image acquisition device acquires images of the head and the nearby area of the driver by adopting a camera fixed on a front windshield in the vehicle and transmits the images to the computing device;

the computing device obtains the characteristics of the driver in the state of connecting and calling the mobile phone and the state of not connecting and calling the mobile phone through training samples, and can classify each image acquired by the image acquisition device, so that whether the driver connects and calls the mobile phone in the driving process can be further determined, and the information of whether the driver connects and calls the mobile phone is transmitted to the warning device;

the warning device sends a warning to the driver after receiving the information that the driver is calling the mobile phone sent by the computing device, and sends a signal to the wireless transmission device if the information that the driver is calling the mobile phone sent by the computing device is still received after the warning is sent;

the wireless transmission device uploads the image of the driver connecting the mobile phone to the monitoring center of the traffic bureau through the wireless network as the evidence of the law violation.

2. An automatic monitoring method for a driver to take a mobile phone call during driving is characterized in that:

the method comprises the following steps: when the vehicle engine is started, the monitoring device is automatically started, the device performs self-checking operation, whether a face exists in a shot video picture or not and whether the face is in a specific area or not are checked, and if the conditions are met, the steering wheel is unlocked; otherwise, the steering wheel is in a locked state, and the driver can not drive the vehicle;

step two: after the device is normally started, monitoring of the head region image is started. The calculating device obtains the positions of the face and the hands of the driver in the image through skin color detection, calculates the gradient histogram characteristics of the face and the hand regions and the lip shape characteristics of the driver, and classifies the face and the hands by a support vector machine so as to determine whether the driver is connecting or disconnecting the mobile phone. If the gesture of the driver for connecting and calling the mobile phone is found to occur for more than a certain time in the running process of the vehicle, the driver is judged to be connecting and calling the mobile phone;

step three: the alarm device gives an alarm, if the driver still does not stop using the mobile phone after the alarm is given out for 3 seconds, the camera uploads an image of the mobile phone used by the driver to a monitoring center of a traffic bureau through a network to serve as an evidence of the law violation, and law enforcement personnel can conveniently process the law violation;

step four: the device is started only during the running of the vehicle, and when the vehicle stops, the device automatically stops working.

Images