CN111325872B - Driver driving abnormity detection method based on computer vision - Google Patents

Driver driving abnormity detection method based on computer vision Download PDF

Info

Publication number
CN111325872B
CN111325872B CN202010071329.2A CN202010071329A CN111325872B CN 111325872 B CN111325872 B CN 111325872B CN 202010071329 A CN202010071329 A CN 202010071329A CN 111325872 B CN111325872 B CN 111325872B
Authority
CN
China
Prior art keywords
target detection
driver
target
detection
driving
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010071329.2A
Other languages
Chinese (zh)
Other versions
CN111325872A (en
Inventor
羊晋
司俊俊
涂波
刘孟奇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hezhixin Shandong Big Data Technology Co ltd
Original Assignee
Hezhixin Shandong Big Data Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hezhixin Shandong Big Data Technology Co ltd filed Critical Hezhixin Shandong Big Data Technology Co ltd
Priority to CN202010071329.2A priority Critical patent/CN111325872B/en
Publication of CN111325872A publication Critical patent/CN111325872A/en
Application granted granted Critical
Publication of CN111325872B publication Critical patent/CN111325872B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07C—TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
    • G07C5/00—Registering or indicating the working of vehicles
    • G07C5/08—Registering or indicating performance data other than driving, working, idle, or waiting time, with or without registering driving, working, idle or waiting time
    • G07C5/0841—Registering performance data
    • G07C5/085—Registering performance data using electronic data carriers
    • G07C5/0866—Registering performance data using electronic data carriers the electronic data carrier being a digital video recorder in combination with video camera
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
    • G06V20/597—Recognising the driver's state or behaviour, e.g. attention or drowsiness
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07C—TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
    • G07C5/00—Registering or indicating the working of vehicles
    • G07C5/08—Registering or indicating performance data other than driving, working, idle, or waiting time, with or without registering driving, working, idle or waiting time
    • G07C5/0841—Registering performance data
    • G—PHYSICS
    • G08—SIGNALLING
    • G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
    • G08B21/00—Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
    • G08B21/18—Status alarms
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/07—Target detection

Abstract

A driver drives unusual check out test set and detection method based on computer vision, wherein, the check out test set includes vehicle carried lens and video record store passback apparatus, the vehicle carried lens is connected with video record store passback apparatus, characterized by that, also include detecting server and data storage server in the detection center, detecting server and data storage server pass the network connection, the detecting server is connected with video record store passback apparatus through the network again; the detection center is also provided with a central video image processing and target detection device, the central video image processing and target detection device is connected with a detection server, and the detection server is connected with a video recording, storing and returning device through a network. The invention solves the problems of large workload and slow response of manual analysis in the existing monitoring means, automatically detects and monitors the driving conditions of drivers and passengers and vehicles in real time, saves a large amount of manpower, is accurate and effective and achieves the purpose of detection.

Description

Driver driving abnormity detection method based on computer vision
Technical Field
The invention discloses a driver driving abnormity detection device and a driver driving abnormity detection method based on computer vision, relates to the field of computer vision and machine learning, and provides a driver driving condition detection method based on computer vision.
Background
The driving condition of a driver is directly related to the generation of vehicle traffic accidents, and good driving behaviors can reduce the generation of the traffic accidents and casualties in the driving process of the vehicle; poor driving behaviors such as fatigue driving, drunk driving, answering and answering short messages or surfing the internet in the driving process, and the two hands being separated from the steering wheel can cause distraction of driver attention and even abnormal vehicle driving, and traffic accidents can be more easily caused.
For the vehicle carrying, the dangerous driving behavior of the driver not only easily causes road traffic accidents, but also causes the sending accidents of personnel in the vehicle, especially the vehicles with more carrying persons and standing passengers, such as buses and the like. In addition to the usual dangerous driving behavior, the driver of the carrier vehicle may also be unable to drive normally due to the occurrence of an event inside the vehicle, such as a passenger interfering with the driver. Therefore, there is a need for an effective method for detecting the driving condition of a driver, whether the driver is driving safely, distracted or disturbed by the people inside the vehicle.
The current driving behavior of a driver is that corresponding sensors are required to be arranged outside a vehicle or on a transmission system when the driving condition of the vehicle is detected by an external sensor, so that the cost is high and more additional acquisition and processing equipment is needed; or install inside camera additional, whether have the condition of abnormal driving based on video data judgement, and video monitoring data analysis is mostly manual monitoring, spot check mode, and analysis work load is great, the coverage is little, the effect is poor, and the unable real time monitoring field situation of manual reexamination, is difficult to realize that the whole control of abnormal driving condition and make warning and quick response.
The advantage of monitoring the driving behavior of the driver based on computer vision is that 1, many carriers, buses and the like, even private cars are provided with an in-car camera, video acquisition and analysis are carried out on the in-car driver and passengers, and available data (marked and unmarked) are rich; 2. with the development of computer vision technology and deep learning, technologies such as target detection, face recognition and the like are rapidly developed, and the target detection rate, the detection efficiency and the like are remarkably improved; 3. and the advanced chip design and manufacture technology reduces the cost of large-scale target detection, and even the pruning model can be deployed at the acquisition equipment end, namely edge calculation.
Disclosure of Invention
The invention aims to provide a driver driving abnormity detection device and a detection method based on computer vision, which can automatically detect driver driving state abnormity through the detection device.
A driver drives the detection equipment of the anomaly based on video monitoring and computer vision, including vehicle carried camera and video record to store and pass back the apparatus, the vehicle carried camera is connected with video record to store and pass back the apparatus, characterized by, also include detecting server and data storage server in the detection center, detecting server and data storage server are through the network connection, the detecting server is connected with video record to store and pass back the apparatus through the network again; the detection center is also provided with a central video image processing and target detection device, the central video image processing and target detection device is connected with a detection server, and the detection server is connected with a video recording, storing and returning device through a network.
And the vehicle-mounted equipment end is also provided with vehicle-mounted video image processing and target detection equipment, the vehicle-mounted video image processing and target detection equipment is connected with video recording, storing and returning equipment, and returning data comprises vehicle-mounted video image data and a target detection result.
A driver driving abnormity detection device and a detection method based on computer vision comprise three parts, namely a target detection model pre-training part, a driving abnormity detection model training part and a driving abnormity detection part.
The main process of pre-training the target detection model is as follows:
firstly, collecting and labeling a target detection data set: collecting training data sets of common target detection objects under a driving scene and a general scene, wherein the training data sets comprise human bodies, human faces, hands and common invaders, supplementing the training data sets of the target detection objects under the specific scene according to requirements, and labeling the training data sets according to the classification of the human bodies, the human faces, the hands and the common invaders; the complete data set should include: the method comprises the steps of detecting picture data and target detection labeling results of human bodies, human faces, hands and common invaders of a target object; the target detection and marking result comprises a target type and an image pixel position of the target; saving the labeling result of the data set;
preprocessing a target data set, namely preprocessing the collected target detection training data, wherein the preprocessing comprises the steps of not limiting noise reduction and resampling, and preprocessing the image sizes M N to M1N 1;
thirdly, carrying out physical multiplication on a target detection data set to serve as a model training data set, and dividing the training set and a verification set according to a certain proportion, wherein the training set is larger than the verification set;
selecting a proper deep learning frame according to the target detection effect and the hardware equipment support of model training and online prediction;
selecting an applicable target detection network structure according to the type and the characteristics of the target to be detected;
step six, carrying out iterative training on the target detection model by using the training set, and evaluating by using the verification set to prevent overfitting and gradient divergence, when the target detection effect of the neural network model on the verification set does not become good any more or reaches the preset iterative training times, stopping training, and exporting the trained model;
seventhly, performing model cutting by using a deep learning model pruning algorithm, performing redundancy deletion on the trunk network, and reducing the size of an input network, wherein the model cutting also comprises convolutional layer cutting and channel pruning;
step eight, outputting the trained target detection model, wherein the input picture size of the target detection algorithm of the model is M1 × N1, and the output result is the target type and the position of a target detection frame;
deploying the trained target detection model to a video acquisition end or a driving behavior detection center server;
the driving abnormity detection model training process comprises the following steps:
collecting driver driving video data and corresponding driving information, wherein the video data are video data shot by a driver, and the driving information comprises vehicle information in the driving process: vehicle position, vehicle speed, vehicle direction information;
eleven, marking the collected driver driving video data, wherein the marking information comprises: normal driving and abnormal driving, and corresponding abnormal driving types;
step twelve, performing preliminary preprocessing, noise reduction, frame reduction and cutting transformation on the video;
thirteen, based on the trained target detection model, carrying out target detection on the video image frames by using a target detection algorithm, identifying human bodies, framing the detected human bodies, and outputting corresponding characteristics;
fourteen, based on a pre-training model, utilizing a target detection algorithm (a model with a good small target effect) to detect human body parts and objects, identifying the positions of human faces and hands, and identifying whether other objects influencing driving exist;
fifteen, calculating characteristic parameters based on the target detection information in the second step and the third step;
sixthly, selecting a proper classifier model, and performing driving abnormity detection model training on the characteristics calculated in the step fifteen and the driving characteristics of the vehicle in the step ten, including vehicle position, vehicle speed and vehicle direction information, by combining the labeling result in the step eleven;
seventhly, deploying the driving behavior detection model to a central server for detection;
after model training is completed, the driving behavior of the driver is detected by using the target detection model and the driving behavior detection model, and the method comprises the following steps:
eighteen, mounting a vehicle-mounted camera and video recording, storing and returning equipment at a vehicle end, wherein the vehicle-mounted camera is connected with the video recording, storing and returning equipment; the detection center is also provided with a central video image processing and target detection device, the central video image processing and target detection device is connected with a detection server, and the detection server is connected with a video recording, storing and returning device through a network;
the vehicle-mounted camera transmits the video monitoring data and the simultaneously acquired running state and running speed of the vehicle to the video recording, storing and returning equipment for data storage, and transmits the stored data to a data storage server of the detection center;
nineteenth, the detection server connected with the data storage server performs preliminary preprocessing, noise reduction, frame dropping and cutting transformation on the video monitoring data;
twenty, carrying out target detection on the collected video image frames by using a target detection algorithm based on a pre-trained target detection model, identifying a human body, and framing the detected human body;
twenty one, based on a pre-training model, utilizing a target detection algorithm, carrying out human body part detection and object detection through video image processing and target detection equipment, identifying the position of a human face and the position of a hand, and identifying whether other target detection information influencing driving of an object exists;
twenty-two, calculating by taking the target detection information as a calculated identification key point parameter to obtain a target detection result value;
twenty-third, according to the set rule, obtaining a target detection result value by simultaneously obtaining the running state, the running speed and the calculation of the vehicle, judging whether the driver has violation behaviors in the driving process and other abnormal behaviors in a specified range, and obtaining the detection result of the driving state of the driver;
and twenty-four steps of transmitting and storing the detection result to a database server for storage.
Wherein, the selected deep learning frame in the step four is Pythrch or Tensorflow; selecting an applicable target detection network structure as YOLOv3, M2Det or CornerNet; sixthly, selecting a random forest or gradient descent decision tree in the classifier model selected by the proper classifier model.
And eighteen, the vehicle-mounted video recording, storing and returning device is further connected with a vehicle-mounted video image processing and target detecting device, the vehicle-mounted video image processing and target detecting device detects the video image transmitted back to the video recording, storing and returning device by the vehicle-mounted camera in real time, transmits the detection result back to the video recording, storing and returning device for storing, and transmits the video monitoring data recorded by the vehicle-mounted camera and the running state and the running speed of the vehicle, which are acquired at the same time, to the data storage server in real time, and simultaneously transmits the detection result to the data storage server for storage.
The real-time detection process of the vehicle-mounted video image processing and target detection equipment comprises the following contents:
a. performing preliminary preprocessing, noise reduction, frame dropping and cutting transformation on video monitoring data;
b. performing target detection on the collected video image frames by using a target detection algorithm based on a pre-training model, identifying a human body, and framing the detected human body;
c. based on a pre-training model, a target detection algorithm is utilized, human body part detection and object detection are carried out through video image processing and target detection equipment, the positions of human faces and hands are identified, whether other target detection information influencing driving objects exists or not is identified, and the target detection information is output to video recording, storing and returning equipment;
d. the vehicle-mounted video image processing and target detection equipment calculates target detection information as a calculated identification key point parameter to obtain a target detection result value;
e. according to a set rule, obtaining a target detection result value by simultaneously obtaining the running state and the running speed of the vehicle and calculating, judging whether a driver has violation behaviors in the driving process and other abnormal behaviors in a specified range, and obtaining a detection result of the driving state of the driver;
f. and transmitting the detection result to the video recording storage returning device.
Wherein, the preliminary preprocessing, noise reduction, frame dropping and cutting transformation of the video monitoring data in the step two comprises the following contents,
a. carrying out intra-frame noise reduction and inter-frame noise reduction on the video image, and processing by using an image noise reduction algorithm;
b. sampling video frames according to the performance of subsequent target detection equipment and actual transmission bandwidth, and reducing the video frame rate to be an original frame rate/N, wherein N is an integer greater than 1;
c. and according to target detection equipment performance and pre-training model parameters, selecting proper resolution to re-crop and transform the video, so that the resolution is transformed from the original resolution m × n to m1 × n1, wherein the m1 × n1 resolution is consistent with the size of the target detection model in the later period.
Performing target detection on the acquired video image frames by using a target detection algorithm based on the pre-training model, identifying a human body, and framing the detected human body; in particular to a method for preparing a high-performance nano-silver alloy,
a. the absolute position of the pixel of the target in the image takes the lower left corner of the image as an origin (0,0), and the number of horizontal pixels and the number of vertical pixels (x, y) relative to the origin;
b. the coordinates (x2, y2) of the lower left corner (x1, y1) and the upper right corner of the target frame are detected by the target, and the pixel area = (y2-y1) × (x2-x1) of the target frame is detected;
c. the object type of the target depends on the data type of the image input by the target detection pre-training model, and the detected object mainly comprises a mobile phone and a non-driver face.
Wherein the key point parameters in the fifth step mainly include,
a. the face, the center position of the driver face, and the comparison with the driver face library is passed;
b. the effective area of the face of the driver, and the target type is the pixel area of the face; c. the center of the hand of the driver is positioned,
d. the abnormal invader types mainly comprise mobile phones and non-driver faces;
e. abnormal invader position, abnormal invader center coordinate;
f. and calculating the change characteristics of the characteristic variables in a certain time window.
g. Time window, standard deviation of variation of image coordinates of the center position B1 of the face of the driver, and maximum displacement;
h. time window, driver face effective area B2, maximum area, minimum area, standard deviation;
j. time window, missing the face detection frame number proportion.
Wherein the abnormal behaviors in the step six and other specified ranges further comprise,
a. during temporary parking, whether the driver drives normally or not;
b. in the vehicle driving process/temporary stop, the driver driving behavior is abnormal: a) the driver's own attention is distracted; b) the driver is disturbed by the persons in the vehicle.
Wherein the features calculated in step fifteen include:
A. based on target detection, identifying single-frame features of the main key point parameters, including:
A1. the center position of the driver face, the target type in the step three is the face, and the target type is compared with a driver face library and passes through the comparison;
A2. the effective area of the driver face is the pixel area of the face based on the target type in the step three;
A3. the center position of the driver's hand and the target type in the third step are the center position of the hand.
A4. The type of the abnormal invader is the type of the target in the third step, namely the mobile phone and the non-driver face.
A5. And (4) detecting the central coordinates of the abnormal invader by the target in the step three.
B. Based on the above a, i.e. single frame features, the feature variables within a certain time window are calculated.
B1. t1 time window (f 1 frames), driver face center position B1 image coordinate variation standard deviation, maximum displacement;
B1. t1 time window (f 1 frames), driver face effective area B2, maximum area, minimum area, standard deviation;
B3. t1 time window (f 1 frames), missing the face detection frame number ratio.
The invention solves the problems of large workload and slow response of manual analysis in the existing monitoring means, and can automatically detect the driving conditions of drivers and passengers and vehicles in real time so as to achieve the monitoring effect. The invention saves a large amount of manpower, is accurate and effective and achieves the aim of detection.
Drawings
FIG. 1 illustrates a method for detecting abnormal driving of a driver based on computer vision in a first stage of the present invention;
FIG. 2(a) is a schematic view of the structure of the device for detecting the target in the central server for detecting the abnormal driving of the driver in the second stage of the present invention;
FIG. 2(b) is a schematic structural diagram of the device for detecting the target at the video capture end for detecting the abnormal driving of the driver in the second stage of the present invention;
FIG. 3 is a flowchart illustrating the detection process performed by the driver driving abnormality detection system in the second stage of the present invention;
FIG. 4 is a pre-training portion of the third stage object detection model of the present invention, model offline training and optimized clipping;
FIG. 5 is a pre-training portion of the third stage object detection model of the present invention, model offline training and optimized clipping;
fig. 6 shows a third stage driving abnormality detection model training section according to the present invention.
Detailed Description
A driver driving abnormity detection device based on computer vision comprises a vehicle-mounted camera and a video recording, storing and returning device, wherein the vehicle-mounted camera is connected with the video recording, storing and returning device; the detection center is also provided with a central video image processing and target detection device, the central video image processing and target detection device is connected with a detection server, and the detection server is connected with a video recording, storing and returning device through a network.
And the vehicle-mounted equipment end is also provided with vehicle-mounted video image processing and target detection equipment, the vehicle-mounted video image processing and target detection equipment is connected with video recording, storing and returning equipment, and returning data comprises vehicle-mounted video image data and a target detection result.
A driver driving abnormity detection method based on computer vision comprises three parts, namely a target detection model pre-training part, a driving abnormity detection model training part and a driving abnormity detection part.
The main process of pre-training the target detection model is as follows:
firstly, collecting and labeling a target detection data set: collecting training data sets of common target detection objects under a driving scene and a general scene, wherein the training data sets comprise human bodies, human faces, hands and common invaders, supplementing the training data sets of the target detection objects under the specific scene according to requirements, and labeling the training data sets according to the classification of the human bodies, the human faces, the hands and the common invaders; the complete data set should include: the method comprises the steps of detecting picture data and target detection labeling results of human bodies, human faces, hands and common invaders of a target object; the target detection and marking result comprises a target type and an image pixel position of the target; saving the labeling result of the data set;
preprocessing a target data set, namely preprocessing the collected target detection training data, wherein the preprocessing comprises the steps of not limiting noise reduction and resampling, and preprocessing the image sizes M N to M1N 1;
thirdly, carrying out physical multiplication on a target detection data set to serve as a model training data set, and dividing the training set and a verification set according to a certain proportion, wherein the training set is larger than the verification set;
selecting a proper deep learning frame according to the target detection effect and the hardware equipment support of model training and online prediction;
selecting an applicable target detection network structure according to the type and the characteristics of the target to be detected;
step six, carrying out iterative training on the target detection model by using the training set, and evaluating by using the verification set to prevent overfitting and gradient divergence, when the target detection effect of the neural network model on the verification set does not become good any more or reaches the preset iterative training times, stopping training, and exporting the trained model;
seventhly, performing model cutting by using a deep learning model pruning algorithm, performing redundancy deletion on the trunk network, and reducing the size of an input network, wherein the model cutting also comprises convolutional layer cutting and channel pruning;
step eight, outputting the trained target detection model, wherein the input picture size of the target detection algorithm of the model is M1 × N1, and the output result is the target type and the position of a target detection frame;
deploying the trained target detection model to a video acquisition end or a driving behavior detection center server;
the driving abnormity detection model training process comprises the following steps:
collecting driver driving video data and corresponding driving information, wherein the video data are video data shot by a driver, and the driving information comprises vehicle information in the driving process: vehicle position, vehicle speed, vehicle direction information;
eleven, marking the collected driver driving video data, wherein the marking information comprises: normal driving and abnormal driving, and corresponding abnormal driving types;
step twelve, performing preliminary preprocessing, noise reduction, frame reduction and cutting transformation on the video;
thirteen, based on the trained target detection model, carrying out target detection on the video image frames by using a target detection algorithm, identifying human bodies, framing the detected human bodies, and outputting corresponding characteristics;
fourteen, based on a pre-training model, utilizing a target detection algorithm (a model with a good small target effect) to detect human body parts and objects, identifying the positions of human faces and hands, and identifying whether other objects influencing driving exist;
fifteen, calculating characteristic parameters based on the target detection information in the second step and the third step;
sixthly, selecting a proper classifier model, and performing driving abnormity detection model training on the characteristics calculated in the step fifteen and the driving characteristics of the vehicle in the step ten, including vehicle position, vehicle speed and vehicle direction information, by combining the labeling result in the step eleven;
seventhly, deploying the driving behavior detection model to a central server for detection;
after model training is completed, the driving behavior of the driver is detected by using the target detection model and the driving behavior detection model, and the method comprises the following steps:
eighteen, mounting a vehicle-mounted camera and video recording, storing and returning equipment at a vehicle end, wherein the vehicle-mounted camera is connected with the video recording, storing and returning equipment; the detection center is also provided with a central video image processing and target detection device, the central video image processing and target detection device is connected with a detection server, and the detection server is connected with a video recording, storing and returning device through a network;
the vehicle-mounted camera transmits the video monitoring data and the simultaneously acquired running state and running speed of the vehicle to the video recording, storing and returning equipment for data storage, and transmits the stored data to a data storage server of the detection center;
nineteenth, the detection server connected with the data storage server performs preliminary preprocessing, noise reduction, frame dropping and cutting transformation on the video monitoring data;
twenty, carrying out target detection on the collected video image frames by using a target detection algorithm based on a pre-trained target detection model, identifying a human body, and framing the detected human body;
twenty one, based on a pre-training model, utilizing a target detection algorithm, carrying out human body part detection and object detection through video image processing and target detection equipment, identifying the position of a human face and the position of a hand, and identifying whether other target detection information influencing driving of an object exists;
twenty-two, calculating by taking the target detection information as a calculated identification key point parameter to obtain a target detection result value;
twenty-third, according to the set rule, obtaining a target detection result value by simultaneously obtaining the running state, the running speed and the calculation of the vehicle, judging whether the driver has violation behaviors in the driving process and other abnormal behaviors in a specified range, and obtaining the detection result of the driving state of the driver;
and twenty-four steps of transmitting and storing the detection result to a database server for storage.
Wherein, the selected deep learning frame in the step four is Pythrch or Tensorflow; selecting an applicable target detection network structure as YOLOv3, M2Det or CornerNet; sixthly, selecting a random forest or gradient descent decision tree in the classifier model selected by the proper classifier model.
And eighteen, the vehicle-mounted video recording, storing and returning device is further connected with a vehicle-mounted video image processing and target detecting device, the vehicle-mounted video image processing and target detecting device detects the video image transmitted back to the video recording, storing and returning device by the vehicle-mounted camera in real time, transmits the detection result back to the video recording, storing and returning device for storing, and transmits the video monitoring data recorded by the vehicle-mounted camera and the running state and the running speed of the vehicle, which are acquired at the same time, to the data storage server in real time, and simultaneously transmits the detection result to the data storage server for storage.
The real-time detection process of the vehicle-mounted video image processing and target detection equipment comprises the following contents:
a. performing preliminary preprocessing, noise reduction, frame dropping and cutting transformation on video monitoring data;
b. performing target detection on the collected video image frames by using a target detection algorithm based on a pre-training model, identifying a human body, and framing the detected human body;
c. based on a pre-training model, a target detection algorithm is utilized, human body part detection and object detection are carried out through video image processing and target detection equipment, the positions of human faces and hands are identified, whether other target detection information influencing driving objects exists or not is identified, and the target detection information is output to video recording, storing and returning equipment;
d. the vehicle-mounted video image processing and target detection equipment calculates target detection information as a calculated identification key point parameter to obtain a target detection result value;
e. according to a set rule, obtaining a target detection result value by simultaneously obtaining the running state and the running speed of the vehicle and calculating, judging whether a driver has violation behaviors in the driving process and other abnormal behaviors in a specified range, and obtaining a detection result of the driving state of the driver;
f. and transmitting the detection result to the video recording storage returning device.
Wherein, the preliminary preprocessing, noise reduction, frame dropping and cutting transformation of the video monitoring data in the step two comprises the following contents,
a. carrying out intra-frame noise reduction and inter-frame noise reduction on the video image, and processing by using an image noise reduction algorithm;
b. sampling video frames according to the performance of subsequent target detection equipment and actual transmission bandwidth, and reducing the video frame rate to be an original frame rate/N, wherein N is an integer greater than 1;
c. and according to target detection equipment performance and pre-training model parameters, selecting proper resolution to re-crop and transform the video, so that the resolution is transformed from the original resolution m × n to m1 × n1, wherein the m1 × n1 resolution is consistent with the size of the target detection model in the later period.
Performing target detection on the acquired video image frames by using a target detection algorithm based on the pre-training model, identifying a human body, and framing the detected human body; in particular to a method for preparing a high-performance nano-silver alloy,
a. the absolute position of the pixel of the target in the image takes the lower left corner of the image as an origin (0,0), and the number of horizontal pixels and the number of vertical pixels (x, y) relative to the origin;
b. the coordinates (x2, y2) of the lower left corner (x1, y1) and the upper right corner of the target frame are detected by the target, and the pixel area = (y2-y1) × (x2-x1) of the target frame is detected;
c. the object type of the target depends on the data type of the image input by the target detection pre-training model, and the detected object mainly comprises a mobile phone and a non-driver face.
Wherein the key point parameters in the fifth step mainly include,
a. the face, the center position of the driver face, and the comparison with the driver face library is passed;
b. the effective area of the face of the driver, and the target type is the pixel area of the face; c. the center of the hand of the driver is positioned,
d. the abnormal invader types mainly comprise mobile phones and non-driver faces;
e. abnormal invader position, abnormal invader center coordinate;
f. and calculating the change characteristics of the characteristic variables in a certain time window.
g. Time window, standard deviation of variation of image coordinates of the center position B1 of the face of the driver, and maximum displacement;
h. time window, driver face effective area B2, maximum area, minimum area, standard deviation;
j. time window, missing the face detection frame number proportion.
Wherein the abnormal behaviors in the step six and other specified ranges further comprise,
a. during temporary parking, whether the driver drives normally or not;
b. in the vehicle driving process/temporary stop, the driver driving behavior is abnormal: a) the driver's own attention is distracted; b) the driver is disturbed by the persons in the vehicle.
Wherein the features calculated in step fifteen include:
A. based on target detection, identifying single-frame features of the main key point parameters, including:
A1. the center position of the driver face, the target type in the step three is the face, and the target type is compared with a driver face library and passes through the comparison;
A2. the effective area of the driver face is the pixel area of the face based on the target type in the step three;
A3. the center position of the driver's hand and the target type in the third step are the center position of the hand.
A4. The type of the abnormal invader is the type of the target in the third step, namely the mobile phone and the non-driver face.
A5. And (4) detecting the central coordinates of the abnormal invader by the target in the step three.
B. Based on the above a, i.e. single frame features, the feature variables within a certain time window are calculated.
B1. t1 time window (f 1 frames), driver face center position B1 image coordinate variation standard deviation, maximum displacement;
B1. t1 time window (f 1 frames), driver face effective area B2, maximum area, minimum area, standard deviation;
B3. t1 time window (f 1 frames), missing the face detection frame number ratio.
The invention provides a driver driving abnormity detection method based on computer vision for solving the technical problems. The method mainly comprises three parts:
the system comprises a target detection model pre-training part, a driving abnormity detection model training part and a driving abnormity detection part.
The main process of pre-training the target detection model is as follows:
the method comprises the steps of collecting and labeling a target detection data set, collecting training data sets of common target detection objects under a driving scene and a common scene, including human bodies, human faces, hands and common invaders (mobile phones and the like), supplementing the target detection data set under a specific scene (a vehicle driving scene and the like) according to requirements, and labeling. The complete data set should include: picture data containing desired target detection objects (human body, human face, hand, common intrusions (mobile phone, etc.)); labeling results of the data set;
preprocessing a target data set, namely preprocessing the collected target detection training data, wherein the preprocessing comprises the steps of not only reducing noise, resampling and the like, but also preprocessing all image sizes (M1N 1);
thirdly, the target detection data set is multiplied as a model training data set, and the training set and the verification set are divided according to a certain proportion (example 9: 1);
selecting a proper deep learning frame (Tensorflow, Pyorch and the like) according to the training & predicting platform and the model characteristics;
step five, selecting an applicable target detection network structure (Yolov 3, CornerNet, M2Det and the like) according to the type and the characteristics (size, range and the like) of the target to be detected;
performing iterative training on the target detection model by using the training set, and evaluating by using the verification set to prevent overfitting and gradient divergence;
seventhly, performing model cutting by using a deep learning model pruning algorithm, performing redundancy deletion on the trunk network, reducing the size of an input network, performing convolutional layer cutting and channel pruning;
step eight, outputting a corresponding target detection pre-training model, inputting the picture size (M1 × N1) by using a target detection algorithm of the model, and outputting the result as the target type and the position of a target detection frame
And step nine, deploying a target detection model, wherein the target detection model comprises two modes of deploying to a video acquisition terminal and a driving behavior detection center server.
The driving abnormity detection model training process comprises the following steps:
collecting driver driving video data and corresponding driving information, wherein the video data are video data shot by a driver, and the driving information comprises vehicle information in the driving process: vehicle position, vehicle speed, vehicle direction information;
eleven, marking the collected driver driving video data, wherein the marking information comprises: normal driving and abnormal driving, and corresponding abnormal driving types;
step twelve, performing preliminary preprocessing, noise reduction, frame reduction and cutting transformation on the video;
and thirteen, carrying out target detection by utilizing the video image frame of the target detection algorithm based on the pre-training model in the step eight. Recognizing a human body, framing the detected human body, and outputting corresponding characteristics;
fourteen, based on a pre-training model, utilizing a target detection algorithm (a model with a good small target effect) to detect human body parts and objects, identifying the positions of human faces and hands, and identifying whether other objects (such as mobile phones and the like) influencing driving exist;
fifteen, calculating characteristic parameters based on the target detection information in the second step and the third step
Sixthly, selecting a proper classifier model (decision tree and the like), and combining the characteristics calculated in the step fifteen and the vehicle driving characteristics (vehicle position, vehicle speed and vehicle direction information) in the step ten with the marked result to perform model pre-training.
Seventhly, deploying a driving behavior detection model to a central server for detection.
After model pre-training is completed, the driving behavior of a driver is detected by using a target detection model and a driving behavior detection model, and the method mainly comprises the following processes:
eighteen, collecting video monitoring data of a driving position of the vehicle, and performing preliminary preprocessing, noise reduction, frame reduction and cutting transformation on the video;
nineteen, carrying out target detection on the collected video image frames by using a target detection algorithm (yolov 3, M2Det, CornerNet and the like) based on the pre-training model deployed in the step nine. Recognizing a human body, framing the detected human body, and outputting a corresponding target detection result;
twenty, based on the pre-training model deployed in the ninth step, a target detection algorithm (a model with a good small target effect) is utilized to detect human body parts and objects, the positions of human faces and hands are identified, whether other objects (mobile phones and the like) influencing driving exist or not is identified, and the position information of a target frame is obtained;
twenty one, calculating characteristic parameters based on the target detection information in the nineteen step and the twenty step, wherein the characteristic parameters are the same as those in the fifteenth step;
twenty-two, acquiring vehicle position, vehicle speed and vehicle direction information;
twenty-third, according to the set rule, judging whether the driver has the violation behaviors in the driving process and the abnormal behaviors in other specified ranges, and realizing the detection of the driving state of the driver by using the model trained in the sixteenth step.
Wherein, the data set labeling result in the step one should include:
A. target type, specific type of detected target;
B. target frame position information (the target detection result is a rectangular frame, the absolute position of pixels in the image, the number of pixels in the horizontal direction and the number of pixels in the vertical direction with the lower left corner of the image as the origin (x, y), the coordinates of the lower left corner (x1, y1) and the coordinates of the upper right corner (x2, y2) of the rectangular window of the target detection result).
The deployment of the target detection model in the ninth step comprises two modes:
A. the method comprises the steps that the method is deployed to a video acquisition end, namely the vehicle-mounted video acquisition equipment also comprises a target detection module and a corresponding target detection model is deployed;
B. and the method is deployed on a central server for driving behavior detection.
Wherein, the preprocessing of the twelve pairs of videos in the step further comprises:
A. carrying out intra-frame noise reduction and inter-frame noise reduction on the video image, and processing by using an image noise reduction algorithm;
B. based on the target detection device characteristics and the pre-trained model parameters, the video is re-cropped and transformed at a resolution selected to transform the resolution from the original resolution M × N to M1 × N1 (corresponding to the input picture size M1 × N1 of the output model in step 8).
Wherein, the human body characteristics output in the step thirteen comprise:
A. the absolute position of the pixel of the human body in the image takes the lower left corner of the image as an original point, and the number of horizontal pixels and the number of longitudinal pixels (x, y) relative to the original point;
B. the target detects the coordinates (x2, y2) of the lower left corner (x1, y1) and the upper right corner of the human body frame, and the human body frame pixel area = (y2-y1) × (x2-x 1).
Wherein, the video features output in the fourteenth step include:
A. the absolute position of the pixel of the target in the image, taking the lower left corner of the image as an origin, and the number of horizontal pixels and the number of longitudinal pixels (x, y) relative to the origin;
B. the coordinates (x2, y2) of the lower left corner (x1, y1) and the upper right corner of the target frame are detected by the target, and the pixel area = (y2-y1) × (x2-x1) of the target frame is detected;
C. the type of the target, the type output in this step includes: human face, hand and abnormal invaders (the object type depends on the data type of the input picture of the target detection pre-training model, and the current main detection objects comprise a mobile phone and a non-driver face).
Wherein the features calculated in step fifteen include:
A. based on target detection, identifying single-frame features of the main key point parameters, including:
A1. the center position of the driver face, the target type in the third step is the face, and the target type is compared with the driver face library to pass.
A2. Effective area of driver face, pixel area based on target type in step three as face
A3. The center position of the driver's hand and the target type in the third step are the center position of the hand
A4. The type of the abnormal invader is the type of the target in the third step, namely the mobile phone and the non-driver face.
A5. And (4) detecting the central coordinates of the abnormal invader by the target in the step three.
D. Based on the above a, i.e. single frame features, the feature variables within a certain time window are calculated.
B1. t1 time window (f 1 frames), driver face center position B1 image coordinate variation standard deviation, maximum displacement;
B1. t1 time window (f 1 frames), driver face effective area B2, maximum area, minimum area, standard deviation;
B3. t1 time window (f 1 frames), missing the face detection frame number ratio.
Wherein the driver driving behavior detection classification detected in the sixteenth step includes:
A. the current driving state of the driver is normal driving
B. Abnormal invaders appear in the images in the driving process of the driver and are mobile phones, and the center distance is smaller than the threshold value, and the mobile phones are operated in the driving process through judgment of the self-learning machine learning model;
C. abnormal invaders appear in the images in the driving process of the driver, and the abnormal invaders are other faces, objects or hands, and the like, and the center distance is smaller than the threshold value, and the abnormal invaders are judged to be interfered in the process of the journey through the self-learning machine learning model;
D. the driver loses detection within a certain time window and exceeds a threshold value, and the suspected attention of the driver is not focused or the driver is out of driving;
the video preprocessing in the eighteenth step further comprises:
A. the method carries out intra-frame noise reduction and inter-frame noise reduction on the video image, and is suitable for processing by an image noise reduction algorithm
B. And sampling the video frame according to the performance of the subsequent target detection equipment and the actual transmission bandwidth, and reducing the video frame rate to be the original frame rate/N (N > 1').
C. And according to the target detection device performance and the parameters of the pre-training model, selecting proper resolution to re-crop and transform the video, so that the resolution is transformed from the original resolution m × n to m1 × n 1.
Wherein, the judgment process for realizing the driving behavior posture of the driver in the twenty-third step is as follows:
A. and detecting the position of the face of the driver based on a pre-training model, comparing the position with a driver face database, and if the face recognition exceeds an abnormal value, determining that the driver is not authenticated and on duty.
B. And twenty-two steps of calculating to obtain characteristics (characteristics in the synchronization step fifteen, A1, the center position of the face of the driver, A2 effective area of the face of the driver and A3 center position of the hand of the driver) meeting a certain threshold range, or judging by a self-learning machine learning model (decision tree) that the driver is in the normal driving operation process.
C. When the driver is in the abnormal driving process, the abnormal driving type of the driver is detected by using the abnormal driving detection model trained in step 16 based on the features (the features in step fifteen, a1. driver face center position, a2 driver face effective area, A3 driver hand center position, a4. abnormal intruding object type, a5. abnormal intruding object position) and the vehicle driving features (vehicle position, vehicle speed, vehicle direction information) acquired in step 22.
Wherein, the abnormal driving behavior detectable in the twenty-third step includes:
A. the abnormal invader is a mobile phone, and the center distance is smaller than the threshold value, and the mobile phone is operated in the driving process after the judgment of the self-learning machine learning model;
B. the abnormal invaders are other faces, objects or hands and the like, and the distance between the centers is smaller than the threshold value, and the abnormal invaders are interfered in the process of travel after being judged by the self-learning machine learning model;
C. the driver loses detection within a certain time window and exceeds a threshold value, and the suspected attention of the driver is not focused or the driver is out of driving;
D. other suspected abnormal driving behaviors.
The invention is to realize the monitoring record and quick response of driver driving abnormity, a whole set of monitoring, recording and analyzing system is established based on camera data, target detection and abnormity analyzing data, the driver driving abnormity detecting system comprises:
1. the vehicle-mounted camera and the video recording, storing and returning device are arranged in front of a driving position of a driver, cover most of human bodies and all human faces of the driver, and have the functions of video data acquisition, storage and data returning.
2. Driver drives video image processing and target detection equipment, divide into two kinds:
a) the camera in the system 1 collects and transmits video data back to the central server in real time for target detection, and transmits a target detection result and sampling frame data to the driver driving behavior detection server;
b) the device is deployed at a video acquisition end, namely the device and the vehicle-mounted camera are deployed together, video data are read and target detection is carried out, and a target detection result and sampling frame data are transmitted back to the driver driving behavior detection server.
3. And the driver driving behavior detection server calculates characteristics based on the target detection result and the vehicle information to perform behavior detection.
4. And the data storage and web page server is responsible for recording and storing various service data in the system and providing web services.
The main work flow of the driver driving abnormity detection system comprises the following steps:
1. and installing a video monitoring system at the vehicle end, acquiring video monitoring data at a driver end in the vehicle, and recording and storing the video monitoring data.
2. And (3) carrying out target detection by using vehicle video data based on a pre-trained target monitoring model and returning data to the central server. There are two ways:
a) a video acquisition end simultaneously performs video acquisition and target detection, and returns a target detection result and sampled video frame data.
b) And returning video acquisition data in real time, and performing target detection and feature calculation and storage at the server side.
3. The driving behavior detection is carried out based on the abnormal driving state detection method, and the basic information of the abnormal driving behavior is recorded, wherein the method comprises the following steps:
c) video acquisition information, time, frame rate, resolution, and the like;
d) vehicle information, license plate number, serial number, vehicle type, etc.;
e) driving state information, current route, speed, location, etc.;
f) driver information, driver registration information, current driver face data;
g) abnormal driving state type, abnormal driving:
i. the detection of the face target of the driver fails, and the suspected attention is not concentrated;
ii, the driver leaves the driving post;
iii, receiving and making calls to operate a mobile phone and the like in the driving process;
iv, the driving process is interfered by others, if the interferers are detected, the image is divided and stored
Other abnormal driving behavior.
4. And according to a preset alarm model and a preset response mechanism, alarming the abnormal driving state and informing a corresponding treating party.
5. And returning the vehicle-mounted video information (in a wired mode, a wireless mode, a cellular network mode, a storage medium mode and the like), extracting corresponding abnormal driving video clips, and performing manual confirmation and rechecking.
6. And training a driving behavior abnormity detection model in an incremental mode based on the abnormal driving data which is manually rechecked.
In order to improve the efficiency of abnormal driving behavior detection, the recognition accuracy and the like, a target detection model and a model for classifying driving behaviors based on target detection extraction characteristics are required to be trained and are based on rechecking data increment training, and the system comprises two parts: the system comprises a target detection model pre-training part and a driving abnormity detection model training part.
The pre-training part of the target detection model in the system comprises four steps of data acquisition and labeling, model off-line training and optimized cutting, and on-line model deployment and prediction:
the data acquisition and labeling stage comprises:
A1. and collecting common target detection training data sets including human bodies, human faces, hands and common invaders (mobile phones and the like) in a driving scene.
A2. For carrier vehicle driver detection, driver face data, including different angle, pose data, is collected under authorization.
A3. And preprocessing the collected target detection training data, including but not limited to noise reduction, resampling and the like, and carrying out manual labeling.
A4. And sorting the original picture and the labeling information to be used as a labeling data set.
The model off-line training and optimizing cutting stage comprises the following steps:
B1. preprocessing the data acquisition and labeling stage, reasonably multiplying a part of data set to be used as a model training data set;
B2. dividing a training set and a verification set of a model training data set according to a certain proportion (example 9: 1)
B3. And selecting an appropriate deep learning frame (Tensorflow, Pyorch and the like) according to the training & predicting platform and the model characteristics, and selecting an applicable target detection network structure (Yolov 3, CornerNet, M2Det and the like) according to the type and characteristics (size, range and the like) of the detected target as required.
B4. And performing iterative training on the target detection model by using the training set, and evaluating by using the verification set to prevent overfitting and gradient divergence.
B5. Model clipping by deep learning model pruning algorithm
The online model deployment and prediction comprises:
C1. deploying a trained target detection model, which comprises two modes: c1a, calculating edges, and deploying the edges to a collection equipment terminal; c1b, central server cluster intensively deployed in driver driving behavior detection system
C2. And preprocessing the collected video, and extracting image data according to frames to perform target detection on-line prediction.
Further, the model pruning in the step B5 includes: and (4) carrying out redundancy deletion on the trunk network, reducing the size of an input network, and carrying out convolutional layer cutting and channel pruning.
In order to optimize a driving abnormity detection system in time and train an abnormity detection model by using production data and abnormal driving rechecking result increment, a driver driving abnormity detection model training part in the system comprises the following main processes:
1. labeling based on current abnormal driver driving video segment data
2. Performing feature extraction based on target detection data, comprising:
1.1 static characteristics
a) The center position of the face of the driver;
b) the effective area of the driver face;
c) the center position of the driver's hand;
d) an abnormal invader type;
e) abnormal invader sites.
f) Other features
1.2 cumulative timing characteristics over a period of time
a) t1 time window (f 1 frames), driver face center position B1 image coordinate variation standard deviation, maximum displacement;
b) t1 time window (f 1 frames), driver face effective area B2, maximum area, minimum area, standard deviation;
c) t1 time window (f 1 frames), losing the face detection frame number proportion;
d) other timing characteristics.
1.3 vehicle Driving features
a) A vehicle position;
b) vehicle speed;
c) vehicle direction information.
3. And selecting a proper classifier model (decision tree and the like), and performing model pre-training by combining the labeled result.
4. And carrying out abnormal driving behavior detection and classification detection on line.
5. And combining the rechecking data with the corresponding features to perform classifier retraining.
The invention utilizes the target detection algorithm to extract the characteristics and combines the machine learning algorithm to carry out the classified recognition of the driving behaviors of the driver so as to realize the real-time comprehensive monitoring of the driving conditions of the driver and the passenger and the vehicle and solve the problems of large workload, slow response and the like of manual analysis of the existing monitoring means.

Claims (9)

1. A driver driving abnormity detection method based on computer vision comprises three parts, namely a target detection model pre-training part, a driving abnormity detection model training part and a driving abnormity detection part;
the main process of pre-training the target detection model is as follows:
firstly, collecting and labeling a target detection data set: collecting training data sets of common target detection objects under a driving scene and a general scene, wherein the training data sets comprise human bodies, human faces, hands and common invaders, supplementing the training data sets of the target detection objects under the specific scene according to requirements, and labeling the training data sets according to the classification of the human bodies, the human faces, the hands and the common invaders; the complete data set should include: the method comprises the steps of detecting picture data and target detection labeling results of human bodies, human faces, hands and common invaders of a target object; the target detection and marking result comprises a target type and an image pixel position of the target; saving the labeling result of the data set;
preprocessing a target detection data set, namely preprocessing the collected target detection training data, wherein the preprocessing comprises the steps of not limiting to noise reduction and resampling, and preprocessing the picture size M N in the target detection to M1N 1;
thirdly, carrying out physical multiplication on a target detection data set to serve as a model training data set, and dividing the training set and a verification set according to a certain proportion, wherein the training set is larger than the verification set;
selecting a proper deep learning frame according to the target detection effect and the hardware equipment support of model training and online prediction;
selecting an applicable target detection network structure according to the type and the characteristics of the target to be detected;
step six, carrying out iterative training on the target detection model by using the training set, and evaluating by using the verification set to prevent overfitting and gradient divergence, when the target detection effect of the neural network model on the verification set does not become good any more or reaches the preset iterative training times, stopping training, and exporting the trained model;
seventhly, performing model cutting by using a deep learning model pruning algorithm, performing redundancy deletion on the trunk network, and reducing the size of an input network, wherein the model cutting also comprises convolutional layer cutting and channel pruning;
step eight, outputting the trained target detection model, wherein the size of an input picture of a target detection algorithm of the model is M1 × N1, and the output result is the target type and the position of a target detection frame;
step nine, transmitting, deploying and storing the trained target detection model to a video acquisition terminal or a driving abnormity detection center server to serve as a model required for subsequent target detection;
the driving abnormity detection model training process comprises the following steps:
step ten, collecting driver driving video data, wherein the video data is the video data shot by the driver
Eleven, marking the collected driver driving video data, wherein the marking information comprises: normal driving and abnormal driving, and corresponding abnormal driving types;
step twelve, performing preliminary preprocessing, noise reduction, frame reduction and cutting transformation on the video;
thirteen, based on the trained target detection model, carrying out target detection on the video image frames by using a target detection algorithm, identifying human bodies, framing the detected human bodies, and outputting corresponding characteristics;
fourteen, based on a trained target detection model, using a target detection algorithm to detect human body parts and objects, identifying the positions of human faces and hands, and identifying whether other objects influencing driving exist;
fifteenth, calculating characteristic parameters based on the target detection information in the thirteenth and fourteenth steps;
sixthly, selecting a proper classifier model, and combining the characteristics calculated in the fifteenth step with the labeling result in the eleventh step to train a driving abnormity detection model;
seventhly, deploying the driving abnormity detection model to a central server for detection;
after model training is completed, the driving behavior of a driver is detected by using a target detection model and a driving abnormity detection model, and the method comprises the following steps:
eighteen, mounting a vehicle-mounted camera and video recording, storing and returning equipment at a vehicle end, wherein the vehicle-mounted camera is connected with the video recording, storing and returning equipment; the detection center is also provided with a central video image processing and target detection device which is connected with a detection server, and the detection server is connected with the video recording, storing and returning equipment through a network;
the vehicle-mounted camera is used for acquiring video monitoring data of a driving position of a vehicle, transmitting the video monitoring data to video recording, storing and returning equipment for data storage and transmitting the stored data to a data storage server of a detection center;
nineteenth, the detection server connected with the data storage server performs preliminary preprocessing, noise reduction, frame dropping and cutting transformation on the video monitoring data;
twenty, carrying out target detection on the collected video image frames by using a target detection algorithm based on the trained target detection model, identifying a human body, and framing the detected human body;
twenty one, based on a trained target detection model, detecting human body parts and detecting objects through a central video image processing and target detection device by using a target detection algorithm, identifying the positions of human faces and hands, and identifying whether other target detection information influencing driving objects exists or not;
twenty-two, calculating by taking the target detection information as a calculated identification key point parameter to obtain a target detection result value;
twenty-third, according to the set rule, judging whether the driver has violation behaviors in the driving process and other abnormal behaviors in a specified range to obtain the detection result of the driving state of the driver;
and twenty-four steps of transmitting and storing the detection result to a database server for recording abnormal driving data of the driver.
2. The computer vision-based driver driving abnormality detection method according to claim 1, wherein the selected deep learning framework of the step four is Pytorch or tensoflow; selecting an applicable target detection network structure as YOLOv3, M2Det or CornerNet; sixthly, selecting a proper classifier model for driving abnormity detection, wherein the classifier model comprises a random forest or gradient descent decision tree.
3. The method for detecting abnormal driving of a driver based on computer vision as claimed in claim 1 or 2, wherein the step eighteen is that the vehicle-mounted camera and the video recording, storing and returning device are further connected with a vehicle-mounted video image processing and target detecting device, the vehicle-mounted video image processing and target detecting device detects the video image transmitted back from the vehicle-mounted camera to the video recording, storing and returning device and transmits the detection result back to the video recording, storing and returning device for storing, and the video recording, storing and returning device transmits the video monitoring data recorded by the vehicle-mounted camera to the data storage server for storage in real time and also transmits the detection result to the data storage server for storage in real time.
4. The method as claimed in claim 3, wherein the real-time detection process of the vehicle-mounted video image processing and target detection device comprises the following steps:
a. performing preliminary preprocessing, noise reduction, frame dropping and cutting transformation on video monitoring data;
b. performing target detection on the acquired video image frames by using a target detection algorithm based on a trained target detection model deployed to a video acquisition end, identifying a human body, and framing the detected human body;
c. detecting human body parts and detecting objects by using a target detection algorithm based on a trained target detection model deployed to a video acquisition end through vehicle-mounted video image processing and target detection equipment, identifying the positions of human faces and hands, identifying whether other target detection information influencing driving objects exists or not, and outputting the target detection information to video recording, storing and returning equipment;
d. the vehicle-mounted video image processing and target detection equipment calculates target detection information as a calculated identification key point parameter to obtain a target detection result value;
e. judging whether a driver has violation behaviors in the driving process and other abnormal behaviors in a specified range according to a set rule to obtain a detection result of the driving state of the driver;
f. and transmitting the detection result to the video recording storage returning device.
5. The method for detecting abnormal driving of a driver based on computer vision as claimed in claim 1, wherein, the step twelve is to perform preliminary preprocessing, noise reduction, frame dropping and clipping transformation on the video;
a. carrying out intra-frame noise reduction and inter-frame noise reduction on the video image, and processing by using an image noise reduction algorithm;
b. sampling video frames according to the performance of subsequent target detection equipment and actual transmission bandwidth, and reducing the video frame rate to be the original frame rate/H, wherein H is an integer greater than 1;
c. and according to the performance of the target detection equipment and based on a trained target detection model, selecting proper resolution to re-crop and transform the video, so that the resolution is transformed from the original size M × N of the video to M1 × N1.
6. The method for detecting abnormal driving of a driver based on computer vision as claimed in claim 1, wherein said twenty steps of detecting the target of the collected video image frames by using the target detection algorithm based on the trained target detection model, recognizing the human body, and framing the detected human body; in particular to a method for preparing a high-performance nano-silver alloy,
a. the absolute position of the pixel of the target in the image takes the lower left corner of the image as an origin (0,0), and the number of horizontal pixels and the number of vertical pixels (x, y) relative to the origin;
b. the coordinates (x2, y2) of the lower left corner (x1, y1) and the upper right corner of the target frame are detected by the target, and the pixel area = (y2-y1) × (x2-x1) of the target frame is detected;
c. the object type of the target depends on the data type of the input picture of the trained target detection model, and the detected object mainly comprises a mobile phone and a non-driver face.
7. The method as claimed in claim 1, wherein the fifteen characteristic parameters mainly include,
a. the face, the center position of the driver face, and the comparison with the driver face library is passed;
b. the effective area of the face of the driver, and the target type is the pixel area of the face; c. the center of the hand of the driver is positioned,
d. the abnormal invader types mainly comprise mobile phones and non-driver faces;
e. abnormal invader position, abnormal invader center coordinate;
f. calculating the change characteristics of the characteristic variables in a certain time window;
g. time window, standard deviation of variation of image coordinates of the center position B1 of the face of the driver, and maximum displacement;
h. time window, driver face effective area B2, maximum area, minimum area, standard deviation;
j. time window, missing the face detection frame number proportion.
8. The computer vision-based driver driving abnormality detection method according to claim 1, wherein the step twenty-three and other prescribed ranges of abnormal behaviors further include,
a. whether the driver drives normally;
b. abnormal driving behavior of the driver: a) the driver's own attention is distracted; b) the driver is disturbed by the persons in the vehicle.
9. The computer vision-based driver driving abnormality detection method according to claim 1, wherein the features calculated in step fifteen include:
A. based on target detection, identifying single-frame features of the main key point parameters, including:
1) the center position of the driver face, the target type in the step fourteen is the face, and the target type is compared with a driver face library to pass;
2) the effective area of the driver face is the pixel area of the face based on the target type in the fourteenth step;
3) the center position of the driver's hand, the target type is the center position of the hand in the fourteenth step;
4) the type of the abnormal invader is that the target type in the fourteenth step is a mobile phone or a non-driver face;
5) the position of the abnormal invader, the central coordinate of the abnormal invader is detected by the target in the step fourteen;
B. based on the above a, i.e. single frame features, the feature variables within a certain time window are calculated:
1) in a time window with the time length of t1, the coordinate of the center position of the driver face B1 image changes standard deviation and maximum displacement;
2) in a time window with the time length of t1, the effective area of the driver face is B2, the maximum area, the minimum area and the standard deviation;
3) and in a time window with the time length of t1, the proportion of the number of the face detection frames is lost.
CN202010071329.2A 2020-01-21 2020-01-21 Driver driving abnormity detection method based on computer vision Active CN111325872B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010071329.2A CN111325872B (en) 2020-01-21 2020-01-21 Driver driving abnormity detection method based on computer vision

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010071329.2A CN111325872B (en) 2020-01-21 2020-01-21 Driver driving abnormity detection method based on computer vision

Publications (2)

Publication Number Publication Date
CN111325872A CN111325872A (en) 2020-06-23
CN111325872B true CN111325872B (en) 2021-03-16

Family

ID=71167223

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010071329.2A Active CN111325872B (en) 2020-01-21 2020-01-21 Driver driving abnormity detection method based on computer vision

Country Status (1)

Country Link
CN (1) CN111325872B (en)

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111765904B (en) * 2020-06-29 2023-12-15 北京百度网讯科技有限公司 Test method and device for automatic driving vehicle, electronic equipment and medium
CN112149511A (en) * 2020-08-27 2020-12-29 深圳市点创科技有限公司 Method, terminal and device for detecting violation of driver based on neural network
CN112070051B (en) * 2020-09-16 2022-09-20 华东交通大学 Pruning compression-based fatigue driving rapid detection method
CN112288280A (en) * 2020-10-30 2021-01-29 大陆投资(中国)有限公司 System and method for evaluating driver providing travel service
CN112347906B (en) * 2020-11-04 2023-06-27 北方工业大学 Method for detecting abnormal aggregation behavior in bus
CN112434564B (en) * 2020-11-04 2023-06-27 北方工业大学 Detection system for abnormal aggregation behavior in bus
CN112782090B (en) * 2020-12-28 2022-01-28 中国科学院长春光学精密机械与物理研究所 Drunk driving automatic monitoring system and detection method
CN112906515A (en) * 2021-02-03 2021-06-04 珠海研果科技有限公司 In-vehicle abnormal behavior identification method and system, electronic device and storage medium
CN115512511A (en) * 2021-06-07 2022-12-23 中移物联网有限公司 Early warning method, early warning device, mobile terminal and readable storage medium
CN113518205A (en) * 2021-06-11 2021-10-19 南京和贤电子科技有限公司 Video patrol processing method based on AI analysis
CN114756211B (en) * 2022-05-13 2022-12-16 北京百度网讯科技有限公司 Model training method and device, electronic equipment and storage medium
CN115565354B (en) * 2022-11-18 2023-03-10 深圳智者行天下科技有限公司 Commercial vehicle driver safety monitoring system based on artificial intelligence
CN116309351B (en) * 2023-02-15 2023-11-21 浙江丽威汽车控制系统有限公司 Automobile engineering material supply processing system
CN116894225B (en) * 2023-09-08 2024-03-01 国汽(北京)智能网联汽车研究院有限公司 Driving behavior abnormality analysis method, device, equipment and medium thereof
CN117076711A (en) * 2023-10-12 2023-11-17 北京汇通天下物联科技有限公司 Training method, recognition method, device and equipment for driving behavior recognition model

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108423006A (en) * 2018-02-02 2018-08-21 辽宁友邦网络科技有限公司 A kind of auxiliary driving warning method and system
CN108875595A (en) * 2018-05-29 2018-11-23 重庆大学 A kind of Driving Scene object detection method merged based on deep learning and multilayer feature
US10255528B1 (en) * 2017-12-06 2019-04-09 Lytx, Inc. Sensor fusion for lane departure behavior detection
CN110147738A (en) * 2019-04-29 2019-08-20 中国人民解放军海军特色医学中心 A kind of driver fatigue monitoring and pre-alarming method and system
CN110641521A (en) * 2019-09-21 2020-01-03 河南蓝信科技有限责任公司 Intelligent recognition system for driver behaviors of motor train unit

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10255528B1 (en) * 2017-12-06 2019-04-09 Lytx, Inc. Sensor fusion for lane departure behavior detection
CN108423006A (en) * 2018-02-02 2018-08-21 辽宁友邦网络科技有限公司 A kind of auxiliary driving warning method and system
CN108875595A (en) * 2018-05-29 2018-11-23 重庆大学 A kind of Driving Scene object detection method merged based on deep learning and multilayer feature
CN110147738A (en) * 2019-04-29 2019-08-20 中国人民解放军海军特色医学中心 A kind of driver fatigue monitoring and pre-alarming method and system
CN110641521A (en) * 2019-09-21 2020-01-03 河南蓝信科技有限责任公司 Intelligent recognition system for driver behaviors of motor train unit

Also Published As

Publication number Publication date
CN111325872A (en) 2020-06-23

Similar Documents

Publication Publication Date Title
CN111325872B (en) Driver driving abnormity detection method based on computer vision
US9881221B2 (en) Method and system for estimating gaze direction of vehicle drivers
US8339282B2 (en) Security systems
WO2020042984A1 (en) Vehicle behavior detection method and apparatus
CN110895662A (en) Vehicle overload alarm method and device, electronic equipment and storage medium
CN103366506A (en) Device and method for automatically monitoring telephone call behavior of driver when driving
CN110222596B (en) Driver behavior analysis anti-cheating method based on vision
Nakashima et al. Passenger counter based on random forest regressor using drive recorder and sensors in buses
CN112434566B (en) Passenger flow statistics method and device, electronic equipment and storage medium
CN110838230B (en) Mobile video monitoring method, monitoring center and system
CN106570444A (en) On-board smart prompting method and system based on behavior identification
CN109919066B (en) Method and device for detecting density abnormality of passengers in rail transit carriage
CN105448105A (en) Patrol police vehicle-based monitoring system
CN112597965A (en) Driving behavior recognition method and device and computer readable storage medium
CN114898297A (en) Non-motor vehicle illegal behavior determination method based on target detection and target tracking
CN115600124A (en) Subway tunnel inspection system and inspection method
CN111985295A (en) Electric bicycle behavior recognition method and system, industrial personal computer and camera
CN114926824A (en) Method for judging bad driving behavior
CN111950499A (en) Method for detecting vehicle-mounted personnel statistical information
CN110909641A (en) Method, device and system for detecting overload of motorcycle
CN114120250B (en) Video-based motor vehicle illegal manned detection method
CN105206060A (en) Vehicle type recognition device and method based on SIFT characteristics
CN112509190B (en) Subway vehicle section passenger flow statistical method based on shielded gate passenger flow counting
CN113989787A (en) Detection method and system for dangerous driving behaviors
KR102479516B1 (en) Multi object detection system using deep running based on closed circuit television image

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant