CN109409173B - Driver state monitoring method, system, medium and equipment based on deep learning - Google Patents
Driver state monitoring method, system, medium and equipment based on deep learning Download PDFInfo
- Publication number
- CN109409173B CN109409173B CN201710716216.1A CN201710716216A CN109409173B CN 109409173 B CN109409173 B CN 109409173B CN 201710716216 A CN201710716216 A CN 201710716216A CN 109409173 B CN109409173 B CN 109409173B
- Authority
- CN
- China
- Prior art keywords
- information
- module
- queue
- video
- video data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/172—Classification, e.g. identification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Multimedia (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
Driver state monitoring methods, systems, media and devices based on deep learning, comprising: initializing image information acquisition equipment and storage equipment, presetting information processing logic, finishing initialization operation and sending prompt information; receiving prompt information, triggering characteristic information detection equipment and image information acquisition equipment to acquire video data of a driver, and extracting single-frame picture information from the video data to obtain a video picture sample; extracting face orientation feature information and attention point angle information in current single-frame picture information, constructing a state detection model according to the face orientation feature information and attention point angle information, performing state detection model deep learning according to a video picture sample, and processing the face orientation feature information and attention point angle information according to preset processing logic to obtain face attention detection information; extracting sign information; and storing the single-frame picture information and the face attention detection information into the detection information and the video data into a queue, and generating and storing driving state judgment log information according to the queue.
Description
Technical Field
The invention relates to a subject three-examination driver monitoring system, in particular to a driver state monitoring method, system, medium and device based on deep learning.
Background
With the advance of time, the number of Chinese drivers is continuously increased, and in addition, in the traditional technology, the driving test of a driving school is monitored by a simple electronic detection reminding device and a system through manual cooperation of a coach, so that the training efficiency of the drivers in the driving school is low, and the learning quality of the driving skills of the drivers cannot be guaranteed, therefore, the problem that the driving skill training resources are increasingly tense is further highlighted along with the non-ideal working efficiency and effect of the education and training of the drivers. Because in the process of daily motor vehicle acquaintance examination detection, an important function requirement is required in the examination detection process of a driver during the state detection of the driver, most examination errors of the driver are closely related to the state change state of an examinee, and in the traditional technology, a coach in a driving school is parallel to the side of the examinee, so that the state attention direction of the examinee cannot be accurately detected.
At present, the following methods are mainly used for detecting drivers: based on the detection of the sensor, the method is mainly based on a wearable sensor, measures acceleration information or angular velocity information of each part of the body of the driver in real time, and then detects the behavior state of the driver according to the measured information. The method has the defects that the wearable sensor needs to be carried about, the equipment cost is high, and the use is very inconvenient. Another type of technology is based primarily on video image analysis detection methods, by directly extracting image features and detecting data. The method has the defects that background modeling is not accurate, the detection error of the extracted feature data is large, more false detection and missing detection are caused, and the feature robustness is low.
The wearable sensor needs to be carried about in the prior art, the equipment cost is high, the use is inconvenient, the detection value error is large, the false detection and the missed detection are more, the hardware cost is high, the feature robustness is low, the information utilization rate is low, and the detection result accuracy is low.
Disclosure of Invention
In view of the technical problems of high hardware cost, low feature robustness, low information utilization rate and low detection result accuracy in the prior art, the invention aims to provide a driver state monitoring method, a driver state monitoring system, a driver state monitoring medium and driver state monitoring equipment based on deep learning. The driver state monitoring method based on deep learning solves the technical problems of high hardware cost, weaker algorithm robustness, low information utilization rate and low accuracy of the facial action monitoring result in the prior art,
to achieve the above and other related objects, the present invention provides a driver condition monitoring method based on deep learning, comprising: initializing image information acquisition equipment and storage equipment, presetting information processing logic, finishing initialization operation and sending prompt information; receiving prompt information, triggering characteristic information detection equipment and image information acquisition equipment to acquire video data of a driver, and extracting single-frame picture information from the video data to obtain a video picture sample; extracting face orientation feature information and attention point angle information in current single-frame picture information, constructing a state detection model according to the face orientation feature information and attention point angle information, performing state detection model deep learning according to a video picture sample, and processing the face orientation feature information and attention point angle information according to preset processing logic to obtain face attention detection information; and storing the single-frame picture information and the face attention detection information into the detection information and the video data into a queue, and generating and storing driving state judgment log information according to the queue.
In an embodiment of the present invention, receiving a prompt message, triggering a system to acquire video data according to the prompt message, extracting single-frame picture information from the video data, and storing the single-frame picture information as a picture sample includes: receiving prompt information, and starting a camera according to the prompt information; acquiring video data of a driver in real time by using a camera; reading video data; extracting current single-frame picture information according to the video data and time; and extracting the single-frame picture information in the queue to obtain a video picture sample.
In an embodiment of the present invention, extracting face orientation feature information and attention point angle information in current single-frame picture information, constructing a state detection model according to the face orientation feature information and attention point angle information, performing deep learning of the state detection model according to a video picture sample, and processing the face orientation feature information and attention point angle information according to preset processing logic to obtain face attention detection information, including: extracting characteristic data in single-frame picture information; normalizing the feature data to obtain a feature vector; constructing a state detection model according to the feature vectors; performing deep learning according to the video picture sample, and updating the video picture sample; comparing the feature vector with eight motion features of observing a left B column, a left rearview mirror, an inner rearview mirror, a right B column of an overlooking instrument panel, a right rearview mirror, a front view, a head-down view and the like in the video picture sample to obtain similar information; the similar information is sorted into face attention detection information.
In one embodiment of the present invention, storing single frame picture information, face attention detection information, and video data in a queue, generating and storing driving state determination log information according to the queue, includes: extracting video data, single-frame picture information and face attention detection information; storing the video data into an image acquisition buffer queue; storing the single-frame picture information into a single-frame picture cache queue; storing the face attention detection information into an algorithm output queue; and generating driving state judgment log information of the driver according to the picture cache queue and the algorithm output queue and storing the driving state judgment log information in a log library.
In one embodiment of the present invention, a driver state monitoring system based on deep learning includes: the system comprises a system preparation module, a video picture sample module, a state detection module and a queue storage module; the system preparation module is used for initializing the image information acquisition equipment and the storage equipment, presetting information processing logic, finishing initialization operation and sending prompt information; the video picture sample module is used for receiving the prompt information, triggering the characteristic information detection equipment and the image information acquisition equipment to acquire the video data of the driver, extracting single-frame picture information from the video data to obtain a video picture sample, and is connected with the system preparation module; the state detection module is used for extracting face orientation feature information and attention point angle information in current single-frame picture information, constructing a state detection model according to the face orientation feature information and attention point angle information, performing state detection model deep learning according to a video picture sample, and processing the face orientation feature information and attention point angle information according to preset processing logic to obtain face attention detection information, and the state detection module is connected with the video picture sample module; the queue storage module is used for storing the single-frame picture information and the face attention detection information into the detection information and the video data into the queue, generating and storing driving state judgment log information according to the queue, and is connected with the state detection module and the characteristic information module.
In an embodiment of the present invention, a video picture sample module includes: the device comprises a camera opening module, a video acquisition module, a video reading module, a single-frame extraction module and a sample generation module; the camera opening module is used for receiving the prompt information and opening the camera according to the prompt information; the video acquisition module is used for acquiring video data of the driver in real time by using the camera and is connected with the camera opening module; the single-frame extraction module is used for extracting the current single-frame picture information according to the video data and time, and is connected with the video reading module; and the sample generation module is used for extracting the single-frame picture information in the queue to obtain a video picture sample, and is connected with the single-frame extraction module.
In one embodiment of the present invention, the state detection module includes: the system comprises a feature extraction module, a feature vector module, a model construction module, a model adjustment module, a feature comparison module and a state data calculation module; the characteristic extraction module is used for extracting characteristic data in the single-frame picture information; the characteristic vector module is used for normalizing the characteristic data to obtain a characteristic vector, and is connected with the characteristic extraction module; the model building module is used for building a state detection model according to the characteristic vector and is connected with the characteristic vector module; the model adjusting module is used for carrying out deep learning according to the video picture samples and updating the video picture samples, and is connected with the model building module; the characteristic comparison module is used for comparing the characteristic vectors with the characteristic vectors and observing eight action characteristics such as a left B column, a left rearview mirror, an inside rearview mirror, a right B column of an overlooking instrument panel, a right rearview mirror, a front view, a head-down view and the like in the video picture sample to obtain similar information, and the characteristic comparison module is connected with the model adjusting module; and the similar information sequencing module is used for sequencing the similar information to obtain the face attention detection information, and the information sequencing module is connected with the feature comparison module.
In one embodiment of the present invention, a queue storage module includes: the system comprises a data extraction module, an image acquisition queue module, a single-frame cache module, an algorithm result module and a log module; the data extraction module is used for extracting video data, single-frame picture information and face attention detection information; the image acquisition queue module is used for storing the video data into an image acquisition cache queue, and the image acquisition queue module is connected with the data extraction module; the single-frame caching module is used for storing single-frame picture information into a single-frame picture caching queue and is connected with the data extraction module; the algorithm result module is used for storing the face attention detection information into an algorithm output queue and is connected with the data extraction module; and the log module is used for generating driving state judgment log information of the driver according to the picture cache queue and the algorithm output queue and storing the driving state judgment log information into a log library, is connected with the image acquisition queue module, is connected with the single-frame cache module, and is connected with the algorithm result module.
In an embodiment of the present invention, the present invention provides a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements the deep learning-based driver status monitoring method provided by the present invention.
In one embodiment, the present invention provides a driver state monitoring device based on deep learning, including: a processor and a memory; the memory is used for storing a computer program, and the processor is used for executing the computer program stored by the memory so as to enable the deep learning based driver state monitoring device to execute the deep learning based driver state monitoring method provided by the invention.
As described above, the method, system, medium, and apparatus for monitoring the state of a driver based on deep learning provided by the present invention have the following advantages: in order to realize the whole-process electronic monitoring and evaluation of the three-test of the driving subjects of the motor vehicle, a driving test visual tracking technical prototype extracts video data such as the posture of a driver through a vehicle-mounted camera, carries out computer visual algorithm processing including face detection, optical flow detection and the like by using tools such as a deep learning neural network and the like, completes the behavior analysis of detecting the attention point of the driver, whether the body extends out of the vehicle or not and the like, improves the objectivity and the accuracy of the three-test of the subjects, and reduces the labor cost.
In summary, the invention solves the technical problems of high hardware cost, weak feature robustness, low information utilization rate and low detection result accuracy in the prior art, each frame of image is checked, 15 frames per second, each frame has one result (one of eight actions) and is transmitted to a triple superior device (timestamp and state corresponding to the current image) after the whole examinee is finished, the whole examinee image and state data are packed into a compressed packet and are transmitted to the superior device, a sample is trained for face tracking, face action features are identified and used for judging actions, a head posture image acquired from a monitoring video is taken as a sample library, features do not need to be designed, the feature robustness is strong, and the actual detection accuracy is high.
Drawings
Fig. 1 shows a flowchart of an embodiment of a deep learning-based driver status monitoring method according to the present invention.
Fig. 2 is a flowchart illustrating step S2 in fig. 1 in an embodiment.
Fig. 3 is a flowchart illustrating step S3 in fig. 1 in an embodiment.
Fig. 4 is a flowchart illustrating step S5 in fig. 1 in an embodiment.
FIG. 5 is a schematic diagram of an embodiment of a deep learning-based driver status monitoring system module according to the invention.
Fig. 6 is a block diagram of the video picture sample block 12 in fig. 5 according to an embodiment.
Fig. 7 is a block diagram illustrating the state detection module 13 in fig. 5 according to an embodiment.
Fig. 8 is a block diagram of the data extraction module 15 in fig. 5 according to an embodiment.
Description of the element reference numerals
Driver state monitoring system based on deep learning
11 System preparation Module
12 video picture sample module
13 state detection module
14 queue storage module
121 camera opening module
122 video acquisition module
123 video reading module
124 single frame extraction module
125 sample generation module
131 characteristic extraction module
132 feature vector Module
133 model building module
134 model adjustment module
135 characteristic comparison module
136 status data calculation module
151 data extraction module
152 image acquisition queue module
153 single frame buffer module
154 algorithm result module
155 log module
Description of step designations
Method steps S1-S5
Method steps S21-S26
Method steps S31-S36
Method steps S51-S55
Detailed Description
The following description of the embodiments of the present invention is provided for illustrative purposes, and other advantages and effects of the present invention will become apparent to those skilled in the art from the present disclosure.
Referring to fig. 1 to 8, it should be understood that the structures shown in the drawings attached to the present specification are only used for understanding and reading the contents disclosed in the specification, and are not used to limit the conditions under which the present invention can be implemented, so that the present invention has no technical essence, and any modification of the structures, changes of the proportional relationship, or adjustment of the size should fall within the scope covered by the technical contents disclosed in the present invention without affecting the efficacy and achievable purpose of the present invention. In addition, the terms "upper", "lower", "left", "right", "middle" and "one" used in the present specification are for clarity of description, and are not intended to limit the scope of the present invention, and the relative relationship between the terms and the terms is not to be construed as a scope of the present invention.
Referring to fig. 1, a flowchart of an embodiment of a deep learning-based driver status monitoring method according to the present invention is shown, as shown in fig. 1, the method includes: a driver state monitoring method based on deep learning relates to a driver state monitoring method based on deep learning, and is characterized by comprising the following steps:
step S1, initializing image information acquisition equipment and storage equipment, presetting information processing logic, finishing initialization operation and sending prompt information, starting the system by pressing a system start button on a system main interface through a client terminal such as a control panel and a computer which are provided with the driver sight monitoring system, automatically carrying out installation detection and setting by the system, and initializing hardware equipment such as a camera, a storage disk and the like;
step S2, receiving prompt information, triggering characteristic information detection equipment and image information acquisition equipment to acquire video data of a driver, extracting single-frame picture information from the video data to obtain a video picture sample, receiving processing trigger information, triggering a system to acquire the video data according to the processing trigger information, extracting the single-frame picture information from the video data and storing the single-frame picture information as an image analysis sample, acquiring eye video data of the driver through a camera installed in a cab, storing the single-frame image information in the video data as the image analysis sample, and storing the video data in an SD card;
step S3, extracting face orientation feature information and attention point angle information in current single-frame picture information, constructing a state detection model according to the face orientation feature information and attention point angle information, performing state detection model deep learning according to a video picture sample, and processing the face orientation feature information and attention point angle information according to preset processing logic to obtain face attention detection information;
step S4, single frame picture information and face attention detection information are stored in a queue, driving state judgment log information is generated according to the queue and stored, sight line detection information obtained through deep neural network processing is converted into a data stream and stored in a response queue, and a driver sight line detection log is generated according to the sight line detection information and stored in a server side.
Please refer to fig. 2, which is a flowchart illustrating step S2 in fig. 1 in an embodiment, which specifically includes:
and S21, receiving the prompt information, starting the cameras according to the prompt information, starting a power supply of system hardware equipment through power-on operation of a main control interface by a user, clicking a cursor of the driver sight line detection system on a main interface of the mobile terminal to start, wherein the system hardware equipment mainly comprises a plurality of cameras arranged on the relative positions of a cab driver seat and a driver, and the driving test system equipment does not provide an operation interface when being deployed. And the system software sets software self-starting configuration under an Autostart catalog of the Ubuntu system during installation. The hardware is electrified, and when the Ubuntu system is started, a starting script is executed, and a driving test system program is automatically started;
step S22, acquiring video data of a driver in real time by using a camera, starting the camera to acquire the video data of the driver, and acquiring original USB camera video data from the camera;
step S23, reading video data, acquiring a video image of a driver in the driving process in real time by a camera through a photosensitive imaging element, and sending the video data acquired by shooting to an image processing logic in a data bus or wireless transmission mode;
step S24, extracting current single-frame picture information according to the video data and time, processing the video information by the driver sight line detection system according to a preset image processing logic to obtain a single-frame original size picture and a compressed format picture, preferably, framing the video data acquired by the camera according to a timestamp, using the generated single-frame picture for corresponding analysis of an image algorithm library, and compressing and storing the picture for report generation;
and S25, extracting the single-frame picture information in the queue to obtain a video picture sample, extracting the single-frame picture information from the image storage queue and aggregating the single-frame picture information into an image analysis sample, wherein the image analysis sample is used for training a deep neural network model.
Please refer to fig. 3, which is a flowchart illustrating step S3 in fig. 1 in an embodiment, which specifically includes:
step S31, extracting feature data in single-frame picture information, preprocessing a head sight image and a posture image to be detected, extracting a head local feature vector and a global head feature vector, and fusing to obtain a global feature vector;
step S32, normalizing the feature data to obtain a feature vector, extracting a local feature vector set from the processed head image data set, and then fusing the local feature vector set to obtain a head posture feature vector;
step S33, constructing a state detection model according to the feature vectors;
step S34, performing deep learning according to the video picture samples, updating the video picture samples, and preprocessing each head posture picture in the image analysis samples to obtain preprocessing information with the picture to be detected;
step S35, comparing the feature vector with eight motion features of observing a left B column, a left rearview mirror, an inside rearview mirror, a right B column of an overlooking instrument panel, a right rearview mirror, a front view, a low head view and the like in the video picture sample to obtain similar information, obtaining a sample global feature vector contained in the sample according to the to-be-detected picture preprocessing information of the image analysis sample, wherein the global feature vector contains the motion data in the bus, comparing the global feature information of the sample with the to-be-detected feature vector to obtain cosine similarity information, and presetting the global feature vector of the image analysis sample mainly aiming at the following subjects, namely three scenes:
scene one: before starting, the inside and outside rearview mirrors are not observed, and the traffic condition behind is observed by returning. Before starting, observing a left rearview mirror and a right rearview mirror: the head deflects 30 degrees to the left, and the driver does not observe the left rearview mirror; the head deflects more than 30 degrees to the left; when the inner rearview mirror is watched, the head deflects rightwards by more than 30 degrees, and the elevation angle on the head is more than 30 degrees; left rear, head deflects more than 60 degrees to the left, which determines violation.
Scene two: the line of sight is more than 2 seconds away from the direction of travel. During the running process of the vehicle, when the sight of the driver leaves the front and the duration time of the driver deviating to one side exceeds two seconds, the violation is judged.
Scene three: the head is lowered to look at the gear during driving. During driving, the head is lowered for more than 2 seconds, when the head is lowered for looking at the gear, the duration of the head which deflects to the right by more than 30 degrees is more than 2 seconds, the head lowering angle is more than 30 degrees, the duration is more than 2 seconds, and violation is judged.
Scene four: in the process of turning the vehicle, the road traffic condition is not observed through a left rearview mirror; after the left turn light is turned on, if the examinee does not observe the left rearview mirror, the head does not deflect left by 30-60 degrees, and violation is judged.
Scene five: in the process of turning the vehicle, the road traffic condition is not observed through the right rearview mirror; and after the examinee opens the right steering and the like, if the examinee does not observe the right rearview mirror, the head does not deflect 45-60 degrees to the right, and violation is judged.
Scene six: before lane changing, the road traffic condition is observed after the observation is carried out in the direction of lane changing without the observation of an inner rearview mirror and an outer rearview mirror; after a voice command of changing lanes is received, or within a certain time after a driver turns on a turn light, if the inside and outside rearview mirrors and the corresponding measured rear part are not observed, and the head deflection is more than 60 degrees, violation is judged.
Scene seven: before stopping, the traffic conditions of the rear and the right side are not observed through the inner rearview mirror and the outer rearview mirror, and in the process that the vehicle speed is reduced to 0 after the right turn lamp is turned on safely through observation and confirmation, if a driver does not observe the inner rearview mirror, the right rearview mirror and the right rear, violation is judged.
And eighth scene: the traffic condition of the left rear part and the right rear part is observed without returning before opening the door when the vehicle needs to get off; when the vehicle speed is 0, if the driver does not observe the left rear part before opening the vehicle door, the violation is judged;
and S36, sequencing the similar information to obtain face attention detection information.
Please refer to fig. 4, which is a flowchart illustrating step S5 in fig. 1 in an embodiment, which specifically includes:
s51, extracting video data, single-frame picture information and face attention detection information, and extracting the video data, the single-frame picture information and video monitoring information from the output end of the camera and the image data processing algorithm;
s52, storing the video data into an image acquisition cache queue, wherein the image acquisition cache queue is used for storing and storing image data processed by an image algorithm in image data storage and is used for summarizing and reporting after driving test is finished and backing up in a validity period, extracting single-frame image information from the image storage queue and aggregating the single-frame image information into an image analysis sample, and the image analysis sample is used for training a deep neural network model;
s53, storing the single-frame picture information into a single-frame picture buffer queue, wherein the single-frame picture buffer queue is mainly an input queue of a volume and depth network model algorithm, and the video data are queue elements of an image acquisition buffer queue and are suitable for being used as input data of an image data processing algorithm according to the sequence of the video data in the queue;
s54, storing the face attention detection information into an algorithm output queue, wherein the algorithm output queue is an image algorithm module processing result queue and is used for reporting to a superior device, and the algorithm output queue is suitable for being transmitted to a server side in an ascending mode according to the sequence of video monitoring information entering the queue for being checked by monitoring personnel;
and S55, generating driving state judgment log information of the driver according to the picture cache queue and the algorithm output queue, storing the driving state judgment log information into a log library, storing and saving a working log generated in the operation of the system by using a system log, managing the driving state judgment log information and sending the driving state judgment log information to a server side.
Referring to fig. 5, a schematic diagram of an embodiment of a deep learning-based driver status monitoring system according to the present invention is shown, wherein the system includes: the system comprises a system preparation module 11, a video picture sample module 12, a state detection module 13 and a queue storage module 14; the system preparation module 11 is used for initializing the image information acquisition device, the sign information detection device and the storage device, presetting information processing logic, finishing initialization operation and sending prompt information, starting the system by pressing a system start button on a system main interface through a client terminal such as a control panel and a computer which are provided with the driver sight monitoring system by a user, automatically carrying out installation detection and setting by the system, and initializing hardware devices such as a camera, a storage disk and the like; the video picture sample module 12 is used for receiving prompt information, triggering the characteristic information detection device and the image information acquisition device to acquire physical sign information and video data of a driver, extracting single-frame picture information from the video data to obtain a video picture sample, receiving processing trigger information, triggering the system to acquire the video data according to the processing trigger information, extracting the single-frame picture information from the video data and storing the single-frame picture information as an image analysis sample, acquiring eye video data of the driver through a camera installed in the driver cab, storing the single-frame image information in the video data as the image analysis sample, and storing the video data in an SD card, wherein the video picture sample module 12 is connected with the system preparation module 11; the state detection module 13 is configured to extract face orientation feature information and attention point angle information in current single-frame picture information, construct a state detection model according to the face orientation feature information and attention point angle information, perform state detection model deep learning according to a video picture sample, process the face orientation feature information and attention point angle information according to preset processing logic to obtain face attention detection information, and the state detection module 13 is connected to the video picture sample module 12; the queue storage module 14 is used for storing single-frame picture information and face attention detection information into a detection information and video data into a queue, generating driving state judgment log information according to the queue and storing the driving state judgment log information, converting sight line detection information obtained through deep neural network processing into a data stream and storing the data stream into a response queue, generating a driver sight line detection log according to the sight line detection information and storing the driver sight line detection log in a server side, and the queue storage module 14 is connected with the state detection module 13.
Please refer to fig. 6, which is a block diagram illustrating an embodiment of the video picture sample module 12 in fig. 5, which specifically includes: the system comprises a camera opening module 121, a video obtaining module 122, a video reading module 123, a single-frame extracting module 124 and a sample generating module 125; the camera opening module 121 is configured to receive the prompt information, open the camera according to the prompt information, enable the power supply of the system hardware device through power-on operation on the main control interface by the user, click a cursor of the driver sight detection system on the main interface of the mobile terminal to open, and provide no operation interface when the driving test system device is deployed. And the system software sets software self-starting configuration under an Autostart catalog of the Ubuntu system during installation. The hardware is electrified, and when the Ubuntu system is started, a starting script is executed, and a driving test system program is automatically started; the video acquisition module 122 is used for acquiring video data of a driver in real time by using a camera, starting the camera to acquire the video data of the driver, and acquiring original USB camera video data from the camera; the video reading module 123 is used for reading video data, the camera acquires a video image of the driver in the driving process in real time through the photosensitive imaging element, and sends the video data acquired by shooting to the image processing logic in a data bus or wireless transmission mode, and the video reading module 123 is connected with the video acquiring module 122; a single frame extraction module 124, configured to extract current single frame picture information according to video data and time, where the driver sight line detection system processes the video information according to a preset image processing logic to obtain a single frame original size picture and a compressed format picture, and preferably, performs framing processing on video data acquired by a camera according to a timestamp, uses the generated single frame picture in an image algorithm library for corresponding analysis, and compresses and stores the picture for report generation, and the single frame extraction module 124 is connected to the video reading module 123; the sample generating module 125 is configured to extract single-frame picture information in the queue to obtain a video picture sample, extract the single-frame picture information from the image storage queue and aggregate the single-frame picture information into an image analysis sample, where the image analysis sample is used to train a deep neural network model, and the sample generating module 125 is connected to the single-frame extracting module 124.
Referring to fig. 7, a schematic diagram of a specific module of the state detection module 13 in fig. 5 in an embodiment is shown, which specifically includes: the system comprises a feature extraction module 131, a feature vector module 132, a model construction module 133, a model adjustment module 134, a feature comparison module 135 and a state data calculation module 136; the feature extraction module 131 is configured to extract feature data in single-frame picture information, preprocess the head sight image and the posture image to be detected, extract a head local feature vector and a global head feature vector, and fuse the head local feature vector and the global head feature vector to obtain a global feature vector; the feature vector module 132 is configured to normalize the feature data to obtain a feature vector, extract a local feature vector set from the processed head image data set, and then fuse the local feature vector set to obtain a head posture feature vector, where the feature vector module 132 is connected to the feature extraction module 131; a model construction module 133 for constructing a state detection model according to the feature vectors, the model construction module 133 being connected 132 with the feature vector module; a model adjusting module 134, configured to perform deep learning according to a video picture sample, update the video picture sample, and preprocess each head pose picture in the image analysis sample to obtain preprocessed information with a to-be-detected picture, where the model adjusting module 134 and the model constructing module 133 are connected to a feature comparing module 135, configured to compare a feature vector with eight motion features of the video picture sample, such as a left B-pillar, a left rearview mirror, an interior rearview mirror, a right B-pillar of an instrument panel, a right rearview mirror, a front view mirror, and a low head view, to obtain similar information, and obtain a sample global feature vector included in the sample according to the to-be-detected picture preprocessed information of the image analysis sample, where the global feature vector includes the above-mentioned motion data in the bar, and compare the global feature information of the sample with the to-be-detected feature vector to obtain cosine similarity information, the feature comparison module 135 is connected with the model adjustment module 134; the similar information sorting module 136 is configured to sort the similar information to obtain the facial attention detection information, and the missing information sorting module 136 is connected to the feature comparison module 135.
Please refer to fig. 8, which is a block diagram of the data extraction module 15 in fig. 5 in an embodiment, specifically including: the system comprises a data extraction module 151, an image acquisition queue module 152, a single-frame buffer module 153, an algorithm result module 154 and a log module 155; the data extraction module 151 is used for extracting video data, single-frame picture information and face attention detection information, and extracting the video data, the single-frame picture information and video monitoring information from the output end of the camera and the image data processing algorithm; the image acquisition queue module 152 is used for storing video data into an image acquisition cache queue, the image acquisition cache queue is used for storing and storing image data processed by an image algorithm in image data storage, and is used for summarizing and reporting the image data after driving test is finished and backing up the image data in a validity period, single-frame image information is extracted from the image storage queue and is aggregated into an image analysis sample, the image analysis sample is used for training a deep neural network model, and the image acquisition queue module 152 is connected with the data extraction module 151; the single-frame buffer module 153 is used for storing single-frame picture information into a single-frame picture buffer queue, the single-frame picture buffer queue is mainly an input queue of a volume and depth network model algorithm, video data are queue elements of an image acquisition buffer queue and are suitable for being used as input data of an image data processing algorithm according to the sequence of the video data entering the queue, and the single-frame buffer module 153 is connected with the data extraction module 151; the algorithm result module 154 is used for storing the face attention detection information into an algorithm output queue, wherein the algorithm output queue is an image algorithm module processing result queue and is used for reporting to upper-level equipment, the algorithm output queue is suitable for being transmitted to a server end in an ascending mode according to the sequence of video monitoring information entering the queue for being checked by monitoring personnel, and the algorithm result module 154 is connected with the data extraction module 151; the log module 155 is used for generating driving state judgment log information of the driver according to the picture cache queue and the algorithm output queue and storing the driving state judgment log information into a log library, storing and storing a working log generated in the operation of the system by the system log, managing the driving state judgment log information and sending the driving state judgment log information to the server side, the log module 155 is connected with the image acquisition queue module 152, the log module 155 is connected with the single-frame cache module 154, and the log module 155 is connected with the algorithm result module 154.
In an embodiment of the present invention, the present invention provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the deep learning-based driver status monitoring method provided by the present invention, and those skilled in the art can understand that: all or part of the steps for implementing the above method embodiments may be performed by hardware associated with a computer program. The aforementioned computer program may be stored in a computer readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.
In one embodiment, the present invention provides a driver state monitoring device based on deep learning, including: a processor and a memory; the memory is used for storing a computer program, and the processor is used for executing the computer program stored in the memory, so as to enable the deep learning-based driver state monitoring device to execute the deep learning-based driver state monitoring method provided by the invention, and the memory may include a Random Access Memory (RAM) or may further include a non-volatile memory (e.g. at least one disk memory). The processor may be a general-purpose processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the integrated circuit may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic device, or discrete hardware components.
In summary, the present invention provides a method, system, medium, and apparatus for monitoring driver status based on deep learning. The invention has the following beneficial effects: in order to realize the whole-process electronic monitoring and evaluation of the three-test of the driving subjects of the motor vehicle, a driving test visual tracking technical prototype extracts video data such as the posture of a driver through a vehicle-mounted camera, carries out computer visual algorithm processing including face detection, optical flow detection and the like by using tools such as a deep learning neural network and the like, completes the behavior analysis of detecting the attention point of the driver, whether the body extends out of the vehicle or not and the like, improves the objectivity and the accuracy of the three-test of the subjects, and reduces the labor cost. In the third test of the motor vehicle driving subjects, the driving test system uses a camera to collect the driving video of an examinee, and the face orientation of the examinee is detected so as to confirm the possible observation target of the examinee; completing the detection of whether an object extends out of the automobile in the left front window area so as to confirm whether a body part of an examinee extends out of the automobile; and finishing the camera imaging quality evaluation to confirm whether other objects block the camera. After detection of a driver focus, whether a body extends out of a window, whether a camera is shielded or not and the like is completed, the driving test system reports a relevant state to the superior device according to a communication protocol agreed with the superior device (the device which finally completes driving test subject compliance judgment) so as to help the superior device to complete driving test subject judgment. In conclusion, the invention solves the technical problems of high hardware cost, weaker feature robustness, low information utilization rate and low detection result accuracy in the prior art, takes the head posture picture acquired from the monitoring video as a sample library, does not need to design features, has strong feature robustness, higher actual detection accuracy and higher commercial value and practicability.
Claims (6)
1. A driver state monitoring method based on deep learning is characterized by comprising the following steps:
initializing image information acquisition equipment and storage equipment, presetting information processing logic, finishing initialization operation and sending prompt information;
receiving the prompt information, triggering characteristic information detection equipment and image information acquisition equipment to acquire video data of a driver, and extracting single-frame picture information from the video data to obtain a video picture sample;
extracting face orientation feature information and attention point angle information in the current single-frame picture information, constructing a state detection model according to the face orientation feature information and attention point angle information, performing deep learning of the state detection model according to the video picture sample, and processing the face orientation feature information and attention point angle information according to the preset information processing logic to obtain face attention detection information;
storing the single-frame picture information, the face attention detection information and the video data into a queue, and generating and storing driving state judgment log information according to the queue;
the method for extracting single-frame picture information from the video data and storing the single-frame picture information as a picture sample comprises the following steps:
receiving the prompt information, and starting a camera according to the prompt information;
acquiring the video data of the driver in real time by using a camera;
reading the video data;
extracting the current single-frame picture information according to the video data and the time;
extracting the single-frame picture information in the queue to obtain the video picture sample;
acquiring the face attention detection information, including:
extracting characteristic data in the single-frame picture information;
normalizing the feature data to obtain a feature vector;
constructing the state detection model according to the feature vector;
performing deep learning according to the video picture sample, and updating the video picture sample;
comparing the feature vector with the feature vector of eight actions of observing a left B column, a left rearview mirror, an inside rearview mirror, an overlooking instrument panel, a right B column, a right rearview mirror, a front part and a head-down looking gear in the video picture sample to obtain similar information;
and sequencing the similar information to obtain the face attention detection information.
2. The method according to claim 1, wherein the storing the single-frame picture information, the face attention detection information, and the video data in a queue, and generating and storing driving state determination log information according to the queue comprises:
extracting the video data, the single-frame picture information, and the facial attention detection information;
storing the video data into an image acquisition buffer queue;
storing the single-frame picture information into a single-frame picture cache queue;
storing the facial attention detection information into an algorithm output queue;
and generating driving state judgment log information of the driver according to the picture cache queue and the algorithm output queue and storing the driving state judgment log information in a log library.
3. A driver state monitoring system based on deep learning, comprising: the system comprises a system preparation module, a video picture sample module, a state detection module, a sign information module and a queue storage module;
the system preparation module is used for initializing the image information acquisition equipment and the storage equipment, presetting information processing logic, finishing initialization operation and sending prompt information;
the video picture sample module is used for receiving the prompt information, triggering the characteristic information detection equipment and the image information acquisition equipment to acquire video data of a driver, and extracting single-frame picture information from the video data to obtain a video picture sample;
the state detection module is used for extracting face orientation feature information and attention point angle information in the current single-frame picture information, constructing a state detection model according to the face orientation feature information and attention point angle information, performing deep learning of the state detection model according to the video picture sample, and processing the face orientation feature information and attention point angle information according to the preset information processing logic to obtain face attention detection information;
the queue storage module is used for storing the single-frame picture information, the face attention detection information and the video data into a queue, and generating and storing driving state judgment log information according to the queue;
wherein the video picture sample module comprises:
the camera opening module is used for receiving the prompt information and opening the camera according to the prompt information;
the video acquisition module is used for acquiring the video data of the driver in real time by using a camera;
the video reading module is used for reading the video data;
the single-frame extraction module is used for extracting the current single-frame picture information according to the video data and the time;
the sample generating module is used for extracting the single-frame picture information in the queue to obtain the video picture sample;
the state detection module includes:
the characteristic extraction module is used for extracting characteristic data in the single-frame picture information;
the characteristic vector module is used for normalizing the characteristic data to obtain a characteristic vector;
the model construction module is used for constructing the state detection model according to the feature vector;
the model adjusting module is used for carrying out deep learning according to the video picture sample and updating the video picture sample;
the characteristic comparison module is used for comparing the characteristic vectors with the characteristic vectors and observing eight action characteristics of a left B column, a left rearview mirror, an inside rearview mirror, an overlooking instrument panel, a right B column, a right rearview mirror, a front view and a low head view in the video picture sample to obtain similar information;
and the similar information sorting module is used for sorting the similar information to obtain the face attention detection information.
4. The system of claim 3, wherein the queue storage module comprises: the system comprises a data extraction module, an image acquisition queue module, a single-frame cache module, an algorithm result module and a log module;
the data extraction module is used for extracting the video data, the single-frame picture information and the face attention detection information;
the image acquisition queue module is used for storing the video data into an image acquisition cache queue;
the single-frame caching module is used for storing the single-frame picture information into a single-frame picture caching queue;
the algorithm result module is used for storing the face attention detection information into an algorithm output queue;
and the log module is used for generating driving state judgment log information of the driver according to the picture cache queue and the algorithm output queue and storing the driving state judgment log information into a log library.
5. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the deep learning-based driver state monitoring method according to any one of claims 1 to 2.
6. A driver state monitoring apparatus based on deep learning, characterized by comprising: a processor and a memory;
the memory is configured to store a computer program, and the processor is configured to execute the computer program stored in the memory to cause the deep learning based driver state monitoring apparatus to perform the deep learning based driver state monitoring method according to any one of claims 1 to 2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710716216.1A CN109409173B (en) | 2017-08-18 | 2017-08-18 | Driver state monitoring method, system, medium and equipment based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710716216.1A CN109409173B (en) | 2017-08-18 | 2017-08-18 | Driver state monitoring method, system, medium and equipment based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109409173A CN109409173A (en) | 2019-03-01 |
| CN109409173B true CN109409173B (en) | 2021-06-04 |
Family
ID=65463290
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710716216.1A Active CN109409173B (en) | 2017-08-18 | 2017-08-18 | Driver state monitoring method, system, medium and equipment based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109409173B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111724119A (en) * | 2019-09-26 | 2020-09-29 | 中国石油大学(华东) | Efficient automatic data annotation auditing method |
| CN110618635A (en) * | 2019-10-08 | 2019-12-27 | 中兴飞流信息科技有限公司 | Train cab operation specification monitoring system based on AI technology |
| CN113901895B (en) * | 2021-09-18 | 2022-09-27 | 武汉未来幻影科技有限公司 | Door opening action recognition method and device for vehicle and processing equipment |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106446811A (en) * | 2016-09-12 | 2017-02-22 | 北京智芯原动科技有限公司 | Deep-learning-based driver's fatigue detection method and apparatus |
| CN106599994A (en) * | 2016-11-23 | 2017-04-26 | 电子科技大学 | Sight line estimation method based on depth regression network |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102830793B (en) * | 2011-06-16 | 2017-04-05 | 北京三星通信技术研究有限公司 | Sight tracing and equipment |
| CN104637246B (en) * | 2015-02-02 | 2017-02-22 | 合肥工业大学 | Driver multi-behavior early warning system and danger evaluation method |
| CN105354986B (en) * | 2015-11-12 | 2017-12-01 | 熊强 | Driver's driving condition supervision system and method |
| CN107123340B (en) * | 2017-06-30 | 2020-06-05 | 无锡合壮智慧交通有限公司 | Method for automatically detecting observation state of driver |
-
2017
- 2017-08-18 CN CN201710716216.1A patent/CN109409173B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106446811A (en) * | 2016-09-12 | 2017-02-22 | 北京智芯原动科技有限公司 | Deep-learning-based driver's fatigue detection method and apparatus |
| CN106599994A (en) * | 2016-11-23 | 2017-04-26 | 电子科技大学 | Sight line estimation method based on depth regression network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109409173A (en) | 2019-03-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109409172B (en) | Driver sight line detection method, system, medium, and apparatus | |
| WO2020042984A1 (en) | Vehicle behavior detection method and apparatus | |
| CN109409173B (en) | Driver state monitoring method, system, medium and equipment based on deep learning | |
| CN112052815B (en) | Behavior detection method and device and electronic equipment | |
| US11798297B2 (en) | Control device, system and method for determining the perceptual load of a visual and dynamic driving scene | |
| CN110889351A (en) | Video detection method and device, terminal equipment and readable storage medium | |
| CN107944382B (en) | Method for tracking target, device and electronic equipment | |
| CN111372037A (en) | Target snapshot system and method | |
| US10860866B2 (en) | Systems and methods of legibly capturing vehicle markings | |
| US20090010499A1 (en) | Advertising impact measuring system and method | |
| CN112016429A (en) | Fatigue driving detection method based on train cab scene | |
| CN111382610B (en) | Event detection method and device and electronic equipment | |
| CN103456168A (en) | System and method for monitoring behaviors of pedestrians at traffic intersection | |
| KR20210014988A (en) | Image analysis system and method for remote monitoring | |
| CN108805184B (en) | Image recognition method and system for fixed space and vehicle | |
| CN110225236B (en) | Method and device for configuring parameters for video monitoring system and video monitoring system | |
| CN110969864A (en) | Vehicle speed detection method, vehicle driving event detection method and electronic equipment | |
| JP4735242B2 (en) | Gaze target object identification device | |
| KR101107120B1 (en) | Device for sound source tracing and object recognition and method for sound source tracing and object recognition | |
| CN109426757B (en) | Driver head posture monitoring method, system, medium and equipment based on deep learning | |
| CN111753587A (en) | Method and device for detecting falling to ground | |
| CN109460077B (en) | Automatic tracking method, automatic tracking equipment and automatic tracking system | |
| CN110807375A (en) | Human head detection method, device and equipment based on depth image and storage medium | |
| CN116823884A (en) | Multi-target tracking method, system, computer equipment and storage medium | |
| Bravo et al. | Outdoor vacant parking space detector for improving mobility in smart cities |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |