CN116469085B

CN116469085B - Monitoring method and system for risk driving behavior

Info

Publication number: CN116469085B
Application number: CN202310322972.1A
Authority: CN
Inventors: 冯波
Original assignee: Wanlian Yida Logistics Technology Co ltd
Current assignee: Wanlian Yida Logistics Technology Co ltd
Priority date: 2023-03-30
Filing date: 2023-03-30
Publication date: 2024-04-02
Anticipated expiration: 2043-03-30
Also published as: CN116469085A

Abstract

The invention relates to the technical field of intelligent traffic data processing, in particular to a method and a system for monitoring risk driving behaviors. The method comprises the following steps: controlling a vehicle-mounted camera to acquire images and acquiring image information of a driver; extracting eye features according to the image information of the driver to obtain eye closing frequency features and eyeball movement track features; according to the eye closing frequency characteristics and the eyeball motion track characteristics, respectively identifying through an eye closing frequency identification model and an eyeball motion track identification model to obtain eye closing frequency data and eyeball motion track data; and performing eye behavior evaluation according to the eye closing frequency data and the eyeball motion track data to obtain multidimensional eye evaluation data of the driver. According to the invention, eye fatigue and distracted dangerous behaviors are identified through eye behavior monitoring of the driver, so that potential traffic safety hazards are avoided.

Description

Monitoring method and system for risk driving behavior

Technical Field

The invention relates to the technical field of intelligent traffic data processing, in particular to a method and a system for monitoring risk driving behaviors.

Background

In long distance or long time driving process, fatigue driving behavior of a driver tends to occur easily, and how to eliminate the fatigue driving behavior, so that potential safety hazards caused by dangerous driving are always one of the problems concerned by traffic industry, generally, in a traditional method, a risk index is obtained through statistical analysis and calculation of static data of the driver to remind, such as comprehensive calculation of age, sex, vehicle type and driving duration of the driver, and risk behavior prediction is performed through a generalized linear model, wherein false reminding is easily sent when the driver is not tired, and the fatigue state of the driver when the driver gets on the vehicle cannot be extracted in time.

With the development of the Internet, an artificial intelligence technology starts to appear, depth association relation information behind data is deeply mined through the artificial intelligence technology, so that objective rules of object back development are timely searched, the problem is better solved, and fatigue information of a driver behind an image is better found by combining the artificial intelligence technology with risk driving behaviors.

Disclosure of Invention

The invention provides a method and a system for monitoring risk driving behaviors to solve at least one of the technical problems.

The invention provides a monitoring method of risk driving behaviors, which comprises the following steps:

step S1: controlling the vehicle-mounted camera to acquire images so as to acquire image information of a driver;

step S2: extracting eye features according to the image information of the driver, so as to obtain eye closing frequency features and eyeball movement track features;

step S3: according to the eye closing frequency characteristics and the eyeball motion track characteristics, the eye closing frequency identification model and the eyeball motion track identification model are respectively used for identification, so that eye closing frequency data and eyeball motion track data are obtained;

step S4: and performing eye behavior evaluation according to the eye closing frequency data and the eyeball motion track data, thereby acquiring multidimensional eye evaluation data of a driver, and sending the multidimensional eye evaluation data to a risk monitoring system for risk early warning operation.

The invention can timely identify dangerous behaviors such as eye fatigue and distraction by monitoring the eye behaviors of the driver, avoid traffic accidents caused by fatigue or distraction of the driver, improve road traffic safety, and can evaluate the eye behaviors of the driver in multiple angles and all directions by adopting the eye closing frequency characteristic and the eyeball movement track characteristic and combining the identification model and the algorithm provided by the invention, thereby improving the accuracy and the reliability of behavior evaluation.

In one embodiment of the present specification, step S2 is specifically:

step S21: performing eye detection operation according to the driver image information to obtain the eye position information of the driver;

step S22: extracting an eye region according to the driver image information and the driver eye position information to acquire eye region image information;

step S23: extracting eye closing frequency characteristics according to the eye region image information, so as to obtain the eye closing frequency characteristics;

step S24: and extracting the characteristics of the eye movement track according to the eye region image information, thereby obtaining the characteristics of the eye movement track.

According to the method, the device and the system, the eye position information of the driver can be accurately obtained through the eye detection technology, so that the problem that feature extraction is inaccurate due to the reasons of posture change, shielding and the like is solved, the accuracy of feature extraction is improved, the eye region extraction technology is adopted, the image information of the eye region can be independently extracted, the influence of interference factors on feature extraction is avoided, the feature extraction is more refined, the processed data volume can be reduced through the eye region extraction technology, the processing speed of the feature extraction is improved, the instantaneity and the reliability of the whole system are improved, and the eye detection and eye region extraction technology can adapt and optimize different driving scenes, such as different light conditions, different driver postures and the like, so that the robustness and the practicability of the system are enhanced.

In one embodiment of the present specification, wherein the step of the eye detection procedure comprises the steps of:

step S211: extracting and converting feature vectors according to the driver image information to obtain edge feature vector data and texture feature vector data;

step S212: image segmentation is carried out through a preset sliding window according to the image information of the driver, so that a cutting image is obtained;

step S213: performing proximity similarity calculation on the cut image according to the edge feature vector data and the texture feature vector data, so as to obtain a proximity fitting index;

step S214: carrying out image combination on the cut images according to the adjacent fitting index so as to obtain a combined image;

step S215: acquiring an image shooting angle and carrying out image overturning on the combined image according to the image shooting angle so as to acquire an overturning image;

step S216: performing bilinear interpolation calculation according to the turnover image through a preset scaling ratio, so as to obtain a scaled image;

step S217: classifying and calculating according to the scaled image through a preset multidimensional linear regression curve, so as to obtain an eye index;

step S218: and carrying out iterative calculation according to the eye indexes until the eye indexes are determined to be larger than the preset eye threshold indexes, generating a correction sliding window according to the combined image corresponding to the eye indexes so as to correct the sliding window, and carrying out coordinate extraction according to the scaled image corresponding to the eye indexes and the standard driver image information to generate eye position information.

According to the embodiment, a plurality of feature vectors and a similarity calculation method are adopted, so that the eye position information of a driver is accurately detected under the condition that the eye shape change is fully considered, and the detection precision and robustness are improved; the drivers with different sizes and postures can be adapted and optimized through a preset sliding window and a multidimensional linear regression curve; meanwhile, the system can also turn over and zoom according to the image shooting angle, so that the system is suitable for more complex image environments, and the processing data volume and the operation time can be reduced by adopting technologies such as image merging and zooming, so that the processing speed of eye detection is accelerated, and the real-time performance and the reliability of the system are improved; by iterative calculation and automatic correction of the sliding window, automation of the eye detection process can be realized, the difficulty and error rate of manual intervention are reduced, and the intelligent and automatic level of the system is improved.

In one embodiment of the present specification, the proximity similarity calculation is calculated by a proximity similarity calculation formula, wherein the proximity similarity calculation formula is specifically:

for adjacent fitting index->To cut the first part of the image>Weighting coefficients of the individual pixel values, +.>To cut the first part of the image >Individual pixel values +.>For the balance adjustment term of the weight coefficient, +.>For fine-tuning offset terms generated from edge feature vector data and texture feature vector data +_>To cut the first part of the image>Weighting coefficients of neighboring pixel values of the individual pixel values, +.>To cut the first part of the image>Individual pixel valuesAdjacent pixel values of +.>For pixel correlation coefficients generated from edge feature vector data and texture feature vectors +.>For initial adjustment item->For the dimension reduction coefficient adjacent to the fitting index, +.>For cutting the total number of pixels of the image +.>Is a correction coefficient adjacent to the fitting index.

The present embodiment provides a proximity similarity calculation formula that sufficiently considers the first cut imageWeight coefficient of individual pixel values +.>Cut the first part of the image>Individual pixel values +.>Balance adjustment item of weight coefficient->Fine tuning offset term generated from edge feature vector data and texture feature vector data>Cut the first part of the image>Weighting system for adjacent pixel values of each pixel valueCount->Cut the first part of the image>Adjacent pixel values of the individual pixel values +.>Pixel correlation coefficients generated from edge feature vector data and texture feature vectors>Initial adjustment item- >The dimension-reducing coefficient adjacent to the fitting index>And the interaction relationship with each other to form a functional relationship +.>The preference of the pixel points along the edge feature vector data and the texture feature vector is considered through the current pixel point and the adjacent pixel points, so that the generated combined image can fully display the eye features and provide accurate data support.

In one embodiment of the present specification, step S22 is specifically:

step S221: extracting an eye region according to the driver image information and the driver eye position information, so as to obtain preliminary eye region image information;

step S222: performing distortion elimination calculation through an optimized epipolar constraint method according to the preliminary eye region image information, so as to obtain adjusted eye image information;

step S223: performing corner detection and focus correction according to the eye image information, and generating corrected eye image information;

step S224: denoising according to the corrected eye image information, thereby obtaining denoising eye image information;

step S225: judging an illumination condition threshold according to the denoising eye image information, so as to generate an illumination image index;

step S226: judging whether the illumination image index is larger than or equal to a preset illumination image threshold index;

Step S227: when the illumination image index is determined to be greater than or equal to a preset illumination image threshold index, denoising eye image information is determined to be eye region image information;

step S228: when the illumination image index is determined to be smaller than a preset illumination image threshold index, performing white balance calculation according to denoising eye image information, so as to obtain white balance image information;

step S229: performing brightness enhancement calculation according to the white balance image information, thereby obtaining enhanced image information;

step S2210: and performing contrast adjustment according to the enhanced image information, so as to obtain eye region image information.

The embodiment can effectively improve the quality of eye images and avoid information loss caused by problems such as image noise, illumination non-uniformity and the like by a plurality of methods such as distortion elimination, denoising, white balance calculation, brightness enhancement, contrast adjustment and the like; the method can adapt to different image environments, such as weak illumination, strong illumination and other conditions through the technologies of illumination condition threshold judgment, white balance calculation, brightness enhancement, contrast adjustment and the like, so that the robustness and the adaptability of the system are improved; through the technologies of angular point detection, focus correction and the like, the eye area can be accurately determined, so that the accuracy and reliability of eye positioning are improved.

In one embodiment of the present specification, the step of optimizing the epipolar constraint method specifically includes:

step S201: acquiring parameter information of a vehicle-mounted camera;

step S202: performing matrix relation calculation according to the vehicle-mounted camera parameter information and preset angle transformation, so as to obtain an imaging matrix relation;

step S203: extracting features according to the preliminary eye region image information so as to obtain region feature information;

step S204: performing polar coordinate conversion according to the regional characteristic information through an imaging matrix relation, so as to obtain polar coordinate information;

step S205: performing optimized image distortion correction according to the polar coordinate information, thereby obtaining distortion correction image information;

step S206: and performing matrix relation calculation according to the distortion correction image information, so as to obtain the eye adjustment image information.

In the embodiment, the distortion correction is performed by optimizing the epipolar constraint method, so that the distortion degree of the image can be effectively reduced, and the quality and the definition of the eye image are improved; through the technologies of feature extraction, polar coordinate conversion and the like, the features of the eye region can be accurately extracted, and the position and the size of the features in the image can be determined, so that the accuracy and the stability of eye positioning are improved; through the automatic processing steps of vehicle-mounted camera parameter information, preset angle transformation and the like, different vehicle-mounted cameras can be adapted and optimized, automatic distortion correction of eye images is realized, and therefore the intelligent and automatic level of the system is improved; by optimizing the epipolar constraint method for distortion correction, recognition errors and instability caused by image distortion can be avoided, so that the reliability and the robustness of the method are enhanced.

In one embodiment of the present specification, the image distortion correction is calculated by optimizing a distortion correction calculation formula, wherein the optimized distortion correction calculation formula is specifically:

correcting pixel coordinates of image information for distortion, < >>For polar information, +.>For the slope of the curve +.>Is curved and is->For curve shape adjustment item +.>For fine tuning the adjustment item->For nonlinear adjustment item->For history frame->Error correction term->And correcting the correction coefficient of the pixel point coordinates of the image information for distortion.

The present embodiment provides an optimized distortion correction calculation formula that fully considers polar coordinate informationSlope of curve->Curve curvature->Curve shape adjustment item->Fine tuning adjustment item->Nonlinear adjustment term->History frame->Error correction term->And the interaction relationship with each other to form a functional relationshipThe curve form is adjusted and controlled through curve slope, curve curvature and curve shape adjustment items, radial distortion and nonlinear distortion are eliminated, fine tuning, translation and nonlinear adjustment of the curve are carried out through fine tuning adjustment items and nonlinear adjustment items, generalization capability is improved, historical values are state information of previous frames, the state of the current frame is predicted, a correction result is further optimized, and finally correction coefficients of pixel point coordinates of image information are corrected through distortion >And correction is carried out, so that the problem of image distortion correction is better solved.

In one embodiment of the present specification, step S23 is specifically:

step S231: preprocessing according to the eye region image information, so as to obtain preprocessed image information;

step S232: gray scale conversion is carried out according to the preprocessed image information, so that gray scale image information is obtained;

step S233: performing minimum estimated value region cutting according to the gray level image information, thereby obtaining estimated value region image information;

step S234: performing binarization processing according to the estimated value area image information, thereby binarizing the information;

step S235: marking the estimated area image information according to the binarization information, thereby obtaining marked area image information;

step S236: judging a marking area threshold according to the marking area image information, so as to generate eye opening and closing information;

step S237: and marking the eye closing information according to the time information corresponding to the eye region image information, so as to obtain the eye closing frequency characteristic information.

The embodiment has the specific beneficial effects that the eye closing frequency characteristic information can be obtained based on the eye region image information, so that the embodiment is used for analyzing the conditions of fatigue, attention level and the like of an individual, and further providing a reference basis for fine management; the minimum estimated value area of the image information is cut, so that the operation load on operation equipment is reduced, and the operation efficiency is improved.

In one embodiment of the present disclosure, the marking area image information includes white marking area image information and black marking area image information, and the eye opening and closing information includes eye opening and closing information, wherein the step of marking area threshold value judgment is specifically;

step S238: carrying out adjacent value statistics and maximum value extraction according to the black mark image area information, thereby obtaining black mark maximum value data;

step S239: judging whether the maximum value data of the black mark is larger than or equal to preset threshold value data of the black mark;

step S240: when the maximum value data of the black mark is determined to be greater than or equal to the preset black mark threshold value data, generating eye closing information;

step S241: and when the maximum value data of the black mark is less than or equal to the preset black mark threshold value data, generating eye opening information.

The embodiment accurately generates the eye opening and closing information through the judgment of the marked area threshold value, so that the eye opening and closing information can be used for identifying and monitoring the eye opening and closing behaviors, and the preparation work is carried out for the next action.

The invention provides a monitoring system for risk driving behavior, which comprises:

At least one processor; the method comprises the steps of,

a memory communicatively coupled to the at least one processor; wherein,

the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform a method of monitoring risk driving behaviour as described above.

According to the invention, through real-time monitoring and evaluation of the eye closing frequency and the eyeball movement track of the driver, the risk driving behavior is identified and early-warned in time, the road traffic safety is improved, and the traffic accidents caused by fatigue, inattention and the like are reduced. Meanwhile, the invention is based on the image acquisition and analysis technology, realizes the monitoring operation without additional equipment, is convenient to use and low in cost, has wide application range, realizes the accurate eye image acquisition and identification of a driver by carrying out depth eye detection and feature extraction identification on the driver image, improves generalization capability, and provides a depth accurate data relationship for realizing the accurate prediction of risk driving behaviors.

Drawings

Other features, objects and advantages of the present application will become more apparent upon reading of the detailed description of non-limiting implementations made with reference to the following drawings in which:

FIG. 1 is a flow chart illustrating steps of a method for monitoring risk driving behavior in accordance with one embodiment;

FIG. 2 is a flow chart illustrating steps of a method for extracting ocular features in accordance with one embodiment;

FIG. 3 is a flowchart illustrating steps of an eye detection operation according to one embodiment;

FIG. 4 is a flow chart illustrating steps of a method of eye region extraction according to one embodiment;

FIG. 5 is a flow chart illustrating steps of an optimized epipolar constraint method according to one embodiment;

FIG. 6 is a flowchart showing steps of a method for extracting frequency characteristics of eye closure in an embodiment;

fig. 7 is a flowchart showing steps of a marking area threshold judging method according to an embodiment.

Detailed Description

The following is a clear and complete description of the technical method of the present patent in conjunction with the accompanying drawings, and it is evident that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, are intended to fall within the scope of the present invention.

Furthermore, the drawings are merely schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus a repetitive description thereof will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. The functional entities may be implemented in software or in one or more hardware modules or integrated circuits or in different networks and/or processor methods and/or microcontroller methods.

It will be understood that, although the terms "first," "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of example embodiments. The term "and/or" as used herein includes any and all combinations of one or more of the associated listed items.

The invention provides a monitoring method of risk driving behavior, please refer to fig. 1 to 7, comprising the following steps:

specifically, for example, a camera is installed in front of the driving position to perform image acquisition, thereby acquiring driver image information.

specifically, for example, the eye region image information of the driver is processed and analyzed by using a computer vision technology, and the characteristics of the eye movement track, including the position, the speed and the acceleration of the eyes, are extracted.

Specifically, for example, the image information of the eye region of the driver is processed and analyzed by using a computer vision technique, the eye closing frequency characteristics of the eye are extracted, the information including the time interval of opening and closing the eye and the duration of closing the eye is extracted, and an image processing method such as a detection method based on characteristic points or a convolutional neural network is used.

specifically, for example, a deep learning-based technique such as Convolutional Neural Network (CNN), cyclic neural network (RNN) is adopted to construct an eye-closing frequency recognition model and an eye movement track recognition model, and training and optimization are performed by using a large-scale data set.

Specifically, statistical analysis such as average, variance, maximum, and minimum is performed, for example, based on the eye-closing frequency data and the eye movement trajectory data, thereby obtaining multidimensional eye evaluation data of the driver;

And taking the eye closing frequency data and the eyeball motion track data as input, constructing an eye behavior evaluation model of the driver, and carrying out risk early warning according to the output result of the model.

In one embodiment of the present specification, step S2 is specifically:

specifically, the driver image information is processed and analyzed, for example, by a Haar cascade classifier, HOG features, SIFT features, or the like, to thereby detect eye position information.

Specifically, the image is cut and scaled using an image processing technique, for example, based on the driver eye position information, thereby extracting the eye region image information.

specifically, for example, the image information of the eye region is preprocessed and analyzed in a binarization and edge detection mode, so that information such as time interval, duration and the like of eye closing is extracted, wherein the binarization is that gray values of the image are binarized according to a preset threshold value, the gray values are smaller than the threshold value and are white, the gray values are larger than the threshold value and are black, and block statistics is carried out on the formed binarized region and threshold value judgment is carried out, so that the eye closing frequency characteristics of the eye are obtained.

Specifically, for example, the eye region image information is processed and analyzed by a detection method based on feature points and a template matching method, so that information such as the position, the speed and the acceleration of the eyeball is extracted, and the movement track characteristics of the eyeball are obtained.

Specifically, for example, # uses the ORB algorithm to extract keypoints

orb = cv2.ORB_create(nfeatures=50)

kp = orb.detect(gray, None)

# drawing key point

img_kp = cv2.drawKeypoints(img, kp, None, color=(0,255,0), flags=0)

Tracking key points using LK optical flow method

lk_params = dict(winSize=(15, 15),

maxLevel=3,

criteria=(cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 0.03))

p0 = np.float32([kp[i].pt for i in range(len(kp))]).reshape(-1, 1, 2)

color = np.random.randint(0, 255, (len(p0), 3))。

Specifically, for example, # extracts edge feature vector data

edges = cv2.Canny(gray, 100, 200)

edge_vector = edges.reshape(-1)

# extraction of texture feature vector data

gray = np.float32(gray) / 255.0

texture = cv2.dct(gray)

texture = texture[:8, :8]

texture_vector = texture.reshape(-1)。

Specifically, for example, the driver image information is processed and analyzed by SIFT, SURF, or HOG methods, thereby extracting edge feature vector data and texture feature vector data.

specifically, parameters such as the size of the sliding window and the stride are set. Then, the positions of the sliding windows are initialized, and the position coordinates of all the sliding windows are generated through circulation. And then, carrying out region growing segmentation on each sliding window in turn to obtain a cutting image.

specifically, the proximity fitting index is obtained by performing similarity calculation on edge feature vector data and texture feature vector data, for example, by euclidean distance or manhattan distance.

Specifically, for example, the proximity similarity calculation is performed by the calculation formulas provided in the remaining embodiments of the present invention, so as to obtain the proximity fitting index.

specifically, for example, the proximity fitting index threshold value is determined according to a preset index, if the proximity fitting index threshold value is greater than or equal to a preset proximity fitting index, for example, 6.24, the corresponding cut images are subjected to image merging, so that a merged image is obtained.

specifically, the image capturing angle is acquired, for example, by using a technique of camera calibration and three-dimensional reconstruction, so as to perform rotation or mirror image transformation on the combined image.

Specifically, for example, the combined image is converted into a grayscale image, then an edge is detected by using a Canny algorithm, then a straight line is detected by using hough transform, the angle of the straight line is calculated, and finally the average value of all angles is calculated. Next, a rotation matrix is obtained from the average angle using the getrotation matrix2D function of OpenCV, and image rotation is performed using the warp affine function. Finally, the original image and the rotated image are displayed.

Specifically, image scaling is performed using, for example, the size function of the OpenCV library. The scaling process can be controlled by setting parameters such as the scaling scale or the target image size, and different interpolation methods such as nearest neighbor interpolation and bilinear interpolation can be selected in the scaling process.

Specifically, for example, # bilinear interpolation scales an image

resized_img = cv2.resize(img, new_size, interpolation=cv2.INTER_LINEAR)。

specifically, for example, a multidimensional linear regression curve is computationally generated according to a support vector machine algorithm.

Specifically, eyes in a face image are detected, for example, using a Haar cascade classifier or a model of a Convolutional Neural Network (CNN), and then an eye region is classified using a classification model.

Step S218: and carrying out iterative calculation according to the eye indexes until the eye indexes are determined to be larger than the preset eye threshold indexes, generating a correction sliding window according to the combined image corresponding to the eye indexes so as to correct the sliding window, and carrying out coordinate extraction according to the scaled image corresponding to the eye indexes and the driver image information to generate eye position information.

Specifically, the sliding window is modified, for example, with a pre-trained classification model, and the eye position is extracted with a regression model. The regression model can adopt CNN-Reg and ResNet models. Through continuous iterative correction, when the eye index is greater than a preset eye threshold index, the eye position can be determined, and coordinate information is output.

for adjacent fitting index->To cut the first part of the image>Weighting coefficients of the individual pixel values, +.>To cut the first part of the image >Individual pixel values +.>For the balance adjustment term of the weight coefficient, +.>For fine-tuning offset terms generated from edge feature vector data and texture feature vector data +_>To cut the first part of the image>Weighting coefficients of neighboring pixel values of the individual pixel values, +.>To cut the first part of the image>Adjacent pixel values of the individual pixel values, < >>For pixel correlation coefficients generated from edge feature vector data and texture feature vectors +.>For initial adjustment item->For the dimension reduction coefficient adjacent to the fitting index, +.>For cutting the total number of pixels of the image +.>Is a correction coefficient adjacent to the fitting index.

The present embodiment provides a proximity similarity calculation formula that sufficiently considers the first cut imageWeight coefficient of individual pixel values +.>Cut the first part of the image>Individual pixel values +.>Balance adjustment item of weight coefficient->Fine tuning offset term generated from edge feature vector data and texture feature vector data>Cut the first part of the image>Weighting factors of neighboring pixel values of the individual pixel values +.>Cut the first part of the image>Adjacent pixel values of the individual pixel values +.>Pixel correlation coefficients generated from edge feature vector data and texture feature vectors>Initial adjustment item->The dimension-reducing coefficient adjacent to the fitting index >And the interaction relationship with each other to form a functional relationship +.>The preference of the pixel points along the edge feature vector data and the texture feature vector is considered through the current pixel point and the adjacent pixel points, so that the generated combined image can fully display the eye features and provide accurate data support.

In one embodiment of the present specification, step S22 is specifically:

specifically, for example, the driver image and eye position information are read, and the position of the eye region is calculated, and the image information of the eye region is extracted by indexing the driver image.

specifically, for example, the left and right eye images are read, camera calibration and correction parameter calculation are performed by using a stereoscopic vision technology, an initunderstator transformation map function in an OpenCV library is used to construct a distortion correction mapping table, distortion elimination processing is performed on the left and right eye images through the map function, and the adjusted left and right eye images are displayed.

specifically, for example, a good featuretotrack function in an OpenCV library is used to perform corner detection, a perspective transformation matrix between the left-eye image and the right-eye image is calculated through a getperspective transformation function, and a perspective transformation operation is performed on the left-eye image by using a warp perspective function, so that corrected left-eye images and corrected right-eye images are displayed.

specifically, gaussian filtering processing is performed using, for example, gaussian blur functions in the OpenCV library.

In particular, for example

# Gaussian filtering left and right eye images

img_left_denoised = cv2.GaussianBlur(img_left_rectified, (5, 5), 0)

img_right_denoised = cv2.GaussianBlur(img_right_rectified, (5, 5), 0)。

specifically, for example, the denoised left and right eye images are converted into gray images by using a cvttcolor function in an OpenCV library, the gray images are binarized by using a global thresholding method, a binarization threshold is determined by using an OTSU method, and an illumination image index is obtained by dividing 255 by calculating the OTSU threshold.

specifically, for example, # defines a preset illumination image threshold index

threshold_illuminance = 0.7

And comparing the illumination image indexes according to a preset illumination image threshold index.

specifically, for example, a cvtdcolor function in an OpenCV library is used to convert a BGR color image into an HSV color space, then an equencehist function is used to perform histogram equalization on a brightness channel, and then the processed image is converted back into the BGR color image and displayed, so that eye image information after white balance is obtained.

specifically, for example, according to the left-eye image and the right-eye image after white balance, through preset defined brightness coefficients and brightness increments, the images are subjected to linear transformation by using a convertScaleAbs function to enhance brightness, so that eye image information after brightness enhancement is obtained.

Specifically, for example, defining a CLAHE parameter, converting a color image into a gray image, creating a CLAHE object by using a createCLAHE function, and performing CLAHE contrast adjustment on the gray image by using an apply function;

# CLAHE contrast adjustment of brightness-enhanced left-right eye images

clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8))

gray_left = cv2.cvtColor(img_left_enhanced, cv2.COLOR_BGR2GRAY)

gray_right = cv2.cvtColor(img_right_enhanced, cv2.COLOR_BGR2GRAY)

clahe_left = clahe.apply(gray_left)

clahe_right = clahe.apply(gray_right)。

step S201: acquiring parameter information of a vehicle-mounted camera;

Specifically, detailed information of the in-vehicle camera, such as an image sensor type, a lens focal length, an exposure time, and the like, is read, for example, using a specific SDK or API.

specifically, for example, according to parameter information of the vehicle-mounted camera, such as focal length, field angle and image sensor size, an internal reference matrix K of the camera is calculated, wherein the internal reference matrix represents a process of converting three-dimensional world coordinates into two-dimensional image coordinates when the camera is imaged, and is a 3×3 matrix; according to the installation position and angle transformation of the vehicle-mounted camera, calculating an external parameter matrix R and T of the camera, wherein the external parameter matrix represents the relation between a camera coordinate system and a world coordinate system, the rotation matrix R represents the rotation relation from the camera coordinate system to the world coordinate system, and the translation vector T represents the position of the camera coordinate system in the world coordinate system; combining the internal reference matrix K and the external reference matrix [ R|T ] to obtain a projection matrix P= [ K|0 ] [ R|T ] of the camera, wherein a zero vector of 3 x 1 is added behind the matrix K and used for representing the depth of an image point, and the projection matrix P converts three-dimensional object coordinates into two-dimensional image coordinates under a camera coordinate system.

specifically, for example, the eye region image is processed using the technique of morphological transformation and edge detection of the OpenCV library, noise and filling holes can be eliminated using morphological operations, and contour lines are extracted using an edge detection algorithm, so that region feature information is obtained.

specifically, for example, distortion correction is performed on the eye region by using an undisitor function of an OpenCV library, an eye region image after distortion correction is cut out according to the position and the size of the eye region in an original image, the eye region image is adjusted to be consistent in size, polar coordinate information obtained by previous calculation is converted into coordinates in a rectangular coordinate system by using a polar_to_cartesian function, and a grid is generated by using the coordinates. The distortion corrected eye region image is polar resampled using the cv2.remap function and the result is output.

Specifically, distortion correction is performed on an input image using an undistor function of the OpenCV library, for example, according to the internal reference matrix K and the distortion coefficient distCoeffs of the in-vehicle camera. The method comprises the steps of removing distortion generated during imaging of a camera;

cutting out an eye area image after distortion correction according to the position and the size of the eye area in an original image, and adjusting to enable the sizes of the eye area image to be consistent;

and (3) according to the polar coordinate information obtained by previous calculation, carrying out polar coordinate resampling on the eye region image after distortion correction.

Specifically, for example, according to polar coordinate information calculated previously, the distortion-corrected eye region image is converted into coordinates in a rectangular coordinate system, and a grid is generated using these coordinates; calculating the corresponding space point coordinates of each grid position under the world coordinate system, and projecting the points onto an image plane according to pose information of the camera and an internal reference matrix to obtain pixel coordinates of the points in the image; and according to the pixel coordinates, performing pixel level adjustment on the original eye region image by using a remap function of an OpenCV library to obtain an optimized eye region image.

correcting pixel coordinates of image information for distortion, < >>For polar information, +.>For the slope of the curve +.>Is curved and is->For curve shape adjustment item +. >For fine tuning the adjustment item->For nonlinear adjustment item->For history frame->Error correction term->And correcting the correction coefficient of the pixel point coordinates of the image information for distortion.

The present embodiment provides an optimized distortion correction calculation formula that fully considers polar coordinate informationSlope of curve->Curve curvature->Curve shape adjustment item->Fine tuning adjustment item->Nonlinear adjustment term->History frame->Error correction term->And the interaction relationship with each other to form a functional relationshipThe curve form is regulated and controlled by the curve slope, curve curvature and curve shape regulating item, and the radial distortion is eliminatedNonlinear distortion, namely fine tuning and translation of curves and nonlinear adjustment are carried out through fine tuning adjustment items and nonlinear adjustment items, generalization capability is improved, a history value is obtained by predicting the state of a current frame according to state information of previous frames, a correction result is further optimized, and finally correction coefficients of pixel point coordinates of image information are corrected through distortion>And correction is carried out, so that the problem of image distortion correction is better solved.

In one embodiment of the present specification, step S23 is specifically:

Specifically, for example, image enhancement operations such as contrast enhancement, histogram equalization are performed to enhance the visual effect and readability of the image; image filtering operations, such as gaussian filtering, median filtering, are performed to remove noise and smooth the image.

specifically, for example, the preprocessed image is subjected to color space conversion, and information of RGB or other color channels is converted into gradation information.

specifically, for example, the gray level image is thresholded to separate the eye area from the background, edge detection operation, such as Canny edge detection, is performed to detect the contour boundary, minimum estimated area cutting is performed according to the contour boundary, the eye area is cut out to obtain an estimated area image, and the minimum estimated area is half of the preset contour area or contour value boundary extraction is performed according to the preset loss value.

Specifically, the estimated value region image is converted into a binary image according to a threshold value, for example, using an Otsu adaptive threshold value method, a fixed threshold value method, or the like.

specifically, for example, the binarized estimated area image is subjected to connected component analysis using a cv2. Connectiedcomponents function, and adjacent pixels are divided into a group to generate a marker matrix.

specifically, for example, if the maximum pixel area in the marker area image information is larger than a preset pixel threshold area, the eye opening information is generated if the marker area image information is displayed in white, and the eye closing information is generated if the marker area image information is displayed in black.

Specifically, for example, the eye closing frequency characteristic information is the number of times of opening and closing eyes within one minute, and is calculated by a statistical method.

specifically, for example, traversing the black marked area to obtain the center coordinate of each marked area; selecting a neighborhood region with a fixed size for each marking region by taking the central coordinate as the center, and counting the gray values of pixel points in the neighborhood region; for the gray values of the pixel points in all the neighborhood regions in each marking region, finding the maximum value; and forming an array by the maximum values of the neighborhood regions in all the marking regions, sequencing the array, and selecting the maximum number in the array as black marking maximum value data.

Specifically, for example, # determines whether the obtained black mark maximum value data is greater than or equal to a preset black mark threshold value data

threshold = 50

if black_value>= threshold:

print('Eye is closed.')

# drawing eye closure prompt information on eye image

cv2.putText(img_eye, 'Eyes Closed', (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 0.8, (0, 0, 255), 2)

else:

print('Eye is open.')

# drawing eye-opening prompt information on eye image

cv2.putText(img_eye, 'Eyes Open', (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 0.8, (0, 255, 0), 2)

A black mark threshold data threshold is predefined for judging eye state. The threshold is usually determined according to actual conditions at the time of algorithm design; judging whether the obtained black mark maximum value data black_value is larger than or equal to a preset black mark threshold value data threshold; if the black_value is greater than or equal to threshold, generating eye closure information; otherwise, generating eye opening information.

at least one processor; the method comprises the steps of,

a memory communicatively coupled to the at least one processor; wherein,

The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.

The foregoing is only a specific embodiment of the invention to enable those skilled in the art to understand or practice the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for monitoring risk driving behavior, comprising the steps of:

Step S2: extracting eye features according to the image information of the driver, so as to obtain eye closing frequency features and eyeball movement track features; the step S2 specifically includes:

step S21: eye detection operation is carried out according to the image information of the driver, and eye position information of the driver is obtained, wherein the eye detection operation specifically comprises:

extracting and converting feature vectors according to the driver image information to obtain edge feature vector data and texture feature vector data;

image segmentation is carried out through a preset sliding window according to the image information of the driver, so that a cutting image is obtained;

performing proximity similarity calculation on the cut image according to the edge feature vector data and the texture feature vector data, so as to obtain a proximity fitting index; the proximity similarity calculation is performed by a proximity similarity calculation formula, wherein the proximity similarity calculation formula specifically comprises:

；

for adjacent fitting index->To cut the first part of the image>Weighting coefficients of the individual pixel values, +.>To cut the first part of the image>Individual pixel values +.>For the balance adjustment term of the weight coefficient, +.>For fine-tuning offset terms generated from edge feature vector data and texture feature vector data +_ >To cut the first part of the image>Weighting coefficients of neighboring pixel values of the individual pixel values, +.>To cut the first part of the image>Adjacent pixel values of the individual pixel values, < >>For pixel correlation coefficients generated from edge feature vector data and texture feature vectors +.>For initial adjustment item->For the dimension reduction coefficient adjacent to the fitting index, +.>For cutting the total number of pixels of the image +.>A correction coefficient that is adjacent to the fit index;

carrying out image combination on the cut images according to the adjacent fitting index so as to obtain a combined image;

acquiring an image shooting angle and carrying out image overturning on the combined image according to the image shooting angle so as to acquire an overturning image;

performing bilinear interpolation calculation according to the turnover image through a preset scaling ratio, so as to obtain a scaled image;

classifying and calculating according to the scaled image through a preset multidimensional linear regression curve, so as to obtain an eye index;

performing iterative computation according to the eye index until the eye index is determined to be greater than a preset eye threshold index, generating a correction sliding window according to the combined image corresponding to the eye index so as to correct the sliding window, and performing coordinate extraction according to the scaled image corresponding to the eye index and the standard driver image information to generate eye position information;

Step S22: extracting an eye region according to the driver image information and the driver eye position information to acquire eye region image information; the step S22 specifically includes:

extracting an eye region according to the driver image information and the driver eye position information, so as to obtain preliminary eye region image information;

performing distortion elimination calculation through an optimized epipolar constraint method according to the preliminary eye region image information, so as to obtain adjusted eye image information; the epipolar constraint optimization method specifically comprises the following steps:

acquiring parameter information of a vehicle-mounted camera;

performing matrix relation calculation according to the vehicle-mounted camera parameter information and preset angle transformation, so as to obtain an imaging matrix relation;

extracting features according to the preliminary eye region image information so as to obtain region feature information;

performing polar coordinate conversion according to the regional characteristic information through an imaging matrix relation, so as to obtain polar coordinate information;

performing optimized image distortion correction according to polar coordinate information so as to obtain distortion correction image information, wherein the image distortion correction is calculated through an optimized distortion correction calculation formula, and the optimized distortion correction calculation formula is specifically as follows:

；

Correcting pixel coordinates of image information for distortion, < >>For polar information, +.>For the slope of the curve +.>Is a curve with the curvature of a curve,for curve shape adjustment item +.>For fine tuning the adjustment item->For nonlinear adjustment item->For history frame->As an error correction term, a correction value is obtained,correcting the correction coefficient of the pixel point coordinates of the image information for distortion;

matrix relation calculation is carried out according to the distortion correction image information, so that eye adjustment image information is obtained;

performing corner detection and focus correction according to the eye image information, and generating corrected eye image information;

denoising according to the corrected eye image information, thereby obtaining denoising eye image information;

judging an illumination condition threshold according to the denoising eye image information, so as to generate an illumination image index;

judging whether the illumination image index is larger than or equal to a preset illumination image threshold index;

when the illumination image index is determined to be greater than or equal to a preset illumination image threshold index, denoising eye image information is determined to be eye region image information;

when the illumination image index is determined to be smaller than a preset illumination image threshold index, performing white balance calculation according to denoising eye image information, so as to obtain white balance image information;

Performing brightness enhancement calculation according to the white balance image information, thereby obtaining enhanced image information;

performing contrast adjustment according to the enhanced image information, thereby obtaining eye region image information;

step S23: extracting eye closing frequency characteristics according to the eye region image information, so as to obtain the eye closing frequency characteristics; the step S23 specifically includes:

preprocessing according to the eye region image information, so as to obtain preprocessed image information;

gray scale conversion is carried out according to the preprocessed image information, so that gray scale image information is obtained;

performing minimum estimated value region cutting according to the gray level image information, thereby obtaining estimated value region image information;

performing binarization processing according to the estimated value area image information, thereby binarizing the information;

marking the estimated area image information according to the binarization information, thereby obtaining marked area image information; the mark area image information comprises white mark area image information and black mark image area information, and the eye opening and closing information comprises eye opening information and eye closing information; the step of judging the threshold value of the marking area comprises the following steps of;

carrying out adjacent value statistics and maximum value extraction according to the black mark image area information, thereby obtaining black mark maximum value data;

Judging whether the maximum value data of the black mark is larger than or equal to preset threshold value data of the black mark;

when the maximum value data of the black mark is determined to be greater than or equal to the preset black mark threshold value data, generating eye closing information;

when the maximum value data of the black mark is less than or equal to the preset black mark threshold value data, generating eye opening information;

judging a marking area threshold according to the marking area image information, so as to generate eye opening and closing information;

marking the eye closing information according to the time information corresponding to the eye region image information, so as to obtain eye closing frequency characteristic information;

step S24: extracting the characteristics of the eye movement track according to the eye area image information, thereby obtaining the characteristics of the eye movement track;

2. A monitoring system for risk driving behavior, comprising:

at least one processor;

a memory communicatively coupled to the at least one processor;

the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the method of monitoring risk driving behaviour as claimed in claim 1.