CN112287861A - Road information enhancement and driving early warning method based on night environment perception - Google Patents

Road information enhancement and driving early warning method based on night environment perception Download PDF

Info

Publication number
CN112287861A
CN112287861A CN202011220517.3A CN202011220517A CN112287861A CN 112287861 A CN112287861 A CN 112287861A CN 202011220517 A CN202011220517 A CN 202011220517A CN 112287861 A CN112287861 A CN 112287861A
Authority
CN
China
Prior art keywords
image
night
road
images
enhancement
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202011220517.3A
Other languages
Chinese (zh)
Inventor
陈梅
王秋铖
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shandong Jiaotong University
Original Assignee
Shandong Jiaotong University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shandong Jiaotong University filed Critical Shandong Jiaotong University
Priority to CN202011220517.3A priority Critical patent/CN112287861A/en
Publication of CN112287861A publication Critical patent/CN112287861A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/588—Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/047—Probabilistic or stochastic networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06T5/73—
    • G06T5/90—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/56—Extraction of image or video features relating to colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/60—Extraction of image or video features relating to illumination properties, e.g. using a reflectance or lighting model
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/09—Arrangements for giving variable traffic instructions
    • G08G1/095—Traffic lights
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30252—Vehicle exterior; Vicinity of vehicle
    • G06T2207/30256—Lane; Road marking
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Abstract

The invention discloses a road information enhancement and driving early warning method based on night environment perception, which belongs to the technical field of intelligent vehicle safety auxiliary driving. In the actual driving process, road video images in the vehicle traveling process are collected, whether the images are at night or not is judged in real time according to a night image classification library trained offline, whether street lamps exist or not is judged, an optimal image enhancement algorithm is selected in real time, the visibility at night is visually enhanced, a night road front and side vehicle identification module based on deep learning is constructed, whether collision risks exist in the front side of the vehicle is judged, and safe driving early warning is carried out.

Description

Road information enhancement and driving early warning method based on night environment perception
Technical Field
The invention belongs to the technical field of intelligent vehicle safety auxiliary driving, and particularly relates to a road information enhancement and driving early warning method based on night environment perception.
Background
At present, a method for improving driving safety at night mainly adopts a fog lamp and judges whether the fog lamp is judged by a driver. However, the existing automobile fog lamp has the problems of limited irradiation range, blind area and the like under the night condition, so that the judgment of the driving condition of a driver is influenced. Therefore, the method has very important significance for sensing the night environment and researching a road information enhancement method to improve the night driving visual field.
At present, in the research of night image enhancement technology, the research on an actual road image as an object is relatively few, and because the colors of the actual night road image are different, some of the actual night road image are faint yellow and some of the actual night road image are gray or black, the difficulty is increased for night image enhancement. In actual processing, the enhanced image is obviously affected by different road lights in the image.
Disclosure of Invention
The invention aims to provide a road information enhancement and driving early warning method based on night environment perception aiming at the defects of the prior art, and the method for recovering road information under the condition of full black and severe night road and the method for enhancing the image quality under the night condition are provided by acquiring real night road images, so that the optimal visual effect of the images under the vision of human eyes is obtained under the condition of not changing light at night.
In order to achieve the purpose, the invention adopts the following technical scheme: a road information enhancement and driving early warning method based on night environment perception is characterized by comprising the following steps:
step one, acquiring a real-time image, establishing a night image classifier and carrying out real-time image classification to obtain a night road image with a street lamp and a full-black night road image;
establishing a full-black road information enhancement model based on a single-scale Retinex method, performing enhancement processing on a full-black night road image, comparing the image of the same content before and after the full-black night road image is enhanced, representing the gray change rate of the line by the edge gray change rate with the maximum number of pixel points at the interval of continuous rising of the gray value of each line, obtaining the average value of the gray change rates of all lines of the image, and performing the full-black night road image enhancement process again if the average value is less than or equal to a threshold value; if the average value exceeds a given threshold value, finishing the quality evaluation of the image enhancement of the full-black night road to obtain an enhanced clear full-black night road image;
step three, establishing a night streetlight image enhancement model based on the McCann99Retinex method, performing enhancement processing on the night streetlight image, comparing the images of the same content before and after the night streetlight image is enhanced, representing the gray change rate of the line by the edge gray change rate with the maximum number of pixels during the continuous rising of the gray value of each line, obtaining the average value of the gray change rates of all the lines of the image, and performing the night streetlight image enhancement process again if the average value is less than or equal to the threshold value; if the mean value exceeds a given threshold value, finishing the enhancement quality evaluation of the street lamp night road image to obtain an enhanced clear street lamp night road image;
fourthly, constructing a night front and side vehicle detection safe driving early warning strategy, thereby realizing safe driving early warning;
the specific process is as follows:
firstly, extracting candidate regions of the road image after being enhanced at night to obtain a rectangular target candidate region database image;
pre-training by using a convolutional neural network (RCNN) structure model in deep learning, specifically, taking a database image of a rectangular target candidate region as an input layer, extracting visual features of the candidate region by using a feature extraction layer for image feature extraction, wherein the visual features comprise edges, corners, textures and colors to form a feature map, and detecting and classifying on the feature map to realize pre-training;
thirdly, real-time classification is carried out by using a Support Vector Machine (SVM), an input layer is a characteristic diagram characteristic output by a candidate region through a convolutional neural network (RCNN) structure model, and an output layer outputs a classification type of a target and detects whether the target belongs to a front vehicle, a side vehicle and prediction frame position information; the predicted frame position information is coordinate information of rectangular frames of front and side vehicles in the four corners of an image coordinate system;
fourthly, when the vehicles appearing in the front and the side of the vehicle are detected, the warning is given to the driver.
Further, in the first step, the process of establishing a night image classifier and performing real-time image classification is as follows:
neural network classifier for establishing night road image
Firstly, an industrial camera collects N night road images and non-night road images, the collected images are transmitted to a night image classification module, wherein the N night road images comprise N1 night road images and N2 non-night road images, N, N1 and N2 are natural numbers, and an image training library of a night image classifier is established in the night image classification module;
secondly, off-line training a probabilistic neural network classifier through an image training library of the night image classifier, classifying the night road image and the non-night road image by the probabilistic neural network classifier according to the extracted night image texture, color and edge characteristics, obtaining the night image characteristics, and finishing the establishment of the night road image neural network classifier;
II establishing different night brightness image classifier
Firstly, an industrial camera collects M road images with different night visibility, wherein the M road images comprise M1 road images with street lamps at night and M2 road images at all black night, M, M1 and M2 are natural numbers, and image training libraries of image classifiers with different night visibility are established in image classification modules with different night visibility;
secondly, off-line training a Gaussian mixture model GMM classifier through an image training library of different night visibility image classifiers, extracting the average gradient characteristics, the contrast values and the edge intensity values of different night visibility images according to the average gradient characteristics, the contrast characteristics and the edge intensity characteristics by the Gaussian mixture model GMM classifier, classifying the night images with street lamps and the all-black night images according to the average gradient characteristics, the contrast values and the edge intensity values, and completing the establishment of the Gaussian mixture model GMM classifier of the different night visibility images;
thirdly, offline training a Support Vector Machine (SVM) classifier through an image training library of different night visibility image classifiers, classifying night images with street lamps and all-black night images by the SVM classifier according to brightness characteristics and image power spectrogram amplitude characteristics of HSV color spaces, extracting brightness values and image power spectrogram amplitude characteristics of the HSV color spaces of the images with different night visibility images, and completing the establishment of the SVM classifier of the different night visibility image support vector machines;
taking a union set of the street lamp night images classified by the Gaussian mixed model GMM classifiers of different night visibility images and the street lamp night images classified by the different night visibility image support vector machine SVM classifiers, and taking a union set of the all-black night images classified by the Gaussian mixed model GMM classifiers of different night visibility images and the all-black night images classified by the different night visibility image support vector machine SVM classifiers to respectively obtain a street lamp night road image and an all-black night road image sample library;
III real-time image classification
Firstly, an industrial camera collects a real-time image;
classifying by using a neural network classifier of the night road image to obtain a night road image;
thirdly, classifying the street lamp night images and the full black night images of the processed night road images by using different night visibility image Gaussian mixture models GMM classifiers and different night visibility image support vector machine SVM classifiers respectively, merging the street lamp night images classified by the different night visibility image Gaussian mixture models GMM classifiers with the street lamp night images classified by the different night visibility image support vector machine SVM classifiers, merging the full black night images classified by the different night visibility image Gaussian mixture models GMM classifiers with the full black night images classified by the different night visibility image support vector machine SVM classifiers, and respectively obtaining the street lamp night road images and the full black night road images;
IV making confidence evaluation standard
The confidence evaluation standard adopts a relative error evaluation standard based on L2 norm, the relative error evaluation standard based on L2 norm is that the obtained real-time streetlight night road image is subtracted from the corresponding pixel value of the streetlight night road image sample in the streetlight night image sample library, the obtained difference is subjected to square summation, and then the square root of the result is obtained; dividing the obtained square root value by the total number of the image pixels to obtain an average error value; taking the threshold value as 0.5, and judging that the image at the moment is a street lamp night road image if the average error value is less than 0.5;
similarly, the confidence evaluation standard adopts a relative error evaluation standard based on an L2 norm, the relative error evaluation standard based on an L2 norm is to subtract the corresponding pixel values of the obtained real-time all-black night road image and all-black night road image samples in an all-black night image sample library, calculate the sum of squares of the obtained difference values, and then calculate the square root of the result; dividing the obtained square root value by the total number of the image pixels to obtain an average error value; the threshold value is 0.5, the average error value is less than 0.5, and the image at that time can be judged to be a completely black night road image.
Further, in the second step, an all-black road information enhancement model based on the single-scale Retinex method is established, and the process of enhancing the all-black night road image is as follows:
firstly, decomposing an original all-black night road image into three RGB color channels based on a single-scale Retinex method model, wherein the form is as follows:
Ii(x,y)＝Ri(x,y)*Li(x,y)；
wherein, Ii(x, y) represents the distribution function of the low visibility all-black night road image to be enhanced, namely the image of the actually collected image at the (x, y) pointGray scale, Li(x, y) represents an incident light component, Ri(x, y) represents the reflected light component, i is a convolution operator, i represents the ith color channel, and i takes the values of 1, 2 and 3;
secondly, carrying out logarithmic conversion on the image of each channel, converting the image into a logarithmic domain for processing, and obtaining the following components by adopting a logarithmic conversion method for solving the incident light component:
logIi(x,y)＝logRi(x,y)+logLi(x,y)；
and thirdly, performing brightness enhancement processing on the original image by adopting convolution of the surrounding function and the original image, wherein the surrounding function has the following form:
F(x,y)＝keps-(x2+y2)/c2；
wherein k is a normalization factor, the surrounding function is subjected to ^ F (x, y) dxdy ═ 1, c is a scale parameter, and the size of c determines the enhancing effect of the final single-scale Retinex method;
fourthly, taking the scale parameter c as a middle scale value 130, and the enhancing effect is optimal; and finally, combining the three color channels, and adopting a single-scale Retinex method to enhance the output image form as follows:
R'i(x,y)＝logIi(x,y)-log(Ii(x,y)*F(x,y))；
night image batch enhancement
In order to realize the enhancement of the road image of the video sequence at the night, the video acquired by the industrial camera in real time is subjected to enhancement processing once every 60 frames on the single-frame image at the night.
Further, the threshold value in the second step is 10.
Further, in the third step, a nighttime streetlight image enhancement model based on the McCann99Retinex method is established, and the process of enhancing the nighttime streetlight image is as follows:
firstly, a conversion process: the McCann99 algorithm uses image pyramid mode to select pixels layer by layer, down-samples the image, the top layer has lowest image resolution and the bottom layer has highest resolution, transforms the original image to logarithmic domain, and the color image performs logarithmic transformation to each channel, because the original image has three channelsGray scale value range of 0,255]Log-logarithmically transformed to [0,1 ]]Range, set original image size as rows 2n×cols·2nThe pyramid layer number is n, the pyramid top layer size is rows × cols, wherein [ rows, cols]∈[1,5](ii) a rows represents column width and cols represents row width;
the pixel comparison process: initializing a constant image matrix R0(x, y), comparing each pixel point with 8 adjacent pixels from the top layer, estimating a reflectivity component R, and performing interpolation operation on the estimated reflectivity component after the previous layer is calculated, namely performing interpolation operation on the operation result R of the nth layer after the nth layer is operatedn(x, y) performing interpolation to double the original size and make the size of the image of the result R of the previous layer be the same as that of the image of the next layer of the pyramid, and performing the same comparison operation again; finally, after 8-neighborhood comparison is carried out on the original image at the bottom layer of the pyramid, a final result R is obtainedm(x, y), i.e. the enhanced image;
and (c) converting and outputting: converting the image obtained after enhancement back to a gray value between 0 and 255 through an exponential function, and outputting the enhanced image;
night image batch enhancement
In order to realize night image enhancement of a video sequence, the single-frame image with the street lamp at night is enhanced once every 60 frames of the video acquired by the industrial camera in real time.
Further, in step three, the threshold is 10.
Further, in the fourth step, for the road image after nighttime enhancement, firstly, the candidate region of the road image after enhancement is extracted, and the process of obtaining the database image of the rectangular target candidate region is as follows: acquiring the enhanced road image by using industrial cameras arranged in front of and on the side of the vehicle, storing the acquired video frame by frame, obtaining a tail part of the vehicle in front of the vehicle, side images of different vehicles and a negative sample image, marking target frames on the positive sample image, marking the front vehicle and the side vehicles respectively, normalizing the candidate areas into the same size, and establishing a rectangular target candidate area database image, wherein the positive sample image is a complete image, a complete or partial left and right side images of the vehicle at the tail part of the vehicle in front of the vehicle, and the negative sample image is a road image without the tail part of the vehicle in front of the vehicle and the left and right side areas of the vehicle.
Through the design scheme, the invention can bring the following beneficial effects:
1. the road information enhancement and driving early warning method based on night environment perception, provided by the invention, comprises the steps of constructing a night image classifier, establishing different night visibility image classifiers, establishing a night streetlight image enhancement model based on a McCann99Retinex method, establishing a black road information enhancement model based on an SSR method, classifying images in real time, and adopting different algorithms according to different categories to obtain the optimal vision enhancement effect. In the actual driving process, road video images in the vehicle traveling process are collected, whether the images are at night or not and whether street lamps exist or not are judged in real time according to a night image classification library trained offline, an optimal image enhancement algorithm is selected in real time, the visibility at night is enhanced, and the influence of low visibility at night on the vision of a driver is weakened.
2. The invention has short signal processing time inside the module and between modules, and can meet the requirement of real-time property.
3. The invention improves the accuracy of nighttime environment perception, can obtain the visual enhancement effect, is beneficial to popularization and application, and can greatly reduce the probability of malignant traffic accidents when a driver drives under the condition of low visibility at night.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limitation and are not intended to limit the invention in any way, and in which:
fig. 1 is a block diagram of a system adopted by a road information enhancement and driving early warning method based on nighttime environment perception in an embodiment of the present invention.
Fig. 2 is a flow chart of a road information enhancement and driving early warning method based on nighttime environment perception in the embodiment of the invention.
The respective symbols in the figure are as follows: the system comprises a vehicle-mounted storage battery 1, a sine wave inverter 2, an industrial camera 3, an image acquisition card 4, an industrial personal computer 5, a vehicle-mounted electronic control unit 6, a night image classification module 7, a night visibility image classification module 8, a night road image enhancement module 9, a night road image enhancement module 10, a black road information enhancement module 11, a night road vehicle identification module 12, an early warning module 13, a vehicle-mounted display screen 14, an alarm lamp device 15, a vehicle sound device 15 and a vehicle-mounted loudspeaker 16.
Detailed Description
The present invention is described in further detail below with reference to fig. 1 and 2. It should be understood that the scope of the present subject matter is not limited to the following examples, and that any techniques implemented based on the teachings of the present invention are within the scope of the present invention. Well-known methods, procedures, components and circuits have not been described in detail so as not to obscure the present invention.
As shown in fig. 1, the system adopted by the road information enhancement and driving early warning method based on night environment sensing provided by the invention comprises a vehicle-mounted storage battery 1, a sine wave inverter 2, an industrial camera 3, an image acquisition card 4, an industrial personal computer 5, a vehicle-mounted electric control unit 6, a vehicle-mounted display screen 13, an alarm lamp device 14, a vehicle sound device 15 and a vehicle-mounted loudspeaker 16, wherein the vehicle-mounted storage battery 1 is connected with the industrial camera 3 through the sine wave inverter 2, and meanwhile, the vehicle-mounted storage battery 1 is connected with the industrial personal computer 5 through the sine inverter 2; the industrial camera 3 is connected with an industrial personal computer 5 through an image acquisition card 4; the vehicle-mounted electronic control unit 6 comprises a night image classification module 7, different night visibility image classification modules 8, a night streetlight image enhancement module 9, a totally black road information enhancement module 10, a night road vehicle identification module 11 and an early warning module 12, wherein the input end of the night image classification module 7 is connected with the industrial personal computer 5, and the output end of the night image classification module 7 is connected with the different night visibility image classification modules 8; the output end of the image classification module 8 for different night visibility is respectively connected with the input ends of the image enhancement module 9 for the street lamp at night and the information enhancement module 10 for the all-black road; the input end of the night road vehicle identification module 11 is simultaneously connected with the output ends of the night streetlight image enhancement module 9 and the all-black road information enhancement module 10; the output end of the night road vehicle identification module 11 is connected with the early warning module 12; the vehicle-mounted display screen 13 is respectively connected with the night streetlight image enhancement module 9 and the full-black road information enhancement module 10; the input end of the alarm lamp device 14 is connected with the early warning module 12; the input end of the automobile sound equipment 15 is connected with the early warning module 12, and the output end of the automobile sound equipment 15 is connected with the vehicle-mounted loudspeaker 16.
A road information enhancement and driving early warning method based on night environment perception comprises the following steps:
step one, acquiring a real-time image, establishing a night image classifier and carrying out real-time image classification to obtain a night road image with a street lamp and a full-black night road image;
the process of establishing the night image classifier and performing real-time image classification is as follows:
neural network classifier for establishing night road image
Firstly, an industrial camera 3 collects N night road images and non-night road images, transmits the collected images to a night image classification module 7, wherein the images comprise N1 night road images and N2 non-night road images, N, N1 and N2 are natural numbers, and an image training library of a night image classifier is established in the night image classification module 7;
secondly, off-line training a probabilistic neural network classifier through an image training library of the night image classifier, classifying the night road image and the non-night road image by the probabilistic neural network classifier according to the extracted night image texture, color and edge characteristics, obtaining the night image characteristics, and finishing the establishment of the night road image neural network classifier;
II establishing different night brightness image classifier
Firstly, an industrial camera 3 collects M road images with different night visibility, wherein the M road images comprise M1 road images with street lamps at night and M2 road images at all black night, M, M1 and M2 are both natural numbers, and an image training library of image classifiers with different night visibility is established in an image classification module 8 with different night visibility;
secondly, off-line training a Gaussian mixture model GMM classifier through an image training library of different night visibility image classifiers, extracting the average gradient characteristics, contrast values and edge intensity values of different night visibility images according to the average gradient characteristics, contrast characteristics and edge intensity characteristics which can represent the night images, classifying the night images with street lamps and the completely black night images according to the average gradient characteristics, the contrast values and the edge intensity values, and completing the establishment of the Gaussian mixture model GMM classifier of the different night visibility images;
thirdly, offline training a Support Vector Machine (SVM) classifier through an image training library of different night visibility image classifiers, classifying the images with street lamps at night and the images with full black at night according to brightness characteristics and image power spectrogram amplitude characteristics of HSV color spaces capable of representing night images, extracting brightness values and image power spectrogram amplitude characteristics of the HSV color spaces of the images with different night visibility images, and completing the establishment of the different night visibility image SVM classifiers;
taking a union set of street lamp night images classified by a Gaussian mixture model GMM classifier of different night visibility images and street lamp images classified by a support vector machine SVM classifier of different night visibility images, taking a union set of all-black night images classified by a Gaussian mixture model GMM classifier of different night visibility images and all-black night images classified by a support vector machine SVM classifier of different night visibility images, respectively obtaining a street lamp night road image and an all-black night road image sample library, reducing errors caused by calculation limitations of a mathematical method, and increasing the capacity of the sample library;
III real-time image classification
Firstly, an industrial camera 3 collects a real-time image;
classifying by using a neural network classifier of the night road image to obtain a night road image;
thirdly, classifying the street lamp night images and the full black night images of the processed night road images by using different night visibility image Gaussian mixture models GMM classifiers and different night visibility image support vector machine SVM classifiers respectively, merging the street lamp night images classified by the different night visibility image Gaussian mixture models GMM classifiers with the street lamp night images classified by the different night visibility image support vector machine SVM classifiers, merging the full black night images classified by the different night visibility image Gaussian mixture models GMM classifiers with the full black night images classified by the different night visibility image support vector machine SVM classifiers, and respectively obtaining the street lamp night road images and the full black night road images;
IV making confidence evaluation standard
The confidence evaluation standard adopts a relative error evaluation standard based on L2 norm, the relative error evaluation standard based on L2 norm is that the obtained real-time streetlight night road image is subtracted from the corresponding pixel value of the streetlight night road image sample in the streetlight night image sample library, the obtained difference is subjected to square summation, and then the square root of the result is obtained; dividing the obtained square root value by the total number of the image pixels to obtain an average error value; the threshold value is taken to be 0.5, the average error value is less than 0.5, the confidence coefficient is high, the result is reliable, and the image at the moment can be judged to be the road image at night with the street lamp;
similarly, the confidence evaluation standard adopts a relative error evaluation standard based on an L2 norm, the relative error evaluation standard based on an L2 norm is to subtract the corresponding pixel values of the obtained real-time all-black night road image and all-black night road image samples in an all-black night image sample library, calculate the sum of squares of the obtained difference values, and then calculate the square root of the result; dividing the obtained square root value by the total number of the image pixels to obtain an average error value; the threshold value is taken to be 0.5, the average error value is less than 0.5, the confidence coefficient is high, the result is reliable, and the image at the moment can be judged to be a completely black night road image;
step two, establishing a Single-Scale Retinex (SSR) method-based all-black road information enhancement model
Firstly, decomposing an original all-black night road image into three RGB color channels, namely a Retinex algorithm model, wherein the form is as follows:
Ii(x,y)＝Ri(x,y)*Li(x,y)；
wherein, Ii(x, y) represents the distribution function of the low visibility all-black night road image to be enhanced, namely the image gray scale of the actually acquired image at the (x, y) point, Li(x, y) represents an incident light component, Ri(x, y) represents the reflected light component, i is a convolution operator, i represents the ith color channel, and i takes the values of 1, 2 and 3;
secondly, carrying out logarithmic conversion on the image of each channel, converting the image into a logarithmic domain for processing, and obtaining the following components by adopting a logarithmic conversion method for solving the incident light component:
logIi(x,y)＝logRi(x,y)+logLi(x,y)；
and thirdly, in order to improve the brightness of the original image, the convolution of the surround function and the original image is adopted to realize, and the form of the surround function is as follows:
F(x,y)＝keps-(x2+y2)/c2；
wherein k is a normalization factor, the surrounding function is subjected to ^ F (x, y) dxdy ═ 1, c is a scale parameter, and the size of c determines the enhancing effect of the final single-scale Retinex method;
inputting the scale size c of the surrounding function, and comprehensively comparing the multiple test results, wherein when the scale parameter c is the middle scale value 130, the enhancement effect is optimal; and finally, combining the three color channels, and adopting a single-scale Retinex method to enhance the output image form as follows:
R'i(x,y)＝logIi(x,y)-log(Ii(x,y)*F(x,y))；
night image batch enhancement
The SSR method (i.e., the single-scale Retinex method) is directed at a single image, and in order to enhance the full-black night road image of the video sequence, the industrial camera 3 performs enhancement processing on the full-black night single-frame image every 60 frames of the video acquired in real time;
comparing the images of the same content before and after the image enhancement of the full black night road, representing the gray change rate of each line by the edge gray change rate with the maximum number of pixel points at the interval of continuous rising of the gray value of each line, obtaining the average value of the gray change rates of all lines of the image, wherein the average value is less than or equal to a threshold value, and the threshold value is 10, and then performing the image enhancement process of the full black night road again; if the mean value exceeds a given threshold value and the threshold value is 10, finishing the quality enhancement evaluation of the full-black night road image to obtain an enhanced clear full-black night road image, and storing the enhanced full-black night road image into a cache of the vehicle-mounted display screen 13;
seventhly, the result after the full black image enhancement is displayed on the vehicle-mounted display screen 13
Pressing a night all-black road information enhancement function key on the vehicle-mounted display screen 13, and synchronously displaying a real-time clear image with similar daytime brightness after the instant night all-black road information enhancement in the cache by the vehicle-mounted display screen 13;
step three, establishing a nighttime streetlight image enhancement model based on McCann99Retinex method
Firstly, a conversion process: to estimate the reflection image, the McCann99 algorithm uses a pyramid-like manner to select pixels layer by layer, and downsamples the image, where the resolution of the top layer is the lowest and the bottom layer is the highest (typically the original image). The original image is transformed to logarithmic domain, the color image performs logarithmic transformation to each channel, and the gray value range of the three channels of the original image is [0,255]]Between, log-logarithmically converting to [0,1 ] is required]Range, set original image size as rows 2n×cols·2nThe pyramid layer number is n, the pyramid top layer size is rows × cols, wherein [ rows, cols]∈[1,5](ii) a rows represents the image line width, cols represents the image column width;
the pixel comparison process: initializing a constant image matrix R0(x, y), comparing each pixel point with 8 adjacent pixels from the top layer, estimating a reflectivity component R, and performing interpolation operation on the estimated reflectivity component after the previous layer is calculated, namely performing interpolation operation on the operation result R of the nth layer after the nth layer is operatedn(x, y) is interpolated to twice the original size of the n +1 layer to make the estimated junction of the previous layerThe image of the fruit R has the same size as the image of the next layer of the pyramid, and the same comparison operation is carried out again; finally, after 8-neighborhood comparison is carried out on the original image at the bottom layer of the pyramid, a final result R is obtainedm(x, y), i.e. the enhanced image;
and (c) converting and outputting: converting the image obtained after enhancement back to the gray value between 0 and 255 through an exponential function, and outputting the enhanced image similar to the daytime;
night image batch enhancement
Because the McCann99Retinex image enhancement algorithm is specific to a single image, in order to enhance the night image of a video sequence, the industrial camera 3 acquires a video in real time and performs enhancement processing on a single frame image with a street lamp at night every 60 frames;
comparing the images of the same content before and after the image enhancement of the street lamp night road, representing the gray change rate of the line by the edge gray change rate with the maximum number of pixel points at the interval of continuous rising of the gray value of each line, obtaining the average value of the gray change rates of all lines of the image, wherein the average value is less than or equal to the threshold value, and the threshold value is 10, and then carrying out the image enhancement process of the street lamp night road again; if the mean value exceeds a given threshold value and the threshold value is 10, finishing the enhancement quality evaluation of the street lamp night road image to obtain an enhanced clear street lamp night road image; storing the enhanced clear image into a cache of the vehicle-mounted display screen 13;
sixthly, the result after the street lamp image enhancement is displayed on the vehicle-mounted display screen 13 at night
Pressing a night non-black road information enhancement function key on the vehicle-mounted display screen 13, and synchronously displaying a real-time clear image with similar daytime brightness after the instant night street lamp road information enhancement in the cache by the vehicle-mounted display screen 13;
step four, constructing a night front and side vehicle detection safe driving early warning strategy
Firstly, extracting candidate regions of the road image after being enhanced at night to obtain a rectangular target candidate region database image;
specifically, the process of obtaining the database image of the rectangular target candidate area comprises the following steps: acquiring the enhanced road image by using industrial cameras arranged in front of and on the side of the vehicle, storing the acquired video frame by frame, obtaining a tail part of the vehicle in front of the vehicle, side images of different vehicles and a negative sample image, marking target frames on the positive sample image, marking the front vehicle and the side vehicles respectively, normalizing the candidate areas into the same size, and establishing a rectangular target candidate area database image, wherein the positive sample image is a complete image, a complete or partial left and right side images of the vehicle at the tail part of the vehicle in front of the vehicle, and the negative sample image is a road image without the tail part of the vehicle in front of the vehicle and the left and right side areas of the vehicle.
Pre-training by using a convolutional neural network (RCNN) structure model in deep learning, specifically, taking a database image of a rectangular target candidate region as an input layer, extracting visual features of the candidate region by using a feature extraction layer for image feature extraction, wherein the visual features comprise edges, corners, textures and colors to form a feature map, and detecting and classifying on the feature map to realize pre-training;
thirdly, real-time classification is carried out by using a Support Vector Machine (SVM), an input layer is a characteristic diagram characteristic output by a candidate region through a convolutional neural network (RCNN) structure model, and an output layer outputs a classification type of a target and detects whether the target belongs to a front vehicle, a side vehicle and prediction frame position information; the predicted frame position information is coordinate information of rectangular frames of front and side vehicles in the four corners of an image coordinate system;
fourthly, when the vehicles appearing in the front and the side of the vehicle are detected, the warning is given to the driver;
specifically, when it is detected that a vehicle appears in front of the vehicle, the warning module 12 sends a warning of "please decelerate, the road condition is abnormal" through the vehicle-mounted speaker 16, and a red warning lamp in the warning lamp device 14 flashes; if a vehicle appears at the side of the host vehicle, the warning module 12 issues a warning of "please keep the vehicle distance" through the vehicle-mounted speaker 16, and the yellow warning lamp in the warning lamp device 14 blinks; if the front and side vehicles are released, the warning module 12 gives a warning of "recovery to normal" through the vehicle-mounted speaker 16, and the warning lamp device 14 stops flashing.
One specific embodiment of the method is given below:
a road information enhancement and driving early warning method based on night environment perception comprises the following steps:
step one, establishing a night image classifier and performing real-time image classification
Neural network classifier for establishing night road image
The data is acquired by the industrial camera 3, the shooting is allowed to be carried out in a completely dark and fuzzy environment, the shot image is allowed to be not parallel to the ground, and the image is allowed to be slightly distorted. Acquiring 1000 night road images and 2000 non-night road images through an industrial camera 3, and establishing an image training library of a night image classifier;
secondly, off-line training a probabilistic neural network classifier through an image training library of the night image classifier, classifying the night road image and the non-night road image by the probabilistic neural network classifier according to the extracted night image texture, color and edge characteristics, obtaining the night image characteristics, and finishing the establishment of the night road image neural network classifier;
II establishing different night brightness image classifier
The industrial camera 3 collects images for training: 3000 road images with different night visibility are collected through an industrial camera 3, wherein 1000 road images with street lamps at night and 2000 road images with full black at night are collected, and an image training library of image classifiers with different night visibility is established through an offline training process;
secondly, off-line training a Gaussian mixture model GMM classifier through an image training library of different night visibility image classifiers, extracting the average gradient characteristics, contrast values and edge strength values of different night visibility images according to the average gradient characteristics, contrast characteristics and edge strength characteristics which can represent the night images, classifying the night images with street lamps and the completely black night images according to the average gradient characteristics, the contrast values and the edge strength values, and completing the establishment of the Gaussian mixture model GMM classifier of the different night visibility images;
thirdly, offline training a Support Vector Machine (SVM) classifier through an image training library of different night visibility image classifiers, classifying night images with street lamps and all-black nights by the SVM classifier according to brightness characteristics and image power spectrogram amplitude characteristics of HSV color spaces capable of representing the night images, extracting brightness values and image power spectrogram amplitude characteristics of the HSV color spaces of the images with different night visibility images, and completing establishment of the SVM classifier of the different night visibility image support vector machines;
taking a union set of night images with street lamps classified by a Gaussian mixture model GMM classifier of different night visibility images and night images with street lamps classified by a support vector machine SVM classifier of different night visibility images, taking a union set of all-black night images classified by a Gaussian mixture model GMM classifier of different night visibility images and all-black night images classified by a support vector machine SVM classifier of different night visibility images, respectively obtaining a sample library of images with street lamps at night roads and all-black night roads, reducing errors caused by calculation limitations of a mathematical method, and increasing the capacity of the sample library;
III real-time image classification
Firstly, an industrial camera 3 collects a real-time image;
classifying by using a neural network classifier of the night road image to obtain a night road image;
thirdly, classifying the street lamp night images and the full black night images of the processed night road images by using different night visibility image Gaussian mixture models GMM classifiers and different night visibility image support vector machine SVM classifiers respectively, merging the street lamp night images classified by the different night visibility image Gaussian mixture models GMM classifiers with the street lamp night images classified by the different night visibility image support vector machine SVM classifiers, merging the full black night images classified by the different night visibility image Gaussian mixture models GMM classifiers with the full black night images classified by the different night visibility image support vector machine SVM classifiers, and respectively obtaining the street lamp night road images and the full black night road images;
step two, establishing a Single-Scale Retinex (SSR) method-based all-black road information enhancement model
Firstly, decomposing an original all-black night road image into three RGB color channels, namely a Retinex algorithm model, wherein the form is as follows:
Ii(x,y)＝Ri(x,y)*Li(x,y)；
wherein, Ii(x, y) represents the distribution function of the low visibility all-black night road image to be enhanced, namely the image gray scale of the actually acquired image at the (x, y) point, Li(x, y) represents an incident light component, Ri(x, y) represents the reflected light component, i is a convolution operator, i represents the ith color channel, and i takes the values of 1, 2 and 3;
secondly, carrying out logarithmic conversion on the image of each channel, converting the image into a logarithmic domain for processing, and obtaining the following components by adopting a logarithmic conversion method for solving the incident light component:
logIi(x,y)＝logRi(x,y)+logLi(x,y)；
and thirdly, in order to improve the brightness of the original image, the convolution of the surround function and the original image is adopted to realize, and the form of the surround function is as follows:
F(x,y)＝keps-(x2+y2)/c2；
wherein k is a normalization factor, the surrounding function is subjected to ^ F (x, y) dxdy ═ 1, c is a scale parameter, and the size of c determines the enhancement effect of the final SSR method;
inputting a Gaussian model, wherein the Gaussian model is represented by the formula F (x, y) keps- (x)2+y2)/c2The scale size c is compared comprehensively through multiple test results, and when the scale parameter c is the middle scale value 130, the enhancement effect is optimal; and finally, combining the three color channels, and adopting an SSR method to enhance the output image form as follows:
R'i(x,y)＝logIi(x,y)-log(Ii(x,y)*F(x,y))；
night image batch enhancement
The SSR method aims at a single image, and in order to realize night image enhancement of a video sequence, the industrial camera 3 performs enhancement processing on a night all-black single-frame image every 60 frames of a video acquired in real time;
and sixthly, evaluating the night enhancement quality of the full-black image, wherein in the night image enhancement process, each step depends on the previous step, so that errors are accumulated, and the quality of the night image after enhancement needs to be evaluated. And comparing the night image before and after enhancement, wherein for the image with the same content, the edge of the enhanced image has the characteristic of larger gray change rate compared with the image before enhancement. And evaluating the image enhancement quality by using the image edge gray scale change rate as an index, namely representing the gray scale change rate of the line by using the edge gray scale change rate with the maximum number of pixel points at the continuous descending interval of the gray scale value of each line, and calculating the average value of the gray scale change rates of all lines of the image. Judging whether the enhancement requirement is met, if the threshold value is lower than the given threshold value, taking the threshold value as 10, carrying out the image enhancement process again, repeating the process until the threshold value requirement is met, taking the threshold value as 10, finishing the enhancement quality evaluation, and storing the enhanced similar daytime image into a cache of a vehicle-mounted display screen 13;
seventhly, the result after the full black image enhancement is displayed on the vehicle-mounted display screen 13
Pressing a night all-black road information enhancement function key on the vehicle-mounted display screen 13, and synchronously displaying a real-time clear image with similar daytime brightness after the latest updated night all-black road information in the cache is enhanced by the vehicle-mounted display screen 13; because the full black image enhancement process is continuously carried out and the cache is continuously updated, the vehicle-mounted display screen 13 continuously displays the vision enhancement image after the latest full black image enhancement. The image flow is stopped at last or the user presses a quit key, the user can press a D key at any time to reset a tracking option, and a driver can acquire the processed road environment in front of the vehicle on the vehicle-mounted display screen 13 by selecting the tracking option, so that the effect of vision enhancement is achieved;
step three, establishing a nighttime streetlight image enhancement model based on McCann99Retinex method
Firstly, a conversion process: to estimate the reflected image, the McCann99 algorithmAnd selecting pixels layer by layer in an image pyramid mode, and downsampling the image, wherein the resolution of the image at the top layer is the lowest, and the image at the bottom layer is the highest (generally the original image). The original image is transformed to logarithmic domain, the color image performs logarithmic transformation to each channel, and the gray value range of the three channels of the original image is [0,255]]Between, log-logarithmically converting to [0,1 ] is required]Range, set original image size as rows 2n×cols·2nThe pyramid layer number is n, the pyramid top layer size is rows × cols, wherein [ rows, cols]∈[1,5](ii) a rows represents the image line width, cols represents the image column width;
the pixel comparison process: initializing a constant image matrix R0(x, y), comparing each pixel point with 8 adjacent pixels from the top layer, estimating a reflectivity component R, and performing interpolation operation on the estimated reflectivity component after the previous layer is calculated, namely performing interpolation operation on the operation result R of the nth layer after the nth layer is operatedn(x, y) performing interpolation to double the original size and make the size of the image of the result R of the previous layer be the same as that of the image of the next layer of the pyramid, and performing the same comparison operation again; finally, after 8-neighborhood comparison is carried out on the original image at the bottom layer of the pyramid, a final result R is obtainedm(x, y), i.e. the enhanced image;
and (c) converting and outputting: converting the image obtained after enhancement back to the gray value between 0 and 255 through an exponential function; outputting the enhanced daytime-like image;
night image batch enhancement
The McCann99Retinex image enhancement algorithm is for a single image, and in order to enhance the night image of the video sequence, the industrial camera 3 performs enhancement processing on the single frame image with street lamps at night every 60 frames of the video acquired in real time.
And fifthly, night image night enhancement quality evaluation of the street lamp exists at night, and because each step depends on the previous step in the night image enhancement process, errors are accumulated, so that the quality of the night image after enhancement needs to be evaluated. And comparing the night image before and after enhancement, wherein for the image with the same content, the edge of the enhanced image has the characteristic of larger gray change rate compared with the image before enhancement. Evaluating the image enhancement quality by using the image edge gray scale change rate as an index, namely representing the gray scale change rate of each line by using the edge gray scale change rate with the maximum number of pixel points at the continuous descending interval of the gray scale value of each line, and calculating the average value of the gray scale change rates of all lines of the image; judging whether the enhancement requirement is met, if the threshold value is lower than the given threshold value, taking the threshold value as 10, carrying out the image enhancement process again, repeating the process until the threshold value requirement is met, taking the threshold value as 10, finishing the enhancement quality evaluation, and storing the enhanced similar daytime image into a cache of a vehicle-mounted display screen 13;
sixthly, the result after the street lamp image enhancement is displayed on the vehicle-mounted display screen 13 at night
The method comprises the steps that a night non-full-black road information enhancement function key on a vehicle-mounted display screen 13 is pressed, the vehicle-mounted display screen synchronously displays a real-time clear image similar to daytime brightness after street lamp road information enhancement is carried out at night in a cache, the non-full-black image enhancement process is continuously carried out, the cache is continuously updated, so that the vehicle-mounted display screen 13 continuously displays a latest vision enhancement image after street lamp image enhancement at night, the image flow is stopped at last or a user presses a quit key, the user can also press a D key to reset a tracking option at any time, and a driver can obtain a processed road environment in front of a vehicle on the vehicle-mounted display screen 13 by selecting the tracking option, so that the vision enhancement effect is achieved;
step four, constructing a night front and side vehicle detection safe driving early warning strategy
Firstly, extracting candidate regions of the road image after being enhanced at night to obtain a rectangular target candidate region database image;
specifically, the process of obtaining the database image of the rectangular target candidate area comprises the following steps: acquiring the enhanced road image by using industrial cameras arranged in front of and on the side of the vehicle, storing the acquired video frame by frame, obtaining a tail part of the vehicle in front of the vehicle, side images of different vehicles and a negative sample image, marking target frames on the positive sample image, marking the front vehicle and the side vehicles respectively, normalizing the candidate areas into the same size, and establishing a rectangular target candidate area database image, wherein the positive sample image is a complete image, a complete or partial left and right side images of the vehicle at the tail part of the vehicle in front of the vehicle, and the negative sample image is a road image without the tail part of the vehicle in front of the vehicle and the left and right side areas of the vehicle.
Pre-training by using a convolutional neural network (RCNN) structure model in deep learning, specifically, taking a database image of a rectangular target candidate region as an input layer, extracting visual features of the candidate region by using a feature extraction layer for image feature extraction, wherein the visual features comprise edges, corners, textures and colors to form a feature map, and detecting and classifying on the feature map to realize pre-training;
thirdly, real-time classification is carried out by using a Support Vector Machine (SVM), an input layer is a characteristic diagram characteristic output by a candidate region through a convolutional neural network (RCNN) structure model, and an output layer outputs a classification type of a target and detects whether the target belongs to a front vehicle, a side vehicle and prediction frame position information; the predicted frame position information is coordinate information of rectangular frames of front and side vehicles in the four corners of an image coordinate system;
when detecting that a vehicle appears in front of the vehicle, the early warning module 12 gives a warning of 'please decelerate, the road condition is abnormal' through the vehicle-mounted loudspeaker 16, and a red warning lamp in the warning lamp device 14 flickers; if a vehicle appears at the side of the host vehicle, the warning module 12 issues a warning of "please keep the vehicle distance" through the vehicle-mounted speaker 16, and the yellow warning lamp in the warning lamp device 14 blinks; if the front and side vehicles are released, the warning module 12 gives a warning of "recovery to normal" through the vehicle-mounted speaker 16, and the warning lamp device 14 stops flashing.
In the above specific embodiment, the number of the collected different road images is 3000, including 1000 different nighttime road images and 2000 non-nighttime road images (including buildings, grasslands, sky, etc.), but the scope of the collected number of the road images in the present invention is not limited to this embodiment, and based on the common general knowledge, the larger the previous data collection amount is, the higher the accuracy of the later data processing is, so this embodiment only gives an example of an end value, i.e. a minimum value; similarly, in the embodiment, the number of the road images with different night brightness collected is 3000, which includes 1000 road images with street lamps and 2000 road images with black night, and only the end value, i.e. the example of the minimum value, is given in the embodiment.

Claims (7)

1. A road information enhancement and driving early warning method based on night environment perception is characterized by comprising the following steps:
step one, acquiring a real-time image, establishing a night image classifier and carrying out real-time image classification to obtain a night road image with a street lamp and a full-black night road image;
establishing a full-black road information enhancement model based on a single-scale Retinex method, performing enhancement processing on a full-black night road image, comparing the image of the same content before and after the full-black night road image is enhanced, representing the gray change rate of the line by the edge gray change rate with the maximum number of pixel points at the interval of continuous rising of the gray value of each line, obtaining the average value of the gray change rates of all lines of the image, and performing the full-black night road image enhancement process again if the average value is less than or equal to a threshold value; if the average value exceeds a given threshold value, finishing the quality evaluation of the image enhancement of the full-black night road to obtain an enhanced clear full-black night road image;
step three, establishing a night streetlight image enhancement model based on the McCann99Retinex method, performing enhancement processing on the night streetlight image, comparing the images of the same content before and after the night streetlight image is enhanced, representing the gray change rate of the line by the edge gray change rate with the maximum number of pixels during the continuous rising of the gray value of each line, obtaining the average value of the gray change rates of all the lines of the image, and performing the night streetlight image enhancement process again if the average value is less than or equal to the threshold value; if the mean value exceeds a given threshold value, finishing the enhancement quality evaluation of the street lamp night road image to obtain an enhanced clear street lamp night road image;
fourthly, constructing a night front and side vehicle detection safe driving early warning strategy, thereby realizing safe driving early warning;
the specific process is as follows:
firstly, extracting candidate regions of the road image after being enhanced at night to obtain a rectangular target candidate region database image;
pre-training by using a convolutional neural network (RCNN) structure model in deep learning, specifically, taking a database image of a rectangular target candidate region as an input layer, extracting visual features of the candidate region by using a feature extraction layer for image feature extraction, wherein the visual features comprise edges, corners, textures and colors to form a feature map, and detecting and classifying on the feature map to realize pre-training;
thirdly, real-time classification is carried out by using a Support Vector Machine (SVM), an input layer is a characteristic diagram characteristic output by a candidate region through a convolutional neural network (RCNN) structure model, and an output layer outputs a classification type of a target and detects whether the target belongs to a front vehicle, a side vehicle and prediction frame position information; the predicted frame position information is coordinate information of rectangular frames of front and side vehicles in the four corners of an image coordinate system;
fourthly, when the vehicles appearing in the front and the side of the vehicle are detected, the warning is given to the driver.
2. The road information enhancement and driving early warning method based on night environment perception according to claim 1, wherein the road information enhancement and driving early warning method comprises the following steps: in the first step, the process of establishing a night image classifier and performing real-time image classification is as follows:
neural network classifier for establishing night road image
Firstly, an industrial camera collects N night road images and non-night road images, the collected images are transmitted to a night image classification module, wherein the N night road images comprise N1 night road images and N2 non-night road images, N, N1 and N2 are natural numbers, and an image training library of a night image classifier is established in the night image classification module;
secondly, off-line training a probabilistic neural network classifier through an image training library of the night image classifier, classifying the night road image and the non-night road image by the probabilistic neural network classifier according to the extracted night image texture, color and edge characteristics, obtaining the night image characteristics, and finishing the establishment of the night road image neural network classifier;
II establishing different night brightness image classifier
Firstly, an industrial camera collects M road images with different night visibility, wherein the M road images comprise M1 road images with street lamps at night and M2 road images at all black night, M, M1 and M2 are natural numbers, and image training libraries of image classifiers with different night visibility are established in image classification modules with different night visibility;
secondly, off-line training a Gaussian mixture model GMM classifier through an image training library of different night visibility image classifiers, extracting the average gradient characteristics, the contrast values and the edge intensity values of different night visibility images according to the average gradient characteristics, the contrast characteristics and the edge intensity characteristics by the Gaussian mixture model GMM classifier, classifying the night images with street lamps and the all-black night images according to the average gradient characteristics, the contrast values and the edge intensity values, and completing the establishment of the Gaussian mixture model GMM classifier of the different night visibility images;
thirdly, offline training a Support Vector Machine (SVM) classifier through an image training library of different night visibility image classifiers, classifying night images with street lamps and all-black night images by the SVM classifier according to brightness characteristics and image power spectrogram amplitude characteristics of HSV color spaces, extracting brightness values and image power spectrogram amplitude characteristics of the HSV color spaces of the images with different night visibility images, and completing the establishment of the SVM classifier of the different night visibility image support vector machines;
taking a union set of the street lamp night images classified by the Gaussian mixed model GMM classifiers of different night visibility images and the street lamp night images classified by the different night visibility image support vector machine SVM classifiers, and taking a union set of the all-black night images classified by the Gaussian mixed model GMM classifiers of different night visibility images and the all-black night images classified by the different night visibility image support vector machine SVM classifiers to respectively obtain a street lamp night road image and an all-black night road image sample library;
III real-time image classification
Firstly, an industrial camera collects a real-time image;
classifying by using a neural network classifier of the night road image to obtain a night road image;
thirdly, classifying the street lamp night images and the full black night images of the processed night road images by using different night visibility image Gaussian mixture models GMM classifiers and different night visibility image support vector machine SVM classifiers respectively, merging the street lamp night images classified by the different night visibility image Gaussian mixture models GMM classifiers with the street lamp night images classified by the different night visibility image support vector machine SVM classifiers, merging the full black night images classified by the different night visibility image Gaussian mixture models GMM classifiers with the full black night images classified by the different night visibility image support vector machine SVM classifiers, and respectively obtaining the street lamp night road images and the full black night road images;
IV making confidence evaluation standard
The confidence evaluation standard adopts a relative error evaluation standard based on L2 norm, the relative error evaluation standard based on L2 norm is that the obtained real-time streetlight night road image is subtracted from the corresponding pixel value of the streetlight night road image sample in the streetlight night image sample library, the obtained difference is subjected to square summation, and then the square root of the result is obtained; dividing the obtained square root value by the total number of the image pixels to obtain an average error value; taking the threshold value as 0.5, and judging that the image at the moment is a street lamp night road image if the average error value is less than 0.5;
similarly, the confidence evaluation standard adopts a relative error evaluation standard based on an L2 norm, the relative error evaluation standard based on an L2 norm is to subtract the corresponding pixel values of the obtained real-time all-black night road image and all-black night road image samples in an all-black night image sample library, calculate the sum of squares of the obtained difference values, and then calculate the square root of the result; dividing the obtained square root value by the total number of the image pixels to obtain an average error value; the threshold value is 0.5, the average error value is less than 0.5, and the image at that time can be judged to be a completely black night road image.
3. The road information enhancement and driving early warning method based on night environment perception according to claim 1, wherein the road information enhancement and driving early warning method comprises the following steps: in the second step, an all-black road information enhancement model based on the single-scale Retinex method is established, and the process of enhancing the all-black night road image is as follows:
firstly, decomposing an original all-black night road image into three RGB color channels based on a single-scale Retinex method model, wherein the form is as follows:
Ii(x,y)＝Ri(x,y)*Li(x,y)；
wherein, Ii(x, y) represents the distribution function of the low visibility all-black night road image to be enhanced, namely the image gray scale of the actually acquired image at the (x, y) point, Li(x, y) represents an incident light component, Ri(x, y) represents the reflected light component, i is a convolution operator, i represents the ith color channel, and i takes the values of 1, 2 and 3;
secondly, carrying out logarithmic conversion on the image of each channel, converting the image into a logarithmic domain for processing, and obtaining the following components by adopting a logarithmic conversion method for solving the incident light component:
logIi(x,y)＝logRi(x,y)+logLi(x,y)；
and thirdly, performing brightness enhancement processing on the original image by adopting convolution of the surrounding function and the original image, wherein the surrounding function has the following form:
F(x,y)＝keps-(x2+y2)/c2；
wherein k is a normalization factor, the surrounding function is subjected to ^ F (x, y) dxdy ═ 1, c is a scale parameter, and the size of c determines the enhancing effect of the final single-scale Retinex method;
fourthly, taking the scale parameter c as a middle scale value 130, and the enhancing effect is optimal; and finally, combining the three color channels, and adopting a single-scale Retinex method to enhance the output image form as follows:
R'i(x,y)＝logIi(x,y)-log(Ii(x,y)*F(x,y))；
night image batch enhancement
In order to realize the enhancement of the road image of the video sequence at the night, the video acquired by the industrial camera in real time is subjected to enhancement processing once every 60 frames on the single-frame image at the night.
4. The road information enhancement and driving early warning method based on night environment perception according to claim 1, wherein the road information enhancement and driving early warning method comprises the following steps: and the threshold value in the second step is 10.
5. The road information enhancement and driving early warning method based on night environment perception according to claim 1, wherein the road information enhancement and driving early warning method comprises the following steps: in the third step, a nighttime streetlight image enhancement model based on the McCann99Retinex method is established, and the process of enhancing the nighttime streetlight image is as follows:
firstly, a conversion process: the McCann99 algorithm uses image pyramid mode to select pixels layer by layer, down-samples the image, the top layer image has lowest resolution and the bottom layer has highest resolution, transforms the original image to logarithmic domain, and the color image performs logarithmic transformation to each channel, because the original image has three channels with gray value range of 0,255]Log-logarithmically transformed to [0,1 ]]Range, set original image size as rows 2n×cols·2nThe pyramid layer number is n, the pyramid top layer size is rows × cols, wherein [ rows, cols]∈[1,5](ii) a rows represents column width and cols represents row width;
the pixel comparison process: initializing a constant image matrix R0(x, y), comparing each pixel point with 8 adjacent pixels from the top layer, estimating a reflectivity component R, and performing interpolation operation on the estimated reflectivity component after the previous layer is calculated, namely performing interpolation operation on the operation result R of the nth layer after the nth layer is operatedn(x, y) performing interpolation to double the original size and make the size of the image of the result R of the previous layer be the same as that of the image of the next layer of the pyramid, and performing the same comparison operation again; finally, after 8-neighborhood comparison is carried out on the original image at the bottom layer of the pyramid, a final result R is obtainedm(x，y) I.e. the enhanced image;
and (c) converting and outputting: converting the image obtained after enhancement back to a gray value between 0 and 255 through an exponential function, and outputting the enhanced image;
night image batch enhancement
In order to realize night image enhancement of a video sequence, the single-frame image with the street lamp at night is enhanced once every 60 frames of the video acquired by the industrial camera in real time.
6. The road information enhancement and driving early warning method based on night environment perception according to claim 1, wherein the road information enhancement and driving early warning method comprises the following steps: in step three, the threshold is 10.
7. The road information enhancement and driving early warning method based on night environment perception according to claim 1, wherein the road information enhancement and driving early warning method comprises the following steps: in the fourth step, for the road image enhanced at night, firstly, the candidate region of the enhanced road image is extracted to obtain a database image of a rectangular target candidate region, and the process is as follows: acquiring the enhanced road image by using industrial cameras arranged in front of and on the side of the vehicle, storing the acquired video frame by frame, obtaining a tail part of the vehicle in front of the vehicle, side images of different vehicles and a negative sample image, marking target frames on the positive sample image, marking the front vehicle and the side vehicles respectively, normalizing the candidate areas into the same size, and establishing a rectangular target candidate area database image, wherein the positive sample image is a complete image, a complete or partial left and right side images of the vehicle at the tail part of the vehicle in front of the vehicle, and the negative sample image is a road image without the tail part of the vehicle in front of the vehicle and the left and right side areas of the vehicle.
CN202011220517.3A 2020-11-05 2020-11-05 Road information enhancement and driving early warning method based on night environment perception Pending CN112287861A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011220517.3A CN112287861A (en) 2020-11-05 2020-11-05 Road information enhancement and driving early warning method based on night environment perception

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011220517.3A CN112287861A (en) 2020-11-05 2020-11-05 Road information enhancement and driving early warning method based on night environment perception

Publications (1)

Publication Number Publication Date
CN112287861A true CN112287861A (en) 2021-01-29

Family

ID=74351465

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011220517.3A Pending CN112287861A (en) 2020-11-05 2020-11-05 Road information enhancement and driving early warning method based on night environment perception

Country Status (1)

Country Link
CN (1) CN112287861A (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113298055A (en) * 2021-07-27 2021-08-24 深兰盛视科技（苏州）有限公司 Vein identification method, vein identification device, vein identification equipment and computer readable storage medium
CN114283288A (en) * 2021-12-24 2022-04-05 合肥工业大学智能制造技术研究院 Method, system, equipment and storage medium for enhancing night vehicle image
CN114494063A (en) * 2022-01-25 2022-05-13 电子科技大学 Night traffic image enhancement method based on biological vision mechanism
CN116894791A (en) * 2023-08-01 2023-10-17 中国人民解放军战略支援部队航天工程大学 Visual SLAM method and system for enhancing image under low illumination condition

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102044151A (en) * 2010-10-14 2011-05-04 吉林大学 Night vehicle video detection method based on illumination visibility identification
CN102231206A (en) * 2011-07-14 2011-11-02 浙江理工大学 Colorized night vision image brightness enhancement method applicable to automotive assisted driving system
CN105512623A (en) * 2015-12-02 2016-04-20 吉林大学 Foggy-day driving visual enhancement and visibility early warning system and method based on multiple sensors
CN109815802A (en) * 2018-12-18 2019-05-28 中国海洋大学 A kind of monitor video vehicle detection and recognition method based on convolutional neural networks

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102044151A (en) * 2010-10-14 2011-05-04 吉林大学 Night vehicle video detection method based on illumination visibility identification
CN102231206A (en) * 2011-07-14 2011-11-02 浙江理工大学 Colorized night vision image brightness enhancement method applicable to automotive assisted driving system
CN105512623A (en) * 2015-12-02 2016-04-20 吉林大学 Foggy-day driving visual enhancement and visibility early warning system and method based on multiple sensors
CN109815802A (en) * 2018-12-18 2019-05-28 中国海洋大学 A kind of monitor video vehicle detection and recognition method based on convolutional neural networks

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
陈梅: "不利视觉下能见度检测及道路信息恢复方法研究", 《中国优秀博硕士学位论文全文数据库(博士) 工程科技Ⅰ辑》 *

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113298055A (en) * 2021-07-27 2021-08-24 深兰盛视科技（苏州）有限公司 Vein identification method, vein identification device, vein identification equipment and computer readable storage medium
CN114283288A (en) * 2021-12-24 2022-04-05 合肥工业大学智能制造技术研究院 Method, system, equipment and storage medium for enhancing night vehicle image
CN114283288B (en) * 2021-12-24 2022-07-12 合肥工业大学智能制造技术研究院 Method, system, equipment and storage medium for enhancing night vehicle image
CN114494063A (en) * 2022-01-25 2022-05-13 电子科技大学 Night traffic image enhancement method based on biological vision mechanism
CN114494063B (en) * 2022-01-25 2023-04-07 电子科技大学 Night traffic image enhancement method based on biological vision mechanism
CN116894791A (en) * 2023-08-01 2023-10-17 中国人民解放军战略支援部队航天工程大学 Visual SLAM method and system for enhancing image under low illumination condition
CN116894791B (en) * 2023-08-01 2024-02-09 中国人民解放军战略支援部队航天工程大学 Visual SLAM method and system for enhancing image under low illumination condition

Similar Documents

Publication Publication Date Title
CN112308799A (en) Offshore road complex environment visibility optimization screen display method based on multiple sensors
CN112287861A (en) Road information enhancement and driving early warning method based on night environment perception
CN109977812B (en) Vehicle-mounted video target detection method based on deep learning
CN101783076B (en) Method for quick vehicle type recognition under video monitoring mode
US9384401B2 (en) Method for fog detection
CN105512623A (en) Foggy-day driving visual enhancement and visibility early warning system and method based on multiple sensors
CN103034843B (en) Method for detecting vehicle at night based on monocular vision
CN106686280A (en) Image repairing system and method thereof
CN112329623A (en) Early warning method for visibility detection and visibility safety grade division in foggy days
CN105184271A (en) Automatic vehicle detection method based on deep learning
JP6723328B2 (en) Vehicle detection method, night-time vehicle detection method and system based on dynamic light intensity
AU2009295350A1 (en) Detection of vehicles in an image
CN109886086B (en) Pedestrian detection method based on HOG (histogram of oriented gradient) features and linear SVM (support vector machine) cascade classifier
CN111553214B (en) Method and system for detecting smoking behavior of driver
CN111161160B (en) Foggy weather obstacle detection method and device, electronic equipment and storage medium
CN110060221B (en) Bridge vehicle detection method based on unmanned aerial vehicle aerial image
CN106407951A (en) Monocular vision-based nighttime front vehicle detection method
CN113903012A (en) Collision early warning method and device, vehicle-mounted equipment and storage medium
CN112419745A (en) Highway group fog early warning system based on degree of depth fusion network
CN115376108A (en) Obstacle detection method and device in complex weather
CN115147450B (en) Moving target detection method and detection device based on motion frame difference image
CN112330559B (en) Early warning method for image information recovery and lane keeping of severe foggy roads
CN111667655A (en) Infrared image-based high-speed railway safety area intrusion alarm device and method
CN115311241B (en) Underground coal mine pedestrian detection method based on image fusion and feature enhancement
CN115100618B (en) Multi-source heterogeneous perception information multi-level fusion characterization and target identification method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20210129