CN109871778B - Lane keeping control method based on transfer learning - Google Patents

Abstract

The invention discloses a lane keeping control method based on transfer learning, which comprises the steps of firstly collecting videos recorded by a front camera of a driving recorder and steering control signals of a vehicle as training data, preprocessing the data by changing brightness, resizing, increasing shadows and the like, using VGGNet trained based on an ImageNet data set as a feature extraction network, and simultaneously adding a full connection layer on the top layer to construct an end-to-end lane line keeping control network model; further training the model through the collected driving video and steering signals, and finally evaluating the robustness of the network model; the VGGNet trained based on the ImageNet data set is used as the feature extraction network, the steering angle of automatic driving can be well fitted under the condition that vehicle-mounted computing resources and data sets are limited, meanwhile, the method has certain effectiveness and reliability in the aspect of generalization of a network model, and can be widely applied to various lane line maintenance task systems related to automatic driving.

Description

Lane Keeping Control Method Based on Transfer Learning

technical field

The invention belongs to the technical field of computers, and in particular relates to a lane keeping control method based on transfer learning.

Background technique

Traditional pattern recognition methods based on computer vision often require artificially designed features. Human-designed features are often prone to omissions. For self-driving cars, a program design flaw that ignores a certain situation may cause serious consequences. In recent years, the deep learning method represented by Convolutional Neural Network (CNN) has become one of the key technologies in the field of autonomous driving because of its adaptability to complex environments. The successful application of convolutional neural networks mainly depends on two factors: one is a large public dataset (such as ImageNet), and the other is a high-performance computing platform (such as a GPU cluster). However, there are still some technical difficulties to be solved in the process of using deep learning for automatic driving.

One of them is that the deep learning algorithm model has high requirements for computing resources, which is not conducive to the training of on-board computing platforms. The training of large-scale deep neural networks requires high computing resources. However, due to the needs of power consumption and volume of the vehicle-mounted computing platform, it is only suitable for building a simple deep learning computing environment, which hinders the widespread application of deep learning in the field of autonomous driving. one of the bottlenecks. For example, using the ImageNet dataset to train a convolutional neural network with a depth of 16 layers, it takes about 10 days to complete the model training using NVIDIA Tesla K80GPU. If you switch to an ordinary PC (take GTX970 as an example), it may take more than a year to complete the training. Another problem is that the collection of driving data sets is difficult, and models trained with smaller data sets do not generalize well in real complex environments.

Therefore, in the case of limited computing resources and data sets, how to efficiently obtain a reliable model is a major difficulty in the application of deep learning technology in the field of autonomous driving. In addition, traditional lane keeping algorithms include, for example, the patent "lane keeping and automatic centering system and method based on deviation prediction algorithm" proposed by Chongqing Changan Automobile Co., Ltd. (application date: 2014-10-31, application number: CN 201410613860, Publication number: CN104442814B) and the patent "A Lane Line Detection Stabilization Algorithm" proposed by the University of Electronic Science and Technology of China (application date: 2017-8-18, application number: CN 201710712894, publication number: CN107516078A). Both of the above two patent-applied methods can effectively detect lane lines to a certain extent. However, these two methods essentially separate lane recognition from the generation of steering control signals, and then maintain lane lines based on specific rules. Therefore, it is difficult for these rule-based algorithms to meet the driving needs under complex conditions.

Contents of the invention

The object of the present invention is to provide a lane keeping control method based on transfer learning, which adopts an end-to-end model and directly outputs the vehicle steering control signal by inputting the front-end video of the original driving recorder, so as to achieve the purpose of keeping lane lines and solve the existing problems. technical problem.

In order to achieve the above object, the technical solution adopted in the present invention is a lane keeping control method based on transfer learning, comprising the following specific steps:

S1, collecting video data and steering data during driving;

S2, segment and convert the data collected by S1; the data set after segmentation and conversion is divided into training set and test set,

S3, build an end-to-end lane keeping control model, specifically:

S31 uses transfer learning to initialize the model, uses VGGNet trained based on the ImageNet dataset as a feature extraction network, and freezes the weights of the first 8 layers of VGG16; adds a Flatten layer between the convolutional layer and the fully connected layer; on this basis , and at the same time change the original 3 fully connected layers of VGG16 to 5 fully connected layers; add another fully connected layer to output the steering angle value, and construct an end-to-end lane keeping control model;

S32, optimize the model constructed in step 31, and optimize the model by using Max-pooling, BatchNormalization, and Truncated Normal methods at the algorithm level;

S4, using the training set data separated by S2 to train the model constructed by S3 to obtain the training model and weight parameters;

S5. During the driving process of the vehicle, the training model and weight parameters obtained in S4 are used to realize real-time control of the driving lane of the vehicle according to the road surface information monitored by the driving recorder of the vehicle.

In S1, the driving video data comes from the video in the bicycle recorder, and the steering data uses the steering control signal during the driving of the vehicle.

In S2, the video data in S1 is encoded in the H264/MKV format with a resolution of 1280×720.

In S2, the video data in S1 is extracted frame by frame and preprocessed by changing brightness, resizing and adding shadows.

In S3, Maxpooling is used to reduce the error of feature extraction; a Bernoulli function is used to randomly generate a vector of 0 or 1 with probability P, so that a certain neuron stops working with probability P;

Batch Normalization is used to add normal normalization processing in the middle layer of the deep network, and at the same time constrain the network to automatically adjust the strength of the normalization during the training process.

In S3, except for the last layer in the network model, all other layers use the correction unit Relu as the activation function.

In S3, TruncatedNormal is used to initialize the truncated Gaussian distribution, discarding data outside the two standard deviations of the mean and re-forming the truncated distributed data.

In S4, the running environment of the model is: NVIDIA Tesla K80, 12GB Memory, 61GB RAM, and Tensorflow-gpu1.10.0.

MSE (Mean Squared Error) is used to evaluate the robustness and accuracy of the training model obtained in S4; MSE reflects the robustness and accuracy of the training model by evaluating the degree of data change, the smaller the value of MSE, the higher the accuracy of the model; MSE is:

为S2得到的测试集中第i个数据的真实转向值，

为S4中得到的模型根据测试集中输入的图像数据对第i个转向信号的预测值，N为数据集的大小；MSE＜3°时，S4所得模型的稳健性和精确度符合要求，当MSE≥3°时，重复S3和S4，直至MSE＜3°。in,

The true steering value of the i-th data in the test set obtained for S2,

is the prediction value of the i-th turn signal by the model obtained in S4 based on the input image data in the test set, N is the size of the data set; when MSE<3°, the robustness and accuracy of the model obtained in S4 meet the requirements, when MSE When ≥3°, repeat S3 and S4 until MSE<3°.

Compared with the prior art, the present invention has at least the following beneficial effects:

First of all, the neural network algorithm adopted in the present invention integrates the feature extraction step into the algorithm, which improves the efficiency of model building; overcomes the traditional pattern recognition method to extract features from the original data, the number of image pixels is too large, and the data dimension High, it will cause a disaster of dimensionality. It is a problem that needs enough experience to design features, and it is becoming more and more difficult when the amount of data is increasing; traditional lane keeping methods usually combine lane line recognition with steering control signal generation. Separate processing, first identify the lane line, and then perform steering control based on specific rules to achieve the effect of lane line keeping. Therefore, these rule-based judgment algorithms are difficult to meet the driving needs under complex conditions, and the present invention adopts end-to-end The control model directly outputs the steering angle value through the last fully connected layer, and inputs the video data collected by the driving recorder to directly output the steering control angle of the vehicle, thus simplifying the control mode; the deep learning algorithm model itself has high requirements for computing resources and To the characteristic of strong dependence on data; To this, the present invention is based on transfer learning in described method, utilizes VGGNet based on ImageNet data set training as feature extraction network, under the all limited situation of on-board computing resource and data set, propose A method for obtaining an efficient and reliable model; simulation results show that the present invention can better fit the steering angle of automatic driving, and has certain effectiveness and reliability in model generalization, and can be widely used in various types of automatic driving Relevant lane keeping task system.

Description of drawings

Fig. 1 is the realization flow diagram of the present invention;

Figure 2(a) and Figure 2(b) are the collected image data, and Figure 2(c) and Figure 2(d) are the data diagrams in the middle of processing.

Figure 3(a) is the general neural network training process, and Figure 3(b) is the training process using the Dropout neural network.

Figure 4 is a schematic diagram of the correction unit activation function ReLU(x)

Fig. 5 is the framework diagram of the algorithm model of the present invention;

Fig. 6 is a comparison chart of the prediction result of the model of the present invention and the performance of the test set.

Detailed ways

The present invention first collects the video recorded by the front camera of the driving recorder and the steering control signal of the vehicle as training data, preprocesses the data by changing the brightness, resetting the size, adding shadows, etc., and uses the VGGNet trained based on the ImageNet data set as training data. Feature extraction network, while adding a fully connected layer on the top layer, thereby constructing an end-to-end lane line keeping control model; then further training the model through the collected driving video and turn signals; finally using MSE to evaluate the robustness of the model .

With reference to accompanying drawing 1, concrete realization steps of the present invention are as follows:

S1: Collect Datasets

First, the video recorded by the front camera of the driving recorder and the steering control signal of the vehicle are collected as training data. The dataset gives the steering angle corresponding to each frame of image. In the present invention, the input independent variable X is a single-frame image captured by the camera, and the output variable Y is the steering angle of the steering wheel. The essence of the problem is to use a convolutional neural network (CNN) to train an end-to-end model F, input a video image x, and use the model to predict the steering wheel steering angle, that is, y=f(x);

S2: Data preprocessing

The video file is encoded in H264/MKV format with a resolution of 1280×720; there are a lot of interference factors in the video, such as the sky, etc., and the data set needs to be segmented and converted.

First, the video will be extracted frame by frame according to the picture, which is convenient for further image processing. Referring to Figure 2, specifically refer to Figure 2(a), Figure 2(b), Figure 2(c) and Figure 2(d) to preprocess the data by changing the brightness, resizing, adding shadows, etc., so that the data When the set is limited, it plays the role of artificially increasing the size of the training set.

In addition to lighting conditions, another possible situation on real road conditions is shadows, such as shadows of buildings, trees, and other vehicles, which will interfere with the algorithm results, so the present invention simulates this situation by randomly adding shadows—— Randomly select 3 to 4 points on the image to darken the tone of this area, which is achieved by adding a layer of shadow mask to the second channel in the HLS color gamut; and then divide the processed data dataset into training sets and the test set.

S3: Model building

S31 model initialization: use VGGNet trained based on the ImageNet dataset as a feature extraction network, freeze the weights of the first 8 layers of VGG16, add a Flatten layer between the convolutional layer and the fully connected layer, and at the same time use the original 3 fully connected layers of VGG16 Change to 5 fully connected layers, and the last fully connected layer directly outputs the steering angle value, thereby constructing an end-to-end lane keeping control model;

Because the automatic driving data set is small, the present invention is based on transfer learning, and uses the means of transfer learning to initialize the network model; the present invention freezes the weights of the first 8 layers of the VGGNet network model, that is, the layers with the strongest generalization ability, Then perform S32 on the following network in order to achieve the desired effect, specifically: first import the VGG16notop weight based on ImageNet training, this network model is suitable for feature extraction; add a regularization layer at the bottom of the network model to improve the operation speed of the network model; freeze The weight parameters of the first 8 layers retain the generalization ability; add the Flatten layer between the convolutional layer and the fully connected layer, and change the original 3 fully connected layers of VGG16 to 5 fully connected layers, and the last fully connected layer directly Output the predicted angle value, so as to realize the function of end-to-end prediction, wherein, the mathematical expression of the fully connected layer network is: Y=WH+b; where W represents the weight matrix between the fully connected layer and the previous layer network, b is the bias value matrix, and Y is the output of this layer; the Flatten layer is used for flattening parameters, that is, multi-dimensional input is one-dimensional.

S32 network model optimization:

The error of S321 feature extraction mainly comes from two aspects: one is the increase in the variance of the estimated value caused by the limitation of the neighborhood size; the other is the deviation of the estimated mean value caused by the parameter error of the convolution layer, and the first error is reduced by using Mean-pooling , more background information of the image is retained, Max-pooling is used to reduce the second error, and more texture information is retained; the present invention needs to identify mainly lane line texture information, and Max-pooling is used to reduce the error of feature extraction .

S322, the large-scale neural network model has two disadvantages: long training time and easy overfitting; referring to Fig. 3, the present invention uses Dropout in the training process of the deep learning network. Discarded from the network model; the present invention uses the Bernoulli function to randomly generate a vector of 0 or 1 with probability P, so that a certain neuron stops working with probability P, that is, the activation value of this neuron becomes 0 .

S323, using Batch Normalization to add normal normalization processing in the middle layer of the network model, appearing as a BN layer, and constraining the network model to automatically adjust the normalization strength during the training process, thereby speeding up the training speed and reducing the cost of weight initialization;

S324, each layer of the network model except the last layer uses the correction unit Relu as the activation function, so that the network model can converge faster and will not be saturated, and is used to combat the problem of gradient disappearance. ReLU is applied to each pixel, and the All pixel values less than 0 in the feature map are set to zero, introducing nonlinearity in ConvNet, and its convergence speed is much faster than sigmoid. In the negative part, ReLU will set it to 0, so the gradient is also 0. During the training process The negative part will not be updated, as shown in Figure 4;

S325, use TruncatedNormal to initialize the truncated Gaussian distribution, and the data located outside the two standard deviations of the mean will be discarded and regenerated to form truncated distributed data; the finally obtained model structure is shown in Figure 5.

S4, training model: use the training set data separated by S2 to train the model built by S3, and obtain the training model (JSON format) and weight parameters (HDF5 format) exported by keras; the operating environment is NVIDIA Tesla K80, 12GB Memory , 61GB RAM and Tensorflow-gpu1.10.0;

S5, during the driving process of the vehicle, use the training model and weight parameters obtained in S4, and realize the real-time control of the vehicle driving lane according to the road surface information monitored by the vehicle's driving recorder, and perform the same processing on the data in the driving recorder as in S2 Use it as the input of the model obtained in S4 to control the driving of the vehicle in the lane in real time.

MSE (Mean Squared Error) is used to evaluate the robustness and accuracy of the training model obtained in S4. The smaller the value of MSE, the better the accuracy of the prediction model describing the experimental data.

为第i个数据的真实转向值

为模型对第i个信号的预测值，N为数据集的大小，MSE＜3°时，训练模型的稳健性和精确度符合要求；本发明模型预测结果与测试集性能比较结果如图6所示，本发明所得训练模型的稳健性和精确度均符合要求。in,

is the true steering value of the i-th data

is the predicted value of the i-th signal by the model, N is the size of the data set, and when MSE<3°, the robustness and accuracy of the training model meet the requirements; the model prediction results of the present invention and the performance comparison results of the test set are shown in Figure 6 It is shown that the robustness and accuracy of the training model obtained by the present invention all meet the requirements.

Claims (6)

1. The lane keeping control method based on the transfer learning is characterized by comprising the following specific steps of:

s1, collecting video data and steering data in a driving process;

s2, segmenting and converting the data collected in the S1; dividing the data set after cutting and conversion into a training set and a testing set;

s3, constructing an end-to-end lane line keeping control model, specifically:

s31, initializing the model by adopting a transfer learning means, and freezing the front 8 layers of weights of the VGG16 by using VGGNet trained on the ImageNet data set as a feature extraction network; adding a Flatten layer between the convolution layer and the full-connection layer; on the basis, the original 3 full connection layers of the VGG16 are changed into 5 full connection layers; adding a full-connection layer output steering angle value to construct an end-to-end lane keeping control model;

s32, optimizing the model constructed in the step 31, and optimizing the model by adopting a Max-posing, batch Normalization and Truncated Normalization method in an algorithm level; specifically, the method comprises the following steps: reducing errors of feature extraction by adopting Maxpooling; a Bernoulli function is adopted to randomly generate a vector of 0 or 1 according to the probability P, so that a certain neuron stops working according to the probability P;

adding normal standardization processing in a middle layer of the deep network by adopting Batch Normalization, and simultaneously restricting the network to automatically adjust the standardized strength in the training process;

except the last layer in the network model, the other layers all use a correction unit Relu as an activation function;

initializing Truncated Gaussian distribution by adopting a Truncated Normal, discarding data beyond two standard deviations of the mean value, and reforming the data of the Truncated distribution;

s4, training the model constructed in the S3 by adopting the training set data separated in the S2 to obtain a training model and weight parameters;

and S5, in the driving process of the vehicle, the training model and the weight parameters obtained in the S4 are adopted, and the real-time control of the driving lane of the vehicle is realized according to the road information monitored by the vehicle data recorder of the vehicle.

2. The method for controlling lane keeping based on transfer learning of claim 1, wherein in S1, the video data in driving is from a video in a drive recorder, and the steering data is a steering control signal in the driving process of the vehicle.

3. The method of claim 1, wherein in S2, the video data in S1 is encoded in H264/MKV format at 1280 x 720 resolution.

4. The method for controlling lane keeping based on transfer learning of claim 1, wherein in S2, the video data in S1 is pre-processed by changing brightness, resizing and increasing shadow after being extracted frame by frame.

5. The transfer learning-based lane-keeping control method according to claim 1, wherein in S4, the operating environment of the model is: NVIDIA Tesla K80, 12GB Memory, 61GB RAM and Tensorflow-gpu1.10.0.

6. The transfer learning-based lane-keeping control method according to claim 1, wherein the robustness and accuracy of the training model obtained in S4 is evaluated using MSE; MSE is:

wherein,

for the true steering value of the ith data in the test set obtained at S2,

the model obtained in S4 is used for predicting the ith steering signal according to the image data input in the test set, N is the size of the data set, when MSE is less than 3 degrees, the robustness and the accuracy of the model obtained in S4 meet the requirements, and when MSE is more than or equal to 3 degreesS3 and S4 are repeated until MSE < 3 deg..

Images