CN113379711B - An Image-Based Method for Obtaining the Adhesion Coefficient of Urban Road Pavement - Google Patents

Abstract

The invention provides an image-based urban road pavement adhesion coefficient acquisition method, which comprises the steps of firstly establishing a pavement image information base, then establishing a pavement image data set, establishing and training a pavement image area extraction network, then establishing and training a pavement type recognition network, and finally acquiring pavement adhesion coefficient information; the method can provide road adhesion coefficient information for the development of an intelligent driving auxiliary system and an unmanned driving system; the method realizes the acquisition of the adhesion coefficient by acquiring the front road image, and can realize the advance acquisition of the road adhesion information; the method designs a method for extracting the network and the road surface identification network serial based on the image road surface area, and then simplifies the structure of the road surface identification network, so that the real-time and rapid acquisition of the front road surface adhesion information can be realized.

Description

An Image-Based Method for Obtaining the Adhesion Coefficient of Urban Road Pavement

technical field

The invention belongs to the technical field of intelligent vehicles, and relates to a method for obtaining a road surface adhesion coefficient, more particularly, to an image-based method for obtaining the road surface adhesion coefficient of an urban road.

Background technique

With the development of automobile intelligence, users have higher and higher requirements for the performance of in-vehicle intelligent driving assistance systems and unmanned driving systems, and the performance improvement of most intelligent driving assistance systems depends on the accuracy of dynamic control, high-performance dynamic The design of the control system needs to obtain real-time and accurate road information. The estimator based on the dynamic model can obtain the real-time and accurate estimated value of the road adhesion coefficient, but this dynamic estimation method largely depends on the accuracy of the vehicle model and the tire model. And it needs to meet certain driving excitation conditions. In addition, the dynamic estimation results reflect the road adhesion coefficient at the contact footprint between the tire and the road surface. There is a certain hysteresis, and it is difficult to obtain the predicted value of the road adhesion coefficient in advance.

At this stage, more and more smart cars are equipped with cameras and other equipment to obtain road information and surrounding vehicle information, which brings new opportunities for the research on road adhesion coefficient identification methods. A certain prediction ability enables intelligent driving vehicles to adjust control strategies in advance in the event of sudden changes in the road surface and improve the ability to cope with dangerous conditions. However, how to use image sensing information to obtain road adhesion coefficient information is still a challenge.

SUMMARY OF THE INVENTION

In order to overcome the above problems existing in the prior art, the present invention provides an image-based method for obtaining the adhesion coefficient of urban road pavement.

This method is achieved through the following technical solutions:

An image-based method for obtaining the adhesion coefficient of urban road pavement, the specific steps are as follows:

Step 1. Establish a road image information database

The precondition of using image-based pavement adhesion coefficient acquisition is that a complete pavement image information database can be established, and the sample images can be properly processed to ensure that the feature information in the images can be fully acquired;

First of all, it is necessary to collect road image data. In the process of road image collection, it is necessary to make up for the factors that are unfavorable to the imaging effect. The image acquisition equipment is not limited to a certain type or type of image acquisition equipment. The equipment performance and installation location requirements are as follows: with 1280 The video resolution of ×720 and above, the video frame rate is 30 frames per second and above, the maximum effective shooting distance is more than 70 meters, and the wide dynamic technology can quickly adapt to changes in light intensity; the installation location of the equipment should ensure that the acquired image information The captured road occupies more than half of the entire image area;

According to the urban road pavement conditions under different weather conditions, through comparative analysis and combined with the types of urban road pavements in my country, the pavement types that need to be identified are specifically defined as asphalt pavement, cement pavement, loose snow pavement, compacted snow pavement and ice slab pavement. Pavement type, the video files of the data collection process are decomposed into pictures at intervals of 10 frames, according to the pavement characteristics in "GB/T 920-2002 Highway Pavement Grade and Surface Type Code" and "Investigation and Analysis of Pavement Adhesion Coefficient in Cold Areas" Organize the pictures according to the above five attribution categories, and store the same type of road images in the same folder to complete the establishment of the road image information database;

Step 2. Create a road image dataset

The original collected images still contain a large number of non-pavement elements, which seriously affects the accuracy of the pavement adhesion coefficient acquisition. Therefore, the image-based pavement adhesion coefficient acquisition method requires image samples and pixel-level labels of the corresponding areas of the pavement. Therefore, step 1 is required. The images in the collected road image information database are used to mark the road range. Labelme in the software Anaconda is selected as the labeling tool, and each picture in the sample set is manually labelled one by one using the labeling tool. The labeling process is to click the create polygons button. Draw points along the boundary of the road area in the picture, so that the label box can completely cover the road area, and the label type is named road; after labeling, save it to generate a json file, and use the json_to_dataset.py script program that comes with the software Anaconda to convert the json file Convert to get the _json folder. The folder contains five files with names and suffixes: img.png, label.png, label_viz.png, info.yaml and label_names.txt. You only need to format the label.png image. The file can be converted to an 8-bit grayscale label image. The above-mentioned labeling process is performed on the pictures in the road image information database with Labelme in Anaconda in turn, and the grayscale label atlas and road image information of the pictures in the road image information database are obtained. The grayscale label atlas of the pictures in the library is the road image dataset;

Step 3. Establish and train the road image region extraction network

The extraction process of pavement image area is implemented in the Anaconda environment through the semantic segmentation network. The entire semantic segmentation network is an encoder-decoder structure, and the specific design is as follows:

3.1. First, scale the image to be recognized into a picture with a size of 769×769×3 as the input of the semantic segmentation network;

3.2. Then set the first layer as the convolution layer, use 32 filters of 3 × 3 size, the stride is 2, the padding is 1, and after batch regularization and ReLU activation function, the obtained convolution layer output feature map The size is 385×385×32;

3.3. Input the output feature map of the convolutional layer to the maximum pooling layer, the size is 3×3, the stride is 2, and the size of the output feature map of the pooling layer is 193×193×32;

3.4. Use the output feature map of the pooling layer as the input of the bottleneck module structure. The detailed implementation process of the bottleneck module structure is as follows: First, copy the input feature channel to increase the feature dimension, and one of the branches directly passes through the size of 3 × 3 and the step size of 2 The depth convolution, the other branch is divided into two sub-branches by channel splitting, one sub-branch undergoes 3×3 depth convolution and 1×1 point-by-point convolution, and the other sub-branch directly adopts feature complex Then, the two sub-branches are connected by channel splicing, and then the channel arrangement order is disrupted by channel cleaning, and after the same depth convolution with the size of 3 × 3 and the stride of 2, it is spliced with the copy channel. , and finally through 1×1 point-by-point convolution to achieve information exchange between groups, so the output feature map size of the entire bottleneck module structure is reduced by half, the number of channels is doubled, and the output feature map size after one bottleneck module structure is 97. ×97×64;

3.5. Take the output result of process 3.4 as input, go through the bottleneck module structure again, and obtain the output feature map size of 49×49×128 after passing through the bottleneck module structure twice. Take this result as input, go through the bottleneck module structure again, and get the The output feature map size of the triple bottleneck module structure is 25×25×256, which is 32 times smaller than the original image, and the whole part above is used as the encoder part of the semantic segmentation network;

3.6. The decoder part adopts the skip structure, and uses the bilinear interpolation method to upsample the output feature map of the third-order bottleneck module structure through 3.5 to obtain a feature map with a size of 49 × 49 × 256. The output feature maps of the two-time bottleneck module structure of 3.5 are added pixel by pixel. In this process, the output feature channels of the two-time bottleneck module structure of process 3.5 need to be copied to ensure that the result is still 256 output channels;

3.7. Use the bilinear interpolation method again to upsample the result of process 3.6 by a factor of 2 to obtain a feature map with a size of 97 × 97 × 256, and perform pixel by pixel with the output feature map of the primary bottleneck module structure of process 3.4. add;

3.8. Pass the result of process 3.7 through the 1×1 convolution layer to make the number of output channels become the number of semantic categories, and set the Dropout layer to reduce the occurrence of overfitting. Finally, the features with the same size as the original image are obtained by upsampling 8 times. Figure, and use the Argmax function to give the semantic category prediction result of each pixel point according to the maximum probability, and finally obtain the entire semantic segmentation network;

Randomly scramble the pavement image data set established in step 2, select 80% of the sample images as the training set, and 20% of the sample images as the validation set; during the training of the semantic segmentation network; read the training set image tensor according to 0.25 steps The length is randomly scaled between 0.5 times and 2 times, and the image tensor is randomly cropped with a size of 769×769 pixels and randomly flipped left and right to achieve the purpose of data enhancement, improve the adaptability of the segmentation network, and change the pixel value from 0 -255 normalized to between 0-1;

When training the semantic segmentation network, the Poly learning rate rule is used, the learning rate decay expression is formula (1), the initial learning rate is 0.001, the number of training iteration steps is iter, the maximum number of training steps max_iter is set to 20K steps, and the power is set to 0.9; Use the Adam optimization algorithm to dynamically adjust the learning rate of each parameter by using the first-order moment estimation and second-order moment estimation of the gradient. According to the computer hardware performance, set the batch size to 16, save the model parameters every 10-30min, and at the same time Use the validation set to evaluate the performance of the network;

After the network training is completed, appropriate semantic segmentation evaluation indicators need to be selected to evaluate the model performance. Before that, the confusion matrix is first introduced. As shown in Table 1, each row of the binary confusion matrix represents the predicted category, and each row of the binary confusion matrix represents the predicted category. One column represents the true attribution category of the data, and the specific value indicates the number of samples predicted to be a certain category;

Table 1 Schematic representation of the two-class confusion matrix

The evaluation index of the semantic segmentation network is the average intersection and union ratio MIoU, which represents the ratio of the intersection and union of each type of prediction result and the real value, summed and averaged, as shown in formula (2):

When the training reaches MIoU>60%, it can be considered that the training is completed. Save the model and model parameters after training to obtain the road image area extraction network, and input the actual collected original image into the road surface image area extraction network to complete the road area in the image. extraction;

Step 4. Establish and train the pavement type recognition network

The process of extracting the road surface area in the real-time image information can be completed through the network in step 3, and step 4 is to complete the identification of the road surface type on the basis of the road surface area extraction result in step 3;

After the image pavement data set is processed by the semantic segmentation network, an image set containing only the pavement area is obtained, which will be used as the final data set for training and evaluating the pavement type recognition network. The specific network structure is designed as follows:

4.1. First, the image to be classified and recognized is scaled into a picture with a size of 224×224×3, which is used as the input of the convolutional neural network;

4.2. Then set the first layer as the convolution layer, use 32 filters of 3 × 3 size, the stride is 2, the padding is 1, and after batch regularization and ReLU activation function, the output feature map of the convolution layer is obtained The size is 112×112×32;

4.3. Input the output feature map of the convolutional layer to the maximum pooling layer, the size is 3×3, the stride is 2, and the output feature map size of the pooling layer is 56×56×32;

4.4. The output feature map of the pooling layer is used as the input of the bottleneck module structure. The detailed implementation process of the bottleneck module structure is as follows: First, copy the input feature channel to increase the feature dimension, and one of the branches directly passes through the size of 3 × 3 and the step size of 2 The depth convolution, the other branch is divided into two sub-branches by channel splitting, one sub-branch undergoes 3×3 depth convolution and 1×1 point-by-point convolution, and the other sub-branch directly adopts feature complex Then, the two sub-branches are connected by channel splicing, and then the channel arrangement order is disrupted by channel cleaning, and is spliced with the copy channel after the same depth convolution with the same size of 3 × 3 and a stride of 2. , and finally through 1×1 point-by-point convolution to achieve information exchange between groups, it can be seen that the output feature map size of the entire bottleneck module structure is reduced by half, and the number of channels is doubled. ×28×64;

4.5. Take the output result of process 4.4 as input, go through the bottleneck module structure again, get the output feature map size of 14×14×128 after passing through the bottleneck module structure twice, take this result as input, go through the bottleneck module structure again, get the The output feature map size of the triple bottleneck module structure is 7×7×256;

4.6. Use a global average pooling layer of size 7×7 to convert the output result in process 4.5 into a feature map of size 1×1×256;

4.7. Use a fully connected layer and Softmax function as the network classifier, convert the output feature map in process 4.6 into the probability value of each category, and use the Argmax function to determine the network classification result according to the maximum probability value;

Then, the images containing only the road surface area obtained in step 3 are made into a data set for training the road surface type recognition network. Different types of road surface images are still classified and stored according to the folder names established in step 1, and the images in different folders are read in turn. data, and add 5-bit 0/1 label information and road adhesion coefficient information, see Table 2, adjust the image size to 224×224 pixels by bilinear interpolation, and normalize the pixel value from 0-255 Change it to between 0 and 1, scramble the road image data set and randomly select 20% of each type of pictures as the validation set, and the rest as the training set;

Table 2 Pavement image category labels

After the training set and validation set of the road surface type recognition network are completed, the network model is trained and evaluated. The batch size is set to 64, the cross entropy loss function is selected, and the Adam optimization algorithm is used. The basic learning rate is 0.0001. When MIoU>80%, it can be considered that the training is completed, and the model and training results are saved according to the number of iterations epoch, and the trained road type recognition network can be obtained;

Step 5. Obtain the information of the pavement adhesion coefficient

The process of obtaining road adhesion coefficient information is as follows: the front road image is captured by the camera during the driving process of the vehicle, and the front road image captured by the camera is transmitted to the road image area extraction network to obtain the road area, and then the image containing only the road area is transmitted to The road surface type identification network performs classification and identification. After the road surface identification is completed, the adhesion coefficient range of the road surface is determined according to the corresponding vehicle speed, see Table 2, and the middle value of the upper and lower limits of the road adhesion coefficient range is the current road adhesion coefficient, and the road surface can be completed. Acquisition of adhesion coefficient information.

Compared with the prior art, the beneficial effects of the present invention are:

The invention discloses an image-based road adhesion coefficient acquisition method, which can provide road adhesion coefficient information for the development of an intelligent driving assistance system and an unmanned driving system; the method realizes the acquisition of the adhesion coefficient by acquiring the road image ahead, and can realize the road surface adhesion coefficient. Advance acquisition of adhesion information; because this method designs a network serial method based on road image area extraction and road type identification network, it can realize real-time and fast acquisition of front road adhesion information.

Description of drawings

Fig. 1 is a schematic flowchart of an image-based method for obtaining the adhesion coefficient of urban road pavement in this method;

Fig. 2 is the network structure diagram of the road surface image area extraction in this method;

Fig. 3 is the bottleneck module structure diagram in this method;

Fig. 4 is the network structure diagram of road type identification in this method;

Detailed ways

The invention proposes an image-based method for obtaining the road surface adhesion coefficient of urban roads for the problem of obtaining the road surface adhesion coefficient information required for the development of intelligent vehicle driving assistance and unmanned driving technology.

An image-based method for obtaining the adhesion coefficient of urban road pavement according to the present invention, the specific steps are as follows:

Step 1. Establish a road image information database

The precondition of using image-based pavement adhesion coefficient acquisition is that a complete pavement image information database can be established, and the sample images can be properly processed to ensure that the feature information in the images can be fully acquired;

First of all, it is necessary to collect road image data. In the process of road image collection, it is necessary to make up for the factors that are unfavorable to the imaging effect. The image acquisition equipment is not limited to a certain type or type of image acquisition equipment. The equipment performance and installation location requirements are as follows: with 1280 The video resolution of ×720 and above, the video frame rate is 30 frames per second and above, the maximum effective shooting distance is more than 70 meters, and the wide dynamic technology can quickly adapt to changes in light intensity; the installation location of the equipment should ensure that the acquired image information The captured road occupies more than half of the entire image area;

According to the urban road pavement conditions under different weather conditions, through comparative analysis and combined with the types of urban road pavements in my country, the pavement types that need to be identified are specifically defined as asphalt pavement, cement pavement, loose snow pavement, compacted snow pavement and ice slab pavement. Pavement type, the video files of the data collection process are decomposed into pictures at intervals of 10 frames, according to the pavement characteristics in "GB/T 920-2002 Highway Pavement Grade and Surface Type Code" and "Investigation and Analysis of Pavement Adhesion Coefficient in Cold Areas" Organize the pictures according to the above five attribution categories, and store the same type of road images in the same folder to complete the establishment of the road image information database;

Step 2. Create a road image dataset

The original collected images still contain a large number of non-pavement elements, which seriously affects the accuracy of the pavement adhesion coefficient acquisition. Therefore, the image-based pavement adhesion coefficient acquisition method requires image samples and pixel-level labels of the corresponding areas of the pavement. Therefore, step 1 is required. The images in the collected road image information database are used to mark the road range. Labelme in the software Anaconda is selected as the labeling tool, and each picture in the sample set is manually labelled one by one using the labeling tool. The labeling process is to click the create polygons button. Draw points along the boundary of the road area in the picture, so that the label box can completely cover the road area, and the label type is named road; after labeling, save it to generate a json file, and use the json_to_dataset.py script program that comes with the software Anaconda to convert the json file Convert to get the _json folder. The folder contains five files with names and suffixes: img.png, label.png, label_viz.png, info.yaml and label_names.txt. You only need to format the label.png image. The file can be converted to an 8-bit grayscale label image. The above-mentioned labeling process is performed on the pictures in the road image information database with Labelme in Anaconda in turn, and the grayscale label atlas and road image information of the pictures in the road image information database are obtained. The grayscale label atlas of the pictures in the library is the road image dataset;

Step 3. Training of the road image area extraction network

The road image area extraction network is implemented in the Anaconda environment through the semantic segmentation network. The structure of the semantic segmentation network is shown in Figure 2. The entire semantic segmentation network is an encoder-decoder structure. The specific design is as follows:

3.1. First, scale the image to be recognized into a picture with a size of 769×769×3 as the input of the semantic segmentation network;

3.2. Then set the first layer as the convolution layer, use 32 filters of 3 × 3 size, the stride is 2, the padding is 1, and after batch regularization and ReLU activation function, the obtained convolution layer output feature map The size is 385×385×32;

3.3. Input the output feature map of the convolutional layer to the maximum pooling layer, the size is 3×3, the stride is 2, and the size of the output feature map of the pooling layer is 193×193×32;

3.4. The output feature map of the pooling layer is used as the input of the bottleneck module structure. The bottleneck module structure is shown in Figure 3. The detailed implementation process of the bottleneck module structure is as follows: First, the input feature channel is copied to increase the feature dimension, and one of the branches directly passes the size It is a 3×3 depth convolution with a stride of 2, and the other branch is divided into two sub-branches by channel splitting. One sub-branch undergoes a 3×3 depth convolution and a 1×1 point-by-point convolution. The other sub-branch directly adopts the method of feature multiplexing, and then connects the two sub-branches through channel splicing, and then disrupts the channel arrangement order through channel cleaning. After depth convolution, it is spliced with the copy channel, and finally the information exchange between groups is realized through 1×1 point-by-point convolution. Therefore, the output feature map size of the entire bottleneck module structure is reduced by half, and the number of channels is doubled. After a bottleneck The output feature map size of the module structure is 97×97×64;

3.5. Take the output result of process 3.4 as input, go through the bottleneck module structure again, and obtain the output feature map size of 49×49×128 after passing through the bottleneck module structure twice. Take this result as input, go through the bottleneck module structure again, and get the The output feature map size of the triple bottleneck module structure is 25×25×256, which is 32 times smaller than the original image, and the whole part above is used as the encoder part of the semantic segmentation network;

3.6. The decoder part adopts the skip structure, and uses the bilinear interpolation method to upsample the output feature map of the third-order bottleneck module structure through 3.5 to obtain a feature map with a size of 49 × 49 × 256. The output feature maps of the two-time bottleneck module structure of 3.5 are added pixel by pixel. In this process, the output feature channels of the two-time bottleneck module structure of process 3.5 need to be copied to ensure that the result is still 256 output channels;

3.7. Use the bilinear interpolation method again to upsample the result of process 3.6 by a factor of 2 to obtain a feature map with a size of 97 × 97 × 256, and perform pixel by pixel with the output feature map of the primary bottleneck module structure of process 3.4. add;

3.8. Pass the result of process 3.7 through the 1×1 convolution layer to make the number of output channels become the number of semantic categories, and set the Dropout layer to reduce the occurrence of overfitting. Finally, the features with the same size as the original image are obtained by upsampling 8 times. Figure, and use the Argmax function to give the semantic category prediction result of each pixel point according to the maximum probability, and finally obtain the entire semantic segmentation network;

Randomly scramble the pavement image data set established in step 2, select 80% of the sample images as the training set, and 20% of the sample images as the validation set; during the training of the semantic segmentation network; read the training set image tensor according to 0.25 steps The length is randomly scaled between 0.5 times and 2 times, and the image tensor is randomly cropped with a size of 769×769 pixels and randomly flipped left and right to achieve the purpose of data enhancement, improve the adaptability of the segmentation network, and change the pixel value from 0 -255 normalized to between 0-1;

When training the semantic segmentation network, the Poly learning rate rule is used, the learning rate decay expression is formula (1), the initial learning rate is 0.001, the number of training iteration steps is iter, the maximum number of training steps max_iter is set to 20K steps, and the power is set to 0.9; Use the Adam optimization algorithm to dynamically adjust the learning rate of each parameter by using the first-order moment estimation and second-order moment estimation of the gradient. According to the computer hardware performance, set the batch size to 16, save the model parameters every 10-30min, and at the same time Use the validation set to evaluate the performance of the network;

After the network training is completed, appropriate semantic segmentation evaluation indicators need to be selected to evaluate the model performance. Before that, the confusion matrix is first introduced. As shown in Table 1, each row of the binary confusion matrix represents the predicted category, and each row of the binary confusion matrix represents the predicted category. One column represents the true attribution category of the data, and the specific value indicates the number of samples predicted to be a certain category;

Table 1 Schematic representation of the two-class confusion matrix

The evaluation index of the semantic segmentation network is the average intersection and union ratio MIoU, which represents the ratio of the intersection and union of each type of prediction result and the real value, summed and averaged, as shown in formula (2):

When the training reaches MIoU>60%, it can be considered that the training is completed. Save the model and model parameters after training to obtain the road image area extraction network, and input the actual collected original image into the road surface image area extraction network to complete the road area in the image. extraction;

Step 4. Train the road recognition network

The process of extracting the road surface area in the real-time image information can be completed through the network in step 3, and step 4 is to complete the identification of the road surface type on the basis of the road surface area extraction result in step 3;

After the image pavement data set is processed by the semantic segmentation network, an image set containing only the pavement area is obtained, which will be used as the final data set for training and evaluating the pavement type recognition network. As shown in Figure 4, the specific network structure is designed as follows:

4.1. First, the image to be classified and recognized is scaled into a picture with a size of 224×224×3, which is used as the input of the convolutional neural network;

4.2. Then set the first layer as the convolution layer, use 32 filters of 3 × 3 size, the stride is 2, the padding is 1, and after batch regularization and ReLU activation function, the output feature map of the convolution layer is obtained The size is 112×112×32;

4.3. Input the output feature map of the convolutional layer to the maximum pooling layer, the size is 3×3, the stride is 2, and the output feature map size of the pooling layer is 56×56×32;

4.4. The output feature map of the pooling layer is used as the input of the bottleneck module structure. The detailed implementation process of the bottleneck module structure is as follows: First, copy the input feature channel to increase the feature dimension, and one of the branches directly passes through the size of 3 × 3 and the step size of 2 The depth convolution, the other branch is divided into two sub-branches by channel splitting, one sub-branch undergoes 3×3 depth convolution and 1×1 point-by-point convolution, and the other sub-branch directly adopts feature complex Then, the two sub-branches are connected by channel splicing, and then the channel arrangement order is disrupted by channel cleaning, and is spliced with the copy channel after the same depth convolution with the same size of 3 × 3 and a stride of 2. , and finally through 1×1 point-by-point convolution to achieve information exchange between groups, it can be seen that the output feature map size of the entire bottleneck module structure is reduced by half, and the number of channels is doubled. ×28×64;

4.5. Take the output result of process 4.4 as input, go through the bottleneck module structure again, get the output feature map size of 14×14×128 after passing through the bottleneck module structure twice, take this result as input, go through the bottleneck module structure again, get the The output feature map size of the triple bottleneck module structure is 7×7×256;

4.6. Use a global average pooling layer of size 7×7 to convert the output result in process 4.5 into a feature map of size 1×1×256;

4.7. Use a fully connected layer and Softmax function as the network classifier, convert the output feature map in process 4.6 into the probability value of each category, and use the Argmax function to determine the network classification result according to the maximum probability value;

Then, the images containing only the road surface area obtained in step 3 are made into a data set for training the road surface type recognition network. Different types of road surface images are still classified and stored according to the folder names established in step 1, and the images in different folders are read in turn. data, and add 5-bit 0/1 label information and road adhesion coefficient information, see Table 2, adjust the image size to 224×224 pixels by bilinear interpolation, and normalize the pixel value from 0-255 Change it to between 0 and 1, scramble the road image data set and randomly select 20% of each type of pictures as the validation set, and the rest as the training set;

Table 2 Pavement image category labels

After the training set and validation set of the road surface type recognition network are completed, the network model is trained and evaluated. The batch size is set to 64, the cross entropy loss function is selected, and the Adam optimization algorithm is used. The basic learning rate is 0.0001. When MIoU>80%, it can be considered that the training is completed, and the model and training results are saved according to the number of iterations epoch, and the trained road type recognition network can be obtained;

Step 5. Obtain the information of the pavement adhesion coefficient

The process of obtaining the road adhesion coefficient information is as follows: during the driving process of the vehicle, the front road image is captured by the camera, and the front road image captured by the camera is transmitted to the road image area extraction network to obtain the road area, and then the image containing only the road area is transmitted to The road surface type identification network performs classification and identification. After the road surface identification is completed, the adhesion coefficient range of the road surface is determined according to the corresponding vehicle speed, see Table 2, and the middle value of the upper and lower limits of the road adhesion coefficient range is the current road adhesion coefficient, and the road surface can be completed. Acquisition of adhesion coefficient information.

Claims (1)

1. An image-based urban road pavement adhesion coefficient acquisition method is characterized by comprising the following specific steps:

step one, establishing a road surface image information base

The precondition for obtaining the road adhesion coefficient based on the image is that a perfect road image information base can be established, and the sample image is properly processed to ensure that the characteristic information in the image is fully obtained;

firstly, acquiring pavement image data, wherein adverse factors on imaging effects need to be made up in the pavement image acquisition process, the image acquisition equipment is not limited to one or a certain type of image acquisition equipment, and the requirements on equipment performance and installation position are as follows: the video resolution of 1280 multiplied by 720 and above is provided, the video frame rate is 30 frames per second and above, the maximum effective shooting distance is more than 70 meters, and the wide dynamic technology is provided to quickly adapt to the light intensity change; the installation position of the equipment should ensure that the road surface shot in the acquired image information occupies more than half of the whole image area;

according to the conditions of urban road surfaces under different weather conditions, through comparative analysis and by combining the types of the urban road surfaces in China, the road surface types to be identified are specifically defined as 5 road surface types including an asphalt road surface, a cement road surface, a loose snow road surface, a compacted snow road surface and an ice plate road surface, a video file in the data acquisition process is decomposed into pictures at intervals of 10 frames, the pictures are sorted according to the 5 attribution types according to road surface characteristics in GB/T920 plus 2002 road surface grade and surface layer type code and pavement adhesion coefficient survey analysis in cold regions, the same type of road surface images are uniformly stored under the same folder, and the establishment of a road surface image information base is completed;

step two, establishing a pavement image data set

The method comprises the steps that an original acquired image still contains a large number of non-road surface elements, and the acquisition precision of a road surface adhesion coefficient is seriously influenced, so that an image sample and a pixel-level label of a region corresponding to a road surface are needed in the road surface adhesion coefficient acquisition method based on the image, the image in a road surface image information base acquired in the step one needs to be subjected to road surface range labeling, Labelme in software Anaccnda is selected as a labeling tool, the labeling tool is used for manually labeling each image in a sample set one by one, a create polygon button is clicked in the labeling process, points are drawn along the boundary of the road surface region in the image, a labeling frame can completely cover the road surface region, and the labeling category is named as road; after the labeling is finished, a json file can be generated and is converted by using a self-contained json _ to _ dataset. py script program in software Anaconda to obtain a json folder, five files with names and suffixes of img.png, label.png, label _ viz.png, info.yaml and label _ names are contained under the folder, only the file with the label.png picture format is required to be converted to obtain a 8-bit gray label image, the labeling process is sequentially carried out on the pictures in the road image information base by using Labelmes in the Anaconda to obtain a gray label image set of the pictures in the road image information base, and the gray label image set of the pictures in the road image information base is a road image data set;

step three, establishing and training a road surface image area extraction network

The extraction network of the pavement image area is realized in an Anaconda environment through a semantic segmentation network, the whole semantic segmentation network is of an encoder-decoder structure, and the specific design is as follows:

3.1, firstly, zooming the image to be recognized into a picture with the size of 769 multiplied by 3, and taking the picture as the input of a semantic segmentation network;

3.2, then setting the first layer as a convolutional layer, adopting 32 filters with the size of 3 × 3, the step length is 2, filling the filter with 1, and obtaining the size of an output characteristic graph of the convolutional layer, which is 385 × 385 × 32, through batch regularization and a ReLU activation function;

3.3, inputting the convolution layer output characteristic diagram into a maximum value pooling layer, wherein the size is 3 multiplied by 3, the step length is 2, and the size of the output characteristic diagram of the pooling layer is 193 multiplied by 32;

3.4, taking the output characteristic diagram of the pooling layer as the input of the bottleneck module structure, wherein the bottleneck module structure is realized in detail as follows: firstly, copying input characteristic channel to increase characteristic dimension, one branch directly passing through depth convolution with size of 3X 3 and step length of 2, another branch equally dividing into two sub-branches by channel splitting, one sub-branch is subjected to 3 x 3 depth convolution and 1 x 1 point-by-point convolution, the other sub-branch is directly subjected to a characteristic multiplexing mode, then the two sub-branches are connected through channel splicing, the channel arrangement sequence is disturbed through channel cleaning, after the same depth convolution with the size of 3 multiplied by 3 and the step length of 2, the data are spliced with a copy channel, finally the information exchange between groups is realized through the point-by-point convolution with the size of 1 multiplied by 1, therefore, the size of the output characteristic diagram of the whole bottleneck module structure is reduced by half, the number of channels is doubled, and the size of the output characteristic diagram passing through the bottleneck module structure once is 97 multiplied by 64;

3.5, taking the output result of the process 3.4 as input, passing through the bottleneck module structure again to obtain the output characteristic graph with the size of 49 × 49 × 128 after passing through the bottleneck module structure twice, taking the result as input, passing through the bottleneck module structure again to obtain the output characteristic graph with the size of 25 × 25 × 256 after passing through the bottleneck module structure three times, and reducing the size by 32 times compared with the original image, wherein the whole part is used as an encoder part of a semantic segmentation network;

3.6, a decoder part adopts a jump structure, 2 times of upsampling is carried out on the output characteristic diagram of the three-time bottleneck module structure which is processed by the 3.5 process by using a bilinear interpolation method to obtain a characteristic diagram with the size of 49 multiplied by 256, the characteristic diagram is added with the output characteristic diagram of the two-time bottleneck module structure which is processed by the 3.5 process pixel by pixel, and output characteristic channels of the two-time bottleneck module structure which is processed by the 3.5 process need to be copied in the process so as to ensure that the result still has 256 output channels;

3.7, performing 2 times of upsampling on the result obtained in the process 3.6 by using a bilinear interpolation method again to obtain a characteristic diagram with the size of 97 multiplied by 256, and adding the characteristic diagram with the output characteristic diagram of the primary bottleneck module structure passing through the process 3.4 pixel by pixel;

3.8, converting the output channel number into a semantic category number by passing the result of the process 3.7 through a 1 × 1 convolutional layer, setting a Dropout layer to reduce the occurrence of an overfitting phenomenon, finally obtaining a feature map with the same size as the original image by 8 times of upsampling, giving a semantic category prediction result of each pixel point according to the maximum probability by using an Argmax function, and finally obtaining the whole semantic segmentation network;

randomly disordering the pavement image data set established in the step two, selecting 80% of sample pictures as a training set, and selecting 20% of sample pictures as a verification set; during semantic segmentation network training; the size of the read training set picture tensor is randomly scaled between 0.5 time and 2 times according to 0.25 step length, the picture tensor is randomly cut according to the size of 769 multiplied by 769 pixels and is randomly turned left and right, the purpose of data enhancement is achieved, the adaptability of a segmentation network is improved, and pixel point values are normalized from 0-255 to 0-1;

selecting a Poly learning rate rule when training a semantic segmentation network, wherein a learning rate attenuation expression is an expression (1), an initial learning rate is 0.001, training iteration steps are iters, the maximum training step max _ iter is set to be 20K steps, and power is set to be 0.9; using an Adam optimization solution algorithm, dynamically adjusting the learning rate of each parameter by using first moment estimation and second moment estimation of the gradient, setting the batch processing size to be 16 according to the performance of computer hardware, storing the model parameters once every 10-30min, and simultaneously using a verification set to perform performance evaluation on the network;

after the network training is finished, a proper semantic segmentation evaluation index is required to be selected for evaluating the performance of the model, before that, a confusion matrix is introduced, as shown in table 1, each row of the two-classification confusion matrix represents a prediction class, each column of the two-classification confusion matrix represents a real attribution class of data, and a specific numerical value represents the number of samples predicted to be a certain class;

TABLE 1 two-class confusion matrix schematic

The evaluation index of the semantic segmentation network is an average intersection-to-union ratio MIoU, which represents the ratio of the intersection and the union of each type of prediction result and the true value, and the result of the sum and the re-averaging is shown in formula (2):

when the MIoU is trained to be more than 60%, the training can be considered to be finished, the trained model and model parameters are stored, a road surface image area extraction network can be obtained, and the actually acquired original image is input into the road surface image area extraction network, so that the extraction of the road surface area in the image can be finished;

step four, establishing and training a road surface type recognition network

The extraction process of the road surface area in the real-time image information can be completed through the network in the third step, and the identification of the road surface type is completed on the basis of the extraction result of the road surface area in the third step;

after the image pavement data set is processed by the semantic segmentation network, an image set only containing a pavement area is obtained and is used as a final data set of a training and evaluation pavement type recognition network, so that the pavement type recognition network is built under the Anaconda environment, and the specific network structure is designed as follows:

4.1, firstly, scaling the image to be classified and identified into a picture with the size of 224 multiplied by 3 as the input of a convolutional neural network;

4.2, then setting the first layer as a convolutional layer, adopting 32 filters with the size of 3 × 3, the step length is 2, filling the filter with 1, and obtaining the size of an output characteristic diagram of the convolutional layer with the size of 112 × 112 × 32 through batch regularization and a ReLU activation function;

4.3, inputting the convolution layer output characteristic diagram into a maximum value pooling layer, wherein the size is 3 multiplied by 3, the step length is 2, and the size of the output characteristic diagram of the pooling layer is 56 multiplied by 32;

4.4, taking the output characteristic diagram of the pooling layer as the input of the bottleneck module structure, wherein the bottleneck module structure is implemented in detail as follows: firstly, copying input characteristic channel to increase characteristic dimension, one branch directly passing through depth convolution with size of 3X 3 and step length of 2, another branch equally dividing into two sub-branches by channel splitting, one sub-branch is subjected to 3 x 3 depth convolution and 1 x 1 point-by-point convolution, the other sub-branch is directly subjected to a characteristic multiplexing mode, then the two sub-branches are connected through channel splicing, the channel arrangement sequence is disturbed through channel cleaning, after the same depth convolution with the size of 3 x 3 and the step length of 2, the data are spliced with a copy channel, finally the information exchange between groups is realized through the point-by-point convolution with the size of 1 x 1, it can be seen that the size of the output characteristic diagram of the whole bottleneck module structure is reduced by half, the number of channels is doubled, and the size of the output characteristic diagram passing through the bottleneck module structure for one time is 28 multiplied by 64;

4.5, taking the output result of the process 4.4 as input, and passing through the bottleneck module structure again to obtain an output characteristic diagram with the size of 14 multiplied by 128 after passing through the bottleneck module structure twice, and taking the result as input, and passing through the bottleneck module structure again to obtain an output characteristic diagram with the size of 7 multiplied by 256 after passing through the bottleneck module structure three times;

4.6, converting the output result in the process 4.5 into a characteristic diagram with the size of 1 × 1 × 256 by using a global average pooling layer with the size of 7 × 7;

4.7, using a layer of full connection layer and a Softmax function as a network classifier, converting the output characteristic diagram in the process 4.6 into probability values belonging to various categories, and determining a network classification result according to the maximum probability value by using an Argmax function;

then, the images only containing the road surface area obtained in the step three are made into data sets for training a road surface type recognition network, the road surface images of different types are still stored according to the folder names established in the step one in a classified mode, the image data in different folders are read in sequence, 5-bit 0/1 label information and road surface adhesion coefficient information are added, see table 2, the image size is adjusted to 224 x 224 pixels in a bilinear interpolation mode, the pixel point value is normalized to be between 0 and 255 and 0 and 1, the road surface image data sets are disturbed, each type of images are randomly extracted according to a proportion of 20 percent to serve as verification sets, and the rest parts serve as training sets;

TABLE 2 pavement image category labels

After the training set and the verification set of the road surface type recognition network are manufactured, training and evaluating a network model are started, the batch processing size is set to 64, a cross entropy loss function is selected, an Adam optimization solving algorithm is used, the basic learning rate is 0.0001, when the training is completed until the MIoU is more than 80%, the training can be considered to be completed, the model and the training result are stored according to the iteration times epoch, and the well-trained road surface type recognition network can be obtained;

step five, obtaining the road surface adhesion coefficient information

The road surface adhesion coefficient information acquisition process is as follows: the method comprises the steps of shooting a front road image through a camera in the driving process of a vehicle, transmitting the front road image shot by the camera to a road image area extraction network to obtain a road area, transmitting the image only containing the road area to a road type identification network for classification and identification, judging the adhesion coefficient range of the road where the road is located according to the corresponding vehicle speed after the road identification is finished, referring to a table 2, and taking the intermediate values of the upper limit and the lower limit of the road adhesion coefficient range as the current road adhesion coefficient to finish the acquisition of the road adhesion coefficient information.

Images