CN111985317A

CN111985317A - Road surface cleanliness evaluation method for intelligent road sweeping

Info

Publication number: CN111985317A
Application number: CN202010663905.2A
Authority: CN
Inventors: 赵健成; 顾昕程; 杨致饶; 徐江; 高传宝
Original assignee: Shanghai Fujie Technology Co ltd
Current assignee: Shanghai Fujie Technology Co ltd
Priority date: 2020-07-10
Filing date: 2020-07-10
Publication date: 2020-11-24
Anticipated expiration: 2040-07-10
Also published as: CN111985317B

Abstract

The invention discloses a pavement cleanliness evaluation method for intelligent road cleaning, which comprises the steps of extracting a pavement area through a Yoloct pavement segmentation network, then obtaining the positioning and classification of large garbage through a Yolov3 target detection network, and simultaneously obtaining the density distribution of small garbage by utilizing an MCNN density estimation network; and finally, weighting and summing all the garbage to obtain the cleanliness index of the road surface. The invention adopts the transfer learning and image enhancement algorithm to ensure that the training model obtains better performance. By exploring the field and combining image segmentation, target detection and density estimation algorithms successfully realize the estimation of the cleanliness of the road surface.

Description

Road surface cleanliness evaluation method for intelligent road sweeping

Technical Field

The invention relates to the field of artificial intelligence, in particular to a pavement cleanliness assessment method for intelligent road cleaning.

Background

The computer vision technology is a technology for enabling a computer to simulate the visual process of a human, and has the ability to feel the environment and the visual function of the human. And (4) integrating technologies such as image processing, artificial intelligence and pattern recognition.

Computer vision began from statistical pattern recognition in the 50's of the 20 th century, and then work was mainly focused on two-dimensional image analysis and recognition, such as optical character recognition, analysis and interpretation of workpiece surfaces, micrographs and aerial pictures, etc. in the 60's, Roberts (1965) extracted three-dimensional structures of polyhedrons such as cubes, wedges, prisms, etc. from digital images by computer programs and described the shapes of objects and their spatial relationships [ Roberts 1965 ]. By the 70 s, some vision application systems "Guzman 1969, Maekworth 1973" have emerged. In the middle of the 70 s, the institute of technology and technology (MIT) Artificial Intelligence (AI) laboratory officially offered a "Machine Vision" (Machine Vision) course, which was taught by professor b.k.p.hom. Since the 80 s, computer vision studies have gone through a phase of development that has moved from the laboratory to practical applications. The rapid improvement of the industrial level of the computer and the development of disciplines such as artificial intelligence, parallel processing, neuron network and the like further promote the practicability of a computer vision system and concern the research of a plurality of complex vision processes. Currently, computer vision technology is widely applied to many fields such as computational geometry, computer graphics, image processing, robotics, and the like.

The goals of computer vision research are two: one is to develop an image understanding system that automatically constructs scene descriptions from input image data, and the other is to understand human vision in order to make work that is difficult or impossible for humans to reach with machines instead of humans all the day. Computer vision is also currently an active and highly effective topic of intense research in artificial intelligence and robotic science.

The neural network technology is an algorithm model which is similar to the behavior characteristics of an animal neural network and enables a computer to learn by 'observing' data. The technology achieves the aim of information processing by adjusting the connection relation between the nodes. Various Neural networks have been studied, such as Deep Neural Network (DNN), Convolutional Neural Network (CNN), Deep Belief Network (DBN), and the like. Can be applied to different fields: image processing, voice recognition, etc.

The convolutional neural network has good characteristics, can identify important features of the picture, and greatly improves the performance of machine vision. The convolutional neural network is mainly studied here, whereby the classical network is simply enumerated:

(1)LeNet5

LeNet5 is one of the earliest convolutional neural networks, and is a classic model suitable for handwriting recognition, and has a small volume and only 7 CNNs. The method carries out feature extraction through operations such as convolution, pooling and the like, simplifies the data calculation amount, and finally utilizes the full-connection layer to classify, thereby providing a framework for building a large-scale neural network of offspring.

(2)AlexNet

AlexNet was born in 2012, and based on the idea of LeNet, 5 convolutional layers and 3 full-connection layers were established, so that ReLu was successfully used as an activation function, and multiple GPUs were used for parallel training, thereby improving algorithm efficiency. A method for Dropout to randomly ignore neurons and data enhancement is also proposed to prevent model training from overfitting.

(3)GoogleNet

GoogleNet is a neural network studied by google, similar to AlexNet, but smaller in size and deeper in hierarchy. The method creatively uses the convolution kernel of 1x1 to carry out dimension lifting, and simultaneously uses the convolution kernels of a plurality of scales to carry out convolution, thereby reducing parameters and calculation amount again. In addition, a softmax function is added to prevent the problems of gradient disappearance and the like.

A city street rubbish detection and cleanliness assessment method integrating mobile edge calculation and deep learning; and detecting the road garbage by adopting a Faster R-CNN target detection network. The detectable categories of trash include: waste paper, plastic bags, plastic bottles, pop cans, etc. But the dense small garbage such as leaves, branches, cigarette ends and the like cannot be effectively detected. And training samples are few, and the network model cannot be fully learned by only adopting 321 garbage images as a training data set. Therefore, the road cleanliness assessment based on the detection result has great limitation

B, Mohammad sweet Rad.A Computer Vision System to locate and Classify waters on the Streets; the target detection network based on deep convolution is adopted to realize the detection and identification of the small garbage on the road surface, including leaves, cigarette ends, leaf piles and the like. The method has good effect on sparse small targets, but cannot effectively sense when the targets are highly overlapped. The algorithm has high requirements on the image resolution, and cannot effectively detect the image with low resolution, and the high-resolution image greatly influences the real-time performance of the algorithm. Meanwhile, in order to eliminate target dimension change caused by perspective transformation of the camera, an imaging plane is required to be parallel to the ground when the camera is installed, and certain complexity is brought.

Because the image information of the small garbage is less, the appearance characteristic change is large after the small garbage is highly overlapped, the conventional target detection models A and B cannot effectively sense the dense small garbage, the garbage data for model training is less, the models cannot be fully learned, and the sensing of the road garbage by the models is not facilitated. Meanwhile, the road cleanliness evaluation has no unified standard temporarily. These all present significant difficulties for computer vision road cleanliness assessment efforts.

Disclosure of Invention

1. Objects of the invention

The invention provides a road surface cleanliness evaluation method for intelligent road cleaning, aiming at solving the problem of effective perception of dense small garbage.

2. The technical scheme adopted by the invention

The invention discloses a pavement cleanliness evaluation method for intelligent road cleaning, which comprises the steps of extracting a pavement area through a Yoloct pavement segmentation network, then obtaining the positioning and classification of large garbage through a Yolov3 target detection network, and simultaneously obtaining the density distribution of small garbage by utilizing an MCNN density estimation network; and finally, weighting and summing all the garbage to obtain the cleanliness index of the road surface.

Preferably, the pavement is detected and segmented by adopting an example segmentation algorithm, including Mask R-CNN, FCIS and Yolact.

Preferably, a plurality of interested targets in the digital image or video are identified and positioned, and a one stage target detection algorithm yolov3 is adopted, so that the positioning and the identification of the targets are realized by laying a priori box on the graph and regressing the central point, the length, the width and the category of the object.

Preferably, the density estimation: the density map provides distribution information for dense small objects in the image; considering that the conventional target detection algorithm cannot effectively position the dense small targets, the density estimation algorithm is adopted to calculate the distribution information of the small targets.

Preferably, the method comprises the steps of collecting road surface garbage image data, labeling targets in each image, wherein two labeling formats are available, one is a rectangular frame labeling format (class, x, y, w, h) for garbage target detection, the first parameter class represents the category of labeled contents, the second parameter x represents the x coordinate of the normalized target center point, the third parameter y represents the y coordinate of the normalized target center point, the fourth parameter w represents the width of the normalized target frame, and the fifth parameter h represents the height of the normalized target frame; the other is a point label format (x1, y1) for garbage density estimation, where the first parameter x1 represents the x1 coordinate of the target center point and the second parameter y1 represents the y1 coordinate of the target center store; the point labeling format generates a density map data format through convolution operation.

Preferably, the pavement segmentation neural network adopts a yolact frame structure; firstly, using ResNet101 as a backbone, wherein the ResNet101 comprises five convolution modules, namely conv1, conv5, an FPN network is a P3 layer-P7 layer, and the C5 layer passes through a convolution layer to obtain a P5 layer; then, carrying out bilinear interpolation on the P5 layer once to amplify the P5 layer, and adding the P5 layer and the convolved C4 layer to obtain a P4 layer; the same procedure gave a layer of P3; besides, the P5 layer is convoluted to obtain a P6 layer, and the P6 layer is convoluted to obtain a P7 layer; following the parallel operation, the P3 level is sent to Protonet, while the P3-P7 level is sent to Prediction Head; the Protonet structure consists of a plurality of 3x3 convolutional layers, an upsampling layer and a 1x1 convolutional layer, and Relu is used as an activation function; a Prediction Head in parallel with the protocol, first a 3x3 convolutional layer shared by the three branches, then a convolutional layer with a respective 3x3 for each branch; finally, in order to generate an instance mask, taking a mask coefficient as a coefficient, and linearly combining results of prototype branches; and generating a mask corresponding to the road surface in the image by using the linear combination result through a sigmoid function, and covering the mask on the original image to extract the road surface area of the image.

Preferably, a Yolov3 framework is adopted by the target detection network for large garbage perception; the method adopts dark net53 as a feature extraction network, comprises a layer 0 to a layer 74, which is composed of a series of convolution layers of 1x1 and 3x3, each convolution layer is followed by a BN layer and a Leaky-ReLU layer, and meanwhile, dark net53 also adopts residual connection; layers 75 to 105 behind Darknet53 are feature interaction layers of the yolo network, and the boundary frame prediction is respectively carried out in three dimensions, wherein in the first dimension, a feature map is subjected to 32 times of downsampling and is suitable for detecting a target with a larger size in an image; in the second scale, the feature map is subjected to 16 times of downsampling, has a medium receptive field and is suitable for detecting a medium-sized target; in the third scale, the feature map is subjected to 8 times of downsampling, the resolution of the feature map is high, and the feature map is suitable for detecting small-sized targets; obtaining a relatively large garbage boundary frame and an affiliated label in the image by predicting under three scales; thereby obtaining the category and distribution information of the large garbage.

Preferably, the density estimation network of the dense small garbage adopts an MCNN framework; three rows of convolutional neural networks are included; the pooling layer is the maximum pooling of 2x2, and the activation function is ReLU; the final column output results are merged together and converted to a density map of dense small garbage in the image using a convolution kernel of 1x 1.

Preferably, the road cleanliness is calculated based on the garbage distribution:

the calculation formula is as follows:

RC is the cleanliness of the road surface, and the lower the value is, the cleaner the road surface is; wherein C is₁Is the sum of the costs generated by all small garbage; c₂Is the sum of the costs of all large garbage; s, the area of the road surface evaluated; α and β are cleanliness correction factors;

C₁is calculated based on the density map, and the calculation formula is as follows:

C₁＝sum(DesityMap)×W₀ (2)

wherein sum (DesityMap) represents the pixel-by-pixel summation of the entire density map, the result of the summation representing an estimate of the amount of small garbage; w₀Is the weight of each small garbage, the weight W of the small garbage₀Is always 1; the result of calculating equation (2) thus represents a weighted sum of all small garbage;

C₂the method is obtained based on a target detection result, and the calculation formula is as follows:

wherein W_nRepresents the weight occupied by the class represented by the nth big garbage; n is the number of the large garbage obtained by target detection; the result of calculating equation (3) thus represents a weighted sum of all large garbage.

Preferably, comparison of cleanliness before and after cleaning verifies effectiveness of the cleaning strategy:

2. advantageous effects adopted by the present invention

(1) The invention adopts the transfer learning and image enhancement algorithm to ensure that the training model obtains better performance. By exploring the field and combining image segmentation, target detection and density estimation algorithms successfully realize the estimation of the cleanliness of the road surface.

(2) At present, in related work in the field, only a single target detection network is often adopted to try to position and identify the pavement rubbish, but the single target detection network cannot have good perception performance on dense small targets, and the small rubbish often occupies a large proportion in the pavement cleanliness estimation process. The invention adopts a combined sensing mode of target detection and density estimation to position and identify the garbage on the road surface. Through verification, the scheme has good accuracy, instantaneity and fault tolerance on garbage with various scales and density degrees.

(3) The invention relates to the calculation of the road surface cleanliness, and defines the standard of the road surface cleanliness due to the temporary lack of a unified standard in the industry. A pavement cleanliness calculation formula is designed based on the weight and the pavement area of the garbage, and the formula has certain rationality through verification.

(4) Based on the road surface cleanliness evaluation algorithm, the road surface cleanliness evaluation algorithm can acquire road surface images before and after cleaning through the front-view camera and the rear-view camera, compare the road surface cleanliness before and after cleaning, and verify the effectiveness of a cleaning strategy. The cleaning process is more intelligent, and the cleaning device has high practical value.

Drawings

FIG. 1 is a schematic view of the overall structure;

FIG. 2 is an image segmentation diagram;

FIG. 3 is a target detection map;

FIG. 4 is a map of a pavement debris image dataset;

FIG. 5 is a view of a yolact pavement partitioning network structure;

FIG. 6 is a diagram of an example of road surface segmentation;

FIG. 7 is a diagram of a Yolov3 network architecture;

FIG. 8 is a diagram of an example of garbage target detection;

FIG. 9 is a diagram of a MCNN density estimation network architecture;

FIG. 10 is a diagram of an example of garbage density estimation;

FIG. 11 is a garbage distribution map for joint target detection and density estimation.

Detailed Description

The technical solutions in the examples of the present invention are clearly and completely described below with reference to the drawings in the examples of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present invention without inventive step, are within the scope of the present invention.

The present invention will be described in further detail with reference to the accompanying drawings.

Example 1

Image segmentation

The image segmentation technology is one of important research directions in the field of computer vision, and is an important part of image semantic understanding. The technologies related to the technology, such as scene object segmentation, human body front background segmentation, human face human body matching, three-dimensional reconstruction and the like, are widely applied to the industries of unmanned driving, augmented reality, security monitoring and the like.

The image segmentation means that an image is divided into a plurality of mutually disjoint areas according to characteristics such as gray scale, color, spatial texture, geometric shape and the like, so that the characteristics show consistency or similarity in the same area and obviously differ among different areas. The image segmentation is suitable for scenes with high understanding requirements, such as segmentation of roads and non-roads in unmanned driving. Conventional image segmentation methods include a threshold-based segmentation method, a region-based segmentation method, an edge-based segmentation method, and the like. Such algorithms tend to have significant limitations in complex scenarios. In recent years, under the influence of the development of deep learning, a plurality of image segmentation methods based on artificial neural networks appear. The artificial neural network is a highly nonlinear dynamic system formed by interconnection of large-scale neurons, and is an engineering system for simulating the structure and intelligent behavior of the human brain organization mechanism on the basis of understanding and understanding the human brain organization mechanism and the operation mechanism. The segmentation method can mine the semantic information of the image at a deeper level, thereby realizing a better segmentation effect.

The image segmentation algorithm can be further divided into: semantic segmentation, instance segmentation, panorama segmentation, and superpixel segmentation, etc. An example segmentation algorithm is used to detect and segment the road surface. Example segmentation is an extension of the detection task, requiring pixel-level contouring of the target to be described. The current classical networks compared in example segmentation are Mask R-CNN, FCIS, Yolact, etc. These are all example segmentation methods based on deep learning.

Target detection

Target detection technology is also a major research direction in the field of computer vision. It is widely applied in the fields of safety, military, traffic, medical treatment and the like.

Object detection is primarily the identification and localization of multiple objects of interest in a digital image or video. Conventional target detection can be divided into three steps: firstly, selecting a candidate region in an image, then extracting visual features such as Haar, HOG and the like, and finally classifying the image based on a support vector machine model, an RF model and other common classifiers. With the development of deep learning, the target detection algorithm is also shifted from the traditional algorithm based on manual features to the detection technology based on artificial neural network. The latter development has mainly focused on two directions: two stage algorithms such as R-CNN series and one stage algorithms such as YOLO, SSD, etc. The real-time performance of the algorithm is considered and the performance is considered. A one stage target detection algorithm yolov3 was used. Yolov3 achieves target location and identification by laying a priori box on the map and regressing the center point, length, width, and category of the object.

Density estimation

Density estimation is yet another direction of computer vision. There is a wide range of applications in target number perception tasks for specific macroscopic scenes (e.g. bird flocks) and the micro-world (e.g. microscopic cells). In 2016, Yingying Zhang et al generated a density map by using a multi-column convolutional neural network, and counted the population in the image by using the density map, thereby achieving a good effect. In 2017, X.Q.Zhou et al successfully used density maps to locate and identify small targets. Their research has shown that density maps can provide good distribution information for dense small objects in an image. In consideration of the fact that the conventional target detection algorithm cannot effectively locate the dense small targets, the density estimation algorithm is adopted to calculate the distribution information of the small targets.

Considering that the existing data set in the field is few, a pavement garbage image data set is automatically collected for data analysis and model training, wherein the pavement garbage image data set comprises 1000 pavement garbage image samples, objects in each image are reliably labeled, two labeling formats are provided, one labeling format is a rectangular frame labeling format (class, x, y, w, h) for detecting the garbage objects, wherein the first parameter class represents the category of labeling content, the second parameter x represents the x coordinate of the normalized object center point, the third parameter y represents the y coordinate of the normalized object center point, the fourth parameter w represents the width of the normalized object frame, and the fifth parameter h represents the height of the normalized object frame. The other is a point label format (x1, y1) for garbage density estimation, where the first parameter x1 represents the x1 coordinate of the target center point and the second parameter y1 represents the y1 coordinate of the target center store. The point label format may generate a density map data format through a convolution operation.

As shown in fig. 5, the whole network adopts yolact framework structure. First, ResNet101 is used as a back bone, which has five convolution blocks, conv 1.., conv5, whose outputs correspond to C1 to C5 on the graph, respectively. P3-P7 are then FPN networks. P5 is obtained by passing C5 through a convolutional layer; then, carrying out bilinear interpolation on the P5 once to amplify the P5, and adding the P5 and the convolved C4 to obtain P4; the same procedure gave P3. Furthermore, P6 was obtained by convolving P5, and P7 was obtained by convolving P6. Next is a parallel operation, P3 is fed into Protonet, while P3-P7 is fed into Prediction Head. The Protonet structure consists of multiple 3x3 convolutional layers, an upsampling layer, and a 1x1 convolutional layer, using Relu as the activation function. The Prediction Head in parallel with the Protonet is first a 3x3 convolutional layer shared by the three branches, and then a convolutional layer with a respective 3x3 for each branch. Finally, in order to generate the instance mask, the results of prototype branches are linearly combined with mask coeffient as a coefficient. And (5) the linear combination result is processed by a sigmoid function to generate final masks. As shown in fig. 6 below.

The mask corresponding to the road surface in the image can be obtained through the road surface segmentation neural network, and the road surface area of the image can be extracted after the mask is covered on the original image.

As shown in fig. 7, the target detection network for garbage sensing (big garbage) employs Yolov3 framework. It uses Darknet53 as the feature extraction network, including from layer 0 to layer 74, it is composed of a series of 1x1 and 3x3 convolutional layers, each convolutional layer is followed by a BN layer and a Leaky-ReLU layer, and Darknet53 also uses residual connection. Layers 75 to 105 behind Darknet53 are feature interaction layers of the yolo network, and the boundary box prediction is carried out in three dimensions respectively, and in the first dimension, the feature map is subjected to 32 times of downsampling and is suitable for detecting a large-size target in an image. In the second scale, the feature map is subjected to 16 times of downsampling, has a medium-sized receptive field, and is suitable for detecting a medium-sized target. In the third scale, the feature map is subjected to 8 times of downsampling, the resolution of the feature map is high, and the feature map is suitable for detecting small-sized targets. By doing prediction at three scales, the final result is output.

The bounding box (bounding box) and the label (label) of the relatively large garbage in the image can be obtained through the target detection neural network. Thereby obtaining the category and distribution information of the large garbage. As shown in fig. 8.

As shown in fig. 9, the density estimation network for garbage perception (dense small garbage) employs an MCNN framework. The MCNN is a full convolution network that includes three convolutional neural networks with different sizes and numbers of convolution kernels in each row, but the network structure is basically the same. The pooling layer is the maximum pooling of 2x2, and the activation function is ReLU. In addition, in order to reduce the calculation amount, the number of convolution kernels corresponding to the columns corresponding to the large convolution kernels is small. The final column output results are merged together and converted to the final density image using a 1x1 convolution kernel.

The density map of the dense small garbage in the image can be obtained through the density estimation neural network. Thereby obtaining the distribution information of the dense small garbage. As shown in fig. 10.

5.5 road cleanliness is calculated based on garbage distribution

The invention provides a pavement cleanliness evaluation standard which is used for measuring the cleanliness of a pavement.

The calculation formula is as follows:

RC is the cleanliness of the road surface, and a lower value indicates a cleaner road surface. Wherein C is₁Is the sum of the costs of all small garbage. C₂Is the sum of the costs of all large garbage. S area of the road surface evaluated. Alpha and beta are cleanliness correction factors.

C₁＝sum(DesityMap)×W₀ (2)

where sum (DesityMap) represents the pixel-by-pixel summation of the entire density map, the result of which represents an estimate of the amount of small garbage. W₀Is the weight occupied by each small garbage, the weight W of the small garbage in the invention₀Always 1. The result of calculating equation (2) therefore represents a weighted sum of all small garbage.

wherein W_nRepresenting the weight occupied by the class represented by the nth large garbage. And n is the number of the large garbage obtained by target detection. The result of calculating equation (3) thus represents a weighted sum of all large garbage.

The weights W are defined as follows:

in order to better quantify the cleanliness of the road surface, a grade-based road surface cleanliness evaluation index is also introduced:

comparison of cleanliness before and after cleaning: and verifying the effectiveness of the cleaning strategy by comparing the cleanliness of the road surface before and after cleaning. The calculation formula is as follows:

in summary, the invention is a road cleanliness assessment algorithm for intelligent cleaning based on image segmentation, target detection and density estimation. According to the algorithm, a road surface area is extracted through a Yoloct road surface segmentation network, then the positioning and the classification of large garbage are obtained through a Yolov3 target detection network, and meanwhile, the density distribution of small garbage is obtained through an MCNN density estimation network. And finally, weighting and summing all the garbage to obtain the cleanliness index of the road surface. The algorithm is small in calculation complexity and high in instantaneity. The target detection and density estimation algorithm are combined, so that garbage can be effectively positioned and identified in various complex road surface scenes, reliable road surface garbage distribution information is obtained, and the stability and the reasonability of cleanliness evaluation are improved. The road surface cleaning is more intelligent, and the practical value is very high.

The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A road surface cleanliness assessment method for intelligent road sweeping is characterized by comprising the following steps:

extracting a road surface area through a Yoloct road surface segmentation network, then obtaining the positioning and classification of large garbage through a Yolov3 target detection network, and simultaneously obtaining the density distribution of small garbage by utilizing an MCNN density estimation network; and finally, weighting and summing all the garbage to obtain the cleanliness index of the road surface.

2. The method for evaluating the cleanliness of a road surface for intelligent sweeping of a road according to claim 1, characterized in that: and detecting and segmenting the road surface by adopting an example segmentation algorithm, wherein the example segmentation algorithm comprises Mask R-CNN, FCIS and Yolact.

3. The method for evaluating the cleanliness of a road surface for intelligent sweeping of a road according to claim 1, characterized in that: the method comprises the steps of identifying and positioning a plurality of interested targets in a digital image or video, adopting a one stage target detection algorithm yolov3, and realizing the positioning and identification of the targets by paving a prior frame on the image and regressing the central point, the length, the width and the category of an object.

4. The road surface cleanliness evaluation method for intelligent sweeping of roads according to claim 1, characterized by density estimation: the density map provides distribution information for dense small objects in the image; considering that the conventional target detection algorithm cannot effectively position the dense small targets, the density estimation algorithm is adopted to calculate the distribution information of the small targets.

5. The road surface cleanliness evaluation method for intelligent road sweeping according to claim 4, characterized in that: collecting pavement rubbish image data, labeling targets in each image, wherein the two labeling formats are rectangular frame labeling formats (class, x, y, w, h) for rubbish target detection, the first parameter class represents the class of labeling content, the second parameter x represents the x coordinate of the normalized target center point, the third parameter y represents the y coordinate of the normalized target center point, the fourth parameter w represents the normalized target frame width, and the fifth parameter h represents the normalized target frame height; the other is a point label format (x1, y1) for garbage density estimation, where the first parameter x1 represents the x1 coordinate of the target center point and the second parameter y1 represents the y1 coordinate of the target center store; the point labeling format generates a density map data format through convolution operation.

6. The method for evaluating the cleanliness of a road surface for intelligent sweeping of a road according to claim 1, characterized in that: the pavement segmentation neural network adopts a yolact frame structure; firstly, using ResNet101 as a backbone, wherein the ResNet101 comprises five convolution modules, namely conv1, conv5, an FPN network is a P3 layer-P7 layer, and the C5 layer passes through a convolution layer to obtain a P5 layer; then, carrying out bilinear interpolation on the P5 layer once to amplify the P5 layer, and adding the P5 layer and the convolved C4 layer to obtain a P4 layer; the same procedure gave a layer of P3; besides, the P5 layer is convoluted to obtain a P6 layer, and the P6 layer is convoluted to obtain a P7 layer; following the parallel operation, the P3 level is sent to Protonet, while the P3-P7 level is sent to Prediction Head; the Protonet structure consists of a plurality of 3x3 convolutional layers, an upsampling layer and a 1x1 convolutional layer, and Relu is used as an activation function; a Prediction Head in parallel with the protocol, first a 3x3 convolutional layer shared by the three branches, then a convolutional layer with a respective 3x3 for each branch; finally, in order to generate an instance mask, taking a mask coefficient as a coefficient, and linearly combining results of prototype branches; and generating a mask corresponding to the road surface in the image by using the linear combination result through a sigmoid function, and covering the mask on the original image to extract the road surface area of the image.

7. The method for evaluating the cleanliness of a road surface for intelligent sweeping of a road according to claim 1, characterized in that: the target detection network for sensing the large garbage adopts a Yolov3 framework; the method adopts the Darknet53 as a feature extraction network, comprises a layer 0 to a layer 74, and consists of a series of 1x1 and 3x3 convolutional layers, wherein each convolutional layer is followed by a BN layer and a Leaky-ReLU layer, and the Darknet53 also adopts residual connection; layers 75 to 105 behind Darknet53 are feature interaction layers of the yolo network, and the boundary frame prediction is respectively carried out in three dimensions, wherein in the first dimension, a feature map is subjected to 32 times of downsampling and is suitable for detecting a target with a larger size in an image; in the second scale, the feature map is subjected to 16 times of downsampling, has a medium receptive field and is suitable for detecting a medium-sized target; in the third scale, the feature map is subjected to 8 times of downsampling, the resolution of the feature map is high, and the feature map is suitable for detecting small-sized targets; obtaining a relatively large garbage boundary frame and an affiliated label in the image by predicting under three scales; thereby obtaining the category and distribution information of the large garbage.

8. The method for evaluating the cleanliness of a road surface for intelligent sweeping of a road according to claim 1, characterized in that: the density estimation network of the dense small garbage adopts an MCNN framework; three rows of convolutional neural networks are included; the pooling layer is the maximum pooling of 2x2, and the activation function is ReLU; the final column output results are merged together and converted to a density map of dense small garbage in the image using a convolution kernel of 1x 1.

9. The road cleanliness evaluation method for intelligent road sweeping according to claim 1, characterized in that the road cleanliness is calculated based on garbage distribution:

the calculation formula is as follows:

RC is the cleanliness of the road surface, and the lower the value is, the cleaner the road surface is; wherein C is₁Is the sum of the costs generated by all small garbage; c₂Is the sum of the costs of all large garbage; s, the area of the road surface evaluated; alpha and beta are cleanliness correction factors；

C₁＝sum(DesityMap)×W₀ (2)

10. The method for evaluating the cleanliness of a road surface for intelligent sweeping of a road according to claim 6, characterized in that: comparing and verifying the cleanliness before and after cleaning to verify the effectiveness of the cleaning strategy: