WO2024060529A1 - 一种道路病害识别方法、系统、设备及存储介质 - Google Patents

一种道路病害识别方法、系统、设备及存储介质 Download PDF

Info

Publication number
WO2024060529A1
WO2024060529A1 PCT/CN2023/080853 CN2023080853W WO2024060529A1 WO 2024060529 A1 WO2024060529 A1 WO 2024060529A1 CN 2023080853 W CN2023080853 W CN 2023080853W WO 2024060529 A1 WO2024060529 A1 WO 2024060529A1
Authority
WO
WIPO (PCT)
Prior art keywords
disease
road
image
sample
images
Prior art date
Application number
PCT/CN2023/080853
Other languages
English (en)
French (fr)
Inventor
关永胜
杨阳
张志祥
刘强
胡永
Original Assignee
中路交科科技股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中路交科科技股份有限公司 filed Critical 中路交科科技股份有限公司
Publication of WO2024060529A1 publication Critical patent/WO2024060529A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30252—Vehicle exterior; Vicinity of vehicle
    • G06T2207/30256—Lane; Road marking

Definitions

  • the present invention relates to the field of artificial intelligence technology, and in particular to a road disease identification method, system, equipment and storage medium.
  • road diseases include cracks, potholes, looseness, etc., and are not a single road crack.
  • Different types of road diseases have different repair methods. The detection and classification of road diseases are very important for road disease repair.
  • the technical problem to be solved by the present invention is to provide a road disease identification method, system, equipment and storage medium, which can perform disease identification analysis on road video data in real time, realize the identification and marking of disease areas, and facilitate subsequent construction personnel to identify and mark the road. Carry out maintenance.
  • a road disease identification method including:
  • the disease is determined to be a crack, its width is calculated, and if the disease is determined to be a pothole or rut, its depth is calculated.
  • characteristic parameters include disease category, relative position of the disease, confidence level and disease instance segmentation area.
  • the disease categories include cracks, ruts, potholes, looseness, oil spills and repairs.
  • the road surface image comes from the video collected by the vehicle-mounted camera.
  • the road surface image is obtained by reading the image in the video frame, cropping the frame image, and extracting features of the cropped image.
  • the calculation method of crack width specifically includes the following steps:
  • A1 If the disease is determined to be a crack, determine the instance segmentation area
  • A2 Determine the edge line and central axis of the instance segmentation area
  • A3 According to the central axis, determine the center point list
  • A4 Take a center point, search for the two nearest edge line coordinate points, and calculate the width value
  • A5 Loop step A4 until the width values corresponding to all center points in the center point list are calculated, and calculate the average width.
  • the calculation method of the pit depth specifically includes the following steps:
  • the invention also discloses a road disease identification system, which includes:
  • the data preprocessing module is used to collect historical road surface images with diseases, detect and segment them sequentially, and generate several visualized sample images, and the sample images have characteristic parameters that mark the sample images;
  • a sample data set creation module uses the sample images to create a sample training set
  • the model generation module builds a discrimination model for judging disease categories, uses the sample training set to train the discrimination model, and obtains a trained discrimination model;
  • a determination module is used to obtain a new road surface image, detect and segment it sequentially, and then input it into the determination model to generate a determination result;
  • the result post-processing module calculates the width if the disease is determined to be a crack, and calculates the depth if the disease is determined to be a pit or rut.
  • an image processing module which is used to read the image in the video frame, crop the frame image, and extract features of the cropped image.
  • the present invention also discloses an electronic device, including at least one processor;
  • the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the methods described above.
  • the present invention also discloses a storage medium.
  • the computer storage medium stores a computer program.
  • the computer program is used to cause the computer to execute any of the methods described above.
  • this invention also adds multiple types of road diseases such as road potholes and loose roads, making the identification of road diseases more specific and complete;
  • the present invention obtains information such as crack width, pit depth and other information by segmenting the disease area, and while identifying road diseases, it also classifies road diseases to facilitate maintenance by construction workers;
  • the present invention adopts the instance segmentation technology of first detection and then segmentation to ensure accurate positioning and complete segmentation information
  • this system is deployed on edge devices and can be installed on vehicles in the form of an all-in-one machine to detect real-time information while the vehicle is driving.
  • the present invention is more flexible and easy to deploy than other road disease identification systems, and can achieve real-time disease visualization.
  • FIG1 is a diagram showing the effect of semantic segmentation in the background technology
  • Figure 2 is a flow chart of a road disease identification method in an embodiment of the present invention.
  • FIG3 is a schematic diagram of a road damage identification method according to an embodiment of the present invention.
  • Figure 4 is an effect diagram after image detection and segmentation in the embodiment of the present invention.
  • Figure 5 is a flow chart of crack width calculation in the embodiment of the present invention.
  • Figure 6 is the first effect diagram used to demonstrate crack width calculation in the embodiment of the present invention.
  • Figure 7 is a second rendering showing crack width calculation in the embodiment of the present invention.
  • Figure 8 is the third effect diagram used to demonstrate crack width calculation in the embodiment of the present invention.
  • Figure 9 is a flow chart of pit depth calculation in the embodiment of the present invention.
  • Figure 10 is an architecture diagram of a road disease identification system in an embodiment of the present invention.
  • the road disease identification methods shown in Figures 2 to 9 include:
  • the disease is determined to be a crack, its width is calculated, and if the disease is determined to be a pothole or rut, its depth is calculated.
  • the recognition method provided by the present invention is implemented by an algorithm, and the algorithm is deployed as a whole on the NVIDIA Jetson NX edge computing platform. It is lightweight and convenient and can be easily deployed on any vehicle model.
  • the road image is collected by real-time shooting through a camera, and a RealSense D435 depth camera is selected as the camera.
  • the identification method provided by the present invention collects historical road surface images with different diseases as a data source, and the detection and segmentation of road surface images is realized through YOLACT.
  • YOLACT is a real-time instance segmentation model. It mainly implements instance segmentation through two parallel sub-networks. Compared with traditional instance segmentation models, there is only a slight loss of accuracy, but it greatly improves the speed of segmentation.
  • the road surface image after detection and instance segmentation becomes several visualized sample images.
  • Each sample image has characteristic parameters used to mark the sample image.
  • the specific model building process is as follows:
  • Step 1 Load the sample image and label information, convert them into a format recognizable by the model, normalize the data, and initialize the model parameters;
  • Step 2 Input the sample training set into the discriminant model and output a feature list, including four types of information: disease category, relative position of the disease, confidence level and disease instance segmentation area;
  • Step 3 The feature list and labels enter the loss function to calculate the loss
  • Step 4 Back propagate the adjusted parameters to verify the model until the model converges.
  • the next step is to apply the recognition model to the actual use environment, use the on-board camera to obtain new road video in real time, read the video frame, and crop the frame image to the size of (640, 640) or (320, 320);
  • the detection module part put it into the Neck module and the PANet module in sequence to obtain the prediction results, and then perform the NMS (non-maximum suppression) operation on the anchor to select the BBox result with the highest quality;
  • Protonet is used to classify the features extracted by the network at the pixel level to obtain the segmentation heat map
  • the obtained BBox results are restored to the segmentation heat map according to the ratio, and the segmentation heat map is cropped according to the BBox results;
  • the corresponding Bbox results and segmentation masks are restored according to the proportion of the original image and placed at the corresponding position of the original image for display.
  • Specific disease categories include cracks, ruts, potholes, looseness, oil spills and repairs. Cracks, ruts and potholes need to be repaired according to their specific degree of damage. For example, if the disease is determined to be a crack, its width will be calculated. If the disease is determined to be a pothole or rut, its depth is calculated. In this application, while identifying the road disease, it also classifies the road disease to facilitate maintenance by construction workers.
  • the calculation method of the crack width specifically includes the following steps:
  • A1 If the disease is determined to be a crack, determine the instance segmentation area
  • A2 Use Sobel operator method and central axis transformation method to determine the edge line and central axis line of the instance segmentation area;
  • A3 Based on the central axis, determine the center point list
  • A4 Take a center point, use the kd-tree algorithm to find the K nearest neighbors of a given point, then use singular value decomposition to calculate the normal vector of the skeleton line, determine the orthogonal slope based on the normal vector, and search for the nearest two edge line coordinate points. To calculate the width value;
  • A5 Loop step A4 until the width values corresponding to all center points in the center point list are calculated, and calculate the average width.
  • Figures 6 to 8 show the entire process of image processing by the algorithm. After determining that it is a crack-type disease, the width value in the entire length direction is obtained through the calculation of the edge line and central axis in Figure 8, and then the average value is calculated. , the average width is obtained, and the crack width is classified according to the average width. Different threshold ranges can be set to feedback different crack damage conditions.
  • the calculation method of the groove depth specifically includes the following steps:
  • the method for calculating the depth of the ruts is the same as the method for calculating the depth of the pits.
  • the average depth of the pits can be calculated, and the depth of the pits can be classified according to the average depth. By setting different threshold ranges, different pit damage conditions can be fed back.
  • the vehicle-mounted positioning device while generating the road surface image, it is also necessary to obtain the geographic coordinate information sent by the vehicle-mounted positioning device, and save the judgment result generated based on the road surface image together with the geographic coordinate information to the vehicle-mounted terminal and/or send it directly to the terminal device of the maintenance personnel by wireless transmission.
  • This method establishes a connection between the image and the coordinate information, thereby facilitating the later search for the specific location of the disease.
  • the specific implementation method can refer to the existing technology and will not be repeated here.
  • the embodiments of the present application may be provided as methods, devices, storage media, or electronic equipment products. Therefore, the embodiments of the present application may be implemented entirely in hardware embodiments, embodiments that combine hardware and software, or in pure software.
  • the real-time monitoring device for moving objects in the embodiments of the present application is introduced below.
  • the device embodiments below correspond to the method embodiments above.
  • Those skilled in the art can perform the implementation process below based on the above description. Understand, no detailed description will be given here;
  • this embodiment also provides a road disease identification system, including:
  • the data preprocessing module is used to collect historical road surface images with diseases, detect and segment them sequentially, and generate several visualized sample images.
  • the sample images have characteristic parameters that mark the sample images;
  • a sample data set building module uses sample images to build a sample training set
  • the model generation module builds a discrimination model for judging disease categories, uses the sample training set to train the discrimination model, and obtains a trained discrimination model;
  • the determination module is used to obtain new road surface images, detect and segment them sequentially, and then input them into the determination model to generate determination results;
  • the result post-processing module calculates the width if the disease is determined to be a crack, and calculates the depth if the disease is determined to be a pit or rut.
  • the image processing module is used to read the image in the video frame, crop the frame image, and extract features of the cropped image.
  • an electronic device including at least one processor; and a memory communicatively connected to the at least one processor;
  • the memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor, so that at least one processor can perform any of the above methods.
  • This kind of electronic device can be deployed on the vehicle on the edge device side NX, AGX and other hosts.
  • This type of host loads the Linux system for development.
  • the edge device side performs the calculation process, since the trained network model may be very large, the parameters There are many, and there are differences in machine performance on the deployment side, which will lead to slow inference speed and high latency. This is fatal for high real-time applications. Therefore, further, this embodiment also discloses TensorRT, which is a high-performance deep learning inference (Inference) optimizer that can provide low-latency, high-throughput deployment inference for deep learning applications. TensorRT can be used to accelerate inference in ultra-large-scale data centers, embedded platforms, or autonomous driving platforms.
  • Inference deep learning inference
  • the deployment phase mainly completes the reasoning process. Kernel Auto-Tuning and Dynamic Tensor Memory are completed here. Deserialize the model file in the previous step first and create a runtime engine. Then you can input data (such as pictures outside the test set or data set), and then output the classification vector result or detection result. After the deployment, optimization, and testing process, the final instance segmentation model can achieve real-time and efficient recognition effects on the edge device.
  • a computer storage medium stores a computer program, and the computer program is used to cause the computer to execute any of the above methods.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Geometry (AREA)
  • Quality & Reliability (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

本发明涉及人工智能技术领域，尤其涉及一种道路病害识别方法、系统、设备及存储介质，其识别方法包括：收集带有病害的历史路面图像，并对其依次进行检测和分割，生成若干个可视化的样本图像，样本图像上具有标记该样本图像的特征参数；利用样本图像建立样本训练集；搭建用于判断病害类别的判别模型，利用样本训练集对判别模型进行训练，得到训练好的判别模型；获取新路面图像，对其依次检测和分割后，输入到判别模型中生成判定结果；若判定病害为裂缝，则计算其宽度，若判定病害为坑槽或车辙，则计算其深度，采用本发明提供的识别方法，能够对道路视频数据进行病害识别分析，实现对病害区域的识别与标记，方便后续施工人员对道路进行养护。

Description

一种道路病害识别方法、系统、设备及存储介质 技术领域
本发明涉及人工智能技术领域，尤其涉及一种道路病害识别方法、系统、设备及存储介质。
 背景技术
现有的道路病害检测技术通常局限于道路裂缝的检测，但道路病害包括裂缝、坑槽、松散等多种类型，并非单一的道路裂缝，不同类型的道路病害对应的修补方法各不一样，多种道路病害的检测与分类对于道路病害修补十分重要。
现有的道路病害识别技术大多使用语义分割，这种对图像像素分类的技术易产生噪声区域，对定位产生干扰，并且，语义信息不够丰富，这样可能导致最终的语义分割结果准确率不高，分割效果不好。
公开于该背景技术部分的信息仅仅旨在加深对本公开总体背景技术的理解，而不应当被视为承认或以任何形式暗示该信息构成本领域技术人员所公知的现有技术。
 发明内容
本发明所要解决的技术问题是：提供一种道路病害识别方法、系统、设备及存储介质，能够实时对道路视频数据进行病害识别分析，实现对病害区域的识别与标记，方便后续施工人员对道路进行养护。
为了达到上述目的，本发明所采用的技术方案是：一种道路病害识别方法，包括：
收集带有病害的历史路面图像，并对其依次进行检测和分割，生成若干个可视化的样本图像，所述样本图像上具有标记该所述样本图像的特征参数；
利用所述样本图像建立样本训练集；
搭建用于判断病害类别的判别模型，利用所述样本训练集对所述判别模型进行训练，得到训练好的判别模型；
获取新路面图像，对其依次检测和分割后，输入到所述判别模型中生成判定结果；
若判定病害为裂缝，则计算其宽度，若判定病害为坑槽或车辙，则计算其深度。
进一步地，所述特征参数包括病害类别、病害相对位置、置信度和病害实例分割区域。
进一步地，所述病害类别包括裂缝、车辙、坑槽、松散、泛油和修补。
进一步地，所述路面图像来源于车载摄像头采集的视频，通过读取视频帧中的图像，并将该帧图像进行裁剪，以及将裁剪后图像进行特征提取，来获得路面图像。
进一步地，裂缝宽度的计算方法具体包括如下步骤：
A1：若判定病害为裂缝，则确定实例分割区域；
A2：确定实例分割区域边缘线和中轴线；
A3:根据所述中轴线，确定中心点列表；
A4:取一中心点，搜索最近的两个边缘线坐标点，计算宽度值；
A5:循环步骤A4，直到中心点列表中所有中心点对应的宽度值计算完毕，并计算宽度平均值。
进一步地，坑槽深度的计算方法具体包括如下步骤：
B1：若判定结果为坑槽类，则确定实例分割区域；
B2：根据实例分割区域，利用相机应用转换深度热度图；
   B3：确定相机外参，计算相机外参矩阵，修正深度系数；
B4：针对所有实例分割区域，判断去除奇异点，获取所有非奇异点坐标以及对应的深度值；
B5：基于所求得的所有非奇异点坐标以及对应的深度值求取该坑槽区域的平均深度。
进一步地，在生成路面图像的同时，还需要获取车载定位设备发送的地理坐标信息，将基于路面图像生成的判定结果连同地理坐标信息保存至车载终端和/或采用无线传输直接发送至养护人员的终端设备。
本发明还公开了一种道路病害识别系统，包括：
数据预处理模块，用于收集带有病害的历史路面图像，并对其依次进行检测和分割，生成若干个可视化的样本图像，所述样本图像上具有标记该所述样本图像的特征参数；
样本数据集建立模块，利用所述样本图像建立样本训练集；
模型生成模块，搭建用于判断病害类别的判别模型，利用所述样本训练集对所述判别模型进行训练，得到训练好的判别模型；
判定模块，用于获取新路面图像，对其依次检测和分割后，输入到所述判别模型中生成判定结果；
结果后处理模块，若判定病害为裂缝，则计算其宽度，若判定病害为坑槽或车辙，则计算其深度。
进一步地，还包括图像处理模块，所述图像处理模块用于读取视频帧中的图像，并将该帧图像进行裁剪，以及将裁剪后图像进行特征提取。
另一方面，本发明还公开了一种电子设备，包括至少一个处理器；
以及与所述至少一个处理器通信连接的存储器；
其中，所述存储器存储有可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行前面任一项所述的方法。
另一方面，本发明还公开了一种存储介质，所述计算机存储介质存储有计算机程序，所述计算机程序用于使计算机执行前面任一项所述的方法。
本发明的有益效果包括以下几点：
（1）、本发明除了道路裂缝检测之外，还加入了道路坑槽、道路松散等多类道路病害类型，对于道路病害的识别更为具体、完善；
（2）、本发明通过对病害区域的分割，获取如裂缝宽度、坑槽深度等信息，在识别道路病害的同时，对道路病害进行分级，方便施工人员进行维护；
（3）本发明区别于现有技术中的语义分割手段，采用先检测，后分割的实例分割技术，保证定位准确，且分割信息完整；
（4）区别于现存的道路病害识别系统大多为桌面客户端的行驶，只能对图片数据进行识别，而本系统部署于边缘设备，以一体机的形式可安装于车辆上，在车辆行驶时实时获取道路视频数据，进行病害识别分析，本发明相比于其它道路病害识别系统更为灵活，部署方便，且能达到实时病害可视化。
 附图说明
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明中记载的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为背景技术中采用语义分割的效果图；
图2为本发明实施例中道路病害识别方法的流程图；
图3为本发明实施例中道路病害识别方法的架构图；
图4为本发明实施例中对图像进行检测与分割后的效果图；
图5为本发明实施例中裂缝宽度计算的流程图；
图6为本发明实施例中用于展示裂缝宽度计算的效果图一；
图7为本发明实施例中用于展示裂缝宽度计算的效果图二；
图8为本发明实施例中用于展示裂缝宽度计算的效果图三；
图9为本发明实施例中坑槽深度计算的流程图；
图10为本发明实施例中道路病害识别系统的架构图。
 具体实施方式
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。
需要说明的是，当元件被称为“固定于”另一个元件，它可以直接在另一 个元件上或者也可以存在居中的元件。当一个元件被认为是“连接”另一个元 件，它可以是直接连接到另一个元件或者可能同时存在居中元件。本文所使用 的术语“垂直的”、“水平的”、“左”、“右”以及类似的表述只是为了说明的目 的，并不表示是唯一的实施方式。
除非另有定义，本文所使用的所有的技术和科学术语与属于本发明的技术 领域的技术人员通常理解的含义相同。本文中在本发明的说明书中所使用的术 语只是为了描述具体的实施例的目的，不是旨在于限制本发明。本文所使用的 术语“及/或”包括一个或多个相关的所列项目的任意的和所有的组合。
如图2至图9所示的道路病害识别方法，包括：
收集带有病害的历史路面图像，并对其依次进行检测和分割，生成若干个可视化的样本图像，所述样本图像上具有标记该所述样本图像的特征参数；
利用所述样本图像建立样本训练集；
搭建用于判断病害类别的判别模型，利用所述样本训练集对所述判别模型进行训练，得到训练好的判别模型；
获取新路面图像，对其依次检测和分割后，输入到所述判别模型中生成判定结果；
若判定病害为裂缝，则计算其宽度，若判定病害为坑槽或车辙，则计算其深度。
本发明提供的识别方法，采用算法实现，算法整体部署在nvidia jetson NX边缘计算平台，轻巧便捷，方便部署于任意车型，而对路面图像的采集，是通过摄像头来进行实时拍摄，摄像头方面选用RealSense D435深度相机。
本发明提供的识别方法，通过采集带有不同病害的历史路面图像作为数据源，路面图像的检测与分割是通过YOLACT来实现。YOLACT是一种实时实例分割的模型，它主要是通过两个并行的子网络来实现实例分割的，和传统的实例分割模型相比，只有轻微的精度损失，但却大大提升了分割的速度。
如图4所示，经过检测和实例分割后的路面图像变成若干个可视化的样本图像，每个样本图像都具有用于标记该样本图像的特征参数，选取部分样本图像来建立样本训练集，搭建判别模型，并利用样本训练集对判别模型进行训练，具体的模型搭建过程如下：
步骤1：加载样本图像和标签中的信息，转换为模型可识别的格式，对数据归一化处理，初始化模型参数；
步骤2：将样本训练集输入到判别模型中，输出特征列表，包含病害类别、病害相对位置、置信度和病害实例分割区域这四类信息；
步骤3：特征列表和标签进入损失函数计算损失；
步骤4：反向传播调整参数对模型进行验证，直到模型收敛。
建立好的识别模型，需要验证模型识别的准确率，准确率的确认可以通过人工复检来筛分出准确和不准确的数据，从而计算准确率，当准确率低时，还需要在选取之前未使用的样本图像进行继续训练，直到识别模型的准确率达标，之后获得训练好的识别模型。
接下来，就是将识别模型运用到实际的使用环境中，使用车载摄像头实时获取新的路面视频，读取视频帧，将该帧图像裁剪为（640,640）或（320,320）的尺寸；
将裁剪后的图像输入网络特征提取模块，提取特征，将其分别放入到YOLACT中的检测模块和分割模块中；
检测模块部分，将其依次放入Neck模块和PANet模块得到预测结果，然后对anchor执行NMS（非极大值抑制）操作，选取质量最高的BBox结果；
分割模块部分，使用Protonet对网络提取的特征进行像素级别分类，得到分割热度图；
将得到的BBox结果根据比例都还原到分割热度图中，并按照BBox结果对分割热度图进行裁剪；
对剪裁下来的分割热度图进行阈值化处理，得到分割mask（蒙板）；
对应的Bbox结果和分割mask都按原图比例还原并放置到原图对应位置进行显示。
具体的病害类别包括裂缝、车辙、坑槽、松散、泛油和修补，而裂缝、车辙和坑槽需要根据其具体损害程度来确定修复方式，比如，若判定病害为裂缝，则计算其宽度，若判定病害为坑槽或车辙，则计算其深度，本申请中在识别道路病害的同时，还对道路病害进行分级，方便施工人员进行维护。
作为上述实施例的具体公开，如图5所示，裂缝宽度的计算方法具体包括如下步骤：
A1：若判定病害为裂缝，则确定实例分割区域；
A2：使用Sobel算子法和中轴变换法确定实例分割区域边缘线和中轴线；
A3:根据中轴线，确定中心点列表；
A4:取一中心点，使用kd-tree算法查找给定点的K近邻点，然后使用奇异值分解计算骨架线的法向量，根据法向量确定正交斜率，搜索最近的两个边缘线坐标点，从而计算宽度值；
A5:循环步骤A4，直到中心点列表中所有中心点对应的宽度值计算完毕，并计算宽度平均值。
图6至图8展示了算法对图像处理的整个过程，在确定是裂缝类病害后，通过图8中边缘线和中轴线的计算，来获得整个长度方向上的宽度值，之后，进行均值计算，得出宽度平均值，根据宽度平均值对裂缝宽度进行等级划分，可以通过设置不同阈值范围，来反馈不同裂缝损害情况。
作为上述实施例的进一步公开，如图9所示，坑槽深度的计算方法具体包括如下步骤：
B1：若判定结果为坑槽类，则确定实例分割区域；
B2：根据实例分割区域，利用相机应用转换深度热度图；
   B3：确定相机外参，计算相机外参矩阵，修正深度系数；
B4：针对所有实例分割区域，判断去除奇异点，获取所有非奇异点坐标以及对应的深度值；
B5：基于所求得的所有非奇异点坐标以及对应的深度值求取该坑槽区域的平均深度。
而针对判定结果为车辙类，其车辙深度计算的方法与坑槽深度计算的方法等同，采用上述计算方法，可以计算出坑槽的平均深度，根据深度平均值对坑槽深度进行等级划分，可以通过设置不同阈值范围，来反馈不同坑槽损害情况。
作为上述实施例的优选，在生成路面图像的同时，还需要获取车载定位设备发送的地理坐标信息，将基于路面图像生成的判定结果连同地理坐标信息保存至车载终端和/或采用无线传输直接发送至养护人员的终端设备，这种方式将图像与坐标信息建立联系，从而方便后期查找病害具体位置，具体的实现方式可以参照现有技术，在此不再赘述。
本领域的技术人员应当知道，本申请实施例中可提供为方法、装置、存储介质或者电子设备产品，因此本申请实施例可以完全采用硬件实施例、硬件与软件结合的实施例或者纯软件实施例，下面对本申请实施例中的运动物体的实时监测装置进行介绍，下文中的装置实施例与上文中的方法实施例相互对应，本领域技术人员可以基于上文的描述对下文的实施过程进行理解，这里不再进行详细描述；
如图10所示，本实施例还提供了一种道路病害识别系统，包括：
数据预处理模块，用于收集带有病害的历史路面图像，并对其依次进行检测和分割，生成若干个可视化的样本图像，样本图像上具有标记该样本图像的特征参数；
样本数据集建立模块，利用样本图像建立样本训练集；
模型生成模块，搭建用于判断病害类别的判别模型，利用样本训练集对判别模型进行训练，得到训练好的判别模型；
判定模块，用于获取新路面图像，对其依次检测和分割后，输入到判别模型中生成判定结果；
结果后处理模块，若判定病害为裂缝，则计算其宽度，若判定病害为坑槽或车辙，则计算其深度。
进一步地，还包括图像处理模块，图像处理模块用于读取视频帧中的图像，并将该帧图像进行裁剪，以及将裁剪后图像进行特征提取。
在本发明实施例的以下部分，对本发明实施例中的计算机存储介质及电子设备实施例进行介绍，下文中的计算机存储介质以及处理器实施例与上文中的方法实施例相互对应，本领域技术人员可以基于上文的描述对下文的实施过程进行理解，这里不再进行详细描述；
在本发明实施例的另一方面，还提供了一种电子设备，包括至少一个处理器；以及与至少一个处理器通信连接的存储器；
其中，存储器存储有可被至少一个处理器执行的指令，指令被至少一个处理器执行，以使至少一个处理器能够上述任一项的方法。
这种电子设备可以是部署于车辆上的在边缘设备端NX、AGX等主机上，该类主机装载linux系统以开发，边缘设备端进行运算过程时，由于训练的网络模型可能会很大，参数很多，而且部署端的机器性能存在差异，就会导致推理速度慢，延迟高。这对于高实时性的应用场合是致命的。所以，进一步地，本实施例还公开了TensorRT，TensorRT是一种高性能的深度学习推理（Inference）优化器，可以为深度学习应用提供低延迟、高吞吐率的部署推理。TensorRT可用于对超大规模数据中心、嵌入式平台或自动驾驶平台进行推理加速。
模型通过TensorRT加速的过程如下：
（1）.TensorRT安装,确认好设备的CUDA版本
（2）.将训练好的模型从pytorch模型转为通用的ONNX格式
（3）.bulid阶段：将ONNX格式的模型转为TensorRT模型进行加速和部署，在模型转换时会完成前述优化过程中的层间融合，精度校准。这一步的输出是一个针对特定GPU平台和网络模型的优化过的TensorRT模型，这个TensorRT模型可以序列化存储到磁盘或内存中。
（4）.deploy阶段：测试engine模型，deploy阶段主要完成推理过程，Kernel Auto-Tuning 和 Dynamic Tensor Memory是在这里完成的。将上面一个步骤中的模型文件首先反序列化，并创建一个runtime engine，然后就可以输入数据（比如测试集或数据集之外的图片），然后输出分类向量结果或检测结果。经过部署、优化、测试的流程，最终实例分割模型可以在边缘设备端达到实时高效的识别效果。
在本发明实施例的另一方面，还提供了一种计算机存储介质，计算机存储介质存储有计算机程序，计算机程序用于使计算机执行上述任一项的方法。
本行业的技术人员应该了解，本发明不受上述实施例的限制，上述实施例和说明书中描述的只是说明本发明的原理，在不脱离本发明精神和范围的前提下，本发明还会有各种变化和改进，这些变化和改进都落入要求保护的本发明范围内。本发明要求保护范围由所附的权利要求书及其等效物界定。

Claims (10)

  1. 一种道路病害识别方法，其特征在于，包括：
    收集带有病害的历史路面图像，并对其依次进行检测和分割，生成若干个可视化的样本图像，所述样本图像上具有标记该所述样本图像的特征参数；
    利用所述样本图像建立样本训练集；
    搭建用于判断病害类别的判别模型，利用所述样本训练集对所述判别模型进行训练，得到训练好的判别模型；
    获取新路面图像，对其依次检测和分割后，输入到所述判别模型中生成判定结果；
    若判定病害为裂缝，则计算其宽度，若判定病害为坑槽或车辙，则计算其深度。
  2. 根据权利要求1所述的道路病害识别方法，其特征在于，所述特征参数包括病害类别、病害相对位置、置信度和病害实例分割区域。
  3. 根据权利要求1所述的道路病害识别方法，其特征在于，所述病害类别包括裂缝、车辙、坑槽、松散、泛油和修补。
  4. 根据权利要求1所述的道路病害识别方法，其特征在于，所述路面图像来源于车载摄像头采集的视频，通过读取视频帧中的图像，并将该帧图像进行裁剪，以及将裁剪后图像进行特征提取，来获得路面图像。
  5. 根据权利要求1所述的道路病害识别方法，其特征在于，裂缝宽度的计算方法具体包括如下步骤：
    A1：若判定病害为裂缝，则确定实例分割区域；
    A2：确定实例分割区域边缘线和中轴线；
    A3:根据所述中轴线，确定中心点列表；
    A4:取一中心点，搜索最近的两个边缘线坐标点，计算宽度值；
    A5:循环步骤A4，直到中心点列表中所有中心点对应的宽度值计算完毕，并计算宽度平均值。
  6. 根据权利要求1所述的道路病害识别方法，其特征在于，坑槽深度的计算方法具体包括如下步骤：
    B1：若判定结果为坑槽类，则确定实例分割区域；
    B2：根据实例分割区域，利用相机应用转换深度热度图；
       B3：确定相机外参，计算相机外参矩阵，修正深度系数；
    B4：针对所有实例分割区域，判断去除奇异点，获取所有非奇异点坐标以及对应的深度值；
    B5：基于所求得的所有非奇异点坐标以及对应的深度值求取该坑槽区域的平均深度。
  7. 一种道路病害识别系统，其特征在于，包括：
    数据预处理模块，用于收集带有病害的历史路面图像，并对其依次进行检测和分割，生成若干个可视化的样本图像，所述样本图像上具有标记该所述样本图像的特征参数；
    样本数据集建立模块，利用所述样本图像建立样本训练集；
    模型生成模块，搭建用于判断病害类别的判别模型，利用所述样本训练集对所述判别模型进行训练，得到训练好的判别模型；
    判定模块，用于获取新路面图像，对其依次检测和分割后，输入到所述判别模型中生成判定结果；
    结果后处理模块，若判定病害为裂缝，则计算其宽度，若判定病害为坑槽或车辙，则计算其深度。
  8. 根据权利要求7所述的道路病害识别系统，其特征在于，还包括图像处理模块，所述图像处理模块用于读取视频帧中的图像，并将该帧图像进行裁剪，以及将裁剪后图像进行特征提取。
  9. 一种电子设备，其特征在于，包括至少一个处理器；
    以及与所述至少一个处理器通信连接的存储器；
    其中，所述存储器存储有可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行权利要求1～6中任一项所述的方法。
  10. 一种存储介质，其特征在于，所述计算机存储介质存储有计算机程序，所述计算机程序用于使计算机执行权利要求1～6中任一项所述的方法。
PCT/CN2023/080853 2022-09-23 2023-03-10 一种道路病害识别方法、系统、设备及存储介质 WO2024060529A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211167654.4 2022-09-23
CN202211167654.4A CN115578326A (zh) 2022-09-23 2022-09-23 一种道路病害识别方法、系统、设备及存储介质

Publications (1)

Publication Number Publication Date
WO2024060529A1 true WO2024060529A1 (zh) 2024-03-28

Family

ID=84581275

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/080853 WO2024060529A1 (zh) 2022-09-23 2023-03-10 一种道路病害识别方法、系统、设备及存储介质

Country Status (2)

Country Link
CN (1) CN115578326A (zh)
WO (1) WO2024060529A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115578326A (zh) * 2022-09-23 2023-01-06 中路交科科技股份有限公司 一种道路病害识别方法、系统、设备及存储介质
CN117274817B (zh) * 2023-11-15 2024-03-12 深圳大学 裂缝自动化识别方法、装置、终端设备及存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109685124A (zh) * 2018-12-14 2019-04-26 斑马网络技术有限公司 基于神经网络的道路病害识别方法及装置
US20190154442A1 (en) * 2016-01-15 2019-05-23 Fugro Roadware Inc. High speed stereoscopic pavement surface scanning system and method
CN112598672A (zh) * 2020-11-02 2021-04-02 坝道工程医院(平舆) 一种基于深度学习的路面病害图像分割方法和系统
CN112966665A (zh) * 2021-04-01 2021-06-15 广东诚泰交通科技发展有限公司 路面病害检测模型训练方法、装置和计算机设备
CN113780200A (zh) * 2021-09-15 2021-12-10 安徽理工大学 基于计算机视觉的路面多种病害面积检测及定位方法
CN115578326A (zh) * 2022-09-23 2023-01-06 中路交科科技股份有限公司 一种道路病害识别方法、系统、设备及存储介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190154442A1 (en) * 2016-01-15 2019-05-23 Fugro Roadware Inc. High speed stereoscopic pavement surface scanning system and method
CN109685124A (zh) * 2018-12-14 2019-04-26 斑马网络技术有限公司 基于神经网络的道路病害识别方法及装置
CN112598672A (zh) * 2020-11-02 2021-04-02 坝道工程医院(平舆) 一种基于深度学习的路面病害图像分割方法和系统
CN112966665A (zh) * 2021-04-01 2021-06-15 广东诚泰交通科技发展有限公司 路面病害检测模型训练方法、装置和计算机设备
CN113780200A (zh) * 2021-09-15 2021-12-10 安徽理工大学 基于计算机视觉的路面多种病害面积检测及定位方法
CN115578326A (zh) * 2022-09-23 2023-01-06 中路交科科技股份有限公司 一种道路病害识别方法、系统、设备及存储介质

Also Published As

Publication number Publication date
CN115578326A (zh) 2023-01-06

Similar Documents

Publication Publication Date Title
WO2024060529A1 (zh) 一种道路病害识别方法、系统、设备及存储介质
Akagic et al. Pothole detection: An efficient vision based method using rgb color space image segmentation
CN109598794B (zh) 三维gis动态模型的构建方法
CN111611861B (zh) 一种基于多尺度特征关联的图像变化检测方法
CN110428490B (zh) 构建模型的方法和装置
CN111709966B (zh) 眼底图像分割模型训练方法及设备
CN113343461A (zh) 自动驾驶车辆的仿真方法、装置、电子设备及存储介质
CN110020650B (zh) 一种基于深度学习识别模型的倾斜车牌的识别方法及装置
CN114821541B (zh) 一种基于网格模型的道路面损伤检测方法及相关设备
US11587299B2 (en) Systems and methods for detection of anomalies in civil infrastructure using context aware semantic computer vision techniques
CN116597270A (zh) 基于注意力机制集成学习网络的道路损毁目标检测方法
CN114998251A (zh) 一种基于联邦学习的空中多视觉平台地面异常检测方法
CN114581886A (zh) 结合语义分割和频域分析的能见度判别方法、装置及介质
CN113326734A (zh) 一种基于YOLOv5的旋转目标检测方法
CN116958837A (zh) 基于无人机的市政设施故障检测系统
Dongye et al. A pavement disease detection method based on the improved mask R-CNN
KR102416714B1 (ko) 3차원 이미지 및 딥러닝을 활용한 도시 규모 나무 지도 작성 시스템 및 방법
CN112861682B (zh) 基于朴素贝叶斯云计算路面图像采集分类方法及其装置
US20230084761A1 (en) Automated identification of training data candidates for perception systems
CN114639013A (zh) 基于改进Orient RCNN模型的遥感图像飞机目标检测识别方法
Kee et al. Cracks identification using mask region-based denoised deformable convolutional network
WO2023007535A1 (ja) 下水管内異状診断支援システム、下水管内異状診断支援システム用のクライアントマシン、サーバマシン、及び関連する方法
CN118247717A (zh) 一种基于可变形注意力-空间多尺度网络模型的路面病害识别方法、存储介质以及电子装置
CN117611431A (zh) 图像生成方法、模型训练方法、装置及存储介质
Liu et al. Enhanced Visual Pipeline Defect Detection Using Partial Convolution Image Restoration

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23866832

Country of ref document: EP

Kind code of ref document: A1