TW202136752A - Detection system and method for detecting road damage distinguishing road damage according to the image and calculating and storing instantaneous coordinates of the GNSS-RTK moving device according to the observation data - Google Patents

Abstract

This invention provides a detection system for detecting road damage. The detection system includes a GNSS-RTK positioning device used for receiving and transmitting observation data; a moving carrier including an image acquisition device for acquiring an image of a road pavement; a GNSS-RTK moving device used for receiving and transmitting the observation data of the GNSS-RTK positioning device; and a calculation unit connected to the image acquisition device and the GNSS-RTK moving device by information for recognizing road damage according to the image, and also calculating and storing instantaneous coordinates of the GNSS-RTK moving device according to the observation data to achieve precise positioning for detecting and recording road damage.

Description

Detection system and method for detecting road damage

The present invention relates to a detection system and method for detecting road damage, especially the use of multi-satellite combination combined with real-time dynamic measurement technology (Global Navigation Satellite System-Real Time Kinematic, GNSS-RTK) and image capture methods to detect road damage, Method and system for recording the location of damage coordinates and estimating the size of damage.

Roads are the lifeblood of logistics in today's society and are closely related to people's daily commuting. Therefore, the smoothness of the roads will not only affect the economy, but also have a major relationship with the people's traffic safety. However, the road has been used for a long time, and the occurrence of damage is inevitable. At present, domestic road maintenance is mostly by means of regular dispatch of engineering vehicles for inspection and people’s notification, but this method is very manpower and time-consuming. In addition, everyone judges the road damage The standards are different, so it is hoped that technology can be used to achieve objective judgments.

With the development of science and technology, the field of AI has gradually matured. Deep learning has been used in many fields, such as cancer diagnosis, equipment diagnosis, financial marketing, etc., but it has not yet been used in road detection. Objects in images are obtained through deep learning. The subtle features of, provide a more accurate way to analyze road damage.

At present, the inspection vehicle is equipped with a GPS positioning device and a camera. The inspector can take pictures of the entire inspection route. If the road is damaged during the inspection, the on-board computer can be used to calibrate the damage in real time to capture the screen and use it. GPS positioning confirms the broken coordinates, and transmits the report information to the monitoring center through wireless transmission. Due to the use of manual judgment, there may be missed judgments and judgment errors. If AI is used for automatic detection, this problem can be avoided and reduced The cost of manpower; in addition, because GPS is affected by the following problems, there will be a large error in the coordinate positioning.

1. Atmospheric influence: The refraction effect of the ionosphere and troposphere in the atmosphere on electromagnetic waves changes the propagation speed of GPS signals, which delays GPS signals.

2. Satellite ephemeris error: Due to the complicated external force during the operation of the satellite, the ground control station and receiving terminal cannot determine and master its laws, so the errors cannot be eliminated.

3. Multi-path effect: GPS signals are received after being reflected on different obstacles, which delays the time.

Due to the above factors, the application of GPS technology is limited to some extent. In order to achieve a certain accuracy, the distance from the ground control station is often required to be not too long (less than 10 kilometers) in actual applications.

In addition, when driving under an overpass, tunnel, underground parking lot and other sheltered environments, the positioning information cannot be obtained in real time, causing problems such as inability to locate.

In order to solve the above-mentioned lack of poor positioning accuracy and identify road damage, the present invention provides a detection system for detecting road damage, which includes: a GNSS-RTK positioning device for receiving and transmitting observation data; and a mobile vehicle , Which includes: an image capturing device for capturing images of road paving; a GNSS-RTK mobile device for receiving and transmitting observation data of the GNSS-RTK positioning device; a computing unit with information connected to the image The capture device and the GNSS-RTK mobile device identify road damage based on the image, and at the same time calculate and store the instantaneous coordinates of the GNSS-RTK mobile device based on the observation data.

As in the above-mentioned detection system for detecting road damage, the mobile vehicle further includes an inertial navigation module, and the information is connected to the calculation unit.

The detection system for detecting road damage as described above, wherein the image capturing device includes at least one camera.

As in the above-mentioned detection system for detecting road damage, the calculation unit uses a deep learning training model to identify whether the road surface in the image is damaged.

Such as the above-mentioned detection system for detecting road damage, wherein the deep learning training model can include a Region-based Convolutional Neural Network (R-CNN) and a Fast Region-based Convolutional Neural Network (R-CNN). based Convolutional Neural Network, Fast R-CNN), Faster Region-based Convolutional Neural Network (Faster R-CNN), Mask Region-based Convolutional Neural Network, Mask R-CNN) one of them.

Such as the above-mentioned detection system for detecting road damage, the deep learning training model can be the detection framework of YOLO (You Only Look Once) or the detection framework of Single Shot Multibox Detector (SSD).

Such as the above-mentioned detection system for detecting road damage, wherein the computing unit information is connected to the cloud or the Internet, the computing unit is replaced by the cloud, the road damage is identified based on the image, and the observation data is calculated and stored at the same time. GNSS-RTK mobile device instantaneous coordinates.

In addition, the present invention provides a detection method for detecting road damage. The steps include: inputting the image captured by the image capturing device into the trained deep learning training model to quickly detect the damage category; using photography The measurement technology estimates the actual area of the damaged area; and uses a combination of multiple satellites and real-time dynamic measurement technology GNSS-RTK to record and detect the instantaneous coordinates when the road is damaged.

The above-mentioned detection method for detecting road damage further includes the following steps: using an inertial navigation module to predict the instantaneous coordinates based on the last coordinate position and movement information in the no-signal condition under no-signal conditions.

The above-mentioned detection method for detecting road damage further includes the following steps: while capturing images, multi-satellite combination combined with real-time dynamic measurement technology GNSS-RTK through differential calculation to calculate the coordinate position.

As the above-mentioned detection method for detecting road damage, the deep learning training model includes a regional convolutional neural network, a fast regional convolutional neural network, a faster regional convolutional neural network, and a masked regional convolutional neural network One of the models of the road model.

As in the above-mentioned detection method for detecting road damage, the deep learning training model can be the detection framework of YOLO or the detection framework of a single multi-frame detector.

The embodiments of the present invention will be described below with reference to schematic diagrams of ideal embodiments of the present invention. The shapes and setting methods in these diagrams may vary due to manufacturing technology, design, and/or tolerances. Therefore, the embodiments described in the text of the present invention should not be regarded as limiting the structure of the present invention to specific elements or shapes, and should include any difference in shape caused by manufacturing.

Please refer to the schematic diagram of the road damage detection system 100 in Figure 1. As shown in the figure, when the mobile vehicle 101 is on the road to detect whether the road is damaged, the darker area is the range of the road damage detection. The range is not limited, so 360-degree road detection is also feasible.

Please refer to the schematic diagram of the GNSS-RTK technology in Figure 2 of the present invention. As shown in the figure, the GNSS-RTK technology includes: a plurality of satellites 300, and a GNSS-RTK positioning device 200 with a plurality of fixed points on the ground (for ease of description, Only a single illustration) and the GNSS-RTK mobile device 103 set on the mobile vehicle 101, where the GNSS-RTK positioning device 200 is best set to be erected on a point with known high-precision coordinates, that is, as an energy source Receive observation data transmitted by a plurality of satellites 300, and transmit the observation data of the GNSS-RTK positioning device 200 to the GNSS-RTK mobile device 103 through the transmission of radio equipment, so the GNSS-RTK mobile device 103 is not limited to settings Just above the mobile vehicle 101, the position that can receive the observation data of the satellite 300 and the GNSS-RTK positioning device 200 is enough. To facilitate the understanding of the content below, only the connection relationship of the GNSS-RTK technology will be explained here. The detailed process of GNSS-RTK technology will be described later.

Next, please refer to the schematic diagram of the mobile vehicle detection in FIG. 3 of the present invention. The mobile vehicle 101 is equipped with an image capturing device 102, such as a camera, a GNSS-RTK mobile device 103, and a computing unit 104, wherein the image capturing device 102. The GNSS-RTK mobile device 103 and the computing unit 104 can be integrated, and the computing unit 104 is connected to the image capturing device 102 and the GNSS-RTK mobile device 103. First, when the mobile vehicle When 101 encounters a damaged road on the road, the image capturing device 102 captures the road image IM received within the detection range, and according to the GNSS-RTK mobile device 103 from the GNSS-RTK positioning device 200 The obtained observation data is then simultaneously transmitted to the calculation unit 104 to perform road image IM identification processing, recording the current coordinates, and estimating the size of the road surface damage.

Next, the details of the steps and methods of the road damage detection system 100 will be described. First, please refer to the working flow chart of the calculation unit 104 in FIG. 4. As shown in the figure, the processing flow of the calculation unit 104 has 5 steps. According to different requirements, the execution order of the steps shown in the flowchart can be adjusted or some steps can be omitted.

In step S101, the computing unit 104 reads the captured road image IM from the image capturing device 102.

Step S102, according to the detected road image IM, the calculation unit 104 uses the deep learning model to quickly and uniformly identify whether the road is damaged. The following describes the damage identification training and prediction model. The best deep learning model configuration of the present invention In order to use the Mask Region based Convolutional Neural Network (Mask R-CNN), of course, the well-known deep learning model can also be used, such as: Region based Convolutional Neural Network (Region based Convolutional Neural Network) , R-CNN), Fast Region based Convolutional Neural Network (Fast R-CNN), Faster Region based Convolutional Neural Network (Faster R-CNN), or Use the detection framework of YOLO (You Only Look Once) or the detection framework of Single Shot Multibox Detector (SSD) for image recognition. Please also refer to Figures 5 to 7. Figure 5 shows the Mask R of the present invention. -The flow chart of the CNN model. Figure 6 is the flow chart of the Mask R-CNN training model of the present invention. Figure 7 is the flow chart of the Mask R-CNN detection model of the present invention. This embodiment uses the flowchart of the Mask R-CNN training model with the Backbone as the Residual Network 101 (Residual Network 101, ResNet101) to detect road damage.

The flowchart of the Mask R-CNN model of the present invention is an object detection and instance segmentation (Instance Segmentation) model. In step S201, the image is input to a pre-trained convolutional neural network for feature extraction to obtain the corresponding feature map ( feature map), step S202, use the region candidate network (Region Proposal Network, RPN) to extract candidate frames (region proposals) from the feature map, and use the candidate frame scores to filter out regions of interest (Region of interest) with higher accuracy. , RoI), step S203, use the RoI Align layer to extract the features of these RoIs, step S204, define a multi-task loss function for each sampled RoI Loss Function = Classification Loss + Bounding box regression Loss ) + Mask Loss. During training, the Faster Region based Convolutional Neural Network (Faster R-CNN) branch and the mask branch are trained separately and in parallel. Using the Mask R-CNN model to perform detection steps is similar to training, please refer to Figures 6 and 7. Steps S201 to S203 are the same as Figure 5, but different from step S204 in Figure 5, the Faster R-CNN branch and mask during training The branches are trained separately and in parallel, and when predicting, first run the Faster R-CNN branch to obtain the specific RoI Classification (such as cracks, potholes, etc.) and Bounding box in step S203, and then run the mask branch to predict each sense The segmentation mask on the area of interest (RoI) can quickly detect the damage category, and at the same time automatically circle the damaged area, and store the damaged image and damage information in the computing unit 104.

Please refer to step S103 of the work flow chart of the calculation unit 104 in FIG. 4, use GNSS-RTK technology to obtain high-precision coordinate positions and record them. Please refer to FIG. 8 and also refer to FIG. The RTK positioning device 200 will know the coordinates and carrier phase observation data of the GNSS-RTK positioning device 200 (the positioning phase generated by itself at the moment of receiving the observation and the phase difference received by the satellite 300), and the observation will be made through the communication device The data is sent to the GNSS-RTK mobile device 103 in real time, and the GNSS-RTK mobile device 103 uses the OTF (On-the-Fly) cycle undetermined value search method to quickly calculate the undetermined value (the satellite 300 and the GNSS-RTK positioning The integer cycle value between the devices 200 is an unknown value, which is called the cycle undetermined value. Among them, OTF generally refers to the algorithm that correctly solves the cycle undetermined value in a moving state, and then uses the differential positioning to calculate the GNSS-RTK movement The instantaneous coordinates of the device 103, which means that the GNSS-RTK mobile device 103 can solve the undetermined value of the frequency in a purely dynamic environment. In addition, the closer the distance between the two stations, the more it can eliminate the GNSS-RTK positioning device 200 and the GNSS-RTK The common error between the mobile devices 103 and the shorter the cycle undetermined value solution time is, the higher the positioning accuracy is.

Therefore, the use of GNSS-RTK technology can remove the common error between the GNSS-RTK mobile device 103 and the GNSS-RTK positioning device 200 and obtain the centimeter-level instantaneous coordinates of the GNSS-RTK mobile station. Compared with GPS, its positioning accuracy It is relatively large, so it can more accurately locate the damaged road, and it will not happen that the construction personnel can not find the damaged road surface on the coordinate of the road damage.

In addition, the system for detecting road damage of the present invention is equipped with an inertial navigation module 105 (not shown). When driving under an overpass, tunnel, underground parking lot, etc., the GNSS positioning accuracy is greatly reduced or cannot be positioned. , Using the inertial navigation module 105, it can be accurately positioned even in the driving process when the GNSS signal is lost. Inertial navigation technology uses a gyroscope to form a navigation coordinate system, so that the measurement axis of the accelerometer is stabilized in the coordinate system , And give the heading and attitude angle; the accelerometer is used to measure the acceleration of the moving body, and the speed is obtained after one integration of time, and the displacement can be obtained after one time integration of the speed. In this way, when the GNSS signal is lost , Can predict the real-time coordinate position.

＝

＝

（1），由式（1）解得方程式：x=

， 𝑧=

，可得像機距P之距離為

，其中f為相機焦距，P（x，z）為目標座標，此外焦距與感光元件長度為已知，因此可以計算得到破損之大小。Step S104, use photogrammetry to estimate the damage size. As shown in FIG. 9, the focal length of the image capturing device 102 and the photosensitive element are proportional to the focal point to damage distance and the size of the road surface damage, wherein the image capturing device 102 and the image The middle road distance is obtained by the dual-lens distance measurement, then please refer to Figure 10, according to the triangle similarity law:

=

=

(1), solve the equation by formula (1): x=

, 𝑧=

, The available distance between the camera and P is

, Where f is the focal length of the camera, P(x, z) is the target coordinate, and the focal length and the length of the photosensitive element are known, so the size of the damage can be calculated.

In step S105, the damaged image, damage coordinates, and damage estimated size information are aggregated and stored in the computing unit 104 to facilitate road damage management, or the information 104 in the computing unit is connected to the cloud or the Internet, 104 in the computing unit It is also possible to upload the observation data and the road image IM to the cloud or the Internet, and the cloud or the Internet replaces 104 in the computing unit to perform the above-mentioned deep learning. For image identification, training, and instantaneous coordinate calculation and storage, it is better What's more, it can do real-time management and cooperate with the real-time management system. When there is a maintenance engineering vehicle nearby, you can directly or calculate the best maintenance schedule for maintenance, so it can reduce the labor time and increase efficiency.

100: Pavement damage detection system 101: Mobile Vehicle 102: Image capture device 103: GNSS-RTK mobile device 104: calculation unit 105: Inertial Navigation Module 200: GNSS-RTK positioning equipment 300: Satellite IM: road image

Figure 1 shows a schematic diagram of the road damage detection system of the present invention. Figure 2 shows a schematic diagram of the GNSS-RTK technology of the present invention. Fig. 3 shows a schematic diagram of the mobile carrier of the present invention during detection. Fig. 4 shows a working flow chart of the computing unit of the present invention. Figure 5 shows a flowchart of the Mask R-CNN model of the present invention. Figure 6 shows a flowchart of the Mask R-CNN training model of the present invention. Figure 7 shows a flowchart of the Mask R-CNN prediction model of the present invention. Figure 8 shows a flow chart of the GNSS-RTK technology of the present invention. Fig. 9 shows a schematic diagram of the method for calculating the damage size of the present invention. Fig. 10 shows a schematic diagram of the dual-lens distance measurement of the present invention.

Claims (13)

A detection system for detecting road damage, which includes: A GNSS-RTK (Global Navigation Satellite System-Real Time Kinematic) positioning device for receiving and transmitting observation data; A mobile vehicle, which includes: An image capture device for capturing an image of a road pavement; A GNSS-RTK mobile device for receiving and transmitting the observation data of the GNSS-RTK positioning device; A calculation unit, which is connected to the image capturing device and the GNSS-RTK mobile device with information, identifies road damage based on the image, and calculates and stores the instantaneous coordinates of the GNSS-RTK mobile device based on the observation data. The detection system according to claim 1, wherein the mobile vehicle further includes an inertial navigation module, and the information is connected to the calculation unit. The detection system according to claim 1, wherein the observation data is a carrier phase observation data. The detection system according to claim 1, wherein the image capturing device includes at least one camera. The detection system according to claim 1, wherein the calculation unit uses a deep learning training model to identify whether the road surface in the image of the road pavement is damaged. The detection system according to claim 5, wherein the deep learning training model may include a regional convolutional neural network (Region based Convolutional Neural Network, R-CNN), and a Fast Region based Convolutional Neural Network (Fast Region based Convolutional Neural Network). , Fast R-CNN), Faster Region based Convolutional Neural Network (Faster R-CNN), Mask Region based Convolutional Neural Network (Mask Region based Convolutional Neural Network, Mask R) -CNN) one of them. According to the detection system described in claim 5, the deep learning training model may be a detection framework of YOLO (You Only Look Once) or a detection framework of Single Shot Multibox Detector (SSD). The detection system according to claim 1, wherein the computing unit information is connected to the cloud or the Internet, the computing unit is replaced by the cloud, the road damage is identified based on the image, and the GNSS is calculated and stored based on the observation data. RTK mobile device instantaneous coordinates. A detection method for detecting road damage, the steps include: Input the real-time image captured by the image capture device into the trained deep learning training model to quickly detect the damage category; Use photogrammetry techniques to estimate the actual area of the damaged area; and Use multi-satellite combination combined with real-time dynamic measurement technology GNSS-RTK to record and detect instantaneous coordinates when the road is damaged. The detection method described in claim 9 further includes the following steps: Using the inertial navigation module, it is used to predict the instantaneous coordinates based on the last coordinate position and movement information in the absence of signal in the absence of signal. The detection method described in claim 9 further includes the following steps: While capturing images, multi-satellite combination combined with real-time dynamic measurement technology GNSS-RTK through differential calculation to calculate the coordinate position. The detection method according to claim 9, wherein the deep learning training model includes a regional convolutional neural network, a fast regional convolutional neural network, a faster regional convolutional neural network, and a masked regional convolutional neural network model One of the models. The detection method according to claim 9, wherein the deep learning training model can be a detection framework of YOLO or a detection framework of a single multi-frame detector.

Images