CN115272300B - Pavement disease detection method, system, device, equipment and medium - Google Patents

Pavement disease detection method, system, device, equipment and medium Download PDF

Info

Publication number
CN115272300B
CN115272300B CN202211140242.1A CN202211140242A CN115272300B CN 115272300 B CN115272300 B CN 115272300B CN 202211140242 A CN202211140242 A CN 202211140242A CN 115272300 B CN115272300 B CN 115272300B
Authority
CN
China
Prior art keywords
road
frame image
vehicle
video frame
positioning data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202211140242.1A
Other languages
Chinese (zh)
Other versions
CN115272300A (en
Inventor
张银河
于美丽
封顺天
邓辉
安昭旭
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Telecom Digital City Technology Co ltd
Original Assignee
China Telecom Digital City Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Telecom Digital City Technology Co ltd filed Critical China Telecom Digital City Technology Co ltd
Priority to CN202211140242.1A priority Critical patent/CN115272300B/en
Publication of CN115272300A publication Critical patent/CN115272300A/en
Application granted granted Critical
Publication of CN115272300B publication Critical patent/CN115272300B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S19/00—Satellite radio beacon positioning systems; Determining position, velocity or attitude using signals transmitted by such systems
    • G01S19/38—Determining a navigation solution using signals transmitted by a satellite radio beacon positioning system
    • G01S19/39—Determining a navigation solution using signals transmitted by a satellite radio beacon positioning system the satellite radio beacon positioning system transmitting time-stamped messages, e.g. GPS [Global Positioning System], GLONASS [Global Orbiting Navigation Satellite System] or GALILEO
    • G01S19/42—Determining position
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/529—Depth or shape recovery from texture
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/01—Detecting movement of traffic to be counted or controlled
    • G08G1/0104—Measuring and analyzing of parameters relative to traffic conditions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence

Landscapes

  • Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Remote Sensing (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Quality & Reliability (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • Traffic Control Systems (AREA)

Abstract

The application discloses a pavement disease detection method, a system, a device, equipment and a medium, which are applied to the technical field of intelligent traffic, and specifically comprise the following steps: extracting pavement characteristic data of each target road video frame image from a road video stream reported by vehicle-mounted terminal equipment, and obtaining a pavement disease detection result of a road area in each target road video frame image by adopting a pavement disease detection model based on the pavement characteristic data; the method comprises the steps of determining geographic space grid codes of road areas in each target road video frame image based on GNSS positioning data of each target road video frame image in a GNSS positioning data stream reported by vehicle-mounted terminal equipment, and determining road geographic positions of the road areas in each target road video frame image based on the geographic space grid codes, so that road surface disease detection results and road geographic positions of each road area are obtained, the problem that road surface disease detection is easy to miss is effectively solved, and the efficiency and timeliness of road surface disease detection are improved.

Description

Pavement disease detection method, system, device, equipment and medium
Technical Field
The application relates to the technical field of intelligent traffic, in particular to a pavement damage detection method, a system, a device, equipment and a medium.
Background
With the increase of population, the expansion of cities and the improvement of economic level, the road construction scale is larger and larger, the total road mileage is increased year by year, the road damage speed is increased day by day, when the road is damaged, the vehicle driving can be influenced, even the traffic safety is endangered, and the road maintenance management is particularly important.
In view of the development situation of the current road maintenance management, a manual visual inspection mode is still adopted for identifying most road pavement diseases such as cracks, depressions, ruts, pits and the like, and the manual visual inspection mode has low detection efficiency, poor timeliness and easy omission, and can influence the traffic efficiency and ensure the safety of inspectors.
Disclosure of Invention
The embodiment of the application provides a pavement disease detection method, a system, a device, equipment and a medium, which are used for solving the problems of low detection efficiency, poor timeliness and easiness in omission of the existing pavement disease detection method.
The technical scheme provided by the embodiment of the application is as follows:
on one hand, the embodiment of the application provides a pavement disease detection method, which comprises the following steps:
receiving a Global Navigation Satellite System (GNSS) positioning data stream and a road video stream corresponding to the GNSS positioning data stream, which are transmitted by a vehicle-mounted terminal device and are collected at the same time when a video reporting mechanism is determined to be triggered in the vehicle running process; the method comprises the steps that GNSS positioning data in a GNSS positioning data stream correspond to road video frame images in a road video stream one by one;
extracting each target road video frame image from the road video stream, and performing road surface disease detection on road areas in each target road video frame image by adopting a road surface disease detection model to obtain a road surface disease detection result corresponding to the road areas in each target road video frame image; the road surface disease detection model is a deep learning model for detecting the type and degree of the road surface disease of the road area in the target road video frame image based on the depth characteristic data and the texture characteristic data of the road area in the target road video frame image;
determining a geographic space grid code corresponding to a road area in each target road video frame image based on GNSS positioning data corresponding to each target road video frame image in the GNSS positioning data stream, and determining a road geographic position corresponding to the road area in each target road video frame image based on the geographic space grid code corresponding to the road area in each target road video frame image;
and performing correlation processing on the road surface disease detection result corresponding to the road area in each target road video frame image and the road geographic position corresponding to the road area in each target road video frame image to obtain the road geographic position corresponding to the road area in each target road video frame image and the road surface disease detection result.
On the other hand, the embodiment of the application provides a pavement damage detection system, which comprises a plurality of vehicle-mounted terminal devices and a road management device;
the vehicle-mounted terminal equipment is used for sending the GNSS positioning data streams acquired at the same time and the road video streams corresponding to the GNSS positioning data streams to the road management equipment when the condition that the video reporting condition is met is determined in the running process of the vehicle; wherein, each GNSS positioning data in the GNSS positioning data stream corresponds to each road video frame image in the road video stream one by one;
the road management device is used for receiving the GNSS positioning data stream sent by the vehicle-mounted terminal device and the road video stream corresponding to the GNSS positioning data stream; extracting each target road video frame image from the road video stream, and performing road surface disease detection on the road area in each target road video frame image by adopting a road surface disease detection model to obtain a road surface disease detection result corresponding to the road area in each target road video frame image; determining a geographic space grid code corresponding to a road area in each target road video frame image based on GNSS positioning data corresponding to each target road video frame image in the GNSS positioning data stream, and determining a road geographic position corresponding to the road area in each target road video frame image based on the geographic space grid code corresponding to the road area in each target road video frame image; performing correlation processing on road surface disease detection results corresponding to road areas in each target road video frame image and road geographic positions corresponding to the road areas in each target road video frame image to obtain road geographic positions corresponding to the road areas in each target road video frame image and road surface disease detection results; the road surface disease detection model is a deep learning model for detecting the type and degree of the road surface disease of the road area in the target road video frame image based on the depth characteristic data and the texture characteristic data of the road area in the target road video frame image.
On the other hand, the embodiment of the application provides a road surface disease detection device, includes:
the data receiving unit is used for receiving a GNSS positioning data stream which is transmitted by the vehicle-mounted terminal equipment and acquired at the same time when the vehicle-mounted terminal equipment determines that the video reporting condition is met in the running process of the vehicle and a road video stream corresponding to the GNSS positioning data stream; wherein, each GNSS positioning data in the GNSS positioning data stream corresponds to each road video frame image in the road video stream one by one;
the system comprises a disease detection unit, a road video frame image acquisition unit and a road disease detection unit, wherein the disease detection unit is used for extracting each target road video frame image from a road video stream, and performing road disease detection on a road area in each target road video frame image by adopting a road disease detection model to obtain a road disease detection result corresponding to the road area in each target road video frame image; the road surface disease detection model is a deep learning model for detecting the type and degree of the road surface disease of the road area in the target road video frame image based on the depth characteristic data and the texture characteristic data of the road area in the target road video frame image;
the road positioning unit is used for determining a geographic space grid code corresponding to a road area in each target road video frame image based on GNSS positioning data corresponding to each target road video frame image in the GNSS positioning data stream, and determining a road geographic position corresponding to the road area in each target road video frame image based on the geographic space grid code corresponding to the road area in each target road video frame image;
and the result correlation unit is used for performing correlation processing on the road surface disease detection result corresponding to the road area in each target road video frame image and the road geographic position corresponding to the road area in each target road video frame image to obtain the road geographic position corresponding to the road area in each target road video frame image and the road surface disease detection result.
In another aspect, an embodiment of the present application provides an electronic device, including: the road surface disease detection method comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein the processor executes the computer program to realize the road surface disease detection method provided by the embodiment of the application.
On the other hand, an embodiment of the present application further provides a computer-readable storage medium, where computer instructions are stored, and when the computer instructions are executed by a processor, the method for detecting a road surface disease provided in the embodiment of the present application is implemented.
The beneficial effects of the embodiment of the application are as follows:
in the embodiment of the application, the GNSS positioning data stream collected at the same time and the road video stream corresponding to the GNSS positioning data stream are reported to the road management device by the vehicle-mounted terminal device installed on the vehicle when the vehicle running process determines that a video reporting mechanism is triggered, so that the road management device can detect road surface diseases of road geographic positions corresponding to the GNSS positioning data stream based on the GNSS positioning data stream collected at the same time and the road video stream corresponding to the GNSS positioning data stream reported by the vehicle-mounted terminal device, thereby realizing automatic identification of the road surface diseases of different road regions, further improving the working efficiency and the intelligent degree of road surface disease detection, greatly saving manpower, material resources and road maintenance cost, and vehicles such as taxies, private cars and the like are huge in cities, can comprehensively cover the whole city, and the GNSS positioning data stream collected at the same time and the GNSS positioning data stream corresponding to the GNSS positioning data stream collected by the vehicle-mounted terminal device can be used for detecting road surface diseases and avoiding waste caused by the effective calculation of the GNSS positioning data stream and the road video stream reporting cost when the GNSS positioning data stream is triggered by the GNSS positioning data stream and the GNSS positioning data stream reported during the running process.
Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
fig. 1 is a schematic system architecture diagram of a pavement damage detection system in an embodiment of the present application;
FIG. 2 is a schematic flow chart of a general road surface disease detection method in the embodiment of the present application;
FIG. 3 is a schematic view of a detailed flow chart of a pavement disease detection method in an embodiment of the present application;
FIG. 4 is a functional structure diagram of a pavement damage detection device in the embodiment of the present application;
fig. 5 is a schematic hardware structure diagram of an electronic device in an embodiment of the present application.
Detailed Description
In order to make the purpose, technical solution and advantages of the present application more clearly and clearly understood, the technical solution in the embodiments of the present application will be described below in detail and completely with reference to the accompanying drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
To facilitate a better understanding of the present application by those skilled in the art, a brief description of the technical terms involved in the present application will be given below.
The road management equipment is cloud equipment which runs with a road management system and can realize various management such as maintenance, construction and the like on a road. In the embodiment of the application, the road management system at least comprises a data fusion module for fusing the road video stream and the GNSS positioning data stream, a pavement disease detection module for detecting pavement diseases of the road video stream and the GNSS positioning data stream through Artificial Intelligence (AI), an acquisition trigger module for realizing data reporting trigger of the vehicle-mounted terminal equipment by setting a data reporting mechanism, a work order control module for automatically generating a road maintenance work order according to an identification result of the pavement disease detection module and performing work order tracking processing of the whole process, a data storage module for storing various data such as the road video stream, the GNSS positioning data stream and the pavement disease detection result, and a data query module for realizing data query and report generation.
The vehicle-mounted terminal equipment is front-end equipment which is installed on vehicles such as buses, taxis and private cars and is internally provided with an embedded operating system and has the functions of mobile communication, high-precision mobile positioning, vehicle state acquisition, information display and voice broadcast, image and video acquisition, vehicle bump induction and the like. In the embodiment of the present application, the vehicle-mounted terminal device at least includes a wireless communication module (e.g., a 4G and/or 5G communication module) for implementing wireless communication, a GNSS positioning module for implementing high-precision positioning, an On Board Diagnostic (OBD) interface module for implementing vehicle operation data acquisition, an information display module for implementing information display, a video acquisition module for implementing road video stream acquisition, and a bump sensing module for implementing vehicle bump sensing by a built-in three-axis gyroscope sensor.
The road surface disease detection model is obtained by training based on road surface feature data (including but not limited to depth feature data and texture feature data), standard road surface disease types and standard road surface disease degrees corresponding to each road image sample in the road image sample set, and is a deep learning model for detecting the road surface disease types and the road surface disease degrees of the road regions in the road video frame images based on the depth feature data and the texture feature data of the road regions in the road video frame images. In the embodiment of the present application, the pavement disease detection model may be, but is not limited to, a neural network model, a convolutional neural network model, and the like.
And the road damage type is the road damage type identified by the road damage detection model. In the embodiment of the present application, the pavement damage types include, but are not limited to, cracks, depressions, ruts, pits, and the like.
And the degree of the pavement diseases is the damage degree corresponding to the pavement disease type identified by the pavement disease detection model. In the embodiment of the application, the degree of the pavement damage can be the grade of the pavement damage, such as general, medium, high, highest and the like, and the higher the grade of the pavement damage is, the higher the damage degree of the pavement is; the degree of the road surface damage may also be a road surface damage score, such as 20 points, 50 points, 80 points, 100 points, etc., with the greater the road surface damage score, the higher the degree of road surface damage.
It should be noted that, in the present application, the terms "first", "second", and the like are used for distinguishing similar objects, and are not necessarily used for describing a specific order or sequence order. It is to be understood that such terms are interchangeable under appropriate circumstances such that the embodiments described herein are capable of operation in other sequences than described or illustrated herein.
After introducing the technical terms related to the present application, the application scenarios and design ideas of the embodiments of the present application are briefly described below.
Along with the rapid development of Chinese economy, the road construction scale is larger and larger, the total road mileage is increased year by year, the maintenance mileage proportion is increased year by year, and the mode of manually visually inspecting road defects cannot adapt to the current road maintenance requirement, therefore, in the embodiment of the application, the vehicle-mounted terminal equipment mounted on vehicles such as buses, taxis, private cars and the like is added into the road maintenance management, the GNSS positioning data stream acquired at the same time and the road video stream corresponding to the GNSS positioning data stream are reported to the road management equipment when the vehicle running process determines to trigger a video reporting mechanism through the vehicle-mounted terminal equipment, so that the road management equipment can automatically identify the road defects in different road areas based on the GNSS positioning data stream acquired at the same time and the road video stream corresponding to the GNSS positioning data stream reported by the vehicle-mounted terminal equipment, the working efficiency and the intelligent degree of detecting the road defects of the different road areas are improved, the manpower and material resources are greatly saved, the cost of the taxis and the cost of the private cars are reduced, the cost of the buses and the cost of the private cars and the vehicle running process of the vehicle maintenance can greatly reduce the cost of the vehicle maintenance and the vehicle maintenance mileage of the vehicle running process of the vehicle The repeated calculation of the GNSS positioning data stream and the road video stream at the same geographic position causes the waste of computing resources.
After introducing the application scenario and the design idea of the embodiment of the present application, the following describes in detail the technical solution provided by the embodiment of the present application.
Referring to fig. 1, a pavement damage detection system 100 provided in an embodiment of the present application at least includes a plurality of vehicle-mounted terminal devices 110 and a road management device 120;
the plurality of vehicle-mounted terminal devices 110 are configured to, when it is determined that a video reporting condition is satisfied during a vehicle driving process, send a GNSS positioning data stream acquired at the same time and a road video stream corresponding to the GNSS positioning data stream to the road management device 120; wherein, each GNSS positioning data in the GNSS positioning data stream corresponds to each road video frame image in the road video stream one by one;
the road management device 120 is configured to receive a GNSS positioning data stream sent by the vehicle-mounted terminal device 110 and a road video stream corresponding to the GNSS positioning data stream; extracting each target road video frame image from the road video stream, and performing road surface disease detection on the road area in each target road video frame image by adopting a road surface disease detection model to obtain a road surface disease detection result corresponding to the road area in each target road video frame image; determining a geographic space grid code corresponding to a road area in each target road video frame image based on GNSS positioning data corresponding to each target road video frame image in the GNSS positioning data stream, and determining a road geographic position corresponding to the road area in each target road video frame image based on the geographic space grid code corresponding to the road area in each target road video frame image; performing correlation processing on road surface disease detection results corresponding to road areas in each target road video frame image and road geographic positions corresponding to the road areas in each target road video frame image to obtain road geographic positions corresponding to the road areas in each target road video frame image and road surface disease detection results; the road surface disease detection model is a deep learning model for detecting the road surface disease based on the depth characteristic data and the texture characteristic data of the road area in the target road video frame image.
In one possible embodiment, the road management device 120 is further configured to send a mode-on instruction to the plurality of in-vehicle terminal devices 110;
the plurality of vehicle-mounted terminal devices 110 are further configured to start a data reporting mode when receiving a mode start instruction sent by the road management device 120; the data reporting mode comprises a first data reporting mode and a second data reporting mode; in the first data reporting mode, the vehicle-mounted terminal device 110 sends GNSS positioning data in real time, determines to trigger a video reporting mechanism when it is monitored that a vehicle bumps, and sends a GNSS positioning data stream acquired at the same time and a road video stream corresponding to the GNSS positioning data stream; in the second data reporting mode, the vehicle-mounted terminal device 110 sends GNSS positioning data in real time, determines to trigger a video reporting mechanism when receiving a video reporting instruction, and sends a GNSS positioning data stream acquired at the same time and a road video stream corresponding to the GNSS positioning data stream.
In a possible implementation, the vehicle-mounted terminal device 110 is further configured to send GNSS positioning data to the road management device 120 in real time;
the road management device 120 is further configured to, when receiving GNSS positioning data sent by the vehicle-mounted terminal device 110 in real time, send a video reporting instruction to the vehicle-mounted terminal device 110 when determining that the vehicle-mounted terminal device 110 meets a video reporting condition based on the GNSS positioning data sent by the vehicle-mounted terminal device 110 in real time; the video reporting condition at least comprises any one or any combination of the following conditions; the method comprises the steps that the reporting times of road video streams corresponding to the road geographic positions represented by GNSS positioning data sent by vehicle-mounted terminal equipment in real time are not higher than a set threshold value, the driving times of a fixed driving route where the road geographic positions represented by the GNSS positioning data sent by the vehicle-mounted terminal equipment in real time are not higher than the set driving times of the vehicle-mounted terminal equipment in a set time range, the road geographic positions represented by the GNSS positioning data sent by the vehicle-mounted terminal equipment in real time are appointed road positions to be detected, suspected road surface diseases exist at the road geographic positions represented by the GNSS positioning data before the GNSS positioning data sent by the vehicle-mounted terminal equipment in real time are received, and a video reporting instruction manually triggered by the GNSS positioning data sent by the vehicle-mounted terminal equipment in real time is received.
In one possible embodiment, the road management device 120 is specifically configured to determine each road video frame image in the road video stream as a target road video frame image; or, frame extraction processing is carried out on the road video stream according to the set frame extraction interval, and each target road video frame image is obtained.
In a possible implementation manner, the road management device 120 is further configured to, for each target road video frame image, based on a road surface defect detection result corresponding to a road region in the target road video frame image, when it is determined that a road region in the target road video frame image has a road surface defect, obtain a road video segment in a set time period including the target road video frame image from a road video stream, generate a road maintenance work order based on the road surface defect detection result, the road geographic position, and the road video segment corresponding to the road region in the target road video frame image, allocate the road maintenance work order to the first road management client for processing, and track a processing state of the road maintenance work order in real time.
In a possible implementation manner, the road management device 120 is further configured to store the road surface disease detection result, the road geographic position, the road video segment, and the road maintenance work order corresponding to the road area in each target road video frame image into the database; when a road data query request sent by a second road management client is received, acquiring a road surface disease detection result, a road geographic position, a road video segment and a road maintenance work order corresponding to each road area meeting the data query condition in the road data query request from a database; and generating a road data report and sending the road data report to a second road management client for displaying based on the road surface disease detection result, the road geographic position, the road video band and the road maintenance work order corresponding to each road area according with the data query condition in the road data query request.
Based on the pavement damage detection system shown in fig. 1, the embodiment of the present application provides a pavement damage detection method applied to a road management device 120, and referring to fig. 2, a general flow of the pavement damage detection method provided in the embodiment of the present application is as follows:
step 201: receiving a GNSS positioning data stream acquired at the same time and a road video stream corresponding to the GNSS positioning data stream, which are sent by the vehicle-mounted terminal device 110 when determining to trigger a video reporting mechanism in the running process of the vehicle; and each piece of GNSS positioning data in the GNSS positioning data stream corresponds to each road video frame image in the road video stream one by one.
In the embodiment of the present application, in order to enable the vehicle-mounted terminal devices 110 installed on vehicles such as buses, taxis, private cars, and the like to report the GNSS positioning data streams collected at the same time and the road video streams corresponding to the GNSS positioning data streams when a video reporting mechanism is determined to be triggered during the running of the vehicle, the road management device 120 may respectively send a mode opening instruction to each vehicle-mounted terminal device 110 added to the road surface disease detection system 100, and each vehicle-mounted terminal device 110 starts a data reporting mode when receiving the mode opening instruction sent by the road management device 120, so that the GNSS positioning data streams collected at the same time and the road video streams corresponding to the GNSS positioning data streams are reported when the video reporting mechanism is determined to be triggered during the running of the vehicle, thereby triggering the reporting processes of the GNSS positioning data streams and the road video streams. In specific implementation, the data reporting mode includes, but is not limited to, a first data reporting mode and a second data reporting mode, based on which, when each vehicle-mounted terminal device 110 receives a mode start instruction sent by the road management device 120, two modes, namely the first data reporting mode and the second data reporting mode, may be started to perform data reporting in a dual mode, and also any one of the first data reporting mode and the second data reporting mode may be started to perform data reporting in a single mode. In the first data reporting mode, the vehicle-mounted terminal device 110 sends GNSS positioning data in real time, and determines to trigger a video reporting mechanism and send a GNSS positioning data stream acquired at the same time and a road video stream corresponding to the GNSS positioning data stream when it is monitored that a vehicle bumps, and in the second data reporting mode, the vehicle-mounted terminal device 110 sends GNSS positioning data in real time, and determines to trigger the video reporting mechanism and send the GNSS positioning data stream acquired at the same time and the road video stream corresponding to the GNSS positioning data stream when a video reporting instruction is received.
In practical applications, when the road management device 120 sends a mode opening instruction to each vehicle-mounted terminal device 110 that joins the road surface disease detection system 100, in one possible embodiment, the road management device 120 may determine all vehicle-mounted terminal devices 110 on all buses, taxis, private cars, and other vehicles with wireless communication functions in the whole area (for example, the whole city) as the vehicle-mounted terminal devices 110 that join the road surface disease detection system 100 and send the mode opening instruction, in another possible embodiment, in consideration of problems such as personal privacy, the road management device 120 may first send a road maintenance joining request to the vehicle-mounted terminal devices 110 with wireless communication functions on all buses, taxis, private cars, and other vehicles in the whole area (for example, the whole city), when each vehicle-mounted terminal device 110 with wireless communication functions in the whole area receives the road maintenance request sent by the road management device 120, display or push inquiry information indicating whether to join the road maintenance is allowed to the driver user and send a feedback result of the road maintenance result indicating that the road maintenance is allowed to the vehicle-mounted terminal devices 110 when each vehicle-mounted terminal device 110 in the whole area (for example, the vehicle-mounted terminal devices 110) receives the road maintenance result indicating that the road maintenance is allowed to join the road maintenance system, and sends feedback data indicating that the road maintenance result indicating that the road maintenance is allowed to the road maintenance device 110, and the road surface disease detection system, thereby realizing that each vehicle-mounted terminal device 110 joins the road maintenance device 100, and sending feedback the road maintenance system joining control method.
Further, the road management device 120 sends a mode starting instruction to each vehicle-mounted terminal device 110 added to the road surface disease detection system 100, so that after each vehicle-mounted terminal device 110 added to the road surface disease detection system 100 starts a data reporting mode, each vehicle-mounted terminal device 110 can report GNSS positioning data in real time, and after the road management device 120 receives the GNSS positioning data sent by the vehicle-mounted terminal device 110 in real time, when it is determined that the vehicle-mounted terminal device 110 meets a video reporting condition based on the GNSS positioning data sent by the vehicle-mounted terminal device 110 in real time, the road management device sends a video reporting instruction to the vehicle-mounted terminal device 110; the video reporting condition at least comprises any one or any combination of the following conditions:
1) The number of reports of the road video stream corresponding to the geographical road position represented by the GNSS positioning data sent by the vehicle-mounted terminal device 110 in real time is not greater than a set threshold (e.g., 5 times);
2) The number of times of driving of the fixed driving route in which the road geographic position represented by the GNSS positioning data transmitted by the vehicle-mounted terminal device 110 in real time is not higher than the set number of times of driving (for example, 1 time or n times) of the vehicle-mounted terminal device 110 within the set time range (for example, the current day);
3) The geographical position of the road represented by the GNSS positioning data sent by the vehicle-mounted terminal device 110 in real time is a specified road position to be detected;
4) Detecting that a suspected road surface disease exists at a road geographic position represented by GNSS positioning data before the GNSS positioning data sent by the vehicle-mounted terminal device 110 in real time is received; for example, before receiving GNSS positioning data sent by the vehicle-mounted terminal device 110 in real time, it is detected that a suspected road surface defect exists at a geographic position of a road represented by the GNSS positioning data, but the suspected road surface defect cannot be confirmed due to object occlusion, image blurring, and the like;
5) And receiving a video reporting instruction manually triggered by GNSS positioning data sent by the vehicle-mounted terminal device 110 in real time.
In specific implementation, when the vehicle-mounted terminal device 110 receives a video reporting instruction sent by the road management device 120, it determines to trigger a video reporting mechanism, and starts to simultaneously acquire the GNSS positioning data stream and the road video stream corresponding to the GNSS positioning data stream, and sends the GNSS positioning data stream acquired at the same time and the road video stream corresponding to the GNSS positioning data stream to the road management device 120, of course, the vehicle-mounted terminal device 110 also determines to trigger the video reporting mechanism when it is monitored that the vehicle bumps, and starts to simultaneously acquire the GNSS positioning data stream and the road video stream corresponding to the GNSS positioning data stream, and sends the GNSS positioning data stream acquired at the same time and the road video stream corresponding to the GNSS positioning data stream to the road management device 120, and the road management device 120 can receive the positioning data stream acquired at the same time and the road video stream corresponding to the positioning data stream, which are sent when the vehicle-mounted terminal device 110 determines to trigger the video reporting mechanism in the driving process of the vehicle.
Step 202: extracting each target road video frame image from the road video stream, and performing road surface disease detection on road areas in each target road video frame image by adopting a road surface disease detection model to obtain a road surface disease detection result corresponding to the road areas in each target road video frame image; the road surface disease detection model is a deep learning model for detecting the type and degree of road surface diseases of the road area in the target road video frame image based on the depth characteristic data and the texture characteristic data of the road area in the target road video frame image.
In particular implementation, the road management device 120 may perform the following steps in step 202:
first, the road management device 120 extracts each target road video frame image from the road video stream. Specifically, in one embodiment, to achieve omni-directional road surface disease detection, the road management device 120 may determine each road video frame image in the road video stream as the target road video frame image. In another embodiment, in order to improve the efficiency of detecting the road surface diseases and reduce the consumption of computing resources, the road management device 120 may also perform frame extraction processing on the road video stream according to the set frame extraction interval to obtain each target road video frame image.
Then, the road management device 120 inputs each target road video frame image into the road surface disease detection model to obtain a road surface disease detection result corresponding to the road area in each target road video frame image. In practical application, the pavement disease detection model is divided into three stages in the pavement disease detection process, wherein the first stage is a road area identification stage, and in the stage, the pavement disease detection model detects road areas in video frame images of all target roads; the second stage is a characteristic extraction stage, in which the pavement disease detection model extracts depth characteristic data and texture characteristic data corresponding to a road area in each target road video frame image as pavement characteristic data; and the third stage is a disease identification stage, wherein in the stage, the pavement disease detection model identifies pavement diseases based on pavement characteristic data such as depth characteristic data and texture characteristic data corresponding to the road area in each target road video frame image to obtain pavement disease detection results such as pavement disease types and pavement disease degrees corresponding to the road area in each target road video frame image.
Step 203: and determining a geographic space grid code corresponding to a road area in each target road video frame image based on the GNSS positioning data corresponding to each target road video frame image in the GNSS positioning data stream, and determining a road geographic position corresponding to the road area in each target road video frame image based on the geographic space grid code corresponding to the road area in each target road video frame image.
In specific implementation, because urban roads are complicated, in order to accurately determine the geographic position of a road, when the road management device 120 locates the geographic position of a road corresponding to a road region in each target road video frame image, in an embodiment, the road management device 120 may convert GNSS locating data corresponding to each target road video frame image into corresponding geospatial mesh codes, and then determine the geographic position of a road corresponding to a road region in each target road video frame image based on the geographic space mesh codes corresponding to the road region in each target road video frame image, so as to accurately locate the geographic position of roads in multiple layers of roads, such as overpasses and viaducts, and improve the locating accuracy of road defect regions, and further achieve the dimension reduction processing from three-dimensional space coordinates to one-dimensional mesh codes, thereby further improving the detection efficiency of road defects. In another embodiment, the road management device 120 may further determine a passing geographic position represented by GNSS positioning data corresponding to each target road video frame image in a fixed driving route of the vehicle in which the vehicle-mounted terminal device 110 is located, as a road geographic position corresponding to a road area in each target road video frame image. Of course, the road management device 120 may also perform comprehensive analysis on the GNSS positioning data, the geospatial mesh code, and the fixed driving route corresponding to each target road video frame image to obtain the road geographic position corresponding to the road area in each target road video frame image.
Step 204: and performing correlation processing on the road surface disease detection result corresponding to the road area in each target road video frame image and the road geographic position corresponding to the road area in each target road video frame image to obtain the road geographic position corresponding to the road area in each target road video frame image and the road surface disease detection result.
In a specific implementation, in one embodiment, the road surface damage detection step in step 202 and the road area positioning step in step 203 may be performed sequentially, in another embodiment, in order to improve the road surface damage detection efficiency, the road surface damage detection step in step 202 and the road area positioning step in step 203 may also be performed in parallel, and after the steps 202 and 203 are performed, the road management device 120 may associate the road surface damage detection result with the road area positioning result through a frame identifier, that is, for each target road video frame image, the road surface damage detection result corresponding to the road area in the target road video frame image and the road geographic position corresponding to the road area in the target road video frame image may be associated based on the frame identifier of the target road video frame image, so as to obtain the road geographic position corresponding to the road area in the target road video frame image and the road surface damage detection result.
In this embodiment, after the road management device 120 associates the road surface defect detection result corresponding to the road region in each target road video frame image with the road geographic position corresponding to the road region in each target road video frame image, for each target road video frame image, and may further determine that there is a road surface defect in the road region in the target road video frame image based on the road surface defect detection result corresponding to the road region in the target road video frame image, obtain a road video segment in a set time period including the target road video frame image from the road video stream, generate and distribute the road maintenance work order to the first road management client for processing based on the road surface defect detection result, the road geographic position, and the road video segment corresponding to the road region in the target road video frame image, and track the processing state of the road maintenance work order in real time, thereby implementing automatic generation and real-time tracking of the road maintenance work order, and further saving manpower and material resources consumed by road surface defect detection.
In addition, the road management device 120 may further store the road surface disease detection result, the road geographic position, the road video segment, and the road maintenance work order corresponding to the road region in each target road video frame image into the database, and when receiving a road data query request sent by the second road management client, after obtaining the road surface disease detection result, the road geographic position, the road video segment, and the road maintenance work order corresponding to each road region that meet the data query condition in the road data query request from the database, generate a road data report based on the road surface disease detection result, the road geographic position, the road video segment, and the road maintenance work order corresponding to each road region that meet the data query condition in the road data query request, and send the road data report to the second road management client for display, thereby implementing a query function of the road surface disease data, and providing a good data support for subsequent road surface disease processing.
In practical application, to avoid the problem of privacy disclosure and improve information security, the road management device 120 may further perform target recognition such as face recognition, pedestrian recognition, license plate recognition, and the like on each road video frame image in the road video frame image corresponding to each target road video frame image before storing the road surface disease detection result, the road geographic position, the road video segment, and the road maintenance work order corresponding to the road region in each target road video frame image in the database, and after obtaining target objects such as face, pedestrian, license plate, and the like in each road video frame image in the road video frame image corresponding to the target road video frame image, code the target objects such as face, pedestrian, license plate, and the like in each road video frame image in the road video frame image corresponding to the target road video frame image. Further, the road management device 120 may store the road surface disease detection result, the road geographic position, the road maintenance work order, and the coded road video segment corresponding to the road area in each target road video frame image to the database, so that when receiving the road data query request sent by the second road management client, a road data report may be generated and sent to the second road management client for display based on the road surface disease detection result, the road geographic position, the road maintenance work order, and the coded road video segment corresponding to each road area in the database that meet the data query condition in the road data query request, thereby implementing privacy protection and improving information security while implementing the query of the road surface disease data.
The pavement damage detection method provided by the embodiment of the present application is further described in detail below with reference to the pavement damage detection system shown in fig. 1, and referring to fig. 3, an interaction flow of the pavement damage detection method provided by the embodiment of the present application is as follows:
step 301: the vehicle-mounted terminal device 110 monitors that a wireless communication connection is established with the road management device 120 when the vehicle is started based on the vehicle operation data acquired through the OBU interface.
Step 302: when the road management device 120 determines that the establishment of the wireless communication connection with the in-vehicle terminal device 110 is completed, it transmits a mode on instruction to the in-vehicle terminal device 110.
Step 303: when receiving the mode starting instruction sent by the road management device 120, the vehicle-mounted terminal device 110 starts the first data reporting mode and the second data reporting mode, and reports the GNSS positioning data in real time in the dual modes.
Step 304: after receiving the GNSS positioning data sent by the vehicle-mounted terminal device 110 in real time, the road management device 120 determines whether the vehicle-mounted terminal device 110 satisfies a video reporting condition based on the GNSS positioning data sent by the vehicle-mounted terminal device 110 in real time; the video reporting conditions are as described above, and the repeated parts are not described again.
Step 305: when the road management device 120 determines that the vehicle-mounted terminal device 110 satisfies the video reporting condition, it sends a video reporting instruction to the vehicle-mounted terminal device 110.
Step 306: when receiving the video reporting instruction sent by the road management device 120, the vehicle-mounted terminal device 110 determines to trigger a video reporting mechanism, and continues to execute step 308.
Step 307: when the vehicle-mounted terminal device 110 monitors that the vehicle bumps through a bump sensor such as a triaxial gyroscope sensor, a video reporting mechanism is determined to be triggered, and the step 308 is continuously executed.
Step 308: the vehicle-mounted terminal device 110 starts to collect the GNSS positioning data stream and the road video stream corresponding to the GNSS positioning data stream, and sends the GNSS positioning data stream collected at the same time and the road video stream corresponding to the GNSS positioning data stream to the road management device 120.
Step 309: the road management device 120 performs frame extraction processing on the road video stream according to the set frame extraction interval to obtain each target road video frame image, and inputs each target road video frame image into the road surface disease detection model to obtain a road surface disease detection result corresponding to the road area in each target road video frame image.
Step 310: the road management device 120 converts the GNSS positioning data corresponding to each target road video frame image into geospatial mesh codes, and determines a road geographic position corresponding to a road area in each target road video frame image based on the geospatial mesh codes corresponding to the road area in each target road video frame image.
Step 311: the road management device 120 associates the road surface defect detection result corresponding to the road region in each target road video frame image with the road geographic position corresponding to the road region in each target road video frame image through the frame identifier, so as to obtain the road geographic position corresponding to the road region in each target road video frame image and the road surface defect detection result.
Step 312: the road management device 120 acquires, for each target road video frame image, a road video segment in a set time period including the target road video frame image from the road video stream when determining that a road region in the target road video frame image has a road surface defect based on a road surface defect detection result corresponding to the road region in the target road video frame image.
Step 313: the road management device 120 generates a road maintenance work order based on the road surface disease detection result, the road geographic position and the road video segment corresponding to the road region in each target road video frame image, allocates the road maintenance work order to the first road management client for processing, and tracks the processing state of the road maintenance work order in real time.
Step 314: the road management device 120 stores the road surface disease detection result, the road geographical position, the road video segment and the road maintenance work order corresponding to the road area in each target road video frame image into the database.
Step 315: when the road management device 120 receives the road data query request sent by the second road management client, the road management device obtains the road surface disease detection result, the road geographic position, the road video segment and the road maintenance work order corresponding to each road area meeting the data query condition in the road data query request from the database.
Step 316: the road management device 120 generates a road data report based on the road surface defect detection result, the road geographic position, the road video segment and the road maintenance work order corresponding to each road region meeting the data query condition in the road data query request, and sends the road data report to the second road management client for display.
Based on the foregoing embodiments, an embodiment of the present application provides a road surface damage detection device applied to a road management device, and referring to fig. 4, a road surface damage detection device 400 provided in an embodiment of the present application at least includes:
the data receiving unit 401 is configured to receive a GNSS positioning data stream acquired at the same time and a road video stream corresponding to the GNSS positioning data stream, which are sent by a vehicle-mounted terminal device when it is determined that a video reporting condition is met during vehicle driving; the method comprises the steps that GNSS positioning data in a GNSS positioning data stream correspond to road video frame images in a road video stream one by one;
a disease detection unit 402, configured to extract each target road video frame image from the road video stream, and perform road disease detection on a road region in each target road video frame image by using a road disease detection model, to obtain a road disease detection result corresponding to the road region in each target road video frame image; the road surface disease detection model is a deep learning model for detecting the road surface disease based on the depth characteristic data and the texture characteristic data of the road area in the target road video frame image;
a road positioning unit 403, configured to determine, based on GNSS positioning data corresponding to each target road video frame image in the GNSS positioning data stream, a geospatial mesh code corresponding to a road region in each target road video frame image, and determine, based on a geospatial mesh code corresponding to a road region in each target road video frame image, a road geographic position corresponding to a road region in each target road video frame image;
a result associating unit 404, configured to perform association processing on the road surface disease detection result corresponding to the road region in each target road video frame image and the road geographic position corresponding to the road region in each target road video frame image to obtain the road geographic position and the road surface disease detection result corresponding to the road region in each target road video frame image.
In a possible implementation manner, the pavement damage detection apparatus 400 provided in the embodiment of the present application further includes:
a mode control unit 405, configured to send a mode starting instruction to instruct the vehicle-mounted terminal device to start a data reporting mode; the data reporting mode comprises a first data reporting mode and a second data reporting mode; in a first data reporting mode, the vehicle-mounted terminal equipment sends GNSS positioning data in real time, determines to trigger a video reporting mechanism when the vehicle is monitored to bump, and sends a GNSS positioning data stream acquired at the same time and a road video stream corresponding to the GNSS positioning data stream; and under a second data reporting mode, the vehicle-mounted terminal equipment sends the GNSS positioning data in real time, determines to trigger a video reporting mechanism when receiving a video reporting instruction, and sends the GNSS positioning data stream acquired at the same time and the road video stream corresponding to the GNSS positioning data stream.
In a possible implementation manner, the pavement damage detection apparatus 400 provided in the embodiment of the present application further includes:
the instruction issuing unit 406 is configured to receive GNSS positioning data sent by the vehicle-mounted terminal device in real time; based on GNSS positioning data sent by the vehicle-mounted terminal equipment in real time, when the vehicle-mounted terminal equipment is determined to meet a video reporting condition, sending a video reporting instruction to the vehicle-mounted terminal equipment; the video reporting condition at least comprises any one or any combination of the following conditions; the method comprises the steps that the reporting times of road video streams corresponding to the road geographic positions represented by GNSS positioning data sent by vehicle-mounted terminal equipment in real time are not higher than a set threshold value, the driving times of a fixed driving route where the road geographic positions represented by the GNSS positioning data sent by the vehicle-mounted terminal equipment in real time are not higher than the set driving times of the vehicle-mounted terminal equipment in a set time range, the road geographic positions represented by the GNSS positioning data sent by the vehicle-mounted terminal equipment in real time are appointed road positions to be detected, suspected road surface diseases exist at the road geographic positions represented by the GNSS positioning data before the GNSS positioning data sent by the vehicle-mounted terminal equipment in real time are received, and a video reporting instruction manually triggered by the GNSS positioning data sent by the vehicle-mounted terminal equipment in real time is received.
In a possible implementation manner, when extracting each target road video frame image from the road video stream, the disease detection unit 402 is specifically configured to:
respectively determining each road video frame image in the road video stream as a target road video frame image; or, performing frame extraction processing on the road video stream according to the set frame extraction interval to obtain each target road video frame image.
In a possible implementation manner, the pavement damage detection apparatus 400 provided in the embodiment of the present application further includes:
and the work order generating unit 407 is configured to, for each target road video frame image, obtain, when it is determined that a road region in the target road video frame image has a road surface defect based on a road surface defect detection result corresponding to the road region in the target road video frame image, a road video segment in a set time period including the target road video frame image from the road video stream, generate, based on the road surface defect detection result, the road geographic location, and the road video segment corresponding to the road region in the target road video frame image, a road maintenance work order, allocate the road maintenance work order to the first road management client, and track a processing state of the road maintenance work order in real time.
In a possible implementation manner, the pavement damage detection apparatus provided in the embodiment of the present application further includes:
the storage query unit 408 is configured to store the road surface disease detection result, the road geographic position, the road video segment, and the road maintenance work order corresponding to the road area in each target road video frame image into the database; when a road data query request sent by a second road management client is received, acquiring a road surface disease detection result, a road geographic position, a road video segment and a road maintenance work order corresponding to each road area meeting the data query condition in the road data query request from a database; and generating a road data report and sending the road data report to a second road management client for displaying based on the road surface disease detection result, the road geographic position, the road video band and the road maintenance work order corresponding to each road area according with the data query condition in the road data query request.
It should be noted that the principle of the road surface damage detection device 400 provided in the embodiment of the present application for solving the technical problem is similar to that of the road surface damage detection method provided in the embodiment of the present application, and therefore, for implementation of the road surface damage detection device 400 provided in the embodiment of the present application, reference may be made to implementation of the road surface damage detection method provided in the embodiment of the present application, and repeated details are not described again.
After the system, the method and the device for detecting a road surface defect provided by the embodiment of the present application are introduced, a brief introduction is performed on the electronic device provided by the embodiment of the present application.
Referring to fig. 5, an electronic device 500 provided in the embodiment of the present application at least includes: the road surface damage detection method includes a processor 501, a memory 502 and a computer program stored in the memory 502 and capable of running on the processor 501, and when the processor 501 executes the computer program, the road surface damage detection method provided by the embodiment of the application is implemented.
The electronic device 500 provided by the embodiment of the present application may further include a bus 503 that connects different components (including the processor 501 and the memory 502). Wherein bus 503 represents one or more of any of several types of bus structures, including a memory bus, a peripheral bus, a local bus, and the like.
The Memory 502 may include readable storage media in the form of volatile Memory, such as Random Access Memory (RAM) 5021 and/or cache Memory 5022, and may further include Read Only Memory (ROM) 5023. The memory 502 may also include a program tool 5025 having a set (at least one) of program modules 5024, the program modules 5024 including, but not limited to, an operating subsystem, one or more application programs, other program modules, and program data, each of which or some combination of which may comprise an implementation of a network environment.
The processor 501 may be a single Processing element or a plurality of Processing elements, for example, the processor 501 may be a Central Processing Unit (CPU) or one or more integrated circuits configured to implement the pavement damage detection method provided in the embodiment of the present application. In particular, the processor 501 may be a general-purpose processor including, but not limited to, a CPU, an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, and the like.
Electronic device 500 may communicate with one or more external devices 504 (e.g., keyboard, remote control, etc.), may also communicate with one or more devices that enable a user to interact with electronic device 500 (e.g., cell phone, computer, etc.), and/or may communicate with any device that enables electronic device 500 to communicate with one or more other electronic devices 500 (e.g., router, modem, etc.). Such communication may be through an Input/Output (I/O) interface 505. Also, the electronic device 500 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public Network such as the Internet) via the Network adapter 506. As shown in FIG. 5, the network adapter 506 communicates with the other modules of the electronic device 500 over a bus 503. It should be understood that although not shown in FIG. 5, other hardware and/or software modules may be used in conjunction with the electronic device 500, including but not limited to: microcode, device drivers, redundant processors, external disk drive Arrays, redundant Array of Independent Disks (RAID) subsystems, tape drives, and data backup storage subsystems, to name a few.
It should be noted that the electronic device 500 shown in fig. 5 is only an example, and should not bring any limitation to the functions and the scope of the application of the embodiments.
The following describes a computer-readable storage medium provided by embodiments of the present application. The computer-readable storage medium provided by the embodiment of the application stores computer instructions, and the computer instructions, when executed by the processor, implement the road surface disease detection method provided by the embodiment of the application. Specifically, the computer instruction may be built in or installed in the processor, so that the processor may implement the pavement damage detection method provided by the embodiment of the present application by executing the built-in or installed computer instruction.
In addition, the road surface disease detection method provided by the embodiment of the present application can also be implemented as a computer program product, where the computer program product includes a program code, and when the program code runs on a processor, the computer program product implements the road surface disease detection method provided by the embodiment of the present application.
The computer program product provided by the embodiments of the present application may be one or more computer-readable storage media, and the computer-readable storage media may be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof, and specifically, more specific examples (a non-exhaustive list) of the computer-readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, a RAM, a ROM, an Erasable Programmable Read-Only Memory (EPROM), an optical fiber, a portable Compact disk Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
The computer program product provided by the embodiment of the application can adopt a CD-ROM and comprises program codes, and can also run on electronic equipment such as road management equipment. However, the computer program product provided by the embodiments of the present application is not limited thereto, and in the embodiments of the present application, the computer readable storage medium may be any tangible medium that can contain or store program code, which can be used by or in connection with an instruction execution system, apparatus, or device.
It should be noted that although in the above detailed description several units or sub-units of the apparatus are mentioned, such a division is merely exemplary and not mandatory. Indeed, the features and functions of two or more units described above may be embodied in one unit, according to embodiments of the application. Conversely, the features and functions of one unit described above may be further divided into embodiments by a plurality of units.
Further, while the operations of the methods of the present application are depicted in the drawings in a particular order, this does not require or imply that these operations must be performed in this particular order, or that all of the illustrated operations must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions.
While the preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all alterations and modifications as fall within the scope of the application.
It will be apparent to those skilled in the art that various changes and modifications may be made in the embodiments of the present application without departing from the spirit and scope of the embodiments of the present application. Thus, if such modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to encompass such modifications and variations.

Claims (8)

1. A pavement disease detection method is characterized by comprising the following steps:
receiving global navigation satellite system GNSS positioning data sent by vehicle-mounted terminal equipment in real time in the vehicle running process; the vehicle-mounted terminal equipment comprises vehicle-mounted terminal equipment installed on at least one type of vehicles in buses, taxis and private cars in a set area;
based on GNSS positioning data sent by the vehicle-mounted terminal equipment in real time, when the vehicle-mounted terminal equipment is determined to meet a video reporting condition, sending a video reporting instruction to the vehicle-mounted terminal equipment; wherein, the video reporting condition at least includes any one or any combination of the following: the reporting times of the road video stream corresponding to the road geographic position represented by the GNSS positioning data sent by the vehicle-mounted terminal equipment in real time are not higher than a set threshold value; the driving times of a fixed driving route in which the road geographic position represented by the GNSS positioning data sent by the vehicle-mounted terminal equipment in real time is not higher than the set driving times of the vehicle-mounted terminal equipment within a set time range; the geographical position of the road represented by the GNSS positioning data sent by the vehicle-mounted terminal equipment in real time is the position of the specified road to be detected; detecting that suspected pavement diseases exist at the geographical position of the road represented by the GNSS positioning data before the GNSS positioning data sent by the vehicle-mounted terminal equipment in real time is received; receiving a video reporting instruction triggered manually by GNSS positioning data sent in real time by the vehicle-mounted terminal equipment;
receiving a GNSS positioning data stream which is transmitted by the vehicle-mounted terminal equipment and acquired at the same time when a video reporting mechanism is determined to be triggered in the vehicle running process and a road video stream corresponding to the GNSS positioning data stream; the vehicle-mounted terminal equipment determines to trigger the video reporting mechanism when receiving a video reporting instruction and/or monitoring that a vehicle bumps; each piece of GNSS positioning data in the GNSS positioning data stream corresponds to each road video frame image in the road video stream one by one;
extracting each target road video frame image from the road video stream, and performing road surface disease detection on a road area in each target road video frame image by adopting a road surface disease detection model to obtain a road surface disease detection result corresponding to the road area in each target road video frame image; the road surface disease detection model is a deep learning model for detecting the type and the degree of the road surface disease of the road area in the target road video frame image based on the depth characteristic data and the texture characteristic data of the road area in the target road video frame image;
determining a geographic space grid code corresponding to a road area in each target road video frame image based on GNSS positioning data corresponding to each target road video frame image in the GNSS positioning data stream, acquiring a passing geographic position represented by the GNSS positioning data corresponding to each target road video frame image in a fixed driving route of a vehicle in which the vehicle-mounted terminal device is located, and determining a road geographic position corresponding to the road area in each target road video frame image based on the geographic space grid code corresponding to the road area in each target road video frame image and the passing geographic position represented by the GNSS positioning data corresponding to each target road video frame image;
performing association processing on the road surface disease detection result corresponding to the road area in each target road video frame image and the road geographic position corresponding to the road area in each target road video frame image to obtain the road geographic position corresponding to the road area in each target road video frame image and the road surface disease detection result;
when the road area in the target road video frame image is determined to have a road surface defect based on a road surface defect detection result corresponding to the road area in the target road video frame image, acquiring a road video segment with a set time period including the target road video frame image from the road video stream, respectively performing target identification on each road video frame image in the road video segment by adopting a target identification model, after a target object in each road video frame image in the road video segment is obtained, coding the target object in each road video frame image in the road video segment, generating a road maintenance work order based on the road surface defect detection result corresponding to the road area in the target road video frame image, the road geographic position and the coded road video segment, distributing the road maintenance work order to a first road management client for processing, and tracking the processing state of the maintenance work order in real time.
2. The method for detecting a road surface disease according to claim 1, wherein before receiving the GNSS positioning data stream collected at the same time and sent by the vehicle-mounted terminal device when determining to trigger the video reporting mechanism in the driving process of the vehicle and the road video stream corresponding to the GNSS positioning data stream, the method further comprises:
sending a mode opening instruction to the vehicle-mounted terminal equipment to indicate the vehicle-mounted terminal equipment to open a data reporting mode; the data reporting mode comprises a first data reporting mode and a second data reporting mode; in the first data reporting mode, the vehicle-mounted terminal equipment sends GNSS positioning data in real time, determines to trigger the video reporting mechanism when the vehicle is monitored to bump, and sends a GNSS positioning data stream acquired at the same time and a road video stream corresponding to the GNSS positioning data stream; and in the second data reporting mode, the vehicle-mounted terminal equipment sends GNSS positioning data in real time, determines to trigger a video reporting mechanism when receiving a video reporting instruction, and sends a GNSS positioning data stream acquired at the same time and a road video stream corresponding to the GNSS positioning data stream.
3. A road surface disease detection method according to claim 1, wherein extracting each target road video frame image from the road video stream comprises:
respectively determining each road video frame image in the road video stream as the target road video frame image;
or;
and performing frame extraction processing on the road video stream according to a set frame extraction interval to obtain each target road video frame image.
4. A pavement disease detection method according to claim 1, further comprising:
storing a road surface disease detection result, a road geographic position, a road video segment and a road maintenance work order corresponding to a road area in each target road video frame image into a database;
when a road data query request sent by a second road management client is received, acquiring a road surface disease detection result, a road geographic position, a road video segment and a road maintenance work order corresponding to each road area meeting the data query condition in the road data query request from the database;
and generating a road data report and sending the road data report to the second road management client for displaying based on the road surface disease detection result, the road geographic position, the road video segment and the road maintenance work order corresponding to each road area according with the data query condition in the road data query request.
5. A pavement damage detection system is characterized by comprising a plurality of vehicle-mounted terminal devices and a road management device; the vehicle-mounted terminal equipment comprises vehicle-mounted terminal equipment arranged on at least one type of bus, taxi and private car in a set area;
the vehicle-mounted terminal devices are used for transmitting Global Navigation Satellite System (GNSS) positioning data to the road management device in real time in the vehicle running process; when the vehicle is determined to meet the video reporting condition in the running process, sending the GNSS positioning data stream acquired at the same time and the road video stream corresponding to the GNSS positioning data stream to the road management equipment; the vehicle-mounted terminal equipment determines to trigger the video reporting mechanism when receiving a video reporting instruction and/or monitoring that a vehicle bumps; each piece of GNSS positioning data in the GNSS positioning data stream corresponds to each road video frame image in the road video stream one by one;
the road management device is used for receiving GNSS positioning data which are sent by the vehicle-mounted terminal device in real time in the vehicle running process, and sending a video reporting instruction to the vehicle-mounted terminal device when the vehicle-mounted terminal device is determined to meet a video reporting condition based on the GNSS positioning data which are sent by the vehicle-mounted terminal device in real time; wherein, the video reporting condition at least includes any one or any combination of the following: the method comprises the steps that the reporting times of a road video stream corresponding to a road geographic position represented by GNSS positioning data sent by vehicle-mounted terminal equipment in real time are not higher than a set threshold value, the driving times of a fixed driving route in which the road geographic position represented by the GNSS positioning data sent by the vehicle-mounted terminal equipment in real time is not higher than the set driving times of the vehicle-mounted terminal equipment in a set time range, the road geographic position represented by the GNSS positioning data sent by the vehicle-mounted terminal equipment in real time is a specified road position to be detected, a suspected road disease exists at the road geographic position represented by the GNSS positioning data before the GNSS positioning data sent by the vehicle-mounted terminal equipment in real time are received, and a video reporting instruction triggered manually by the GNSS positioning data sent by the vehicle-mounted terminal equipment in real time is received; receiving the GNSS positioning data stream sent by the vehicle-mounted terminal equipment and a road video stream corresponding to the GNSS positioning data stream, extracting each target road video frame image from the road video stream, and performing road surface disease detection on a road area in each target road video frame image by adopting a road surface disease detection model to obtain a road surface disease detection result corresponding to the road area in each target road video frame image; determining a geographic space grid code corresponding to a road area in each target road video frame image based on GNSS positioning data corresponding to each target road video frame image in the GNSS positioning data stream, acquiring a passing geographic position represented by GNSS positioning data corresponding to each target road video frame image in a fixed driving route of a vehicle in which the vehicle-mounted terminal device is located, and determining a road geographic position corresponding to a road area in each target road video frame image based on the geographic space grid code corresponding to the road area in each target road video frame image and the passing geographic position represented by the GNSS positioning data corresponding to each target road video frame image; performing association processing on the road surface disease detection result corresponding to the road area in each target road video frame image and the road geographic position corresponding to the road area in each target road video frame image to obtain the road geographic position corresponding to the road area in each target road video frame image and the road surface disease detection result; the road surface disease detection model is a deep learning model for detecting the type and the degree of the road surface disease of the road area in the target road video frame image based on the depth characteristic data and the texture characteristic data of the road area in the target road video frame image; when the road region in the target road video frame image has a road fault, acquiring a road video segment with a set time period including the target road video frame image from the road video stream, respectively performing target identification on each road video frame image in the road video segment by using a target identification model to obtain a target object in each road video frame image in the road video segment, coding the target object in each road video frame image in the road video segment, generating a road maintenance work order based on the road fault detection result, the road geographic position and the coded road video segment corresponding to the road region in the target road video frame image, distributing the road maintenance work order to a first road management client side for processing, and tracking the processing state of the road maintenance work order in real time.
6. A pavement damage detection device, characterized in that includes:
the system comprises an instruction issuing unit and a video reporting unit, wherein the instruction issuing unit is used for receiving Global Navigation Satellite System (GNSS) positioning data which is sent by vehicle-mounted terminal equipment in real time in the vehicle running process, and sending a video reporting instruction to the vehicle-mounted terminal equipment when the vehicle-mounted terminal equipment meets video reporting conditions based on the GNSS positioning data which is sent by the vehicle-mounted terminal equipment in real time; the vehicle-mounted terminal equipment comprises vehicle-mounted terminal equipment installed on at least one type of vehicles in buses, taxis and private cars in a set area; the video reporting condition at least comprises any one or any combination of the following conditions: the reporting times of the road video stream corresponding to the road geographic position represented by the GNSS positioning data sent by the vehicle-mounted terminal equipment in real time are not higher than a set threshold value; the driving times of a fixed driving route in which a road geographic position represented by GNSS positioning data sent by the vehicle-mounted terminal equipment in real time is not higher than the set driving times of the vehicle-mounted terminal equipment within a set time range; the geographical position of the road represented by the GNSS positioning data sent by the vehicle-mounted terminal equipment in real time is the position of the specified road to be detected; detecting that suspected pavement diseases exist at the geographical position of the road represented by the GNSS positioning data before the GNSS positioning data sent by the vehicle-mounted terminal equipment in real time is received; receiving a video reporting instruction triggered manually by GNSS positioning data sent in real time by the vehicle-mounted terminal equipment;
the data receiving unit is used for receiving a GNSS positioning data stream which is transmitted by the vehicle-mounted terminal equipment and acquired at the same time when the vehicle-mounted terminal equipment determines that a video reporting condition is met in the running process of the vehicle and a road video stream corresponding to the GNSS positioning data stream; the vehicle-mounted terminal equipment determines to trigger the video reporting mechanism when receiving a video reporting instruction and/or monitoring that a vehicle bumps; each piece of GNSS positioning data in the GNSS positioning data stream corresponds to each road video frame image in the road video stream one by one;
a disease detection unit, configured to extract each target road video frame image from the road video stream, and perform road disease detection on a road area in each target road video frame image by using a road disease detection model to obtain a road disease detection result corresponding to the road area in each target road video frame image; the road surface disease detection model is a deep learning model for detecting the type and the degree of the road surface disease of the road area in the target road video frame image based on the depth characteristic data and the texture characteristic data of the road area in the target road video frame image;
a road positioning unit, configured to determine, based on GNSS positioning data corresponding to each target road video frame image in the GNSS positioning data stream, a geospatial mesh code corresponding to a road area in each target road video frame image, and obtain a passing geographic position represented by the GNSS positioning data corresponding to each target road video frame image in a fixed travel route of a vehicle in which the vehicle-mounted terminal device is located, and determine, based on the geospatial mesh code corresponding to the road area in each target road video frame image and the passing geographic position represented by the GNSS positioning data corresponding to each target road video frame image, a road geographic position corresponding to the road area in each target road video frame image;
a result association unit, configured to perform association processing on a road surface disease detection result corresponding to the road region in each target road video frame image and a road geographic position corresponding to the road region in each target road video frame image to obtain a road geographic position corresponding to the road region in each target road video frame image and a road surface disease detection result;
the work order generating unit is used for acquiring a road video segment of a set time period including a target road video frame image from the road video stream when determining that the road region in the target road video frame image has a road surface disease based on a road surface disease detection result corresponding to the road region in the target road video frame image, respectively performing target identification on each road video frame image in the road video segment by adopting a target identification model to obtain a target object in each road video frame image in the road video segment, then coding the target object in each road video frame image in the road video segment, generating a road work order based on the road surface disease detection result corresponding to the road region in the target road video frame image, the road geographic position and the coded road video segment, distributing the road work order to a first road management client side for processing, and tracking the processing state of the road maintenance work order in real time.
7. An electronic device, comprising: a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method of detecting a road damage according to any one of claims 1-4 when executing the computer program.
8. A computer-readable storage medium storing computer instructions which, when executed by a processor, implement the road surface disease detection method according to any one of claims 1 to 4.
CN202211140242.1A 2022-09-20 2022-09-20 Pavement disease detection method, system, device, equipment and medium Active CN115272300B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211140242.1A CN115272300B (en) 2022-09-20 2022-09-20 Pavement disease detection method, system, device, equipment and medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211140242.1A CN115272300B (en) 2022-09-20 2022-09-20 Pavement disease detection method, system, device, equipment and medium

Publications (2)

Publication Number Publication Date
CN115272300A CN115272300A (en) 2022-11-01
CN115272300B true CN115272300B (en) 2023-01-06

Family

ID=83757421

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211140242.1A Active CN115272300B (en) 2022-09-20 2022-09-20 Pavement disease detection method, system, device, equipment and medium

Country Status (1)

Country Link
CN (1) CN115272300B (en)

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109685124A (en) * 2018-12-14 2019-04-26 斑马网络技术有限公司 Road disease recognition methods neural network based and device
CN110910354A (en) * 2019-11-07 2020-03-24 安徽乐道信息科技有限公司 Road detection vehicle and road detection method and device
CN111080620B (en) * 2019-12-13 2024-03-19 中远海运科技股份有限公司 Road disease detection method based on deep learning
CN111191803A (en) * 2020-03-21 2020-05-22 中设设计集团北京民航设计研究院有限公司 Information management platform system for airport flight area pavement
CN112580833B (en) * 2020-12-07 2022-04-19 武汉羿畅科技有限公司 Rural highway maintenance informatization management system
CN113255605A (en) * 2021-06-29 2021-08-13 深圳市城市交通规划设计研究中心股份有限公司 Pavement disease detection method and device, terminal equipment and storage medium
CN113506261A (en) * 2021-07-07 2021-10-15 北京市商汤科技开发有限公司 Road disease detection method, device, equipment and system
CN114882367A (en) * 2022-05-26 2022-08-09 上海工程技术大学 Airport pavement defect detection and state evaluation method

Also Published As

Publication number Publication date
CN115272300A (en) 2022-11-01

Similar Documents

Publication Publication Date Title
US10304260B2 (en) Automated vehicle identification and inspection data processing system
US9256995B2 (en) Apparatus for diagnosing driving behavior, method for diagnosing driving behavior, and program thereof
WO2018122802A1 (en) Multimodal road traffic anomaly detection method
EP4239614A2 (en) Systems and methods for image-based location determination and parking monitoring
CN113822285A (en) Vehicle illegal parking identification method for complex application scene
CN114492022A (en) Road condition sensing data processing method, device, equipment, program and storage medium
CN115272222A (en) Method, device and equipment for processing road detection information and storage medium
CN115272300B (en) Pavement disease detection method, system, device, equipment and medium
CN106128109A (en) A kind of different territories based on Big Dipper space-time traffic signal slag-soil truck is broken rules and regulations analytical equipment and method
Bhandari et al. Fullstop: A camera-assisted system for characterizing unsafe bus stopping
CN114596704A (en) Traffic event processing method, device, equipment and storage medium
CN109685027B (en) Newly added road identification method and system based on block chain technology
CN114093155A (en) Traffic accident responsibility tracing method and device, computer equipment and storage medium
CN106781470B (en) Method and device for processing running speed of urban road
CN109859488A (en) A kind of clone's taxi detection method and detection system
WO2022233099A1 (en) Networked adas-based method for investigating spatial-temporal characteristics of road area traffic violation behavior
CN111402575A (en) Method and device for evaluating traffic running state
Simeunović et al. Smart Parking Technologies
Adibhatla et al. Street quality identification device
Babu et al. Smart Transport System for Passenger Comfort using IoT
Hou Exploration of Real-Time Traffic Information System Based on Big Data Analysis
CN106875712A (en) A kind of traffic jam coordinates system and control method
Vanitha Survey on Identification of Potholes using Deep Learning Techniques
Wu et al. Detecting Street Signs in Cities Based on Object Recognition with Machine Leaning and GIS Spatial Analysis
Demiyanushko et al. Application of Artificial Intelligence Technology to Determine the Need for Repair Work of Road Elements

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant