CN116778396A

CN116778396A - Road scanning data processing method and device, electronic equipment and storage medium

Info

Publication number: CN116778396A
Application number: CN202310634389.4A
Authority: CN
Inventors: 孙沉; 张磊; 邵栩; 张广杰; 莫若龙
Original assignee: Shenzhen Intellifusion Technologies Co Ltd
Current assignee: Shenzhen Intellifusion Technologies Co Ltd
Priority date: 2023-05-31
Filing date: 2023-05-31
Publication date: 2023-09-19

Abstract

The embodiment of the invention provides a road scanning data processing method, which is used for acquiring scanning information of a road to be processed; detecting whether a section with incomplete scanning result exists in the scanning information; if the section with incomplete scanning result exists in the scanning information, determining the section with incomplete scanning result as a section to be processed, and acquiring the historical repair data and the historical scanning result of the section to be processed; predicting the scanning result of the section to be processed based on the historical repair data, the historical scanning result and the scanning results corresponding to the upper section and the lower section of the section to be processed to obtain a prediction result of the section to be processed; and determining a scanning result of the road to be processed based on the prediction result of the section to be processed. The scanning result of the section to be processed is predicted based on the data such as the historical repair data, so that the prediction result of the section to be processed is obtained, the scanning result of the road to be processed is determined by using the prediction result, and then the scanning times are reduced, and the influence on road traffic is reduced.

Description

Road scanning data processing method and device, electronic equipment and storage medium

Technical Field

The present invention relates to the field of intelligent traffic technologies, and in particular, to a method and apparatus for processing road scan data, an electronic device, and a storage medium.

Background

With the popularization of vehicles and the increase of road paving mileage, more and more families can select vehicles such as self-driving vehicles or buses, so as to reduce the potential safety hazards of road traveling, the maintenance of the road becomes more and more important, and the diseases of the road need to be found in time and the found road diseases need to be maintained. In the existing road inspection method, a remote inspection vehicle or a remote inspection unmanned aerial vehicle is generally used for road inspection, but the existing remote inspection vehicle needs to occupy a lane when the road inspection is carried out, or the problem of scanning failure occurs at a place where some temporary shielding objects are shielded is encountered, in the existing inspection method, the method for solving the problem is generally to temporarily stop the remote inspection vehicle, wait for the temporary shielding objects to evacuate and then scan again, or carry out secondary inspection, but the expressway is provided with a speed limit, the remote inspection vehicle temporarily stops on the expressway, traffic resources are influenced, potential safety hazards are caused, the secondary scanning also needs to occupy more traffic resources, the limit of the unmanned aerial vehicle is also larger, the height needs to be kept for shooting, and the problem of the temporary shielding objects is more compared with that of the remote inspection vehicle.

Disclosure of Invention

The embodiment of the invention provides a road scanning data processing method, which aims to solve the problem that when a remote inspection vehicle or a remote inspection unmanned aerial vehicle is used for scanning inspection in the existing road inspection, the remote inspection vehicle needs to occupy a lane in the scanning process or a place shielded by temporary shielding objects is subjected to scanning failure, and the problem that traffic resources are greatly occupied when the remote inspection vehicle, the remote inspection unmanned aerial vehicle or secondary scanning is stopped temporarily. The method comprises the steps of predicting the scanning results of the section to be processed based on the historical repair data and the scanning results corresponding to the upper section and the lower section of the section to be processed of the historical scanning results, obtaining the prediction results of the section to be processed, determining the scanning results of the road to be processed by using the prediction results, further reducing the scanning times and reducing the influence on road traffic.

In a first aspect, an embodiment of the present invention provides a road scan data processing method, including:

acquiring scanning information of a road to be processed, wherein the scanning information comprises scanning results of all sections;

detecting whether a section with incomplete scanning results exists in the scanning information;

if a section with incomplete scanning results exists in the scanning information, determining the section with incomplete scanning results as a section to be processed, and acquiring historical repair data and historical scanning results of the section to be processed;

Predicting the scanning result of the section to be processed based on the historical repair data, the historical scanning result and the scanning results corresponding to the upper section and the lower section of the section to be processed to obtain a prediction result of the section to be processed;

and determining a scanning result of the road to be processed based on the prediction result of the section to be processed.

Optionally, the scanning result includes scanning an image frame, and the detecting whether a section with incomplete scanning result exists in the scanning information includes:

performing target detection on the scanned image frames of the scanning results corresponding to each section, and determining the blocked scanned image frames;

determining an occluded rate for each section based on the occluded scanned image frames;

and for one section, if the blocked rate is larger than a preset blocked rate threshold value, determining that the corresponding section is a section with incomplete scanning results.

Optionally, before the step of determining that the corresponding section is the section with incomplete scanning result if the blocked rate is greater than the preset blocked rate threshold for one section, the method further includes:

acquiring historical disease data of a target section, wherein the historical disease data comprises disease quantity, disease type, disease degree and disease position;

Determining probability distribution of each disease in the target segment based on the number of diseases, the disease type, the disease extent, and the disease location;

and determining a blocked rate threshold of the target section based on probability distribution of each disease in the target section.

Optionally, before predicting the scan result of the to-be-processed section based on the historical repair data, the historical scan result, and the scan results corresponding to the upper and lower sections of the to-be-processed section, to obtain the predicted result of the to-be-processed section, the method further includes:

determining the number of upper and lower sections of the section to be treated based on probability distribution of each disease in the section to be treated;

and obtaining scanning results corresponding to the upper section and the lower section of the section to be processed.

Optionally, the predicting the scan result of the to-be-processed section based on the historical repair data, the historical scan result, and the scan results corresponding to the upper and lower sections of the to-be-processed section, to obtain the prediction result of the to-be-processed section includes:

aligning the historical repair data, the historical scanning result and the scanning result of the section to be processed to obtain the data to be processed of the section to be processed in the time dimension;

Splicing the scanning result of the section to be processed with the scanning result corresponding to the upper section and the lower section of the section to be processed to obtain the data to be processed of the section to be processed in the space dimension;

and predicting a scanning result of the section to be processed based on the data to be processed of the section to be processed in the time dimension and the data to be processed of the section to be processed in the space dimension, so as to obtain a prediction result of the section to be processed.

Optionally, the predicting the scanning result of the section to be processed based on the data to be processed of the section to be processed in the time dimension and the data to be processed of the section to be processed in the space dimension to obtain the prediction result of the section to be processed includes:

performing feature extraction processing on the data to be processed of the section to be processed in the time dimension to obtain disease time sequence features of the section to be processed;

performing feature extraction processing on the data to be processed of the section to be processed in the space dimension to obtain disease space features of the section to be processed;

performing feature fusion on the disease time sequence features and the disease space features to obtain fusion features;

And predicting the scanning result of the section to be processed based on the fusion characteristic to obtain a prediction result of the area to be processed.

Optionally, the feature fusion is performed on the disease time sequence feature and the disease space feature to obtain a fusion feature, which includes:

performing linear transformation on the disease time sequence characteristic and the disease space characteristic to obtain disease time sequence characteristic and disease space characteristic with the same dimension;

and carrying out channel fusion on the disease time sequence characteristics and the disease space characteristics with the same dimension to obtain fusion characteristics.

In a second aspect, an embodiment of the present invention further provides a road scanning data processing apparatus, including:

the first acquisition module is used for acquiring scanning information of a road to be processed;

the first detection module is used for detecting whether a section with incomplete scanning results exists in the scanning information;

the first processing module is used for determining the section with incomplete scanning results as a section to be processed if the section with incomplete scanning results exists in the scanning information, and acquiring historical repair data and historical scanning results of the section to be processed;

The first prediction module is used for predicting the scanning result of the section to be processed based on the historical repair data, the historical scanning result and the scanning results corresponding to the upper section and the lower section of the section to be processed, so as to obtain a prediction result of the section to be processed;

and the first determining module is used for determining the scanning result of the road to be processed based on the prediction result of the section to be processed.

In a third aspect, an embodiment of the present invention provides an electronic device, including: the road scanning data processing method comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the steps in the road scanning data processing method provided by the embodiment of the invention are realized when the processor executes the computer program.

In a fourth aspect, an embodiment of the present invention provides a computer readable storage medium, where a computer program is stored, where the computer program when executed by a processor implements steps in a road scanning data processing method provided by the embodiment of the present invention.

In the embodiment of the invention, the scanning information of the road to be processed is obtained; detecting whether a section with incomplete scanning result exists in the scanning information; if the section with incomplete scanning result exists in the scanning information, determining the section with incomplete scanning result as a section to be processed, and acquiring the historical repair data and the historical scanning result of the section to be processed; predicting the scanning result of the section to be processed based on the historical repair data, the historical scanning result and the scanning results corresponding to the upper section and the lower section of the section to be processed to obtain a prediction result of the section to be processed; and determining a scanning result of the road to be processed based on the prediction result of the section to be processed. And predicting the section to be processed with incomplete scanning results in the time dimension and the space dimension by using the historical repair data, the historical scanning results and the scanning results corresponding to the upper section and the lower section to obtain a prediction result of the section to be processed, and determining the scanning result of the road to be processed by using the prediction result, thereby reducing the scanning times and reducing the influence on road traffic.

Drawings

In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flowchart of a road scan data processing method according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a road scanning data processing apparatus according to an embodiment of the present invention;

fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

As shown in fig. 1, fig. 1 is a flowchart of a road scanning data processing method according to an embodiment of the present invention. The road scanning data processing method comprises the following steps:

101. and acquiring scanning information of the road to be processed.

In the embodiment of the invention, the road scanning data processing method can be deployed on a server or a remote terminal, the server can be a server or a server cluster used by a road inspection platform, scanning information sent back to the road inspection platform through the remote terminal is processed and analyzed, the remote terminal can be electronic equipment used for processing and analyzing the scanning information on a remote inspection vehicle or a remote inspection unmanned aerial vehicle, and an image shooting device is arranged on the remote inspection vehicle or the remote inspection unmanned aerial vehicle, and the corresponding scanning information can be obtained by scanning and shooting the road surface of the road through the image shooting device. When the remote inspection vehicle or the remote inspection unmanned aerial vehicle runs, the image shooting device can perform scanning shooting at a certain frame rate to obtain continuous scanning images.

The road inspection platform can be an intranet platform built in a road inspection department through a special local area network, and the road inspection vehicle or the road inspection unmanned aerial vehicle can communicate with the road inspection platform through a data encryption protocol. The road inspection platform can be an electronic device with the functions of storing, processing and analyzing the scanning information, such as a server or a server cluster with the functions of storing, processing and analyzing the scanning information.

The road can be expressways or common national roads, provincial roads, urban roads and the like for providing traffic means to pass through, the road to be processed can be a road selected by a user through an interactive interface, and after the user selects a road to be scanned through the interactive interface, a remote inspection vehicle or a remote inspection unmanned aerial vehicle is put into the road selected by the user through the interactive interface to execute scanning work, so that scanning information corresponding to the road is obtained as scanning information of the road to be processed; or the road to be processed can be a road selected by the road inspection platform through analysis methods such as big data analysis, after the road inspection platform selects a road to be scanned through the big data analysis method, the selected result is displayed through an interactive interface, and a user puts a remote inspection vehicle or a remote inspection unmanned aerial vehicle on the road selected by the road inspection platform through the big data analysis method to obtain scanning information corresponding to the road as scanning information of the road to be processed; or the above-mentioned road to be treated can be the road that needs to carry out the inspection, and every road sets up the periodic inspection time, when reaching the periodic inspection time, confirm the road that corresponds as the road to be treated to by the user with long-range inspection car or long-range unmanned aerial vehicle of patrolling and examining put in the road that reaches the periodic inspection time, obtain the scanning information that this road corresponds as the scanning information of the road to be treated.

The scan information of the roads to be processed includes scan results of each section, the scan results may include a scan image sequence including continuous scan image frames, each road to be processed may be divided into a plurality of sections, and each section corresponds to one scan result, that is, each section corresponds to one scan image sequence. The scan result may further include a scan track, where the scan track corresponds to a scan image sequence, where one scan track corresponds to one scan image sequence, and where each scan image frame in the scan image sequence corresponds to one capturing time and capturing position, and also corresponds to one track point in the scan track.

The road to be treated can be divided equally into n sections, for example one section every 200 meters; the road to be treated can be divided into n sections according to the historical disease distribution of the road to be treated, and the length of each section is inversely related to the disease number in the section, namely, the smaller the disease number of one section is, the longer the length of the section is; or may be distinguished by topography, e.g. road dividing an uphill section into one section, a downhill section into one section, and a straight section into one section.

After the image shooting device collects the scanning information, the scanning image can be transmitted to a server or a remote terminal through a data transmission protocol for processing and analyzing the scanning image.

102. And detecting whether a section with incomplete scanning result exists in the scanning information.

In the embodiment of the invention, the scanning result comprises a scanning image sequence of a corresponding section, the scanning image sequence can be subjected to target detection through a pre-trained detection model, whether the scanning result of each section is complete or not is detected, if the scanning result of each section is complete, the section with incomplete scanning result is determined to be absent in the scanning information, and if the scanning result of one section is incomplete, the section with incomplete scanning result is determined to be present in the scanning information.

The detection model can be a deep convolutional neural network, and specifically, the detection model can be a model constructed based on the deep convolutional neural network such as ResNet, yolov, faster R-CNN, SSD, retinaNet and the like. The trained detection model can be obtained through supervised training of a large number of sample image sequences, the sample image sequences can be an image sequence with shielding scanning image frames and an image sequence without shielding scanning image frames, the labels corresponding to the image sequences with shielding scanning image frames are incomplete, the labels corresponding to the image sequences without shielding scanning image frames are complete, in the training process, the sample image sequences are input into the detection model to be trained, loss calculation is conducted between an output result of the detection model to be trained and the labels corresponding to the sample image sequences, a loss function is obtained, the minimum loss function is used as an optimization target, parameter adjustment is conducted on the detection model to be trained through a back propagation algorithm, the parameter adjustment process is iterated until the detection model to be trained converges at the minimum loss function or the iteration times reach the preset times, and training is stopped, so that the trained detection model is obtained. After the trained detection model is obtained, the scanning image sequences corresponding to all the sections are respectively input into the trained detection model, and for the scanning image sequence of one section, if the output of the trained detection model is complete, the scanning result of the section is complete, and if the output of the trained detection model is incomplete, the scanning result of the section is incomplete, and the section is the section with incomplete scanning result.

In a possible embodiment, the detection model is a frame-by-frame detection model, after the scan image sequence is input into the detection model, whether the blocked scan image frame exists in the scan image sequence is detected by the detection model, and if the blocked scan image frame exists, it can be determined that the scan result of the section is incomplete. In the training process of the frame-by-frame detection model, a sample image is provided with an occlusion image and a non-occlusion image, a label corresponding to the occlusion image is incomplete, a label corresponding to the non-occlusion image is complete, the sample image is input into the detection model to be trained, loss calculation is carried out between an output result of the detection model to be trained and the label corresponding to the sample image, a loss function is obtained, the minimum loss function is used as an optimization target, parameter adjustment is carried out on the detection model to be trained through a back propagation algorithm, the parameter adjustment process is iterated until the detection model to be trained converges at the minimum loss function, or the iteration times reach the preset times, training is stopped, and the trained detection model is obtained. After the trained detection model is obtained, the scanning image sequences corresponding to all the sections are sequentially input into the trained detection model according to frames, for one scanning image frame, if the output of the trained detection model is complete, the scanning image frame is not blocked, if the output results of all the scanning image frames in the scanning image sequence are complete, the scanning results of the corresponding sections are complete, the sections are sections with complete scanning results, and if the output of the trained detection model is incomplete, the scanning image frames are blocked, and then the scanning results of the corresponding sections are incomplete.

The scanning result further comprises scanning tracks, each scanning track comprises continuous track points, each track point corresponds to an attribute value, the attribute value can be 0,1 and a null value, 0 can indicate no disease, 1 can indicate disease, and the null value indicates that the detection result is unknown. When there is a null value in the scan trajectory, it can be determined that the scan result of the section is incomplete.

The incomplete section may be a blank section which is found to be directly missing by the detection model according to the comparison of the historical detection data of the road, or may be a blocked section which is detected by the detection model according to a preset blocked rate threshold and in which the blocked rate in the corresponding photographed image is greater than the preset blocked rate threshold.

After the road inspection platform receives the scanning information transmitted by the remote terminal, the scanning information is detected through a pre-trained detection model, and the detected result can be displayed through an interactive interface. In a possible embodiment, the detection model may extract a key frame in the scanned image sequence according to the scanned image sequence of each section, and perform target detection according to the key frame, so as to determine whether the scanning result of the section is complete, so that the model depth of the detection model may be reduced, and the content to be detected may be reduced.

103. If the section with incomplete scanning result exists in the scanning information, determining the section with incomplete scanning result as a section to be processed, and acquiring the historical repair data and the historical scanning result of the section to be processed.

In the embodiment of the invention, the historical repair data and the historical scanning result may be data information stored in a road inspection platform, the historical repair data refers to repair data before the current scanning result, the historical repair data may include historical crack repair data, historical bone stone repair data, historical pothole repair data, historical abrasion repair data and historical settlement repair data, and the historical scanning result may be historical scanning information scanned and shot by an image shooting device. The history scan result may include a history scan image sequence of the corresponding section, where the history scan image sequence includes continuous history scan image frames, and the history scan result refers to a scan result before the current scan result. Each history scanned image frame in the history scanned information comprises a corresponding shooting time and a corresponding shooting position.

The historical crack repair data may be data recorded on a road inspection platform after the road crack is repaired by corresponding repair measures, the historical bone stone repair data may be data recorded on the road inspection platform after the road bone stone is repaired by corresponding repair measures, the historical pit repair data may be data recorded on the road inspection platform after the road abrasion is repaired by corresponding traffic volume, high-speed road or wind sand, the historical settlement repair data may be data recorded on the road inspection platform after the road abrasion is repaired by corresponding repair measures, the road inspection platform is subsided by corresponding road inspection measures.

In one possible embodiment, when the detection model detects that a section with incomplete scanning result exists in the scanning information, which indicates that the section has a condition that the road surface is blocked, the section with incomplete scanning result may be extracted for marking, and the incomplete section is determined as the section to be processed. And after the road inspection platform determines the section to be processed, extracting historical repair data and a historical scanning result of the section to be processed.

104. And predicting the scanning result of the section to be processed based on the historical repair data, the historical scanning result and the scanning results corresponding to the upper section and the lower section of the section to be processed, so as to obtain the prediction result of the section to be processed.

In the embodiment of the present invention, the upper and lower sections of the section to be processed may be understood as an upstream section and a downstream section of the section to be processed, wherein the upstream section is a section that is scanned by a scanning vehicle before the section to be processed and is adjacent to the upstream section, and the downstream section is a section that is scanned by a scanning vehicle after the section to be processed and is adjacent to the downstream section. After determining the section to be processed, determining the upper section and the lower section of the section to be processed at the same time, and extracting corresponding scanning results according to the upper section and the lower section of the section to be processed, thereby obtaining the scanning results corresponding to the upper section and the lower section of the section to be processed. Each road scanning information corresponds to one road, and each section scanning result corresponds to one section of one road. The upper and lower sections of the section to be treated may be specifically divided into an upper section of the section to be treated and a lower section of the section to be treated, for example, the north-facing section adjacent to the geographic position of the section to be treated is defined as an upper section of the section to be treated, and the south-facing section adjacent to the geographic position of the section to be treated is defined as a lower section of the section to be treated. The upper section or the lower section of the section to be treated may be one section or a plurality of sections, and is not limited herein.

And after the road inspection platform extracts time-level information such as historical repair data and historical scanning results of the section to be processed, and then extracts space-level information such as the upper section of the section to be processed and the lower section of the section to be processed at the current moment, predicting the scanning information of the section to be processed according to the time-dimension information and the space-dimension information by using a trained prediction model, and finally obtaining a prediction result. The prediction model may be a deep convolutional neural network, and specifically, the prediction model may be a model constructed based on a deep convolutional neural network such as ResNet, yolov, faster R-CNN, SSD, retinaNet, and the like.

Specifically, a data set of a training prediction model can be constructed by collecting sample data, the sample data can include sample historical repair data, sample historical scanning results and scanning results corresponding to upper and lower sections of a sample, each sample data corresponds to one real scanning result (serving as a label), the prediction results are obtained by inputting the sample historical repair data, the sample historical scanning results and the scanning results corresponding to the upper and lower sections of the sample into a prediction model to be trained, a loss function between the prediction results and the corresponding real scanning results is calculated, the minimum loss function is used as an optimization target, parameter adjustment is performed on the prediction model to be trained through a back propagation algorithm, and the parameter adjustment process is iterated until the prediction model to be trained converges at the minimum of the loss function or the iteration times reach the preset times, so that the trained prediction model is obtained. After the trained prediction model is obtained, the historical repair data corresponding to the section to be processed, the historical scanning result and the scanning results corresponding to the upper section and the lower section can be input into the trained prediction model, and the prediction result of the section to be processed is output.

105. And determining a scanning result of the road to be processed based on the prediction result of the section to be processed.

In the embodiment of the present invention, the prediction result of the section to be processed may be a prediction result of one section, or may be prediction results of a plurality of sections, and specifically, the prediction results of how many sections need to be determined according to how many sections with incomplete scanning results in the road to be processed. And after all the sections to be processed are predicted through a prediction model, obtaining prediction results of all the sections to be processed, wherein the prediction results can be a prediction image sequence and a prediction scanning track, each prediction scanning track comprises continuous prediction track points, each prediction track point corresponds to a prediction attribute value, the prediction attribute value can be 0,1 and 0, the prediction attribute value can be used for indicating no disease, the 1 can be used for indicating the existence of the disease, and the null value can be used for indicating that the detection result is unknown. When there is a null value in the scan trajectory, it can be determined that the scan result of the section is incomplete. If all the sections to be processed in the road to be processed are predicted successfully, all the predicted results can be spliced with the scanning results of the sections not to be processed in the road to be processed, and finally the scanning results of the road to be processed are obtained.

Optionally, the scan result includes a scan image frame, and in the step of detecting whether there is a section with incomplete scan result in the scan information, target detection may be performed on the scan image frame of the scan result corresponding to each section, to determine a blocked scan image frame; determining an occluded rate of each section based on the occluded scanned image frames; for one section, if the blocked rate is larger than a preset blocked rate threshold value, determining that the corresponding section is a section with incomplete scanning results.

In the embodiment of the present invention, the scanning result includes a scanning image sequence, and the scanning image sequence is formed by arranging successive scanning image frames according to a shooting time sequence.

The target detection may be a detection performed by a pre-trained target detection model, and the target detection may be performed on a shielding object, which may be a shielding object for a road surface, and the shielding object may be a vehicle, garbage, water accumulation, or the like.

The target detection model can be a model constructed based on a deep convolutional neural network such as ResNet, yolov, faster R-CNN, SSD, retinaNet and the like. And sequentially inputting the scanned image frames in the scanned image sequence into a target detection model to obtain a target detection result, and outputting the target detection result through a shielding object detection frame (x, y, w, h and u) if the scanned image frames are shielded scanned image frames, wherein (x, y) represents the central position of the shielding object detection frame, w represents the width of the shielding object detection frame, h represents the height of the shielding object detection frame, u represents the confidence of shielding objects in the shielding object detection frame, and the higher the confidence is, and the higher the credibility is of shielding objects in the shielding object detection frame.

For a section, the corresponding shielding rate can be determined according to each shielded scanned image frame, specifically, the shielded rate can be calculated by the following formula:

wherein s is the blocked rate of the section, K is the number of blocked scanned image frames, and N is the sequence of scanned imagesThe number of scanned image frames in a column, the above (x ₀ ，y ₀ ) For the center coordinates of the picture, the above (x _1,j ，y _1,j ) For the center position of the shielding object detection frame in the j-th shielded scanning image frame, A _i ×B _i For the image resolution of the scanned image frames, w represents the width of the occlusion detection frame in the j-th occluded scanned image frame, h represents the height of the occlusion detection frame in the j-th occluded scanned image frame, and u represents the confidence that the occlusion in the occlusion detection frame in the j-th occluded scanned image frame is an occlusion.

The above-mentioned blocked rate threshold may be set manually or determined according to the historical disease data of each section, and different sections have different historical disease data, so that different sections may correspond to different blocked rate thresholds.

In one possible embodiment, the blocked scanned image frame may be a blocked scanned image frame that occurs due to a blocking object such as another vehicle or a temporary obstacle on the road when the remote inspection vehicle performs the inspection scanning, the blocking rate may be a percentage of the blocked scanned image frame that is blocked by a specific obstacle, and the blocked rate threshold may be determined according to the historical disease data of the target area.

In one possible embodiment, after the scan information is input to the detection model, the detection model extracts a scan image key frame from the scan result corresponding to each section by using the scan information, performs object detection on the scan image key frame, determines the blocked rate of each section, and marks the section corresponding to the scan image key frame with the blocked rate greater than the preset blocked rate threshold value in the scan result as a section with incomplete scan result.

Optionally, before determining that the corresponding section is the section with incomplete scanning result if the blocked rate is greater than the preset blocked rate threshold value for one section, historical disease data of the target section can be obtained, wherein the historical disease data comprises the disease number, the disease type, the disease degree and the disease position; determining probability distribution of each disease in the target section based on the number of diseases, the type of the diseases, the degree of the diseases and the positions of the diseases; and determining the shielding rate threshold value of the target section based on the probability distribution of each disease in the target section.

In the embodiment of the present invention, the target zone is a zone with incomplete disease results, the historical disease data may be historical disease data of the target zone stored in the road inspection platform, the disease number may be the number of historical diseases occurring in the target zone, the disease types may include a crack disease type, a bone disease type, a pothole disease type, a wear disease type, and a settlement disease type, the disease degree may be formulated according to a potential safety hazard caused by road traffic, for example, may be formulated as a slight disease, a moderate disease, and a severe disease, specifically, a disease which does not affect normal vehicle traffic in a short period may be defined as a slight disease, a disease which has affected normal vehicle traffic in the target zone may be defined as a moderate disease, a disease which severely affects normal vehicle traffic in the target zone may be defined as a severe disease, and the disease position may be a position corresponding to longitude and latitude in reality according to the scanning information.

In one possible embodiment, when it is required to detect whether the occlusion rate of the target segment is greater than the occlusion rate threshold, historical disease data stored in the road inspection platform is acquired, where the historical disease data is historical disease data of the target segment, probability distribution of each disease in the target segment is determined based on the extracted historical disease data, and the occlusion rate threshold of the target segment is determined based on the probability distribution of each disease in the target segment.

It should be noted that different sections have different historical disease data, different sections can correspond to different blocked rate thresholds, after the number of diseases, the type of diseases, the degree of diseases and the positions of diseases are obtained, the historical disease data can be clustered according to the type of diseases and the positions of diseases to obtain clustering results, each clustering result is a set of disease types in a certain position range, and probability distribution of each disease can be determined based on the clustering results.

Specifically, the probability distribution of the disease can be expressed by the following formula:

wherein r represents a probability distribution of a certain disease type, Q represents the number of clustering results corresponding to a certain disease type, P represents the number of diseases in one clustering result, and (x _2,i ，y _2,i ) The cluster center coordinates (x) of the ith cluster result _3,i,j ，y _2,i,j ) The disease position of the jth disease in the ith clustering result is p _i,j The disease degree of the jth disease in the ith clustering result. The probability distribution r can be understood as a probability distribution that a disease falls within (x _2,i ，y _2,i ) Probability of the vicinity.

After the probability distribution of each disease is obtained, the average value of the probability distribution of each disease may be determined as the blocked rate threshold of the target section, or the median of the probability distribution of each disease may be determined as the blocked rate threshold of the target section, or the sum of the probability distributions of each disease may be determined as the blocked rate threshold of the target section.

Optionally, in the step of predicting the scanning result of the section to be processed based on the history repair data, the history scanning result and the scanning results corresponding to the upper section and the lower section of the section to be processed to obtain the prediction result of the section to be processed, the number of the upper section and the lower section of the section to be processed can be determined based on the probability distribution of each disease in the section to be processed; and obtaining scanning results corresponding to the upper section and the lower section of the section to be processed.

In the embodiment of the invention, when the scanning result of the section to be processed is required to be predicted, the number of the upper sections and the lower sections of the section to be processed is also required to be determined based on the probability distribution of each disease in the section to be processed before. If the probability distribution of each disease in the section to be treated is higher, the number of the upper and lower sections of the section to be treated is required to be correspondingly increased, the upper limit of the number is not limited, and the number and the position of the sections of the road to be treated are determined specifically. If the probability distribution of each disease in the section to be treated is low, the number of the upper and lower sections of the section to be treated is correspondingly reduced to 1 at the minimum, namely, each section to be treated at least acquires the scanning result of one upstream section and the scanning result of one downstream section. After the number of the upper and lower sections of the section to be processed is determined, the scanning result corresponding to the upper and lower sections of the section to be processed can be obtained according to the number of the upper and lower sections of the section to be processed.

Optionally, in the step of predicting the scanning result of the section to be processed based on the historical repair data, the historical scanning result and the scanning results corresponding to the upper section and the lower section of the section to be processed to obtain the prediction result of the section to be processed, the historical repair data, the historical scanning result and the scanning result of the section to be processed may be aligned to obtain the data to be processed of the section to be processed in the time dimension; splicing the scanning result of the section to be processed with the scanning result corresponding to the upper section and the lower section of the section to be processed to obtain the data to be processed of the section to be processed in the space dimension; and predicting the scanning result of the section to be processed based on the data to be processed of the section to be processed in the time dimension and the data to be processed of the section to be processed in the space dimension, so as to obtain the prediction result of the section to be processed.

In the embodiment of the invention, when the scanning result of the section to be processed is required to be predicted, the historical repair data and the historical scanning result can be aligned with the scanning result of the section to be processed, so as to obtain the data to be processed of the section to be processed in the time dimension. Specifically, the historical repair data, the historical scanning result and the scanning result of the section to be processed can be sequenced according to time sequence, and the positions are aligned to obtain the data to be processed of the section to be processed in the time dimension.

And splicing the scanning result of the section to be processed with the scanning result corresponding to the upper section and the lower section of the section to be processed to obtain the data to be processed of the section to be processed in the space dimension, and predicting the scanning result of the section to be processed based on the data to be processed of the section to be processed in the time dimension and the data to be processed of the section to be processed in the space dimension to obtain the prediction result of the section to be processed.

And predicting the data to be processed of the section to be processed in the time dimension and the data to be processed of the section to be processed in the space dimension through a prediction model to obtain a prediction result of the section to be processed. The input of the prediction model is the data to be processed of the section to be processed in the time dimension and the data to be processed of the section to be processed in the space dimension.

Specifically, sample data pairs may be collected, each sample data pair includes sample data of one sample section in a time dimension and sample data of one sample section in a space dimension, each sample data pair may correspond to a real scan result (as a label) of the sample section, the sample data of the sample section in the time dimension and the sample data of the sample section in the space dimension are respectively input into two input modules of the model to be trained, a prediction result of the sample data pair is obtained, a loss function between the prediction result and the corresponding real scan result is calculated, the loss function is used as an optimization target to minimize, parameter adjustment is performed on the model to be trained through a back propagation algorithm, the parameter adjustment process is iterated until the model to be trained converges at a minimum of the loss function or the iteration number reaches a preset number, and the trained prediction model is obtained. After the trained prediction model is obtained, the data to be processed of the section to be processed in the time dimension and the data to be processed of the section to be processed in the space dimension can be input into the trained prediction model, and a prediction result of the section to be processed is output.

In one possible embodiment, in the time dimension, if the historical scan result of the to-be-processed section shows that a hollow with a moderate disease degree appears before five years, the historical repair data shows that a hollow with a moderate disease degree appears again at the same position of the to-be-processed section before three years, the historical repair data shows that the repair is performed again before four years and eleven months, the prediction result can predict the to-be-processed section according to the alignment data in the time dimension, the prediction result is that the to-be-processed section has a hollow with a disease degree identical to the geographic position before two years and eleven months, or the scan result of the to-be-processed section corresponding to the upper and lower sections of the to-be-processed section is spliced in the space dimension, if a crack appears in the upper section of the to-be-processed section, the crack is located in the upper section of the to-be-processed section adjacent to the to-be-processed section in the real world, the crack is predicted to be connected to the predicted section in the space dimension, and the predicted section is not to be-processed section, and the predicted result is the crack appears in the space dimension.

Optionally, in the step of predicting the scanning result of the section to be processed based on the data to be processed of the section to be processed in the time dimension and the data to be processed of the section to be processed in the space dimension to obtain the prediction result of the section to be processed, feature extraction processing can be performed on the data to be processed of the section to be processed in the time dimension to obtain the disease time sequence feature of the section to be processed; performing feature extraction processing on the data to be processed of the section to be processed in the space dimension to obtain disease space features of the section to be processed; carrying out feature fusion on the disease time sequence features and the disease space features to obtain fusion features; and predicting the scanning result of the section to be processed based on the fusion characteristic to obtain a prediction result of the area to be processed.

In the embodiment of the present invention, the prediction model may include two input modules, a feature fusion module, and an output module. The device comprises an input module, an output module, a feature fusion module and a prediction result obtaining module, wherein the input module is used for inputting data to be processed of a section to be processed in a time dimension and extracting corresponding disease time sequence features, the input module is used for inputting the data to be processed of the section to be processed in the space dimension and extracting corresponding disease space features, the feature fusion module is used for carrying out feature fusion on the disease time sequence features and the disease space features after the disease time sequence features and the disease space features are extracted, the fusion features are obtained, and the output module carries out linear regression on the fusion features and outputs the prediction result obtaining the section to be processed.

The feature extraction can utilize convolutional neural networks to extract disease time sequence features and disease space features, or can use a feature extraction method of an attention mechanism to extract disease time sequence features and disease space features. The fusion characteristic can be that the disease time sequence characteristic and the disease space characteristic with different scales are subjected to pooling operation by a pyramid pooling or maximum pooling method, and the pooled results are spliced together.

Optionally, feature fusion is performed on the disease time sequence feature and the disease space feature to obtain a fusion feature, including: performing linear transformation on the disease time sequence characteristics and the disease space characteristics to obtain disease time sequence characteristics and disease space characteristics with the same dimension; and carrying out channel fusion on the disease time sequence characteristics and the disease space characteristics with the same dimension to obtain fusion characteristics.

In the embodiment of the invention, the linear transformation can be performed by convolution, normalization, PCA (principal component analysis) dimension reduction and other methods so as to achieve the purpose of identical disease time sequence characteristic and disease space characteristic dimension. The channel fusion can be to splice disease time sequence characteristics and disease space characteristics with the same dimension as two channels to obtain a double-channel fusion characteristic. The disease time sequence features and the disease space features with the same dimension can be used as two channels to be overlapped, so that the single-channel fusion features can be obtained.

It should be noted that, the road scanning data processing method provided by the embodiment of the invention can be applied to devices such as shooting devices, smart phones, computers, servers and the like capable of processing road scanning data.

As shown in fig. 2, an embodiment of the present invention provides a road scanning data processing apparatus, including:

a first obtaining module 201, configured to obtain scan information of a road to be processed;

a first detection module 202, configured to detect whether a section with incomplete scanning result exists in the scanning information;

the first processing module 203 is configured to determine, if a section with incomplete scan result exists in the scan information, the section with incomplete scan result as a section to be processed, and obtain historical repair data and a historical scan result of the section to be processed;

a first prediction module 204, configured to predict a scan result of the to-be-processed section based on the historical repair data, the historical scan result, and scan results corresponding to upper and lower sections of the to-be-processed section, to obtain a predicted result of the to-be-processed section;

a first determining module 205 is configured to determine a scanning result of the road to be processed based on the prediction result of the section to be processed.

Optionally, the first detection module 202 includes:

the second detection sub-module is used for carrying out target detection on the scanned image frames of the scanning results corresponding to each section and determining the blocked scanned image frames;

a second determination sub-module for determining the blocked rate of each section based on the blocked scanned image frames;

and a third determining submodule, for one section, if the blocked rate is larger than a preset blocked rate threshold value, determining that the corresponding section is a section with incomplete scanning result.

Optionally, the first detection module 202 further includes:

the second acquisition submodule is used for acquiring historical disease data of the target section, wherein the historical disease data comprises the number of diseases, the type of the diseases, the degree of the diseases and the positions of the diseases;

a fourth determination submodule for determining probability distribution of each disease in the target segment based on the number of diseases, the disease type, the disease degree, and the disease position;

and a fifth determination submodule, configured to determine an occlusion rate threshold value of the target segment based on probability distribution of each disease in the target segment.

Optionally, the first prediction module 204 includes:

A fifth determining submodule, configured to determine the number of upper and lower sections of the section to be processed based on probability distribution of each disease in the section to be processed;

and the third acquisition sub-module is used for acquiring scanning results corresponding to the upper section and the lower section of the section to be processed.

Optionally, the first prediction module 204 further includes:

the second processing sub-module is used for aligning the historical repair data, the historical scanning result and the scanning result of the section to be processed to obtain the data to be processed of the section to be processed in the time dimension;

the third processing sub-module is used for splicing the scanning result of the section to be processed with the scanning result corresponding to the upper section and the lower section of the section to be processed to obtain the data to be processed of the section to be processed in the space dimension;

and the second prediction submodule is used for predicting the scanning result of the section to be processed based on the data to be processed of the section to be processed in the time dimension and the data to be processed of the section to be processed in the space dimension to obtain a prediction result of the section to be processed.

Optionally, the first prediction module 204 further includes:

The first extraction submodule is used for carrying out feature extraction processing on the data to be processed of the section to be processed in the time dimension to obtain disease time sequence features of the section to be processed;

the second extraction submodule is used for carrying out feature extraction processing on the data to be processed of the section to be processed in the space dimension to obtain disease space features of the section to be processed;

the first fusion submodule is used for carrying out feature fusion on the disease time sequence features and the disease space features to obtain fusion features;

and the third prediction submodule is used for predicting the scanning result of the section to be processed based on the fusion characteristic to obtain a prediction result of the area to be processed.

Optionally, the first fusion sub-module includes:

the first transformation unit is used for carrying out linear transformation on the disease time sequence characteristics and the disease space characteristics so as to obtain disease time sequence characteristics and disease space characteristics with the same dimension;

and the first fusion unit is used for carrying out channel fusion on the disease time sequence characteristics and the disease space characteristics with the same dimension to obtain fusion characteristics.

Optionally, the apparatus further includes:

a sixth determining module, configured to determine a first data set and a first model to be trained, where the data set includes a detection tag scan image sequence corresponding to a detection sample scan video set and each detection sample scan video set includes at least two detection sample scan videos, and a detection scan image quality of the detection tag scan image sequence is higher than a scan image quality of any one detection sample scan video in the corresponding detection sample scan video set, and the first model to be trained is a model required by the first detecting module 202;

And the first training module is used for carrying out the generated type countermeasure training on the first model to be trained based on the first data set to obtain a trained detection model.

A seventh determining module, configured to determine a second data set and a second model to be trained, where the data set includes a prediction tag scan image sequence corresponding to a prediction sample scan video set and each prediction sample scan video set includes at least two prediction sample scan videos, and a prediction scan image quality of the prediction tag scan image sequence is higher than a scan image quality of any one of the prediction sample scan videos in the corresponding prediction sample scan video set, and the second model to be trained is a model required by the first predicting module 204;

and the second training module is used for carrying out the generated type countermeasure training on the second model to be trained based on the second data set to obtain a trained prediction model.

It should be noted that the road scanning data processing device provided in the embodiment of the present invention may be applied to devices such as a photographing device, a smart phone, a computer, and a server, which may perform road scanning data processing.

The road scanning data processing device provided by the embodiment of the invention can realize each process realized by the road scanning data processing method in the method embodiment, and can achieve the same beneficial effects. In order to avoid repetition, a description thereof is omitted.

Referring to fig. 3, fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, as shown in fig. 3, including: a memory 302, a processor 301 and a computer program stored on the memory 302 and executable on the processor 301 for a road scanning data processing method, wherein:

the processor 301 is configured to call a computer program stored in the memory 302, and perform the following steps:

Optionally, the scan result executed by the processor 301 includes a scanned image frame, and the detecting whether a section with incomplete scan result exists in the scan information includes:

Optionally, before the step of determining that the corresponding section is a section with incomplete scan result if the occlusion rate is greater than a preset occlusion rate threshold for one section, the method executed by the processor 301 further includes:

Optionally, before predicting the scan result of the to-be-processed section based on the historical repair data, the historical scan result, and the scan results corresponding to the upper and lower sections of the to-be-processed section, the method executed by the processor 301 further includes:

Optionally, the predicting the scan result of the to-be-processed section based on the historical repair data, the historical scan result, and the scan result corresponding to the upper and lower sections of the to-be-processed section performed by the processor 301, to obtain a predicted result of the to-be-processed section includes:

Optionally, the predicting, by the processor 301, the scan result of the to-be-processed section based on the to-be-processed data of the to-be-processed section in the time dimension and the to-be-processed data of the to-be-processed section in the space dimension, to obtain a predicted result of the to-be-processed section includes:

Optionally, the feature fusion of the disease timing feature and the disease spatial feature performed by the processor 301, to obtain a fused feature includes:

It should be noted that, the electronic device provided by the embodiment of the invention can be applied to devices such as a smart phone, a computer, a server and the like which can perform the road scanning data processing method.

The electronic equipment provided by the embodiment of the invention can realize each process realized by the road scanning data processing method in the embodiment of the method, and can achieve the same beneficial effects. In order to avoid repetition, a description thereof is omitted.

The embodiment of the invention also provides a computer readable storage medium, on which a computer program is stored, which when executed by a processor, implements each process of the road scanning data processing method or the application-side road scanning data processing method provided by the embodiment of the invention, and can achieve the same technical effects, so that repetition is avoided, and no further description is given here.

Those skilled in the art will appreciate that the processes implementing all or part of the methods of the above embodiments may be implemented by a computer program for instructing relevant hardware, and the program may be stored in a computer readable storage medium, and the program may include the processes of the embodiments of the methods as above when executed. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM) or the like.

The foregoing disclosure is illustrative of the present invention and is not to be construed as limiting the scope of the invention, which is defined by the appended claims.

Claims

1. A road scan data processing method, characterized in that the method comprises the steps of:

2. The road scan data processing method of claim 1, wherein the scan result includes a scanned image frame, and the detecting whether there is a section of incomplete scan result in the scan information includes:

3. The method for processing road scan data according to claim 2, wherein before the step of determining that the corresponding section is a section whose scan result is incomplete if the blocked rate is greater than a preset blocked rate threshold for the one section, the method further comprises:

4. The road scan data processing method as set forth in claim 3, wherein, before said predicting the scan result of the section to be processed based on the history restoration data, the history scan result, and the scan results corresponding to the upper and lower sections of the section to be processed, the method further comprises:

5. The method for processing road scan data according to any one of claims 1 to 4, wherein predicting the scan result of the section to be processed based on the history restoration data, the history scan result, and the scan results corresponding to the upper and lower sections of the section to be processed, to obtain the prediction result of the section to be processed, comprises:

6. The method for processing road scan data according to claim 5, wherein predicting the scan result of the section to be processed based on the data to be processed of the section to be processed in the time dimension and the data to be processed of the section to be processed in the space dimension to obtain the prediction result of the section to be processed comprises:

7. The method of claim 6, wherein the feature fusion of the disease timing feature and the disease spatial feature to obtain a fused feature comprises:

8. A road scanning data processing apparatus, characterized in that the road scanning data processing apparatus comprises:

the acquisition module is used for acquiring the scanning information of the road to be processed;

the detection module is used for detecting whether a section with incomplete scanning results exists in the scanning information;

the processing module is used for determining the section with incomplete scanning result as a section to be processed if the section with incomplete scanning result exists in the scanning information, and acquiring the history repair data and the history scanning result of the section to be processed;

the prediction module is used for predicting the scanning result of the section to be processed based on the historical repair data, the historical scanning result and the scanning result corresponding to the upper section and the lower section of the section to be processed, so as to obtain a prediction result of the section to be processed;

and the determining module is used for determining the scanning result of the road to be processed based on the prediction result of the section to be processed.

9. An electronic device, comprising: memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps in the road scan data processing method according to any one of claims 1 to 7 when the computer program is executed.

10. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon a computer program which, when executed by a processor, implements the steps in the road scanning data processing method according to any one of claims 1 to 7.