CN104916131B - Freeway incident detection data cleaning method - Google Patents

Abstract

The invention belongs to the technical field of road traffic detection. A method comprises the steps that a condition with delay time tolerance ta is used to determine whether freeway incident detection data are missing; a condition screening method and a screening method based on statistics and traffic flow theory are used to determine whether the freeway incident detection data are abnormal; if non morning time data in the missing data and the abnormal data are all zero, a moving average method is used for repairing; and if occupancy and average speed in the abnormal data are high at the same time and partial traffic parameters are zero, an inverse proportion method is used for repairing. According to the invention, the method is applicable to freeway incident detection system practical application; the screening methods can dynamically adapt to different flow changes and meet a traffic flow mechanism; a repairing algorithm can retain partial real information of the abnormal data of the current period; a repairing result is closer to real data; and the method has the advantages of low computational complexity and small system overhead.

Description

Data cleaning method for expressway event detection

Technical Field

The invention belongs to the technical field of road traffic detection, and particularly relates to a data cleaning method for highway incident detection.

Background

In recent years, the problems of traffic congestion, traffic accidents and the like frequently occur on expressways, the operation efficiency of an expressway network is influenced, and with the development of intelligent traffic technologies, an expressway event detection system is being applied and researched successively, so that opportunities are brought to the management of expressway events.

The highway incident detection system relies on a large amount of data acquired from sensors, the reliability of the data directly influences the credibility and reliability of highway incident detection, but due to the limits of technical level and engineering conditions, the reliability of the data cannot meet the actual engineering requirements. Because the intelligent traffic system in China starts relatively late, the construction of supporting facilities (a traffic information acquisition system, a transmission system, hardware facilities and the like) is not perfect, and the reliability of traffic detection data uploaded to an expressway event detection system is also greatly insufficient. Engineering practices show that the detection effect of the highway incident detection system is not ideal due to various unreliable data factors.

Therefore, data cleansing work is a necessary work to improve the reliability of the highway event detection system. The data cleansing work has two main contents: firstly, screening data (including data loss, data abnormity and the like) which causes the system to be incapable of working normally; second, these abnormal data are repaired.

The existing data screening methods are mainly divided into two types of methods based on statistics and traffic flow theories. The data screening method based on statistics (such as a time series method, an exponential smoothing method and the like) has the advantages that the current data can be evaluated through the historical information of the data, and the historical data of the data in each period are different, so that the method has dynamic property, can adapt to different flow characteristic changes, and has the defect that the basic three-parameter relation of a traffic flow theory is not considered; a traffic data screening method based on a traffic flow theory (such as a traffic flow three-parameter relation method, a flow conservation method and the like) has the advantages that a traffic flow mechanism is considered, the standard for evaluating data is certain to meet the traffic flow mechanism, and the defect is lack of dynamic property.

The related solution method in the aspect of data restoration is mainly based on a statistical method, comprises a time sequence, a regression model and the like, and mainly utilizes historical data to predict and restore.

On the one hand, two types of data screening methods have advantages and disadvantages, and no better method for combining the advantages and avoiding the disadvantages exists; on the other hand, the data repairing method only repairs through historical data, and partial real information reserved by the current period data is ignored. Therefore, how to combine the advantages of the two types of data screening methods, avoid the defects, effectively grasp the real traffic flow information retained by the current period data in the repair stage, and have important significance for improving the data reliability of the highway incident detection system.

Disclosure of Invention

In view of the above, the present invention provides a data cleaning method for detecting an expressway event, so as to reduce the influence on the operation of an expressway event detection system caused by data loss and data anomaly, and improve the operation reliability of the system.

In order to achieve the purpose, the invention provides the following technical scheme:

the data cleaning method for highway incident detection comprises the following steps:

1) using a delay time tolerance t_aJudging whether the data detected by the highway incident is missing or not according to the condition; judging data of highway incident detection by adopting condition screening methodWhether the data are abnormal or not is judged, wherein the conditions comprise that the data of non-morning time are all 0, the speed or the occupancy rate is higher than a threshold value, and partial traffic parameters are 0;

2) the method comprises the steps of repairing missing data and abnormal data, adopting a moving average method to repair the data missing data and the abnormal data when the data in the non-early-morning time is all 0, and adopting an inverse proportion method to repair the abnormal data when the occupancy rate and the average speed are high and part of traffic parameters are 0.

Further, the step 1) specifically comprises the following steps:

101) setting tolerance t of data delay time_a；

102) Delay t_aTime reading current period data;

103) reading current period data;

104) if the current period has data, executing step 105), if the current period has no data, marking the current period as data missing, and ending the screening process of data missing;

105) judging whether the read data are all 0, if yes, skipping to execute step 106), and if not, skipping to execute step 107);

106) judging whether the data is early morning, if so, determining the data is normal data, if not, determining the data is abnormal data, finishing screening, marking the data which are not early morning to be all 0, and skipping to execute the step 2);

107) reading data n cycles before the current cycle and calculating data n cycles beforeThe value:

wherein q' (t) is shownShowing data flow values n periods before the current period, o '(t) showing occupancy values n periods before the current period, v' (t) showing average vehicle speed values n periods before the current period, and simultaneously arranging from small to largeA value;

108) calculating the index p_i：

p_i＝(i/n)*100

Wherein i is the sequence number of the first n periods of data from small to large;

109) if p is_i>25, of the ith dataThe value is the first quartile Q1, and if p_i>75, the ith dataThe value is the third quartile Q3;

110) calculating the quartering distance IQR, wherein the calculation formula of the IQR is Q3-Q1;

111): computingValue range ofWhereinAndare respectively asUpper value limit and lower value limit;

112) calculating the value range [ q (t) min, q (t) max ] of q (t)]，Wherein q (t) _ min is a value lower limit of flow, q (t) _ max is a value upper limit of flow, q (t) represents a current period data flow value, o (t) represents a current period occupancy value, and v (t) represents a current period average vehicle speed value;

113) and judging whether q (t) is less than q (t) min or greater than q (t) max, and if so, considering the data as abnormal data.

Further, the step 2) specifically comprises the following steps:

201) judging whether the current data belongs to the data missing condition, if not, entering 202), if so, adopting a weighted moving average method to carry out data repair and finishing the repair process;

202) judging whether the current data is the situation that the data is all 0 at non-early morning time, if not, directly finishing the repairing process, and if so, adopting a weighted moving average method to repair the data and finishing the repairing process;

203) judging whether the data is abnormal and needs to be repaired, if not, finishing the repairing process, and if so, executing the step 204);

204) calculating the beta value:

judging whether the flow is higher than a threshold value, if so, determining that beta is Q (t) -max, and if not, determining that beta is Q (t) -min-Q (t), wherein Q (t) is a current periodic flow value, and Q _ min and Q _ max are a flow value lower limit and a flow value upper limit in the data screening process respectively;

205) and (3) repairing data, wherein a repairing formula is as follows:

wherein,for the repaired traffic flow, average vehicle speed or occupancy, y (t-1) and y (t-2) are respectively the flow and average vehicle speed or occupancy of the previous cycle and the previous two cycles, 1/β is called inverse proportion coefficient, α is weighting coefficient, and generally 0.4-0.8.

Further, the formula for data recovery by the weighted moving average method is as follows:

whereinFor the repaired traffic flow, average vehicle speed or occupancy, y (t-1) and y (t-2) are respectively the flow, average vehicle speed or occupancy of the previous cycle and the previous two cycles, α is a weighting coefficient, and is taken as 0.4-0.8.

Compared with the prior art, the invention has the following advantages: the method is suitable for practical engineering application of an expressway event detection system, the screening method can dynamically adapt to different flow changes and meet the traffic flow mechanism, the repair algorithm can keep part of real information of abnormal data in the current period, the repair result is closer to the real data, the calculation amount is lower, and the system overhead is small.

Drawings

FIG. 1 is a schematic flow diagram illustrating data screening in a data cleansing method for highway incident detection;

fig. 2 shows a schematic flow chart of data recovery in the data cleaning method of highway event detection.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in further detail below.

The data cleaning method for detecting the expressway events comprises the following steps of:

1) using a delay time tolerance t_aJudging whether data detected by the highway incident is missing or not, wherein the data detected by the highway incident comprises traffic flow, average speed and occupancy; judging whether the data detected by the highway incident is abnormal or not by adopting a condition screening method; the conditions comprise that the non-morning time data is all 0, the speed or occupancy is higher than a threshold value, and part of traffic parameters are 0; since the highway incident detection system is a real-time detection system, the data transmission and detection system cannot achieve absolute real-time due to the existence of various factors (asynchronous clocks, transmission delay, calculation delay and the like of the systems), and therefore, for missing data screening, the tolerance t to the data delay time is determined firstly_aI.e. delay time at t_aThe range is acceptable, generally, the tolerance of the delay time is set according to the detection period T, and generally, the range is T/2 to 2 × T, which can be set by the user of the highway event detection system.

Referring to fig. 1, step 1) specifically includes the following steps:

101) setting tolerance t of data delay time_a；

102) Delay t_aTime reading current period data;

103) reading current period data;

104) if the current period has data, executing step 105), if the current period has no data, marking the current period as data missing, and ending the screening process of data missing;

105) judging whether the read data are all 0, if yes, skipping to execute step 106), and if not, skipping to execute step 107);

106) judging whether the data is early morning, if so, determining the data is normal data, if not, determining the data is abnormal data, finishing screening, marking the data which are not early morning to be all 0, and skipping to execute the step 2);

107) reading data n cycles before the current cycle and calculating data n cycles beforeThe value:

wherein q ' (t) represents the data flow value n periods before the current period, o ' (t) represents the occupancy value n periods before the current period, v ' (t) represents the average vehicle speed value n periods before the current period, and the values are arranged from small to largeA value;

108) calculating the index p_i：

p_i＝(i/n)*100

Wherein i is the sequence number of the first n periods of data from small to large;

109) if p is_i>25, of the ith dataThe value is the first quartile Q1, and if p_i>75, the ith dataThe value is the third quartile Q3;

110) calculating the quartering distance IQR, wherein the calculation formula of the IQR is Q3-Q1;

111): computingValue range ofWhereinAndare respectively asUpper value limit and lower value limit;

112) calculating the value range [ q (t) min, q (t) max ] of q (t)]，Wherein q (t) _ min is a value lower limit of flow, q (t) _ max is a value upper limit of flow, q (t) represents a current period data flow value, o (t) represents a current period occupancy value, and v (t) represents a current period average vehicle speed value;

113) and judging whether q (t) is less than q (t) min or greater than q (t) max, and if so, considering the data as abnormal data.

2) The method comprises the steps of repairing missing data and abnormal data, adopting a moving average method to repair the data missing data and the abnormal data when the data in the non-early-morning time is all 0, and adopting an inverse proportion method to repair the abnormal data when the occupancy rate and the average speed are high and part of traffic parameters are 0.

Referring to fig. 2, the step 2) specifically includes the following steps:

201) judging whether the current data belongs to the data missing condition, if not, entering 202), if so, adopting a weighted moving average method to carry out data repair and finishing the repair process;

202) judging whether the current data is the situation that the data is all 0 at non-early morning time, if not, directly finishing the repairing process, and if so, adopting a weighted moving average method to repair the data and finishing the repairing process;

203) judging whether the data is abnormal and needs to be repaired, if not, finishing the repairing process, and if so, executing the step 204);

204) calculating the beta value:

judging whether the flow is higher than a threshold value, if so, determining that beta is Q (t) -max, and if not, determining that beta is Q _ min-Q (t), wherein Q (t) is a current periodic flow value, and Q _ min and Q _ max are a flow value lower limit and a flow value upper limit in the data screening process respectively;

205) and (3) repairing data, wherein a repairing formula is as follows:

wherein,for the repaired traffic flow, average vehicle speed or occupancy, y (t-1) and y (t-2) are respectively the flow and average vehicle speed or occupancy of the previous cycle and the previous two cycles, 1/β is called inverse proportion coefficient, α is weighting coefficient, and generally 0.4-0.8.

The formula for data repair by the weighted moving average method is as follows:

whereinFor the repaired traffic flow, average vehicle speed or occupancy, y (t-1) and y (t-2) are respectively the flow, average vehicle speed or occupancy of the previous cycle and the previous two cycles, α is a weighting coefficient, and is taken as 0.4-0.8.

At present, the data restoration method is the most widely used data restoration method based on statistics, most typically a method of performing moving average by using a plurality of period data, and the restoration method mainly uses historical period data for restoration.

The existing moving average method mainly utilizes historical data to repair, but does not consider partial real information retained by current period data, so that in order to retain partial real information of current period abnormal data, an inverse proportion method is adopted to repair.

The physical significance of the inverse proportion restoration coefficient is that the farther the data deviates from the normal data value range, the more unreliable the data is, and the less information the data retains; when the data deviates from the normal data value range, the closer the data deviates from the normal data value range, the higher the credibility of the data is, the closer the data is to the true value.

The repairing method is characterized in that: in the abnormal data repairing process, the actual measurement data in the current period is adopted, and partial real information of the abnormal data is reserved. Although the current period data is abnormal, part of the information is real, if a traffic event occurs, the occupancy rate should be continuously high, at this time, if the data is repaired by only using the previous two periods, the data cannot keep the current traffic event condition, so that the alarm of the highway event detection system is released, or when the data is recovered to be normal and the traffic event continuously occurs, the alarm is repeated.

Finally, the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all of them should be covered in the claims of the present invention.

Claims (2)

1. The data cleaning method for highway incident detection is characterized by comprising the following steps: the method comprises the following steps:

1) using a delay time tolerance t_aJudging whether the data detected by the highway incident is missing or not according to the condition; judging whether the data detected by the highway incident are abnormal or not by adopting a condition screening method, wherein the conditions comprise that the data of non-early morning time is all 0, the speed or occupancy is higher than a threshold value and part of traffic parameters are 0; the method specifically comprises the following steps:

101) setting tolerance t of data delay time_a；

102) Delay t_aTime reading current period data;

103) reading current period data;

104) if the current period has data, executing step 105), if the current period has no data, marking the current period as data missing, and ending the screening process of data missing;

105) judging whether the read data are all 0, if yes, skipping to execute step 106), and if not, skipping to execute step 107);

106) judging whether the data is early morning, if so, determining the data is normal data, if not, determining the data is abnormal data, finishing screening, marking the data which are not early morning to be all 0, and skipping to execute the step 2);

107) reading data n cycles before the current cycle and calculating data n cycles beforeThe value:

wherein q ' (t) represents the data flow value n periods before the current period, o ' (t) represents the occupancy value n periods before the current period, v ' (t) represents the average vehicle speed value n periods before the current period, and the values are arranged from small to largeA value;

108) calculating the index p_i：

p_i＝(i/n)*100；

Wherein i is the sequence number of the first n periods of data from small to large;

109) if p is_i>25, of the ith dataThe value is the first quartile Q¹And if p is_i>75, the ith dataThe value is the third quartile Q³；

110) Calculating the quartering distance IQR, wherein the calculation formula of the IQR is Q3-Q1;

111): computingValue range ofWhereinAndare respectively asUpper value limit and lower value limit;

112) calculating the value range [ q (t) min, q (t) max ] of q (t)]，Wherein q (t) _ min is a value lower limit of flow, q (t) _ max is a value upper limit of flow, q (t) represents a current period data flow value, o (t) represents a current period occupancy value, and v (t) represents a current period average vehicle speed value;

113) judging whether q (t) is less than q (t) min or greater than q (t) max, if so, considering the data as abnormal data;

2) repairing missing data and abnormal data, adopting a moving average method to repair the data missing data and the abnormal data when the data in the non-early morning time is all 0, and adopting an inverse proportion method to repair the abnormal data when the occupancy rate and the average speed are simultaneously high and part of the traffic parameters are 0; the method specifically comprises the following steps:

201) judging whether the current data belongs to the data missing condition, if not, entering 202), if so, adopting a weighted moving average method to carry out data repair and finishing the repair process;

202) judging whether the current data is the situation that the data is all 0 at non-early morning time, if not, directly finishing the repairing process, and if so, adopting a weighted moving average method to repair the data and finishing the repairing process;

203) judging whether the data is abnormal and needs to be repaired, if not, finishing the repairing process, and if so, executing the step 204);

204) calculating the beta value:

judging whether the flow rate is higher than a threshold value, if so, determining that the flow rate is higher than the threshold value, if not, determining that the flow rate is higher than the threshold value, and if not, determining that the flow rate is higher than the threshold value, and if not, determining that the flow rate;

205) and (3) repairing data, wherein a repairing formula is as follows:

$\hat{y}\left(t\right)=\frac{1}{\mathrm{\β}}y\left(t\right)+\frac{\mathrm{\α}*\mathrm{\β}-\mathrm{\α}}{\mathrm{\β}}y(t-1)+\frac{\mathrm{\β}-\mathrm{\α}*\mathrm{\β}-1+\mathrm{\α}}{\mathrm{\β}}y(t-2);$

wherein,the traffic flow, the average vehicle speed or the occupancy after restoration is obtained, y (t) is the traffic flow, the average vehicle speed or the occupancy before restoration, y (t-1) and y (t-2) are respectively the flow, the average vehicle speed or the occupancy of the previous period and the previous two periods, 1/β is called inverse proportion coefficient, α is weighting coefficient, and 0.4-0.8 is taken.

2. The method of highway event detection data cleansing as recited in claim 1, further comprising: the formula for data repair by the weighted moving average method is as follows:

$\hat{y}\left(t\right)=\mathrm{\α}y(t-1)+(1-\mathrm{\α})y(t-2);$

whereinFor the repaired traffic flow, average vehicle speed or occupancy, y (t-1) and y (t-2) are respectively the flow, average vehicle speed or occupancy of the previous cycle and the previous two cycles, α is a weighting coefficient, and is taken as 0.4-0.8.