CN100535955C - Method for recognizing outlier traffic data - Google Patents

Abstract

The present invention provides a method of identifying outlier traffic data, characterized in that the method firstly collects the traffic data, calculating data average local outlier factor, then judging the outlier data by one of the following two norms: the maximal m data of the average local outlier factor is outlier data, or the average local outlier factor more than the preset threshold is outlier data, finally the identified outlier factor is corrected by deleting or adopting filter method, or analyzing the concealed message contained in the outlier data. The method effectivily detect outlier data of boundary and interior, the effect precedes the outlier detecting method based on the statistics.

Description

A kind of method of discerning outlier traffic data

Technical field

The present invention proposes a kind of method of discerning outlier traffic data, relates to the quality control of the traffic data that intelligent transportation system gathers, and belongs to intelligent information processing technology field in the intelligent transportation system.

Background technology

Traffic data occupies an important position in intelligent transportation system, one of core technology of intelligent transportation system (ITS) is the real-time estimation and the forecasting techniques of traffic parameter, because sampling distortion, measuring error, burst traffic events and other influence factor that may exist, the traffic data of gathering is concentrated the sample that exists the general behavior of not following data model usually, and these abnormity point are the data that peel off.When the traffic data of gathering was used for modeling, these abnormity point did not possess representativeness, effectively modeling and descriptive system.For accuracy and the reliability that improves dynamic information, guarantee the result of use of traffic model, at first need abnormal data is discerned and handled accordingly.

At present, mainly discern the data that peel off based on statistical method in the traffic engineering field, this method is calculated simple, but its application need is known the distribution of data in advance, this is often difficult, and real data does not often meet the mathematical distribution of any perfect condition yet.In addition, based on the statistics the detection algorithm that peels off only be suitable for excavating univariate numeric type data mostly, to higher-dimension, periodic data, grouped data then difficulty discern, this has limited its application.

In order to overcome the defective of said method, the present invention proposes to use the unusual traffic data of outlier data digging algorithm identified based on density.

Summary of the invention

Technical matters: unusual traffic data can make the main points of institute's established model thicken, the essence that can not reflect real system, the invention provides a kind of method of the unusual traffic data of identification based on density, this method can effectively detect border and the inner data that peel off, and its effect is better than the detection method that peels off based on statistics.

Technical scheme: the method for identification outlier traffic data of the present invention is the average part of the computational data factor that peels off at first, then use one of following two criterions to judge outlier: average part m the highest data of the factor that peel off are outlier, or on average the part factor that peels off is an outlier greater than the data of given threshold value.

The average part of the described computational data factor method that peels off is, under a certain natural number k value, the k-part of calculating each data factor that peels off, with certain step-size change k value, the k-part of each data of the double counting factor that peels off, then by on average obtaining the average part of each data factor that peels off, its computing method are:

$\mathrm{lof}\left(p\right)=\frac{\underset{k}{\mathrm{\Σ}}{\mathrm{lof}}_k\left(p\right)}{\frac{k_2-k_1}{s}+1}--1$

Wherein, k ₁And k ₂Be respectively the bound of k, k ₁Be to be not less than 10 natural number, s is a step-length, lof _k(p) for appointing-the k-part of the data p factor that peels off.

The k-part of each data factor computing method that peel off are, the k-part of all data can reach the ratio that density mean value and the k-part of p can reach density in the k-neighborhood of p, that is:

${\mathrm{lof}}_k\left(p\right)=\frac{\underset{o\∈N_k\left(p\right)}{\mathrm{\Σ}}\frac{{\mathrm{lrd}}_k\left(o\right)}{{\mathrm{lrd}}_k}}{\left|N_k\left(p\right)\right|}--2$

Wherein, k is a natural number, N _k(p) be the k-neighborhood of data p, | N _k(p) | be the element number that this neighborhood contains; The k-neighborhood of p is made up of the data that the distance between all and the p is not more than the k-distance of p, and the k-distance of p is data p and the distance between its k nearest data; Lrd _k(p) for the k-part of p can reach density, o is the interior arbitrary data of the k-neighborhood of p, lrd _k(o) can reach density for the k-part of o.

The k-part of arbitrary data p can reach density and be these data and its k-inverse apart from the average reach distance of neighborhood, and its computing method are:

${\mathrm{lrd}}_k\left(p\right)=1/\frac{\underset{o\∈N_k\left(p\right)}{\mathrm{\Σ}}\mathrm{reach}\_\mathrm{dis}{p}_k(p,o)}{\left|N_k\left(p\right)\right|}--3$

Reach_disp _k(p o) is the reach distance that p arrives arbitrary data o in its k-neighborhood, and p is the higher value of distance between the k-distance of o and p and the o with respect to the reach distance of o, that is:

reach_disp _k(p，o)＝max{k_distance(o)，d(p，o)} --4。

Beneficial effect: in the method, the degree that peels off of a point is relevant with the point around it, and this has embodied the notion of " part ", and this is it and the definition difference that in the past peeled off, and also is the advantage place.In addition, use the mean value of the local factor that peels off to judge outlier, make testing result more stable, the variation with parameter k value does not have than cataclysm.Can find the local outlier of other method omission its distinctive feature is arranged based on the mining algorithm that peels off of density, have better application to be worth.

Description of drawings

Fig. 1 is a flow chart of steps of the present invention.Wherein have: k minimum value k _Min, k maximal value k _Max, k changes step-length k _Step

Fig. 2 is traffic flow arrival rate and the density relationship and the data that peel off thereof,

Fig. 3 is the surface evenness test data and the data that peel off.

Embodiment

Specify the working of an invention mode below with reference to the accompanying drawings.Step is as follows:

1. the data acquisition equipment in the utilization intelligent transportation system obtains traffic data as vehicle detection coil, video detector, moving vehicle, radar, ultrasound wave etc., and as the speed of a motor vehicle, vehicle flowrate, occupation rate, hourage etc., establishing institute's image data collection is D;

2. the k-part of calculating each data among the D can reach density;

A given natural number k calculates the k-distance (k-distance (p)) of each data p, its value for p and between its k nearest neighbours o ∈ D apart from d (p, o),

The k neighborhood of p is defined as

N _k(p)＝{q∈D\{p}|d(p，q)≤k_distance(p)} (1)

P with respect to the reach distance of o is

reach_disp _k(p，o)＝max{k_distance(o)，d(p，o)} (2)

It is data p and its k-inverse apart from the average reach distance of neighborhood that the k-part of p can reach density,

${\mathrm{lrd}}_k\left(p\right)=1/\frac{\underset{o\∈N_k\left(p\right)}{\mathrm{\Σ}}\mathrm{reach}\_\mathrm{dis}{p}_k\left(o\right)}{\left|N_k\left(p\right)\right|}---\left(3\right)$

3. the k-part of the computational data factor that peels off then;

The k-part of the p factor that peels off is defined as

${\mathrm{lof}}_k\left(p\right)=\frac{\underset{o\∈N_k\left(p\right)}{\mathrm{\Σ}}\frac{{\mathrm{lrd}}_k\left(o\right)}{{\mathrm{lrd}}_k\left(p\right)}}{\left|N_k\left(p\right)\right|}---\left(4\right)$

4. with certain step-size change k value, repeating step 2 and 3, the k-part of calculating each data factor that peels off.The part factor that peels off has illustrated that the degree that peels off of data, the part of the data factor that peels off is big more, and it might be the data that peel off more.

5. the average part of calculating each data factor that peels off is to eliminate the influence of parameter k to testing result.

6. judge outlier based on the average part factor that peels off.Can use following two criterions: average part m the highest data of the factor that peel off are outlier, or on average the part factor that peels off all is an outlier greater than the data of given threshold value.

7. the deletion or the data that peel off that adopt the filtering technique correction to be identified are perhaps analyzed these data that peel off and are obtained and hide Info.

Embodiment 1: the traffic flow modeling

The traffic flow of highway is described with average speed, arrival rate, density usually, and the relation between arrival rate and the density can be described with figure, is referred to as the traffic flow fundamental figure.Checkout equipment or the transmission equipment traffic events of makeing mistakes, happen suddenly all may make traffic flow data generation abnormal change, no matter be sample error or the data that peel off that unusual traffic events produced, the aspect of model will be thickened, really the inherent law of reactive system.Therefore, before setting up model, need to find out and remove the data that peel off,, improve the accuracy and the reliability of institute's established model with the peel off influence of data of minimizing.

Now collect 709 of the traffic flow datas that the commonplace mouthful of airport expressway in Nanjing arranged, the sampling period is 1 minute, plans to build the model between upright arrival rate and the density.Utilization is sought special sample based on the detection method LOF of density, makes k=20, calculates the local anomaly factor of all samples.With 10 is that step-length increases the k value, and the local anomaly factor of all samples of double counting is until k=150.Then, calculate the average local anomaly factor of all samples, be averaged 12 the highest data of the local anomaly factor and be the data that peel off.Fig. 2 is institute's traffic flow arrival rate and density relationship basic scheme, wherein adds the data that peel off that are of circle.As can be seen, border and the inner data that peel off all effectively detect.

Delete the above-mentioned data that peel off, then adopt the data set that does not contain the data that peel off to set up the traffic flow model of highway.

Embodiment 2: the surface evenness Test Application

Surface evenness is an important indicator of road surface function, and it has not only reflected the driving comfort on road surface, also reflects the health status on road surface from the side.International roughness index IRI (International RoughnessIndex) is extensively adopted by countries in the world, is defined as the total displacement (m of unit) of basic body suspension and the ratio of operating range (km of unit), and unit is m/km.8000 in existing IRI sample, data acquisition every one meter once, test gained with Australian import surface evenness test carriage.

Utilization is sought special sample based on the detection method LOF of density, makes k since 50 for initial value, increases with step-length 10, calculates the local anomaly factor of all samples.Then obtain the average local anomaly factor of all samples.Here, suppose the average local anomaly factor greater than 1.8 all be the data that peel off, 28 the strongest points of degree of then finding to peel off as shown in Figure 3, wherein add the data that peel off that are of circle.

Compare with other check points, these peel off this place's pavement roughness of data declaration or road surfaces are damaged more serious, also may be sampling error or noise.For each detected exceptional value, need manually to participate in, at that time road surface, checkout equipment etc. are further analyzed, correctly distinguish the reason of its generation.

Claims (4)

1. method of discerning outlier traffic data, it is characterized in that this method is at first gathered obtains traffic data, the average part of the computational data factor that peels off, then use one of following two criterions to judge to peel off data: average part m the highest data of the factor that peel off are to peel off data, or the average part factor that peels off is the data that peel off greater than the data of given threshold value, deletion at last or the data that peel off that adopt the filtering method correction to be identified are perhaps analyzed the hiding Info that data comprised that peel off.

2. a kind of method of discerning outlier traffic data according to claim 1, the average part that the it is characterized in that described computational data factor method that peels off is, under a certain natural number k value, the k-part of calculating each data factor that peels off, with certain step-size change k value, the k-part of each data of the double counting factor that peels off, then by on average obtaining the average part of each data factor that peels off, its computing method are:

$\mathrm{lof}\left(p\right)=\frac{\underset{k}{\mathrm{\Σ}}{\mathrm{lof}}_k\left(p\right)}{\frac{k_2-k_1}{s}+1}--1$

Wherein, k ₁And k ₂Be respectively the bound of k, k ₁Be to be not less than 10 natural number, s is a step-length, lof _k(p) be the k-part of the arbitrary data p factor that peels off.

3. a kind of method of discerning outlier traffic data according to claim 2, the k-part that it is characterized in that each data factor computing method that peel off are, the k-part of all data can reach the ratio that density mean value and the k-part of p can reach density in the k-neighborhood of p, that is:

${\mathrm{lof}}_k\left(p\right)=\frac{\underset{o\∈N_k\left(p\right)}{\mathrm{\Σ}}\frac{{\mathrm{lrd}}_k\left(o\right)}{{\mathrm{lrd}}_k\left(p\right)}}{\left|N_k\left(p\right)\right|}--2$

Wherein, k is a natural number, N _k(p) be the k-neighborhood of data p, | N _k(p) | be the element number that this neighborhood contains; The k-neighborhood of p is made up of the data that the distance between all and the p is not more than the k-distance of p, and the k-distance of p is data p and the distance between its k nearest data; Lrd _k(p) for the k-part of p can reach density, o is the interior arbitrary data of the k-neighborhood of p, lrd _k(o) can reach density for the k-part of o.

4. a kind of method of discerning outlier traffic data according to claim 3, the k-part that it is characterized in that arbitrary data p can reach density and be these data and its k-inverse apart from the average reach distance of neighborhood, and its computing method are:

${\mathrm{lrd}}_k\left(p\right)=1/\frac{\underset{o\∈N_k\left(p\right)}{\mathrm{\Σ}}\mathrm{reach}\_{\mathrm{disp}}_k(p,o)}{\left|N_k\left(p\right)\right|}--3$

Reach_disp _k(p o) is the reach distance that p arrives arbitrary data o in its k-neighborhood, and its value is the higher value of distance between the k-distance of o and p and the o, that is:

reach_disp _k(p，o)＝max{k_distance(o)，d(p，o)} --4。

Images