JP5430702B2 - Specific data detection method, specific data detection program, and specific data detection apparatus - Google Patents

Description

  The present invention relates to a technique for determining whether each data is unique data in a data group including a plurality of data.

  The data cleansing process is a process for evaluating a plurality of attribute values associated with the data to be cleansed and detecting data having significantly different properties from other data as unique data. In the data cleansing process, singular data is detected based on the “degree of deviation” indicating how far each attribute value of the data deviates from the standard.

  An example of such data cleansing processing is described in Patent Document 1. In the method of Patent Document 1, cleansing is performed by deleting records whose abnormal value ratio is equal to or greater than a predetermined threshold.

JP 2004-29971 A

However, the conventional technique has a problem that the detection of specific data cannot always be performed appropriately.
For example, in the method of Patent Document 1, the ratio of abnormal values is taken into account, but the degree of deviation of each abnormal value is not taken into consideration. Therefore, data that contains a large number of slightly abnormal attribute values is determined to be singular data, while data that contains only one abnormal value that deviates greatly is determined to be normal data regardless of the degree of deviation. There is a high possibility that it will end.

  The present invention has been made to solve such problems, and an object of the present invention is to provide a specific data detection method, a specific data detection program, and a specific data detection device that can detect specific data more appropriately. .

In order to solve the above-described problem, the singular data detection method according to the present invention is a calculation device provided in a computer that determines whether or not data including attribute values representing values related to a plurality of attribute items is singular data. determining, a specific data detection method for each attribute value of the data, based on a plurality of data, based-out on the edge degree from the reference range calculated in advance for each of the attribute items, the attribute value is a reference If it is within the range, the outlier evaluation score is calculated as 0, and if the attribute value is out of the reference range, the outlier evaluation score is calculated according to the degree of outlier, and the step of determining the outlier evaluation score and the outlier are determined. The step of counting the number of outliers representing the number of attribute values whose evaluation score is equal to or greater than a predetermined threshold, and the number of specificities is determined based on the outlier evaluation score and the number of outlier attributes for each of the data Based on step and, specifically power, for each of the data, possess determining whether specific data, and in the step of determining the specificity power, computing device sums the out evaluation point of the data A step of calculating an outlier total evaluation score, determining the number of singularities by multiplying the outlier total evaluation point and a weight determined based on the number of outlier attributes, and determining whether the data is singular data Determining a specificity level reference range based on the first and third quartiles of all specificity numbers, or determining a specificity number reference range based on a predetermined confidence interval for all specificity numbers And a step of determining whether or not each specificity number belongs to the specificity number reference range .

A step of accepting an input of the multiple data, each data may have a step of outputting whether the specific data.
Criteria range be determined based on the first quartile and the third quartile of the corresponding attribute items, or may be determined based on a predetermined confidence interval in the corresponding attribute item .
Further comprising the step of generating a box-and-whisker plot for each attribute item, the step of outputting whether the specific data, in box plots, at positions corresponding to the attribute values, including the attribute values A step of outputting a symbol indicating whether or not the data is singular data may be included.

  The specific data detection program according to the present invention causes a computer to execute the above-described specific data detection method.

  The singular data detection apparatus according to the present invention is constituted by a computer that executes the above-described singular data detection method.

  According to the specific data detection method, the specific data detection program and the specific data detection apparatus according to the present invention, the specific data is detected based on both the degree of deviation from the reference range and the number of attributes outside the reference range. Specific data can be detected more appropriately.

It is a figure which shows the structure of the specific data detection apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the example of the data group of FIG. It is a flowchart which shows the flow of a process of the specific data detection apparatus of FIG. It is a figure which shows the relationship between a box-and-whisker diagram, a reference | standard range, etc. FIG. FIG. 3 is a diagram illustrating outlier evaluation points corresponding to attribute values of the data group in FIG. 2. It is a figure showing the example of the correspondence of the number of detachment attributes and a weight.

Embodiments of the present invention will be described below with reference to the accompanying drawings.
Embodiment 1 FIG.
FIG. 1 shows the configuration of a singular data detection apparatus 10 according to Embodiment 1 of the present invention. The singular data detection device 10 is a device that detects singular data by determining whether or not each data is singular data for a plurality of data.

  The singular data detection apparatus 10 has a configuration as a well-known computer, and includes a calculation unit 20 that performs a calculation and a storage unit 30 that stores information. The computing means 20 includes a CPU (Central Processing Unit), and the storage means 30 includes a storage medium such as a semiconductor memory and an HDD (Hard Disk Drive).

  Although not particularly shown, the singular data detection apparatus 10 includes an input unit, an output unit, and a network interface. The input means is means used by the user to input information, and is, for example, a mouse or a keyboard. The output means is means for outputting information to the user, and is a display device such as a liquid crystal display, but may be a printing device such as a printer. The network interface is a means for inputting / outputting information to / from a communication network or another computer.

  The storage means 30 stores a data group 40 composed of a plurality of data. The singular data detection apparatus 10 performs singular data detection processing on the data included in the data group 40. The storage unit 30 stores a unique data detection program 50. The singular data detection program 50 is executed by the computing means 20 to cause the computer to realize the function as the singular data detection device 10.

  FIG. 2 shows an example of the data group 40. The data group 40 includes a plurality of data. In FIG. 2, each row represents one piece of data, and 20 pieces of data are shown. Each data includes, for a plurality of items (hereinafter referred to as “attribute items”), quantitative measurement values (hereinafter referred to as “attribute values”) representing the properties of the data.

  FIG. 2 shows five types of attribute items “attribute A” to “attribute E”. For example, the data 01 has a value “0.18” as an attribute value for the attribute item named “attribute A”. In the present embodiment, all attribute values are numerical values. Each data represents, for example, one system development project, and each attribute value represents, for example, a numerical value related to man-hour, cost, quality, and the like. In FIG. 2, a surrounding line and an annotation indicating “out-of-range attribute value” are added for later explanation, but this is not included in the actual data group 40.

  FIG. 2 also shows the first quartile Q1, the third quartile Q3, and the quartile range IQR of each attribute item. In the example of attribute E, the first quartile Q1 = 4.00, the third quartile Q3 = 8.25, and the quartile range IQR = Q3-Q1 = 4.25. IQR corresponds to a box portion of a box plot corresponding to each attribute item. In the example of FIG. 2, the values of the respective quartiles are obtained by linear interpolation. However, for these values, any attribute value may be used as it is without interpolation, and may be rounded as appropriate. .

FIG. 3 is a flowchart showing a process flow of the singular data detection apparatus 10. The singular data detection apparatus 10 executes the method according to this flowchart as a computer.
First, the computing means 20 receives input of a plurality of data constituting the data group 40 and stores it in the storage means 30 (step S1). This input may be performed, for example, via a keyboard, may be performed via a network, or may be performed via a portable storage medium.

Next, the computing means 20 determines a reference range for each attribute item based on the data group 40 (step S2).
FIG. 4 is a diagram illustrating a relationship between a box plot and a reference range. In the present embodiment, the reference range is determined as follows.
First, the first quartile Q1 and the third quartile Q3 are calculated. Also, the quartile range IQR = Q3-Q1 is calculated. The reference range is determined as a range having Q1−1.5 × IQR as a lower limit and Q3 + 1.5 × IQR as an upper limit.

  Next, the computing means 20 determines an outlier evaluation score for each attribute value of each data based on the degree of deviation from the reference range (step S3). As shown in FIG. 4, if the attribute value belongs to the reference range, the degree of detachment is 0, and the evaluation score for detachment is also 0. Further, if the attribute value is out of the reference range and the distance from the upper limit or the lower limit is in the range of 1.5 × IQR or less, the degree of detachment is small and the outlier evaluation score is 10. Further, if the attribute value is out of the reference range and the distance from the upper limit or the lower limit is in a range exceeding 1.5 × IQR, the degree of detachment becomes large and the outlier evaluation score is 20.

In FIG. 2, a thick broken line indicates an outlier attribute value with a large degree of deviation (that is, a outlier evaluation point of 20), and a thin one-dot chain line indicates an outlier attribute value with a small degree of deviation (that is, a outlier evaluation point of 10). Hereinafter, an attribute value whose degree of detachment is not 0 (that is, an attribute value having a detachment evaluation score of 10 or more) is referred to as an “outlier attribute value”.
FIG. 5 shows outlier evaluation points corresponding to the attribute values of the data group of FIG.

The criteria for determining the above-described outlier evaluation points are expressed as inequalities as follows.
・ If X (n, a) <Q1 (a) -3.0 × IQR (a): Degree of outage, outage score 20
・ If Q1 (a) -3.0 × IQR (a) ≦ X (n, a) <Q1 (a) -1.5 × IQR (a): Degree of detachment, evaluation score 10
・ In case of Q1 (a) -1.5 × IQR (a) ≦ X (n, a) ≦ Q3 (a) + 1.5 × IQR (a): Deflection degree 0, Outlier evaluation score 0
・ When Q3 (a) + 1.5 × IQR (a) <X (n, a) ≦ Q3 (a) + 3.0 × IQR (a): Degree of detachment, evaluation score of detachment 10
・ In case of Q3 (a) + 3.0 × IQR (a) <X (n, a): Degree of outlier, outage score 20
Here, the attribute value a of the nth data (1 ≦ n ≦ 20 in the example of FIG. 2) (in the example of FIG. 2, a is one of the attributes A to E) is represented by X (n, a ), And Q1, Q3, and IQR of the attribute item a are represented by Q1 (a), Q3 (a), and IQR (a), respectively.

  Next, the computing means 20 counts the number of outlier attribute values for each data (step S4). In FIG. 5, this number is shown as “number of outlier attributes”.

Next, the calculating means 20 determines the number of specificities for each data (step S5). The determination of the number of specificities is performed as follows, for example.
In the present embodiment, for each data, first, an outlier comprehensive evaluation score is calculated based on the outlier evaluation score of each attribute value of the data. This outlier overall evaluation score is calculated, for example, by summing up all outlier evaluation points for the data. When the outlier comprehensive evaluation score of a certain data is high, this indicates that the data is different in nature from other data in the data group.
The computing means 20 determines the weight for each data based on the number of outlier attributes. Then, the number of specificities is calculated based on the total evaluation score and weight. For example, the number of singularities is calculated as a value obtained by multiplying the total evaluation score and the weight.
Thus, according to the present embodiment, the singular data is detected based on both the degree of deviation from the reference range and the number of deviation attributes.

FIG. 6 shows an example of the correspondence between the number of outlier attributes and the weight. In the case of data 01 in FIG. 5, since the number of outlier attributes is two, the weight is 1.25. By multiplying this weight by the total of outlier evaluation points 40, the number of specificities 40 × 1.25 = 50 is calculated. Is done. In this embodiment, since the weight is defined so as to increase as the number of outlier attributes increases, the number of specificities increases as the number of outlier attributes increases even if the outlier comprehensive evaluation score is the same. For example, in the example of FIG. 5, the data 01 and the data 16 are both out of place and the total evaluation score is 40, but the data 16 having a larger number of out-of-attributes has a larger number of specificities.
6 can be stored in advance in the storage means 30, for example.

Next, the calculation means 20 determines whether each data is singular data based on the number of singularities of all data (step S6). This step corresponds to the step of detecting singular data.
The determination of whether or not the data is singular data may be any method as long as it is a determination by a known statistical method, but can be performed by a method similar to steps S2 and S3, for example. That is, Q1 and Q3 are obtained using the specificity number of all data as a population, the specificity number reference range is determined based on these, and it is determined whether or not each specificity number belongs to this specificity number reference range. Data having a specificity number that does not belong to the range is determined to be unique data. When this method is applied to the example of FIG. 5, it is determined that data 01, data 05, and data 16 are singular data.

Next, the calculation means 20 generates a boxplot for each attribute item (step S7. Note that this step S7 may be executed at any time after step S1). The box-and-whisker chart can be generated by a known method based on Q1, Q3, IQR and the reference range of each attribute item. For example, a rectangle indicating the range with the position of Q1 as the lower limit and the position of Q3 as the upper limit is generated, and a whisker is generated with the position of the attribute value that is the maximum within the reference range as the upper end, and is the minimum within the reference range Generates a beard with the attribute value position at the bottom. Further, a symbol indicating each attribute value may be generated.
Similarly, the calculating means 20 may generate a box plot for the specificity number.

  Next, the calculation means 20 outputs whether each data is singular data (step S8). This output is performed, for example, by displaying on the display means or recording a new file in the storage means 30. Any output format may be used. For example, data determined to be singular data may be output in a list, and an indication of whether the data is singular data may be output in association with each data, or the box generated in step S7 You may output to a whisker. When outputting to the box plot, a symbol indicating whether or not the data including the attribute value is singular data may be output at a position corresponding to each attribute value in the box plot.

  Further, when a box plot is generated for the number of singularities, a similar output may be made in the box plot. For example, you may output the symbol which shows whether the data which have the said specificity number are specific data in the position corresponding to each specificity number.

  According to the first embodiment of the present invention described above, the singular data is detected based on both the degree of deviation from the reference range and the number of outlier attributes. Therefore, the detection is based on only one of these. Compared with the technique to do, specific data can be detected more appropriately. Therefore, more appropriate data cleansing can be realized.

In the first embodiment described above, the following modifications can be made.
The generation of a box plot (step S7) may be omitted. Further, a graph representing each attribute value may be generated instead of or in addition to the boxplot.

  The upper and lower limits of the reference range are not limited to those in the first embodiment, and may be other ranges determined based on Q1 and Q3. Further, it may be based on other statistical values without being based on Q1 and Q3. Further, the reference range may be based on the median Q2, or may be based on quantiles other than quartiles or percentage points. Further, it may be based on an average value and a standard deviation.

The reference range may be determined based on a confidence interval corresponding to a predetermined confidence width, and the confidence interval may be used as the reference range. For example, when at least two attribute items are correlated with each other (or possibly), the confidence width and confidence interval of another attribute value may be determined based on one attribute value.
Such a method can also be used when determining the specificity number reference range in step S6. Further, the reference range for the attribute item and the specificity number reference range may be determined by different methods.

  Further, the degree of deviation from the reference range may be expressed not by the distance from the upper limit or the lower limit of the reference range but by a function of the distance. Further, the detachment evaluation score may be determined by a method other than that shown in FIG. 4. For example, the detachment degree may be used as the detachment evaluation score as it is. Further, a criterion for determining whether or not a certain attribute value is an outlier attribute value may be different from that of the first embodiment. For example, the threshold value of the outlier evaluation score may be a value other than 10.

  The correspondence relationship between the number of outlier attributes and the weight may not be as shown in FIG. 6, and for example, the content may be such that the weight decreases as the number of outlier attributes increases. In this way, it is easier to determine that data including a more extreme detachment attribute value is singular data than data including a large number of detachment attribute values.

  In the first embodiment, the number of singularities is calculated by multiplying the divergence comprehensive evaluation point and the weight. However, this may be another calculation as long as the calculation is based on the detachment comprehensive evaluation point and the weight. For example, the number of singularities may be calculated using addition, power, exponential function, or the like.

In the first embodiment, the correspondence between the number of outlier attributes and the weight is common to all the attribute items, but this may be different for each attribute item. For example, a correspondence relationship corresponding to FIG. 6 may be defined for each attribute item.
The singular data detection method according to the first embodiment can be used for checking basic data for estimation and for assisting in setting a target value related to quality.

  10 singular data detection device, 20 computing means, 30 storage means, 40 data group (data), 50 singular data detection program, Q1, Q3 quartile, IQR quartile range.

Claims (6)

An arithmetic device provided in a computer is a singular data detection method for determining whether or not data including attribute values representing values related to a plurality of attribute items is singular data ,
For each said attribute value before Symbol data, based on a plurality of the data, based-out on the edge degree from previously calculated reference range for each of the attribute items, the attribute value is out if it is within the reference range An evaluation score is calculated as 0, and if the attribute value is out of the reference range, calculating an outlier evaluation score according to the outlier degree, and determining an outlier evaluation score;
For each of the data, counting the number of outliers representing the number of attribute values for which the outlier evaluation score is greater than or equal to a predetermined threshold;
Determining the number of specificities for each of the data based on the outlier evaluation score and the number of outlier attributes;
Determining whether each of the data is singular data based on the number of singularities;
I have a,
In the step of determining the number of singularities, the arithmetic unit calculates an outlier overall evaluation score by summing out the outlier evaluation scores of the data, and is determined based on the outlier overall evaluation score and the number of outlier attributes. Determining the number of singularities by multiplying by
The step of determining whether or not it is singular data includes:
Determining the specificity number reference range based on the first and third quartiles of the total specificity number, or determining the specificity number reference range based on a predetermined confidence interval in the total specificity number Steps,
And a step of determining whether each specificity number belongs to the specificity number reference range or not . A step of accepting the input of the data of multiple,
The method according to claim 1, further comprising: outputting whether each of the data is unique data. The reference range is determined based on a first quartile and a third quartile in a corresponding attribute item, or determined based on a predetermined confidence interval in the corresponding attribute item. Item 3. The specific data detection method according to Item 1 or 2 . Generating a boxplot for each of the attribute items;
The step of outputting whether or not it is singular data is a step of outputting a symbol indicating whether or not the data including the attribute value is singular data at a position corresponding to each attribute value in the boxplot. specific data detection method of claim 2 comprising or a specific data detection method of claim 3 when quoting claim 2,. The specific data detection program which makes a computer perform the method as described in any one of Claims 1-4 . The specific data detection apparatus comprised with the computer which performs the method as described in any one of Claims 1-4 .

Images