KR20210085886A - Data profiling method and data profiling system using attribute value quality index - Google Patents

Abstract

In accordance with one embodiment of the present invention, provided is a data profiling method using an attribute value quality index. The data profiling method includes the following steps of: calculating at least one first statistical value regarding each attribute based on data included in each attribute defined in a data set; determining a weighted value assigned to the first statistical value in accordance with a preset first condition; calculating an attribute value quality index regarding each attribute based on a result of calculating the first statistical value and the weighted value assigned to the first statistical value; selecting at least one of the attributes, of which the attribute value quality index corresponds to a threshold condition, as a profiling target attribute; and calculating a quality index for the data set based on a result of calculating the first statistical value corresponding to the profiling target attribute and the weighted value assigned to the first statistical value. Therefore, the present invention is capable of increasing the accuracy of the attribute value quality index.

Description

Data profiling method and data profiling system using attribute value quality index {DATA PROFILING METHOD AND DATA PROFILING SYSTEM USING ATTRIBUTE VALUE QUALITY INDEX}

The present invention relates to a data profiling method and a data profiling system, and more particularly, to a data profiling method using an attribute value quality index capable of efficiently profiling the quality of data by utilizing the attribute value quality index of the data. and to a data profiling system.

As we enter the era of the 4th industrial revolution, more people and things are connected to the Internet, and a large amount of data (big data) is being produced in the process. (See Fig. 1)

In particular, the amount of digital information produced and distributed worldwide is rapidly increasing and is expected to reach 90 zettabytes (ZB) by 2020. In addition, the amount of digital information in Korea is expected to increase by an average of 57% annually.

A typical example of a technology that utilizes such big data is artificial intelligence (AI) technology. Major companies around the world are focusing on securing artificial intelligence (AI) technology, the core technology of the 4th industrial revolution, to strengthen competitiveness.

For example, global IT companies such as Google, Facebook, Microsoft, and IBM are adopting artificial intelligence algorithm development as their main research field.

Machine learning, the core technology of artificial intelligence technology, is a method that allows a computer to learn data on its own to acquire high-level features.

At this time, for effective machine learning, not only a large amount of data but also high quality data must be secured. This is because the higher the quality of data provided for machine learning, the higher the accuracy of judgments made using artificial intelligence technology.

However, the development of data profiling technology for securing reliable and high-quality data is still insufficient.

In particular, in recent years, as the amount of data in digital format has increased significantly, a suitable profiling method is required. However, a method for quickly and specifically profiling the quality of such data has not yet been prepared.

Therefore, it is required to develop a technology that can efficiently profile the data quality by systematically calculating the data quality index.

An object of the present invention is to provide a data profiling method and a data profiling system using an attribute value quality index capable of efficiently profiling the quality of data by utilizing the attribute value quality index of the data.

The technical problems to be achieved by the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by those of ordinary skill in the art to which the present invention belongs from the description below. There will be.

In order to achieve the above technical object, an embodiment of the present invention provides the steps of: a) calculating at least one first statistical value for each attribute based on data included in each attribute defined in a data set; b) determining a weight assigned to the first statistical value according to a first preset condition; c) calculating an attribute value quality index for each attribute based on the calculation result of the first statistical value and the weight assigned to the first statistical value; d) selecting at least one of the attributes whose attribute value quality index corresponds to a threshold condition as a profiling target attribute; and e) calculating a quality index for the data set based on an operation result of the first statistical value corresponding to the profiling target attribute and a weight assigned to the first statistical value. A data profiling method using

The first condition includes at least one detailed condition, the first statistical value is calculated for each detailed condition belonging to the first condition, and the weight given to the first statistical value is the first statistical value. It may be determined differently for each corresponding detailed condition.

The weight may be determined to be higher as the detailed condition assuming a high data error rate.

before step c), calculating at least one second statistical value calculated in a different manner from the first statistical value for each attribute; and determining a weight to be given to the second statistical value according to a second preset condition, wherein in step c), the attribute value quality index is based on the first statistical value and the first statistical value. The calculation may be performed based on a calculation result of the assigned weight and a calculation result of the second statistical value and the weight assigned to the second statistical value.

The first statistical value is based on at least one of Z-Score, missing value, minimum value, maximum value, mode, mean value, variance, standard deviation, five-value summary, outlier, and near zero variance. can be calculated as

The method may further include, before step a), normalizing the data belonging to the data set for each attribute defined in the data set.

A computer-readable recording medium recording a computer program for executing the method according to the above method may be provided.

In order to achieve the above technical object, an embodiment of the present invention provides a statistical value calculator for calculating at least one first statistical value for each attribute based on data included in each attribute defined in a data set; a weighting unit configured to determine a weight to be given to the first statistical value according to a preset first condition; an attribute value quality index calculator configured to calculate an attribute value quality index for each attribute based on the calculation result of the first statistical value and the weight assigned to the first statistical value; a profiling target attribute determination unit for selecting at least one of the attributes whose attribute value quality index corresponds to a threshold condition as a profiling target attribute; and a data profiling unit configured to calculate a quality index for the data set based on a result of calculating the first statistical value corresponding to the profiling target attribute and the weight given to the first statistical value. A data profiling system using a quality index is provided.

The first condition includes at least one detailed condition, the first statistical value is calculated for each detailed condition belonging to the first condition, and the weight given to the first statistical value is the first statistical value. It may be determined differently for each corresponding detailed condition.

The weight may be determined to be higher as the detailed condition assuming a high data error rate.

The statistical value calculating unit calculates at least one second statistical value calculated in a different way from the first statistical value, and the weighting unit determines a weight to be given to the second statistical value according to a preset second condition, , the attribute value quality index may be calculated based on a calculation result of the first statistical value and a weight assigned to the first statistical value, and a calculation result of the second statistical value and a weight assigned to the second statistical value. .

The statistical value calculating unit may include a descriptive statistics analyzing unit for normalizing data included in each attribute for each attribute.

According to an embodiment of the present invention, data profiling can be performed efficiently because an attribute with a high need for profiling can be selected based on the attribute value quality index calculated for each attribute defined in the data set.

According to an embodiment of the present invention, since the weight applied to the statistical value calculated for each attribute defined in the data set is differently assigned to each of a plurality of conditions, the accuracy of the attribute value quality index may be increased.

It should be understood that the effects of the present invention are not limited to the above-described effects, and include all effects that can be inferred from the configuration of the invention described in the detailed description or claims of the present invention.

1 is a diagram for explaining a conventional data set.
2 is a block diagram schematically illustrating an overall system for providing data profiling according to an embodiment of the present invention.
3 is a block diagram schematically illustrating a data profiling system according to an embodiment of the present invention.
4 is a block diagram specifically illustrating a profiling target attribute selector according to an embodiment of the present invention.
5 is a diagram for explaining an overall process of a data profiling system according to an embodiment of the present invention.
6 is a diagram for explaining a data profiling method according to an embodiment of the present invention.
7 is a diagram for explaining an example of a process performed in each step of a data profiling method according to an embodiment of the present invention.

Hereinafter, the present invention will be described with reference to the accompanying drawings. However, the present invention may be embodied in several different forms, and thus is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be “connected (connected, contacted, coupled)” with another part, it is not only “directly connected” but also “indirectly connected” with another member interposed therebetween. "Including cases where In addition, when a part "includes" a certain component, this means that other components may be further provided without excluding other components unless otherwise stated.

The terms used herein are used only to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as "comprises" or "have" are intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

2 is a block diagram schematically illustrating an overall system for providing data profiling according to an embodiment of the present invention.

As shown in Fig. 2, the entire system for providing data profiling according to an embodiment of the present invention includes an external server (A), a communication network (B), a data profiling system 1000 and a user device (C). may include

The external server A may store a data set to be subjected to data profiling. In this case, the data set may be in the form of a file or a database.

For example, the data set may include a plurality of data records. For example, the data set may include a plurality of predefined properties.

For example, the data set may be structured data such as reference information, transaction information, and aggregate information. For example, the data set may be semi-structured data such as HTML, XML, or GIS. For example, the data set may be unstructured data such as a moving picture or an image.

In addition, the external server A may provide a data set to be subjected to data profiling to the data profiling system 1000 or the user device C through the communication network B.

In this case, the communication network B may be a known communication network B including wired communication and wireless communication.

For example, the communication network B includes a local area network (B) (LAN), a metropolitan area network (B) (MAN), a wide area network (B) (WAN; wide area network), and the like. can For example, the communication network B may be a well-known Internet or World Wide Web (WWW).

In addition, the communication network B may connect the external server A, the communication network B, the data profiling system 1000, and the user device C included in the entire system according to an embodiment of the present invention to each other via wire or wireless. .

The data profiling system 1000 may analyze the data set provided from the external server A to calculate a quality index for the data set.

In more detail, the data profiling system 1000 may calculate an attribute value quality index for each attribute defined in the data set.

In addition, the data profiling system 1000 may select at least one of the properties for which the calculated property value quality index corresponds to a threshold condition as the profiling target property. Also, the data profiling system 1000 may calculate a quality index for a data set based on data belonging to a profiling target attribute.

A detailed operation and configuration of the data profiling system 1000 will be described in detail with reference to FIGS. 3 to 7 .

In addition, the user device C may be connected to the external server A and the data profiling system 1000 through the communication network B.

In addition, the user device C may receive the data profiling result, that is, the quality index for the data set, from the data profiling system 1000 , and may provide it to the user.

For example, the user device C may be various electronic devices such as a desktop, a laptop computer, a tablet, and a smart phone.

Meanwhile, the user device C may upload a data set to be subjected to data profiling to the data profiling system 1000 .

3 is a block diagram schematically illustrating a data profiling system 1000 according to an embodiment of the present invention. 4 is a block diagram specifically illustrating the profiling target attribute selector 200 according to an embodiment of the present invention.

As shown in FIG. 3 , the data profiling system 1000 includes a data set management unit 100 , a profiling target property selection unit 200 , a data profiling performing unit 300 , a memory 400 , and a communication unit 500 . ) and a control unit 600 .

The data set management unit 100 may obtain a data set to be subjected to data profiling from the external server (A). Also, the data set manager 100 may convert the collected data set into a data set having a format suitable for data profiling.

In addition, the profiling target property selector 200 may select at least one of a plurality of properties defined in the data set as the profiling target property.

To this end, the profiling target attribute selecting unit 200 includes a statistical value calculating unit 210, a weighting unit 220, an attribute value quality index calculating unit 230 and profiling as shown in FIG. 4(a). It may include a target attribute determiner 240 .

The statistical value calculating unit 210 may classify data belonging to a data set according to a preset classification criterion for each attribute, and may calculate statistical values for the classified data.

Specifically, the statistical value calculation unit 210 performs the descriptive statistics analysis unit 211, the first statistical value analysis unit 212, and the second statistical value analysis unit 213 as shown in FIG. 4(b). may include

The descriptive statistics analyzer 211 may normalize data included in each attribute defined in the data set for each attribute. In this case, normalization means standardizing the distribution of data belonging to a data set for each attribute.

As an example, the descriptive statistics analyzer 211 may calculate an index value for data belonging to each attribute predefined in the data set. For example, the index value may be any one of mean, variance, standard deviation, and mean deviation.

For example, the descriptive statistics analyzer 211 may convert data belonging to each attribute into a Z-Score based on a value and an index value of data belonging to each attribute predefined in the data set.

Z-Score can be calculated by Equation 1 below.

[Equation 1]

는 제1 속성에 속한 데이터들 중 i번째 데이터의 Z-Score이다.

는 제1 속성에 속한 데이터들 중 i번째 데이터의 값이다.

는 제1 속성에 속한 데이터들의 평균값이다. s는 제1 속성에 속한 데이터들의 표준편차이다.In Equation 1 above

is the Z-Score of the i-th data among data belonging to the first attribute.

is a value of i-th data among data belonging to the first attribute.

is an average value of data belonging to the first attribute. s is the standard deviation of data belonging to the first attribute.

In addition, the first statistical value analyzer 212 may calculate a first statistical value by analyzing data belonging to each attribute predefined in the data set for each attribute.

As an example, the first statistical value is at least one of Z-Score, missing value, minimum value, maximum value, mode, mean value, variance, standard deviation, five-value summary, outlier, and near-zero variance. It may be a statistical value calculated based on .

As an example, the first statistical value analyzer 212 may calculate a first statistical value by analyzing a missing value for each attribute previously defined in the data set.

In this case, the weight assigning unit 220 may determine a weight to be given to the first statistical value based on the first condition.

For example, when the first statistical value satisfies the first condition, the weight assigning unit 220 may assign a preset weight to the first statistical value.

Meanwhile, the first condition may include a plurality of detailed conditions.

In addition, the first statistical value may be calculated for each of a plurality of detailed conditions belonging to the first condition.

In addition, the weighting unit 220 may assign different weights to first statistical values corresponding to each of a plurality of detailed conditions belonging to the first condition.

For example, when the first statistical value is a statistical value related to a missing value, a weight given to the first statistical value may be set as shown in Table 1.

division Weight application criteria by first condition weight Condition 1 - A missing value = 0 0.0 1st condition - B 0 < number of missing values ≤5% 1.2 Condition 1 - C 5% < number of missing values ≤ 15% 1.5 1st condition - D Number of missing values > 15% 2.0

Table 1 shows a case where four detailed conditions (A, B, C, and D) belong to the first condition. As shown in Table 1, the weight applied to each detailed condition is the above detailed condition assuming a high data error rate. The higher the condition, the higher may be determined.

According to this, since the weight given to the first statistical value is differently assigned to each of the plurality of detailed conditions belonging to the first condition, the quality index of each attribute value included in the data set can be more accurately calculated.

Meanwhile, the second statistical value analysis unit 213 may calculate a second statistical value by analyzing data belonging to each attribute predefined in the data set for each attribute.

In this case, the second statistical value may be a statistical value calculated in a different way from the first statistical value.

For example, when the first statistical value is a statistical value related to a missing value, the second statistical value may be a statistical value related to an outlier. In other words, the second statistical value analyzer 213 may analyze an outlier for each attribute predefined in the data set and calculate it as a second statistical value.

In this case, the weight assigning unit 220 may determine a weight to be given to the second statistical value based on the second condition.

For example, when the second statistical value satisfies the second condition, the weight assigning unit 220 may assign a preset weight to the second statistical value.

Meanwhile, the second condition may include a plurality of detailed conditions.

In addition, the second statistical value may be calculated for each of a plurality of detailed conditions belonging to the second condition.

In addition, the weighting unit 220 may assign different weights to the second statistical values corresponding to each of a plurality of detailed conditions belonging to the second condition.

For example, when the second statistical value is a statistical value related to an outlier, a weight given to the second statistical value may be set as shown in Table 2.

division Weight application criteria for each second condition ( β ) weight ( α ) 2nd condition - A` Z-Score ≤ | 2 | 0.0 2nd condition - B` | 2 | Z-Score ≤ | 3 | 1.2 2nd condition - C` | 3 | Z-Score ≤ | 4 | 1.5 2nd condition - D` Z-Score | 4 | 2.0

Table 2 shows a case where four detailed conditions (A', B', C', D') belong to the second condition. According to this, the weight assigned to the second statistical value is a plurality of sub-conditions belonging to the second condition. As different conditions are given for each detailed condition, the quality index of each attribute value included in the data set can be more accurately calculated.

In addition, the attribute value quality index calculation unit 230 may calculate the attribute value quality index based on the first statistical value, the second statistical value, and a weight assigned to each statistical value.

As an example, the attribute value quality index may be calculated by Equation 2 below.

[Equation 2]

는 i번째 속성의 속성 값 품질 지수(Attribute Value Quality Index, AVQI)를 의미한다. k는 측정 항목을 의미한다. n은 측정 항목의 수를 의미한다.

는 k번째 측정 항목의 가중치 적용 기준에 해당하는 가중치를 곱한 값이다.

는 측정 항목의 가중치가 0 이상인 데이터 레코드의 수, 즉 측정항목의 수이다. (단, 가중치가 0이외의 가중치 적용 기준에 해당하는 레코드가 없는 경우, 해당 속성의 속성 값 품질 지수는 0)in Equation 2

denotes an Attribute Value Quality Index (AVQI) of the i-th attribute. k means a measurement item. n means the number of measurement items.

is the value multiplied by the weight corresponding to the weight application criterion of the k-th measurement item.

is the number of data records with a weight of 0 or more, that is, the number of metrics. (However, if there is no record corresponding to a weighting criterion other than 0, the attribute value quality index of the attribute is 0)

는 아래의 수학식 3으로 산출될 수 있다.For example,

can be calculated by Equation 3 below.

[Equation 3]

는 아래의 수학식 4로 산출될 수 있다.For example,

can be calculated by Equation 4 below.

[Equation 4]

In Equations 3 and 4, the number of records extracted by the β criterion is the number of records in which the weight α of the weighting criterion β is other than 0.0.

In addition, the profiling target attribute determiner 240 may select at least one of the attributes whose attribute value quality index corresponds to a threshold condition as the profiling target attribute.

As an example, the profiling target attribute determiner 240 may determine an attribute whose attribute value quality index is greater than a threshold value (eg, 0) as the profiling target attribute.

On the other hand, the data profiling performing unit 300 determines the quality of the data set based on the calculation result of the statistical value corresponding to the profiling target property selected by the profiling target property selecting unit 200 and the weight assigned to each statistical value. index can be calculated.

As an example, the quality index for the data set may be calculated by Equation 5 below.

[Equation 5]

(Structured Data Value Quality Index)는 데이터 집합에 대한 품질 지수를 의미한다. 그리고, i는 i번째 속성을 의미한다. 그리고, n은 프로파일링 대상 속성의 수이다. 그리고,

는 i번째 속성의 가중치 적용 기준(β)에 해당하는 가중치(α)를 i번째 속성의 통계값에 곱한 값이다. 그리고,

는 i번째 속성의 가중치 적용 기준(β)에 해당하는 데이터 레코드의 수, 즉 측정 항목의 수이다.in Equation 5

(Structured Data Value Quality Index) means a quality index for a data set. And, i means the i-th attribute. And, n is the number of profiling target attributes. And,

is a value obtained by multiplying the statistical value of the i-th attribute by the weight (α) corresponding to the weight application criterion (β) of the i-th attribute. And,

is the number of data records corresponding to the weight application criterion (β) of the i-th attribute, that is, the number of measurement items.

As an example, SDVQI may be a quality index for a structured data set.

Hereinafter, data quality efficiency of the data profiling system 1000 according to an embodiment of the present invention will be described with reference to the embodiment of Table 3.

Table 3 is a table in which the first statistical value (missing value) and the second statistical value (outlier) are calculated using the Deli Weather data set registered in Kaggle.

division Criteria for weighting by condition weight Dewptm fog hail hum pressure missing value missing value = 0 0.0 0<Number of missing values ≤5% 1.2 621 757 232 5%<Number of missing values ≤15% 1.5 Number of missing values >15% 2.0 outlier Z-Score ≤ |2| 0.0 |2| < Z-Score ≤ |3| 1.2 826 780 |3| < Z-Score ≤ |4| 1.5 26 7038 One Z-Score > |4| 2.0 5 13 2 One AVQI 0.028 0.50 1.0 0.201 0.203 Rain snow tempm thunder tonado vism wdird wspdm date time 673 4428 2358 14755 3080 9 294 One 34 2652 One 4 952 2 One 3 133 1.000 1.000 0.201 1.000 1.000 0.200 0.500 0.241 0.000 0.000

As shown in Table 3, 15 attributes (dewptm, fog, hail, hum, pressurem, rain, snow, tempm, thunder, tornado, vism, wdird, wspdm, date, time) can be defined within the Deli Weather data set. have.

And, as described in Equations 2 to 4, among 15 attributes, 13 attributes having AVQI greater than 0 may be selected as profiling target attributes.

Meanwhile, a Data Quality Efficiency Measurement Value (DQEM) may be calculated by Equation 6 below.

[Equation 6]

In Equation 6, S is the product of the number of data attributes and the number of data records. And, m is the product of the number of properties to be profiled and the number of data records. In this case, the closer the data quality efficiency measurement value is to 100%, the better the performance.

Accordingly, in the embodiment of Table 3, the measured data quality efficiency may be calculated as 13.333%.

That is, it can be seen that the data quality efficiency measurement value (13.333%) of the embodiment of Table 3 is superior to the data quality efficiency measurement value (0%) when the profiling target attribute is not selected based on the attribute value quality index.

In addition, the memory 400 stores a data set, a type of attribute defined in the data set, statistical values for attributes defined in the data set, weights assigned to statistical values for attributes defined in the data set, and a profiling target. You can store attributes, attribute value quality indexes, quality indexes for data sets, and more.

For example, the memory 400 may be a computer- or human-readable recording medium.

And, the communication unit 500 may communicate with the external server or the user device (C) through the communication network (B).

In addition, the controller 600 may control the operation of the components included in the data profiling system 1000 and the data flow between the components.

5 is a diagram for explaining the overall process of the data profiling system 1000 according to an embodiment of the present invention.

As shown in FIG. 5 , the data set management unit 100 may obtain a data set to be subjected to data profiling from the external server A. As shown in FIG.

In this case, the data set management unit 100 may acquire one or more data sets. That is, the data set management unit 100 may acquire a plurality of data sets 101 and 102 .

In addition, the statistical value calculation unit 210 may calculate a statistical value for the data set obtained by the data set management unit 100 .

In this case, the statistical value calculating unit 210 may include a plurality of statistical value calculating units for calculating statistical values related to different types of data.

As an example, the statistical value calculating unit 210 may include a first statistical value calculating unit 210a and a second statistical value calculating unit 210b.

As an example, the first statistical value calculating unit 210a may calculate a statistical value for the structured data among data belonging to the data set.

Specifically, the first statistical value calculation unit 210a uses the above-described descriptive statistics analysis unit 211 , the first statistical value analysis unit 212 , and the second statistical value analysis unit 213 to obtain statistics on the structured data. value can be calculated.

As an example, the second statistical value calculating unit 210b may calculate a statistical value for unstructured data among data belonging to a data set.

Specifically, the second statistical value calculation unit 210b may include a Term Frequency - Inverse Document Frequency (TF-IDF) analysis unit 214 and a word embedding analysis unit 215 .

The TF-IDF analysis unit 214 may analyze the importance of a specific word included in the data set.

For example, the TF-IDF analyzer 214 may calculate a term frequency (TF) value in the data set.

In addition, the TF-IDF analyzer 214 may calculate a document frequency (DF) value and an inverse document frequency (IDF) value based on the word frequency value. In addition, the TF-IDF analysis unit 214 may calculate the TF-IDF value based on the word frequency value, the document frequency value, and the inverse document frequency value.

In addition, the word embedding analyzer 215 may analyze the relationship between words included in the data set using an artificial neural network method. For example, the word embedding analyzer 215 may analyze the relationship between words included in the data set based on the Word2Vec algorithm.

And, as described with reference to FIG. 4 , the quality index for the data set may be calculated based on the statistical values calculated by the respective statistical value calculating units.

As an example, the first data set quality index may be calculated based on the statistical value calculated by the first statistical value calculating unit 210a.

In addition, the second data set quality index may be calculated based on the statistical value calculated by the second statistical value calculating unit 210b.

In addition, the comprehensive data set quality index may be calculated based on the first data set quality index and the second data set quality index.

6 is a diagram for explaining a data profiling method according to an embodiment of the present invention. 7 is a diagram for explaining an example of a process performed in each step of the data profiling method according to an embodiment of the present invention.

6 and 7 , a step of preparing a data set including a plurality of data records may be performed. (S100)

As an example, the data set may be transmitted from an external server A or an external storage device (not shown) to the data profiling system 1000 via a communication network B.

Specifically, the data set may be transmitted to the data set management unit 100 included in the data profiling system 1000 , and may be stored in the memory 400 .

For example, the data record may be composed of data divided by properties defined in the data set.

In addition, the step of calculating at least one first statistical value for each attribute based on data included in each attribute defined in the data set may be performed. (S200)

For example, the first statistical value may be calculated for each attribute.

As an example, a first statistical value for the first attribute may be calculated. In addition, the first statistical value may be calculated for each of a plurality of detailed conditions (first condition-A, first condition-B, first condition-C) belonging to the first condition.

In this case, a second statistical value for the first attribute may be further calculated according to a required level of profiling. The second statistical value may be calculated for each of a plurality of detailed conditions (second condition - A, second condition - B, and second condition - C) belonging to the second condition.

Meanwhile, although FIG. 7 illustrates that the same condition and weight are set to the first attribute and the second attribute, the present invention is not limited thereto. That is, it goes without saying that the conditions and weights applied to the first attribute and the second attribute may be different.

Then, the step of determining the weight to be given to the first statistical value according to the first preset condition may be performed. (S300)

When the first statistical value satisfies the first condition, a preset weight may be assigned.

For example, when the first condition includes a plurality of detailed conditions, weights a1 , a2 , a3 given to the first statistical values corresponding to each detailed condition may be different for each detailed condition.

Also, when the second statistical value is calculated in step S200, a preset weight may be given to the second statistical value based on the second condition.

Also, when the second condition includes a plurality of detailed conditions, the weights b1, b2, and b3 given to the second statistical values corresponding to each condition may be different for each detailed condition.

In addition, the step of calculating the attribute value quality index for each attribute based on the calculation result of the first statistical value and the weight assigned to the first statistical value may be performed. (S400)

The attribute value quality index may be calculated for each attribute belonging to the data set.

In this case, when the second statistical value is calculated, the calculation result of the second statistical value and the weight assigned to the second statistical value may be utilized to calculate the attribute value quality index.

In addition, the step of selecting at least one attribute among attributes corresponding to the threshold condition of the attribute value quality index as a profiling target attribute may be performed. (S500)

For example, an attribute whose attribute value quality index is greater than a threshold value (ex. 0) may be selected as a profiling target attribute.

Then, the step of calculating the quality index for the data set based on the calculation result of the first statistical value corresponding to the profiling target attribute and the weight given to the first statistical value may be performed. (S600)

In this case, when the second statistical value is calculated, the calculation result of the second statistical value corresponding to the profiling target attribute and the weight assigned to the second statistical value may be utilized to calculate the quality index for the data set.

As an example, the quality index for the data set may be calculated with reference to Equation 5.

The above description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention.

100: data set management unit 200: profiling target property selection unit
210: statistical value calculation unit 220: weighting unit
230: attribute value quality index calculation unit 240: profiling target attribute determination unit
300: data profiling system 400: memory
500: communication unit 600: control unit

Claims (12)

a) calculating at least one first statistical value for each attribute based on data included in each attribute defined in the data set;
b) determining a weight assigned to the first statistical value according to a first preset condition;
c) calculating an attribute value quality index for each attribute based on the calculation result of the first statistical value and the weight assigned to the first statistical value;
d) selecting at least one of the attributes whose attribute value quality index corresponds to a threshold condition as a profiling target attribute; and
e) calculating a quality index for the data set based on the calculation result of the first statistical value corresponding to the profiling target attribute and a weight assigned to the first statistical value
A data profiling method using an attribute value quality index comprising a.
According to claim 1,
The first condition includes at least one detailed condition,
The first statistical value is calculated for each detailed condition belonging to the first condition,
The data profiling method using the attribute value quality index, wherein the weight given to the first statistical value is determined differently for each of the detailed conditions to which the first statistical value corresponds.
3. The method of claim 2,
The data profiling method using the attribute value quality index, wherein the weight is determined to be higher in the detailed condition assuming a high data error rate.
According to claim 1,
Prior to step c),
calculating at least one second statistical value calculated in a different manner from the first statistical value for each attribute; and
Determining a weight to be given to the second statistical value according to a preset second condition; further comprising,
In step c),
The attribute value quality index is calculated based on the calculation result of the first statistical value and the weight assigned to the first statistical value, and the calculation result of the second statistical value and the weight assigned to the second statistical value
Data profiling method using attribute value quality index.
According to claim 1,
The first statistical value is based on at least one of Z-Score, missing value, minimum value, maximum value, mode, mean value, variance, standard deviation, five-value summary, outlier, and near zero variance. A data profiling method using an attribute value quality index calculated as .
According to claim 1,
Before step a),
normalizing the data belonging to the data set for each attribute defined in the data set;
Data profiling method using the attribute value quality index further comprising.
A computer-readable recording medium recording a computer program for executing the method according to any one of claims 1 to 6.
a statistical value calculation unit for calculating at least one first statistical value for each attribute based on data included in each attribute defined in the data set;
a weighting unit configured to determine a weight to be given to the first statistical value according to a preset first condition;
an attribute value quality index calculator configured to calculate an attribute value quality index for each attribute based on a result of calculating the first statistical value and the weight assigned to the first statistical value;
a profiling target attribute determination unit for selecting at least one of the attributes whose attribute value quality index corresponds to a threshold condition as a profiling target attribute; and
and a data profiling performing unit for calculating a quality index for the data set based on an operation result of the first statistical value corresponding to the profiling target attribute and the weight given to the first statistical value; Data profiling system using exponents.
9. The method of claim 8,
The first condition includes at least one detailed condition,
The first statistical value is calculated for each detailed condition belonging to the first condition,
The data profiling system using the attribute value quality index, wherein the weight given to the first statistical value is determined differently for each of the detailed conditions to which the first statistical value corresponds.
10. The method of claim 9,
The data profiling system using the attribute value quality index, wherein the weight is determined to be higher in the detailed condition assuming a high data error rate.
9. The method of claim 8,
The statistical value calculating unit calculates at least one second statistical value calculated in a different way from the first statistical value,
The weighting unit determines a weight to be given to the second statistical value according to a preset second condition,
wherein the attribute value quality index is calculated based on a calculation result of the first statistical value and a weight assigned to the first statistical value, and a calculation result of the second statistical value and a weight assigned to the second statistical value Data profiling system using value quality index.
9. The method of claim 8,
The data profiling system using the attribute value quality index, wherein the statistical value calculation unit includes a descriptive statistics analysis unit for normalizing data included in each attribute for each attribute.

Images