KR100798577B1 - Analyzing system and analyzing method for data quality problem - Google Patents

Abstract

A system and a method for analyzing a data quality problem are provided to analyze the data quality problem of a database deeply, quickly, and easily by generating and displaying an SQL(Structured Query Language) automatically based on metadata defined to the database of a legacy system without database for analyzing a data flow analysis and a detail analysis. A data quality management system(300) includes the target data(310) of which data quality is analyzed. A meta system(200) includes the metadata for the target data. The data quality management system analyzes causes of the data quality program by utilizing the metadata for the target data. The target data is extracted from the database of which the data quality is analyzed. The data quality management system includes the data of a predetermined table for the target data, points a target table to display a flow tracing result, and inquires a target data list by utilizing the metadata related to the data flow.

Description

Analysis System and Analyzing Method for Data Quality Problem

1 is a conceptual diagram of a data quality problem analysis system according to an embodiment;

2 is a flowchart of a data quality problem analysis method using a data quality problem analysis system according to an embodiment;

3 is a conceptual diagram of a data flow analysis method of a data quality problem analysis system according to an embodiment;

4 is a conceptual diagram of a detailed data analysis method of a data quality problem analysis system according to an embodiment.

An embodiment relates to a data quality problem analysis system and a data quality problem analysis method.

In the prior art, the Data Quality Management System (DQMS) generally consists of a definition step, a measurement step, an analysis step, an improvement step, and a control step according to a six sigma technique.

The definition step is to define the range, measurement index, and business rules for quality measurement.The measurement step is to execute quality measurement by converting business rules into measurement scripts.The analysis step has measured results. Analyze the cause of data quality problems (error data), and the improvement phase is the step of performing improvement activities based on the cause analysis result, and the control phase is the control of the above activities such as authority control, procedure control, and monitoring. It's a step.

However, according to the prior art, currently commercialized technologies for data quality management at home and abroad are concentrated on definition, measurement, and control.

In other words, according to the prior art, although the actual cause of the data quality problem is in the data, it only tracks the data model.

Therefore, according to the related art, in order to analyze the cause of the data quality problem, the user analyzes the relationship of data and then directly writes and executes a structured query language (SQL) script to perform an analysis activity. There is a limit.

In addition, according to the prior art, there are some products of the concept of quality control, but there is a lack of systematic support for data analysis.

The embodiment relates to a technique for analyzing the cause when there is a quality problem in the database (DB) data, and in particular, a technique capable of tracking the data flow for data whose questionable data quality is suspected, and an interconnection analysis with surrounding data. It aims to provide a data quality problem analysis system and data quality problem analysis method that can provide the technology to do this.

Data quality problem analysis system according to an embodiment includes a data quality management system including the data to be analyzed quality; And a meta system including metadata about the quality analysis target data, wherein the data quality management system analyzes a cause of a data quality problem by using metadata about the quality analysis target data. .

In addition, the data quality problem analysis method according to the embodiment comprises the steps of extracting the quality analysis target data; And analyzing the cause of the data quality problem by using the metadata for the quality analysis target data.

According to the data quality problem analysis system and the data quality problem analysis method according to the embodiment, in the case of data flow analysis and detailed analysis, the legacy system (Lagacy System) without establishing a separate database (DB) for such analysis SQL (Structured Query Language) can be automatically generated and displayed using metadata defined for the database. As a result, the user can easily and quickly perform in-depth analysis on data quality problems occurring in the database.

Hereinafter, a data quality problem analysis system and a data quality problem analysis method according to an embodiment will be described in detail with reference to the accompanying drawings.

(Example)

1 is a conceptual diagram of a data quality problem analysis system according to an embodiment.

Data quality problem analysis system according to the embodiment is the quality management target database (100); A data quality management system 300 including quality measurement object data 310 extracted from the quality control object database 100 and quality analysis object data 320 extracted from the quality measurement object data 310; And a meta system 200 including meta data about the quality analysis target data 320.

The data quality management system 300 is characterized by analyzing the cause of the data quality problem by using the metadata for the quality analysis target data 320.

In the case of data flow analysis and detailed analysis, the data quality problem analysis system according to the embodiment does not establish a separate database (DB) for such an analysis, but defines a meta-data for the database of the legacy system (DB). SQL (Structured Query Language) can be automatically generated and displayed using metadata. As a result, the user can easily and quickly perform in-depth analysis on data quality problems occurring in the database.

For example, the data flow analysis by the data quality problem analysis system 500, having a data of a specific table for the quality analysis target data 320, designates a target table to see the flow trace result, You can search the target data list by using metadata related to the data flow.

Alternatively, for example, the detailed analysis by the data quality problem analysis system 500 may designate an analysis target column and an associated column for the quality analysis target data 320, and perform the analysis on the quality analysis target data 320. By using metadata about the relationship between tables, data of the analysis column may be grouped with the associated column to determine characteristics of the grouped data.

A description of the data flow analysis and detailed analysis by the data quality problem analysis system 500 will be described later with reference to FIGS. 3 and 4.

Meanwhile, in the embodiment, the quality control target database 100 may include internal data 110 or external data 120. The internal data 110 may include product information, customer information, transaction information, and the like.

The meta system 200 may include meta data about the quality control target data 100. In particular, the meta system 200 may include metadata about the quality analysis target data 320.

In an embodiment, the metadata is structured data of the quality control object data 100 or the quality analysis object data 320, and the data describing the quality control object data 100 or the quality analysis object data 320. to be.

There may be a lot of data in the quality management object data 100 or the quality analysis object data 320. For example, this data is in a table, which contains the table name, column names, relationships between the tables, and valid values for the columns. Meta data includes table names, column names, relationships between tables, and valid values. Of course, the actual metadata contains much more information than this.

The quality analysis target data 320 is a quality measurement of the quality measurement target data 310 extracted from the quality control target database 100 and classified as error data or data quality suspected in the quality score or detailed results. Can be.

In this case, the data quality management system 300 may work by defining existing information for quality measurement such as predetermined quality verification rules, data quality indicators (DQI), critical to quality (CTQ), and the like. For example, the CTQ may refer to a quality standard in which a process of manufacturing a product, a service, and a product meets the standards of TQM (Total Quality Managment) or meets the purchaser's purchasing quality.

The quality of predetermined data is determined by the quality measurement criteria. In this case, the embodiment uses metadata defined for the database of the legacy system (DB) without establishing a separate database (DB) for analyzing the cause of the data quality problem (using basic configuration information). To generate and show SQL (Structured Query Language) automatically.

Meanwhile, the embodiment may further include a management and utilization system 400 for analyzing the status or trend of the measured data quality and performing management and utilization on the measured data.

For example, the analysis of the measurement data may be performed by Online Analytical Processing (OLAP).

For example, OLAP automatically generates a structured query language (SQL) in a predetermined data warehouse (DW) and sends it to the quality analysis target data 320, and the analysis target data 320 sends the SQL. If you send the result to OLAP, OLAP will display the data on the screen.

In this case, it will be described with an example whether OLAP can automatically generate SQL. Since OLAP has meta data in itself, it has information about DW data model and user when building DW (Data Warehouse). When clicked, OLAP queries the metadata to see what information the user wants in which DB table, and the metadata looks at the table name, column name, and join type. The form is sent to OLAP, and OLAP receives it, creates the SQL, and sends it to DW DB.

The data quality problem analysis system 500 according to the embodiment is an activity for analyzing the root cause of the data derived as an error through data quality management activities through various viewpoints and techniques. Was selected as the implementation area.

Next, a data quality problem analysis method using the data quality problem analysis system will be described with reference to FIGS. 1 to 3.

First, the quality measurement target data 310 is extracted from the quality control target data 100 as shown in FIG. 1.

Next, the quality analysis data 320 is extracted from the quality measurement data 310.

Next, the cause of the data quality problem is analyzed by using metadata about the quality analysis target data 320.

2 is a flowchart illustrating a data quality problem analysis method using a data quality problem analysis system according to an embodiment.

In the data quality problem analysis method using the data quality problem analysis system according to the embodiment, the relationship between the tables of the quality measurement target data 310 is reversed and the rule-based data flow is defined to define the quality measurement target. Detailed analysis and data flow analysis may be performed on the data 310.

In other words, detailed analysis by grouping and data flow analysis that tracks data flow forward or backward on the analysis target data (error data or suspicious data) are performed by using data flow and inter-column business rule metadata. .

Specifically, in the analyzing of the cause of the data quality problem, the column naming information on the quality measurement target data 310 is decomposed into morphemes, the reference relationship is determined by analyzing the reference relationship candidate, and then the reference relationship is confirmed by the data. The reverse step S 552 may be performed.

Thereafter, the determined reference relationship may be registered as meta information of the quality control target data 100.

Next, analyzing the cause of the data quality problem, by complementing the reversed reference relationship (redefine) the front and rear relationship of the reference relationship in terms of the data flows on the database (redefine) the overall data flow To define the data flow definition step (S554) can proceed.

Next, in analyzing the cause of the data quality problem, the relationship between columns may be defined (S530) by identifying and defining the relationship of the columns in the table according to the redefined data flow.

Next, in analyzing the cause of the data quality problem, based on the relationship defined between the columns to analyze the business rule to register based on the rule (rule) to manage the rule as the meta information of the quality management target data 100 The base rule registration step (S540) may proceed.

Next, in analyzing the cause of the data quality problem, using the data flow metadata for the quality analysis target data 320 may proceed to the data flow analysis step (S 552) to track the data flow forward or backward. have.

Also, in analyzing the cause of the data quality problem, a detailed analysis step (S 554) by grouping the quality analysis target data may be performed by using metadata of the inter-column business rule. .

The data flow analysis step (S 552) and the detailed analysis step (S 554) step may be performed separately or together.

3 is a conceptual diagram of a data flow analysis method (S 552) of a data quality problem analysis system according to an embodiment.

The data flow analysis step (S 552) has a data of a specific table for the quality analysis target data 320, and designates a target table (tracking target data) for which a flow tracking result is to be viewed.

Thereafter, the target data list (tracking result data) may be inquired by using the metadata related to the data flow.

For example, in the case of data flow analysis as shown in FIG. 3, a target table (in FIG. 3) having data of a specific table (Table B in FIG. 3) to analyze the cause of the problem of data quality and to view the flow trace result F table), you can search the target data list by utilizing the metadata related to the data flow secured internally.

The data flow analysis (S 552) can track the quality analysis target data (error data or suspicious data) according to the business flow of the data generation, the data flow analysis function is a table relationship and work for the flow If only a rule is defined, it can be traced forward or backward from any location. In addition, the data flow analysis (S 552) can be easily used for financial flow tracking, logistics flow tracking on the system.

4 is a conceptual diagram of a detailed data analysis method (S554) of the data quality problem analysis system according to the embodiment.

The detailed analysis step (S 554) is a step of specifying an analysis target column for the quality analysis target data 320, the step of specifying an association column for the quality analysis target data 320, the quality analysis target data ( And grouping data with respect to the analysis column using the association column by using metadata about the relationship between tables in step 320) and identifying characteristics of the grouped data.

Identifying the characteristics of the grouped data refers to confirming whether the analyzed grouped data set has a significant difference compared to the entire data distribution.

For example, the detailed analysis step (S 554) designates an analysis target column (contract table-contract amount in FIG. 4) to be analyzed as shown in FIG. 4, and an associated column (customer table-customer ID and customer in FIG. If you specify table-gender, customer table-region, department table-department, account table-account, etc., it has metadata about internally secured tables and grouping analysis columns as related columns. ) Can show the result.

In an embodiment, the detailed analysis method (S 554) may perform a variety of analysis using the data rules around the business rule or column to analyze the cause of the error, the table owning the value of the analysis target business rule or column You can grasp the detailed distribution of errors by grouping them into other columns in the table or columns in the surrounding tables that can form join relationships. As a result, error data can be spotted and improvement points can be identified.

According to the data quality problem analysis system and the data quality problem analysis method according to the embodiment, in the case of data flow analysis and detailed analysis, the database of the legacy system (Lagacy System) is not established for such analysis. By using metadata defined for (DB), SQL (Structured Query Language) can be automatically generated and displayed. As a result, the user can easily and quickly perform in-depth analysis on data quality problems occurring in the database.

The data quality problem analysis system and the data quality problem analysis method according to the embodiment may be variously applied as follows.

First, by using the data flow tracing concept according to the embodiment, audit tracing can be performed before or after regardless of the positions of tables and columns in the legacy DB.

In addition, after applying the embodiments to define the monitoring indicators from a business or business perspective, if the abnormality in the indicators can be performed data tracing for cause analysis (Data Tracing).

In addition, data mining for inferring meaningful information may be performed by analyzing the correlation between the data on the legacy DB using the detailed analysis concept of the embodiment.

In addition, the embodiment may perform before and after data debugging (Data Debugging) for analyzing the cause of the data that is different from the expected during the system development process when performing the integrated test.

In addition, the embodiment may perform a user report (User Report) function to generate a related Query statement by utilizing the detailed analysis and data flow analysis function when the user generates a report for a specific business purpose.

In addition, in the data quality problem analysis system and the data quality problem analysis method according to the embodiment, the data flow information generated through the data quality problem analysis system may include various data in information management such as metadata, master data, etc. It can be applied in a technical configuration.

The present invention is not limited by the above-described embodiments and drawings, and various other embodiments are possible within the scope of the claims.

According to the data quality problem analysis system and the data quality problem analysis method according to the embodiment, in the case of data flow analysis and detailed analysis, the database of the legacy system (Lagacy System) is not established for such analysis. By using metadata defined for (DB), SQL (Structured Query Language) can be automatically generated and shown. As a result, the user can easily and quickly perform in-depth analysis on data quality problems occurring in the database.

In addition, the data quality problem analysis system and the data quality problem analysis method according to the embodiment may be made in various applications.

Claims (16)

A data quality management system including quality analysis target data; And And a meta system including meta data about the quality analysis target data. The data quality management system, Data quality problem analysis system, characterized in that for analyzing the cause of the data quality problem using the metadata for the quality analysis target data. According to claim 1, The data quality management system, It further includes quality measurement target data extracted from the quality management target database, And the quality analysis target data are extracted from the quality measurement target data. The method of claim 2, The data quality problem analysis system, Data quality problem, characterized in that with the data of the specific table for the quality analysis target data, to specify a target table to see the flow tracking results, and to look up the target data list using the metadata associated with the data flow Analysis system. The method according to claim 2 or 3, The data quality problem analysis system, Specifying an analysis target column and an association column for the quality analysis target data, and grouping data for the analysis column with the association column by using metadata about a relationship between tables for the quality analysis target data. Data quality problem analysis system, characterized in that to identify the characteristics of the grouped data. Extracting quality analysis target data; And And analyzing a cause of a data quality problem by using metadata about the quality analysis target data. The method of claim 5, Extracting the quality analysis target data, Extracting quality measurement target data from the quality control target data; And And extracting the quality analysis target data from the quality measurement target data. The method of claim 6, Analyzing the cause of the data quality problem, A reference relationship reverse step of decomposing the column naming information of the data to be measured to quality by morphemes, analyzing the reference relationship candidates, and verifying the reference relationships by verifying the data; And And registering the determined reference relationship as meta information of the data subject to quality control. The method of claim 7, wherein Analyzing the cause of the data quality problem, A data flow definition step of defining the overall data flow by redefining the procedural relationship of the reference relationship in terms of complementing the reversed reference relationship to the data flowing on the database; Analytical Method. The method of claim 8, Analyzing the cause of the data quality problem, And defining a relationship between the columns to identify and define the relationship of the columns in the table according to the redefined data flow. The method of claim 9, Analyzing the cause of the data quality problem, Analyzing the data quality problem further comprises a rule-based rule registration step of analyzing the business rules based on the relationship defined between the columns to register based on the rule (rule) to manage as meta information of the quality control data; Way. The method of claim 10, Analyzing the cause of the data quality problem, And a data flow analysis step of tracking the data flow forward or backward by using data flow metadata for the quality analysis target data. The method of claim 10, Analyzing the cause of the data quality problem, And a detailed analysis step by grouping the quality analysis target data by using the metadata of the inter-column business rule. The method of claim 10, Analyzing the cause of the data quality problem, A data flow analysis step of tracking a data flow forward or backward by using data flow metadata for the quality analysis target data; And And a detailed analysis step by grouping the quality analysis target data by using the metadata of the inter-column business rule. The method according to claim 11 or 13, The data flow analysis step, Designating a target table for which a flow trace result is to be viewed, with data of a specific table for the quality analysis target data; And And querying a target data list by using metadata related to the data flow. The method according to claim 12 or 13, The detailed analysis step, Designating an analysis target column for the quality analysis target data; Designating an association column for the quality analysis target data; Grouping data with respect to the analysis column using the association column using metadata about a relationship between tables of the quality analysis target data; And Identifying the characteristics of the grouped data; data quality problem analysis method comprising a. The method of claim 15, Identifying the characteristics of the grouped data, A method for analyzing data quality problems, characterized in that the analyzed grouped data set is compared with the total data distribution to see if there is a significant difference.

Images