KR20100068733A - Data cleansing system and method for developing integrated database, and recording medium therefor - Google Patents

Abstract

PURPOSE: A data cleansing system and a method for developing integrated database, and a recording medium therefor are provided to collect data of scattered databases and detect data error and refine detected error. CONSTITUTION: A data collecting unit(220) collects data from a plurality of agency systems. A data error detector(230) detects an error which does not correspond to a standard through a comparison between the collected data and preset detection rules. According to the preset purification rules, a data purifying unit(240) refines the detected data. A user interface unit(280) provides information related to a data purification system(200) to user.

Description

Data Cleansing System and Method for Developing Integrated Database, and Recording Medium Therefor}

The present invention relates to a data purification system and method for building an integrated database, and a recording medium therefor. More specifically, an error is detected by collecting data from a database distributed in various places to build an integrated database. The present invention relates to a data purification system and method for constructing an integrated database for resolving errors, and a recording medium therefor.

Conventionally developed commercial products or systems developed by coding programming languages for specific purposes only partially implement integrated database construction related technologies.

For example, certain software defines the type of error for the data, while another software focuses only on that feature on one system, such as implementing only the ability to transform and transmit the data, so that the contents of a distributed database There is no proposed technique for more systematic management.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems, and to collect a data of a distributed database for building an integrated database, to detect an error, and to implement a technique for refining a detected error on a system. There is a technical problem to provide a data purification system and method for building an integrated database and a recording medium therefor.

Data purification system of the present invention for achieving the above object is a communication interface for performing communication through a communication network; A data collector configured to collect data from a plurality of engine systems; A data error detector for detecting an error that does not correspond to a standard by comparing the data collected by the data collector with a preset detection rule; A data refiner for refining data according to a predetermined refinement rule with respect to the data detected by the data error detector; A database that stores information related to the data purification system; And a user interface for providing a user with information related to the data refining system.

The data purification system,

A data inspecting unit for converting the data purified by the data refining unit, calling a common service, or comparing the pre-transmission location data value with the post-transmission location data value to display updated data; It is desirable to.

The data purification system,

It is preferable that the data inspecting unit further comprises a data transmission unit for transmitting the data whose data inspection procedure is completed to a user terminal or a system in which an integrated database is constructed.

The data purification system,

When receiving a service request including a data search from the user terminal, it is preferable to further include a data management unit for outputting and providing information associated with the requested data through the user interface unit.

The data error detection unit,

Detection rule defining means for setting a detection rule including at least one of a data content criterion, a data structure criterion or a linkage criterion between tables; Data error checking means for checking whether there is a data value that does not match the detection rule by comparing the collected data with the detection rule; And data error detection means for detecting the error type after matching the corresponding error type with the data value checked because the data error check means does not match the detection rule.

The data content criterion includes a typo and a meta information criterion set to a meaningless value, and the data error checking means includes a data error that includes a typo in the collected data value or does not match the meta information criterion. It is desirable to check.

The data structure criteria include data format criteria,

The data error checking means,

It is preferable to check a data value of the collected data that does not match the data format criteria as an error.

The linkage criterion between tables is a criterion for confirming whether a primary key (PK) and a reference key (FK) are matched between linked tables.

The data error checking means,

It is recommended to check whether the primary key and the reference key specified in the base table and the side table of the collected data match, and if not, check the data value as an error.

The data refiner,

Refinement rule definition means for setting a refinement rule for refining the value of the data judged to be an error;

Automatic refining processing means for performing a refining process on the data value detected as an error by said data error detecting unit according to said refining rule; And

And a user refining processing means for outputting a data value for each error type through the user interface unit according to a user's request, and refining the data value with a value input by the user.

Another data purification method of the present invention comprises the steps of: a) the data purification system collecting data from a plurality of engine systems; b) comparing the collected data with a preset detection rule to detect an error that does not correspond to a standard; c) purifying the data according to a predetermined refining rule for the data value detected as an error, and refining the data to a value input by the user; And d) transmitting the data purified in step c) to a user terminal or a system in which an integrated database is constructed.

After step c) and before step d),

The data refining system converts the data refined in step c), calls a common service, or displays the updated data by comparing the position data value before transmission with the position data value after transmission. It is preferable to include.

After step d),

When receiving a service request including a data search from the user terminal, outputting and providing information related to the requested data; preferably.

B),

b-1) setting a detection rule including at least one of a data content criterion, a data structure criterion or a linkage criterion between tables; b-2) comparing the collected data with the detection rule and checking whether there is a data value that does not match the detection rule; b-3) matching the error type to the checked data value because it does not match the detection rule, and detecting the error as an error.

The data content criterion includes a typo and a meta information criterion set to a meaningless value.

In step b-2),

It is preferable to check a data value that contains a typo or a data value that does not match the meta information criteria as an error.

The data structure criterion includes a data format criterion, and in step b-2), it is preferable to check a data value that does not match the data format criterion among the values of the collected data as an error.

The linkage criterion between tables is a criterion for confirming whether a primary key (PK) and a reference key (FK) are matched between linked tables.

In step b-2),

It is recommended to check whether the primary key and the reference key specified in the base table and the side table of the collected data match, and if not, check the data value as an error.

C),

c-1) performing a purification process on the data value detected as an error according to a preset purification rule; c-2) outputting a data value detected as an error according to a user's request for each error type, and refining the data value to a value input by the user.

Before step c-1),

It is preferable to further include a; setting a purification rule for purifying the value of the data determined to be an error.

Yet another recording medium of the present invention comprises: a) a function of collecting data from a plurality of engine systems; b) comparing the collected data with a preset detection rule to detect an error that does not correspond to a standard; c) refining the data according to a predetermined refining rule for the data value detected as an error, and refining the data with the value input by the user; And d) a function of transmitting the data refined in step c) to a user terminal or a system in which an integrated database is built.

B) function,

b-1) setting a detection rule including at least one of a data content criterion, a data structure criterion or a linkage criterion between tables; b-2) comparing the collected data with the detection rule and checking whether there is a data value that does not match the detection rule; b-3) a function of matching a corresponding error type to a checked data value because it does not match the detection rule, and detecting the error as an error.

C) function above;

Setting a refinement rule for refining a value of data determined to be an error; Performing a refinement process on the data value detected as an error according to the refinement rule; And a function of outputting a data value detected as an error according to a user's request for each error type and purifying the data value to a value input by the user.

As described above, a data purification system and method for constructing an integrated database of the present invention and a recording medium therefor include a technique for detecting an error and refining the detected error by collecting data from a distributed database for constructing an integrated database. Since it is implemented on one system, it can be expected that the data stored in each of the plurality of databases can be managed more systematically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

First, FIG. 1 is a diagram showing a connection relationship between a data purification system and a communication network according to the present invention.

As shown, the data purification system 200 integrates databases distributed in a plurality of institutions, such as an institution A system 100, an institution B system, and an institution C system, respectively, to construct a single integrated database. To support this, perform procedures including error detection and purification of data collected from each institutional system. The institution includes all universities, corporations, the public, research institutions, and the like.

2 is a diagram illustrating a configuration of a data purification system according to the present invention.

As shown in the drawing, the data purification system 200 includes a communication interface 210, a data collector 220, a data error detector 230, a data refiner 240, a data inspector 250, and a data transmitter. 260, a data manager 270, a user interface 280, and a database 290.

In more detail, the communication interface 210 is a component for performing communication through a communication network.

The data collection unit 220 collects data from a plurality of engine systems (engine A system, engine B system, engine C system of FIG. 1) 100.

The data error detector 230 detects an error that does not correspond to a standard by comparing the data collected by the data collector 220 with a preset detection rule.

Here, data error means that the data is out of standard. The detection rule for determining the standard of the data is composed of the content and structural aspects, so that the data whose contents are different from the standard data is an error, and the data that is out of the data format or the standard code value is determined as an error.

In addition, the data value detected as the error data value by the data error detection unit 230 is displayed (parts that are determined to be an error) to be displayed for future reference.

The data refiner 240 refines the data according to a preset refinement rule for the data detected by the data error detector 230.

Here, data refinement refers to a process of improving quality by determining inaccurate, incomplete or illogical data and correcting errors and omissions.

The data inspecting unit 250 converts the data purified by the data purifying unit 240, calls a common service, or compares the position data value before transmission with the position data value after transmission and updates the updated data. Display.

Here, the common service call means to call a separate system (not shown) which assigns an identification number so that the data can be identified for each data, thereby identifying the identification number of the purified data. That is, the purified data transmitted through the data transmitter 260 may include an identification number received through a common service call.

When the data input to the data refining system 200 is updated with existing data, the data value of the position before transmission and the data value of the position after transmission are compared to display the updated data. As a result of the comparison, if there is an updated part, the data inspecting unit 250 displays the corresponding part in table units or record units.

The data transmitter 260 transmits the data whose data inspection procedure is completed by the data inspector 250 to a user terminal (not shown) or a system (not shown) in which an integrated database is constructed.

Here, data transmission means passing data from one repository to another. The data transmission disclosed in the present invention does not mean the movement of bits composed of <0> and <1>, but rather the movement of data stored in a database. During data transmission, only the location where the same data is stored may be changed, and at the same time, the data may be changed.

When the data manager 270 receives a service request including a data search from the user terminal, the data manager 270 outputs and provides information related to the requested data through the user interface 280.

The user interface unit 280 is a component for providing a user to check information related to the data purification system.

Database 290 stores information related to data purification system 200.

3 is a diagram illustrating a configuration of a data error detection unit according to the present invention.

As shown, the data error detecting unit 230 includes a detection rule defining means 231, a data error checking means 233, and a data error detecting means 235.

In more detail, the detection rule defining means 231 sets a detection rule including at least one of a data content criterion, a data structure criterion, or a linkage criterion between tables.

Here, the data content criterion includes a typo and a meta information criterion set to a meaningless value, the data structure criterion includes a data format criterion, and the linkage criterion between tables refers to a primary key (PK) associated with the link. It includes a criterion for checking whether a key (FK: foreign key) matches.

The data error checking means 233 compares the collected data with the detection rule and checks whether there is a data value that does not match the detection rule.

In more detail, the data error checking means 233 checks for a data value that contains a typo or does not match the meta information criteria as an error among the values of the collected data.

For example, the data error checking means 233 may have meaningless <----->, <*****>, <......>, <$% @ ^ @@ ^ Check values such as @ & @% # $ @ # $ @>.

If the research manager is required to exist in the meta information of the research project, the data error check means 233 checks whether or not there is one A value corresponding to the research director code in the data value of the corresponding column.

In addition, the data error checking means 233 checks the data value of the collected data that does not match the data format criteria as an error.

For example, when the standard of the date format among the data format criteria is set in the form of YYYY-MM-DD, the data error checking means 233 may actually use various types of date formats (collected or input into the data refining system 200). <20081208>, <081208>, <December 8>, etc.) Check the data value that does not match the YYYY-MM-DD type.

In addition, the data error checking means 233 checks whether the primary key and the reference key specified in the base table and the side table of the collected data match, and if not, checks the data value as an error.

In more detail, the fact that two tables are linked can be determined by a link linking the two tables, in which the primary key and the reference key serve as the link.

For example, the task information is composed of a plurality of other tables (additional tables) related to the task basic information table (basic table) (business information table, research fund table, participating researcher table, etc.) It is associated with a plurality of related other tables. In this case, the column that is the primary key is assigned to the task basic information table, and the column that is the reference key is assigned to a plurality of other tables related to the task, so that records existing in each of the plurality of other tables related to the task are associated with a task. To make sure that you can search.

The task information refers to the tasks classified in detail for the promotion of national R & D projects. This task includes meta-information that describes the task, such as the task name, budget, task implementation agency, task manager, and so on.

On the other hand, the task information disclosed in the present invention shows an example of data, and includes all other types of data in addition to the task information.

The data error detecting means 235 matches the corresponding error type to the data value checked because the data error checking means 233 does not match the detection rule, and then detects it as an error.

When detecting a data error disclosed in FIG. 3, a data type or a null value condition defined in a SQL query statement or a column of a database may be applied.

4 is a diagram illustrating a configuration of a data purification unit according to the present invention.

As illustrated, the data refiner 240 includes a refinement rule defining means 241, an automatic refinement processing means 243, and a user refinement processing means 245.

In more detail, the refinement rule defining means 241 sets a refinement rule for refining the value of the data determined to be an error.

The automatic refining processing means 243 performs the refining process on the data value detected as an error by the data error detecting unit 230 according to the refining rule.

For example, the automatic refining processing means 243 purifies the date type values, such as <20081208>, into a type such as <2008-12-08>, by inserting a hyphen in the middle. In addition, Null column is basically included in the number 1 to refine.

The user refining processing unit 245 outputs a data value for each error type through the user interface unit 280 according to a user's request, and refines the data value to a value input by the user.

For example, when a specific request is input from the user, the user congestion processing unit 245 outputs a data value detected as an error through the user interface unit 280 according to a predefined error type, and the user inputs / The data is refined to the value entered by checking through the modified user interface screen.

5 is a flowchart illustrating a data purification method according to the present invention.

First, the data collection unit 220 of the data purification system 200 collects data from a plurality of engine systems (engine A system, engine B system, engine C system of FIG. 1) (S101).

The data error detector 230 detects an error that does not correspond to a standard by comparing the collected data with a preset detection rule (S103).

The data refiner 240 refines the data according to a predetermined refinement rule with respect to the data value detected as an error, and refines the data to a value input by the user (S105).

The data inspecting unit 250 converts the purified data in step S105, calls a common service, or compares the position data value before transmission with the position data value after transmission and displays the updated data (S107). .

The data transmission unit 260 transmits the purified and inspected data through the steps S105 and S107 to the user terminal or the system in which the integrated database is established (S109).

If the data inspection procedure is omitted according to the needs of the operator, the data transmission unit 260 transmits the purified data to the user terminal or the system in which the integrated database is established through step S105.

Although not shown, after receiving a service request including data search from the user terminal, the data manager 270 outputs and provides information related to the requested data.

6 is a flowchart illustrating a data error detection method according to the present invention in more detail, and step S103 of FIG. 5 will be described in more detail.

First, the detection rule defining means 231 of the data error detection unit 230 sets a detection rule including at least one of a data content criterion, a data structure criterion, and a linkage criterion between tables (S201).

Then, the data error checking means 233 compares the collected data with the detection rule and checks whether there is a data value that does not match the detection rule (S203).

In more detail, the data content criterion includes a typo and a meta information criterion set to a meaningless value, and the data error checking means 233 includes a typo among values of the collected data or does not match the meta information criterion. Check the data value as an error.

In addition, the data structure criterion includes a data format criterion, and the data error checking means 233 checks a data value which does not match the data format criterion among the collected data values as an error.

In addition, the linkage criterion between tables is a criterion for confirming whether a primary key (PK) and a reference key (FK) are matched with each other. Check whether the primary key and the reference key specified in the base table and the side table match, and if not, check the data value as an error.

Thereafter, the data error detecting means 235 matches the corresponding error type to the checked data value because it does not match the detection rule, and detects it as an error (S205).

7 is a flowchart illustrating the data purification method according to the present invention in more detail.

First, although not shown, the refining rule defining means 241 of the data refining unit 240 sets a refining rule for refining the value of the data determined to be an error.

Subsequently, the automatic congestion processing means 243 performs the refining process on the data value detected as an error according to the refining rule (S301).

The user refining processing means 245 outputs the data value detected as an error according to the user's request for each error type, and refines the data value to the value input by the user (S303 to S307).

The a), b), c), etc. described in the claims to be described later do not mean a specific order, and the order may be changed as required by the operator.

In addition, the configuration and function of the data purification system disclosed in the present invention can be implemented in the form of a computer-readable recording medium recording a program.

As such, those skilled in the art will appreciate that the present invention can be implemented in other specific forms without changing the technical spirit or essential features thereof. Therefore, the embodiments described above are to be understood in all respects as illustrative and not restrictive. The scope of the present invention is shown by the following claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present invention. do.

1 is a diagram showing a connection relationship between a data purification system and a communication network according to the present invention;

2 is a diagram showing the configuration of a data purification system according to the present invention;

3 is a view showing the configuration of a data error detection unit according to the present invention;

4 is a view showing the configuration of a data purification unit according to the present invention;

5 is a flowchart illustrating a data purification method according to the present invention;

6 is a flowchart illustrating a data error detection method according to the present invention in more detail.

7 is a flowchart illustrating the data purification method according to the present invention in more detail.

<Explanation of symbols for the main parts of the drawings>

100: institutional system 200: data purification system

210: communication interface unit 220: data collection unit

230: data error detection unit 231: detection rule definition means

233: data error check means 235: data error detection means

240: data refining unit 241: refining rule definition means

243: automatic tablet processing means 245: user tablet processing means

250: data inspecting unit 260: data transmitting unit

270: data management unit 280: user interface unit

290: database

Claims (21)

A communication interface unit for performing communication through a communication network; A data collector configured to collect data from a plurality of engine systems; A data error detector for detecting an error that does not correspond to a standard by comparing the data collected by the data collector with a preset detection rule; A data refiner for refining data according to a predetermined refinement rule with respect to the data detected by the data error detector; A database that stores information related to the data purification system; And A user interface for providing a user with information related to the data refining system; Data purification system for building an integrated database comprising a. The method of claim 1, The data purification system, A data inspecting unit converting the data purified by the data refining unit, calling a common service, or comparing the pre-transmission position data value with the post-transmission position data value to display updated data; Data purification system for building an integrated database, further comprising. The method of claim 2, The data purification system, A data transmission unit for transmitting the data whose data inspection procedure is completed by the data inspection unit to a user terminal or a system in which an integrated database is constructed; Data purification system for building an integrated database, further comprising. The method of claim 1, The data purification system, A data management unit that outputs and provides information related to the requested data through the user interface unit when receiving a service request including a data search from a user terminal; Data purification system for building an integrated database, further comprising. The method of claim 1, The data error detection unit, Detection rule defining means for setting a detection rule including at least one of a data content criterion, a data structure criterion or a linkage criterion between tables; Data error checking means for checking whether there is a data value that does not match the detection rule by comparing the collected data with the detection rule; And Data error detection means for detecting the error type after matching the corresponding error type with the data value checked because the data error check means does not match the detection rule; Data purification system for building an integrated database comprising a. The method of claim 5, The data content criterion includes a typo and a meta information criterion set to a meaningless value. The data error checking means, The data purification system for building an integrated database, characterized in that errors are checked for data values that contain typos or do not match the meta-information criteria among the collected data values. The method of claim 5, The data structure criteria include data format criteria, The data error checking means, The data purification system for building an integrated database, characterized in that for checking the data value that does not match the data format criteria of the collected data as an error. The method of claim 5, The linkage criterion between tables is a criterion for confirming whether a primary key (PK) and a reference key (FK) are matched between linked tables. The data error checking means, A data purification system for building an integrated database, comprising checking whether a primary key and a reference key specified in the base table and the side table of collected data match, and checking the corresponding data value as an error if they do not match. The method of claim 1, The data refiner, Refinement rule definition means for setting a refinement rule for refining the value of the data judged to be an error; Automatic refining processing means for performing a refining process on the data value detected as an error by said data error detecting unit according to said refining rule; And User purification processing means for outputting a data value for each error type through the user interface unit according to a user's request, and for refining the data value to a value input by the user; Data purification system for building an integrated database comprising a. a) the data purification system collecting data from the plurality of institutional systems; b) comparing the collected data with a preset detection rule to detect an error that does not correspond to a standard; c) purifying the data according to a predetermined refining rule for the data value detected as an error, and refining the data to a value input by the user; And d) transmitting the data purified in step c) to a user terminal or a system in which an integrated database is constructed; Data purification method for building an integrated database comprising a. The method of claim 10, After step c) and before step d), The data refining system converting the data purified in step c), calling a common service, or comparing the position data value before transmission with the position data value after displaying the updated data; Data purification method for building an integrated database, characterized in that it further comprises. The method of claim 10, After step d), Outputting and providing information related to the requested data when receiving a service request including a data search from a user terminal; Data purification method for building an integrated database, characterized in that it further comprises. The method of claim 10, B), b-1) setting a detection rule including at least one of a data content criterion, a data structure criterion or a linkage criterion between tables; b-2) comparing the collected data with the detection rule and checking whether there is a data value that does not match the detection rule; b-3) matching a corresponding error type to a checked data value that does not match the detection rule, and then detecting it as an error; Data purification method for building an integrated database comprising a. The method of claim 13, The data content criterion includes a typo and a meta information criterion set to a meaningless value. In step b-2), The data purification method for building an integrated database, which includes a typo in the collected data value or checks a data value that does not match the meta information standard as an error. The method of claim 13, The data structure criteria include data format criteria, In step b-2), The data purification method for building an integrated database, characterized in that for checking the data value that does not match the data format criteria of the collected data as an error. The method of claim 13, The linkage criterion between tables is a criterion for confirming whether a primary key (PK) and a reference key (FK) are matched between linked tables. In step b-2), A method of refining data for building a unified database, comprising checking whether a primary key and a reference key specified in the collected data base tables and side tables match each other, and checking the corresponding data values as errors. The method of claim 13, C), c-1) performing a purification process on the data value detected as an error according to a preset purification rule; c-2) outputting a data value detected as an error according to a user's request for each error type, and refining the data value to a value input by the user; Data purification method for building an integrated database comprising a. The method of claim 17, Before step c-1), Setting a refinement rule to refine the value of the data determined to be an error; Data purification method for building an integrated database, characterized in that it further comprises. a) collecting data from a plurality of institutional systems; b) comparing the collected data with a preset detection rule to detect an error that does not correspond to a standard; c) refining the data according to a predetermined refining rule for the data value detected as an error, and refining the data with the value input by the user; And d) transmitting the data refined in step c) to a user terminal or a system for establishing an integrated database; Computer-readable recording medium recording a program comprising a. The method of claim 19, B) function, b-1) setting a detection rule including at least one of a data content criterion, a data structure criterion or a linkage criterion between tables; b-2) comparing the collected data with the detection rule and checking whether there is a data value that does not match the detection rule; b-3) a function of matching an error type with a checked data value that does not match the detection rule and then detecting it as an error; Computer-readable recording medium recording a program comprising a. The method of claim 19, C) function above; Setting a refinement rule for refining a value of data determined to be an error; Performing a refinement process on the data value detected as an error according to the refinement rule; Outputting a data value detected as an error for each error type according to a user's request, and refining the data value to a value input by the user; Computer-readable recording medium recording a program comprising a.

Images