KR101964454B1 - Apparatus and method for refining data by eliminating problems that is inherent in data - Google Patents

Abstract

According to an embodiment of the present invention, there is provided an information processing apparatus including: a data acquisition unit for reading data; A test unit for detecting a problem inherent in the data by testing the data; A data problem display unit for displaying the detected problem to a user; And a data processing unit for performing post-processing of the data, wherein the test is performed by rules found in the data itself through analysis of the data.

Description

TECHNICAL FIELD [0001] The present invention relates to an apparatus and method for data refinement by eliminating problems inherent in data,

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus and method for purifying data, and more particularly, to an apparatus and method for purifying data with a test performed by a rule found in the data itself through analysis of data.

As the Internet is becoming more popular in countries around the world and the importance of the data delivered to them is becoming increasingly important, a variety of technologies are being developed rapidly to better utilize data.

However, depending on the source of the data, it is often the case that various kinds of error values are included in the data group, the data in different formats are included, or structural problems are inherent in many cases.

Data groups with these problems need to be refined prior to use.

However, since the conventional data refinement techniques use a method of detecting an error value through comparison of a data reference or a standard set in advance with an input data group, the kinds of data groups that can be refined are limited . In other words, it was possible to perform refinement only for data groups of a type that can correspond to a preset data criterion or a standard.

Therefore, there is a need for a data refinement technique that can detect problems inherent in a data group in a new way and resolve them, regardless of the type and format of the data group.

Korean Patent Publication No. 2001-0041449 (May 25, 2001)

It is an object of the present invention to provide an apparatus and method capable of refining data with a test performed by a rule found in the data itself through analysis of the data .

In order to accomplish the above object, one aspect of the present invention provides a data acquisition unit for reading data; A test unit for detecting a problem inherent in the data by testing the data; A data problem display unit for displaying the detected problem to a user; And a data processing unit for performing post-processing of the data, wherein the test is performed by rules found in the data itself through analysis of the data.

In one embodiment, the test unit includes a data value error determination module, wherein the rule performed by the error value determination module of the data value is configured to set a region in which the test is performed in the data, A field type of the entire internal cells is recognized, a field type of the minimum ratio is selected from among the field types, and data inputted to a cell having the field type of the minimum ratio is classified as an error value, Lt; / RTI >

In one embodiment, the test unit includes a data format consistency determination module, wherein the rules performed by the consistency determination module of the data format are configured such that one area in which the test is performed in the data is set, Wherein the format of the entire internal cells is recognized and data entered into the cell having a format different from the format of the maximum rate is selected as the problem in the format of the maximum rate format selected .

In one embodiment, the test unit includes a data structure problem determination module, wherein the rule performed by the data structure problem determination module is configured such that one region in which the test is performed in the data is set, A discontinuous blank area recognized in the data is detected as the problem.

In one embodiment, the test unit includes a data copy module, a data value error determination module, a data format consistency determination module, and a data structure problem determination module. The data copy module generates a plurality of data copies, Wherein the data copy is allocated to each of a data value error determination module, a data format consistency determination module, and a data structure problem determination module, and the data determination module, the data format consistency determination module, And a test is performed.

In one embodiment, the rule performed by the error determination module of the data value may include an area where a test is performed in the data, a field type of the entire one of the inner cells is recognized, The data type of the minimum rate field type is selected and the data input to the cell having the field type of the minimum rate is classified as an error value to thereby detect the problem. A rule is set such that one area in which the test is performed in the data is set, a format of all the cells in one area is recognized, a maximum ratio format is selected from the format, and a format different from the maximum ratio format The data input to the cell is detected as the problem, and the data structure problem determination module The rules that line, the one area, the test is performed on the data are set, a discontinuous free area is recognized from the work area inside the cells may be in the data cleaning device it will be detected by the above problems.

In one embodiment, the post-processing further comprises deriving a method for eliminating the problem, providing the user with the feedback, and receiving the feedback from the user to at least partially reflect the feedback to eliminate the problem. Device.

In one embodiment, the post-processing derives a method for eliminating the problem and removes the problem by the method and pre-stored rules to provide the final data to the user. .

In one embodiment, the post-processing may be a data refining device that selects a field type with a maximum ratio among the field types and replaces the error value with a specific value having the field type of the maximum rate .

In one embodiment, the post-processing may be a data refining device that changes the detected data format to the maximum rate format.

In one embodiment, the post-processing may be a data cleansing device that removes the discontinuous blank areas detected as the problem.

In order to accomplish the above object, another aspect of the present invention provides a method of manufacturing a semiconductor device, comprising: a) obtaining data for reading data; b) a data copying step of generating a plurality of copies of data; c) analyzing a field type of the data cell with respect to one area by receiving the data copy, classifying the data input into the cell having the minimum ratio field type as an error value, ; d) analyzing a format of a data cell with respect to one area, and detecting data inputted to a cell having a format different from a maximum ratio format as a problem; e) a data structure problem determination step of detecting a discontinuous blank area as a problem for one area; f) displaying a data problem for displaying the problem to a user in a visual manner; g) deriving a data problem removal method for deriving a method for eliminating the problem; And h) providing the user with a method for removing the data problem derived from the data.

In one embodiment, after step h), the method may further comprise: i) removing the data problem by obtaining and at least partially reflecting the feedback of the user.

In one embodiment, instead of the step h), h ') removing the data problem by a pre-stored rule and providing the final data to the user.

According to an aspect of the present invention, since the test based on rules found in the data itself is performed through analysis of data to detect a problem inherent in the data, the data can be refined regardless of the type and format of the data have.

According to another aspect of the present invention, field type errors inherent in data, inconsistency in format consistency, and structure problems can be detected and removed by rules found in the data itself.

According to another aspect of the present invention, a field type error inherent in data, an inconsistency in format consistency, and a structure problem can be simultaneously detected and displayed or removed in a visually recognizable manner by a user.

It should be understood that the effects of the present invention are not limited to the above effects and include all effects that can be deduced from the detailed description of the present invention or the configuration of the invention described in the claims.

1 is a view showing a data purifying apparatus according to an embodiment of the present invention.
2 is a flowchart for explaining a data refinement method according to an embodiment of the present invention.
3 is a flowchart for explaining a data refining method according to another embodiment of the present invention.
4 is a flowchart for explaining a data refinement method according to another embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described with reference to the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification, when a part is referred to as being "connected" to another part, it includes not only "directly connected" but also "indirectly connected" . Also, when an element is referred to as "comprising ", it means that it can include other elements, not excluding other elements unless specifically stated otherwise.

Prior to explanation, the meaning of terms used in this specification will be briefly described. It should be noted that the description of the term is provided for the purpose of helping understanding of the specification and is not used to limit the technical idea of the present invention unless explicitly stated as a limitation of the present invention.

- Cell

Is a basic area into which data is input in a spreadsheet program or the like and is a rectangular area formed by mutual intersection of rows and columns which are horizontal and vertical grid shapes constituting a basic form of spreadsheet.

- Field

Means a range of fixed positions in a set of data such as a computer command. For example, it may refer to a designated area used for a certain kind of data in one record.

- Format

It is also referred to as a data format, and is a rule regarding the form of storing or inputting / outputting data. For example, to indicate how many bytes of data are treated as real or integer in a file or record.

- Computer

It is possible to implement an engine that performs the data purification described in the detailed description of the present invention. In one embodiment, the computer obtains data from a local or remote storage device, such as a memory or server, the Internet, or the like. In another embodiment, data is generated in a computer using an application program running on the computer.

The computer may be configured to receive input or feedback from a user or other entity and apply it. Feedback can be interpreted in the computer to understand the preferences of the user or other entity for the output. Input or feedback may be obtained from a response to one or more prompts that may be obtained directly from a user or other entity and / or generated by a computer. Based on this input or feedback, the computer can extract the final data for which the purification has been completed. In addition, when there are a plurality of methods for eliminating the problems for later use, the computer can weight each method and store it in a memory.

Although the subject matter described herein is presented in the general context of program modules that are implemented in connection with the execution of an operating system and an application program on a computer system, one of ordinary skill in the art will understand that other implementations It will be appreciated that the examples may be performed. Generally, program modules include routines, programs, components, data structures, and other types of structures that perform particular tasks or implement particular abstract data types. Further, those skilled in the art will appreciate that the subject matter described herein may be practiced in other computer systems, including hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, It will be understood that the invention may be practiced in various other forms.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a view showing a data purifying apparatus according to an embodiment of the present invention.

Referring to FIG. 1, a data refining apparatus 1000 according to an embodiment of the present invention includes a data acquiring unit 100, a testing unit 200, a data problem display unit 300, and a data processing unit 400.

The data acquiring unit 100 can read data from outside the data refining apparatus 1000.

By testing the data, the test unit 200 can detect problems inherent in the data.

Tests can be performed by rules found in the data itself through analysis of the data. Thus, the data can be refined regardless of the type and format of the data.

The data problem display unit 300 can display the problem detected through the test to the user. A problem display method may be a method of changing the color of a data area in which a problem exists or a method of displaying a square in a cell in the case of data stored in units of cells. However, the present invention is not limited thereto, and it is possible to use a visual method.

The data processing unit 400 may perform post-processing of data. The post-processing can derive a method for eliminating the problem detected through the test and provide it to the user.

Further, the present invention is not limited to this, and may be performed in a manner of receiving the feedback from the user and at least partly reflecting the feedback to eliminate the problem.

Meanwhile, when a plurality of methods for eliminating a problem are derived, each method may be weighted and ranked, and the method may be provided to the user in a state in which the methods and ranking are matched.

The test unit 200 according to an exemplary embodiment of the present invention includes a data value error determination module 202.

The error judgment module 202 of the data value can detect the rule in the data itself through the analysis of the data and perform the test based on the rule.

A rule to be performed by the error judgment module 202 of the data value is such that one area in which data is to be tested is set, a field type of all the cells in one area is recognized, and among the recognized field types, A field type is selected, and data input to a cell having the field type of the minimum ratio is classified as an error value, thereby being detected as a problem.

Thereby, the test can be performed with rules found in the data itself through data cell analysis without contrasting the data with the preset reference or standard.

Therefore, the data can be refined by classifying the data cell having the erroneous field type irrespective of the type and format of the data.

The test unit 200 includes a data format consistency determination module 203 according to an embodiment of the present invention.

The data format consistency determination module 203 also analyzes the data to find rules in the data itself and can perform tests based on the rules.

The rules performed by the consistency determination module 203 of the data format are such that one area in which data is tested is set, the format of all the cells in one area is recognized, and the format of the maximum ratio is selected from among the formats , Data input to a cell having a format different from the maximum rate format may be detected as the problem.

This allows the test to be performed with rules found in the data itself through data cell analysis, without having to contrast the format of the input data with the preset reference or standard format.

Therefore, the test can be performed, and a problem can be detected, without considering the case where the input data includes a format incapable of being contrasted.

The test unit 200 according to an exemplary embodiment of the present invention includes a data structure problem determination module 204.

The data structure problem determination module 204 can also detect the rules in the data itself through analysis of the data and perform tests based on the rules.

A rule performed by the data structure problem determination module 204 may be one in which a test is performed on data, and a discontinuous blank area recognized in the one-area internal cells may be detected as the problem.

As a result, data structure problems can be detected and detected as problems through testing on the data itself.

Thus, the test can be performed on data for which an unexpected structural problem exists, and a problem can be detected.

The test unit 200 includes a data copy module 201, a data value error determination module 202, a data format consistency determination module 203, and a data structure problem determination module 204. [ .

A plurality of data copies may be generated by the data copy module 201 and the data copies may include a data value error determination module 202, a data format consistency determination module 203, and a data structure problem determination module 204. [ Respectively.

In this case, a test according to each module can be simultaneously performed by the error determination module 202, the data format consistency determination module 203, and the data structure problem determination module 204 at the same time.

Thus, problems anticipated in the data can be detected simultaneously and quickly.

Then, all the problems detected from the data are displayed on the data problem display unit 300, so that the user can recognize the problems on the data at first sight.

The rule performed by the error judgment module 202 of the data value is such that one field in which data is tested in the data is set, a field type of all the cells in one area is recognized, and a minimum ratio And the data input to the cell having the field type of the minimum ratio is classified as an error value, thereby being detected as the problem.

The rules performed by the consistency determination module 203 of the data format are set such that one area in which data is to be tested is set, a format of all the cells in one area is recognized, And the data input to the cell having the format different from the format of the maximum rate may be detected as the problem.

The rule performed by the data structure problem determination module 204 may be such that one area in which data is tested is set in the data and a discontinuous blank area recognized in the one area internal cells is detected as the problem .

The post-processing performed in the data processing unit 400 according to another embodiment of the present invention can be achieved by deriving a method for eliminating the problem detected through testing and eliminating the problem by the method and pre-stored rules, It may be to provide to the user.

This allows the user to obtain refined data that can be utilized without further processing.

The post-processing performed by the data processing unit 400 according to an exemplary embodiment of the present invention selects a field type of the maximum ratio among the field types of the data cells and outputs the error detected by the error determination module 202 Value with a specific value having the field type of the maximum rate.

Thereby, a data cell having a field type error value can be refined.

The post-processing performed in the data processing unit 400 according to the embodiment of the present invention may be to change the data format detected as a problem through the test to the maximum ratio format determined according to the test.

This allows the data lacking format consistency to be refined.

The post-processing performed in the data processing unit 400 according to an embodiment of the present invention may be to remove a discontinuous blank area detected as a problem through testing.

In this way, data in which structural problems exist can be refined.

2 is a flowchart for explaining a data refinement method according to an embodiment of the present invention.

Referring to FIG. 2, a data refining method according to an embodiment of the present invention includes: a) data acquisition step (S101a) of reading data; b) a data copying step (S102a) of generating a plurality of copies of data; c) analyzing a field type of the data cell with respect to one area by receiving the data copy, classifying the data input into the cell having the minimum ratio field type as an error value, (S201a); d) determining a consistency of data format (S202a) for analyzing a format of a data cell with respect to one area and detecting data inputted to a cell having a format different from the maximum rate format; e) a data structure problem determination step (S203a) for detecting a discontinuous blank area as a problem for one area; f) displaying a data problem (S300a) for displaying the problem to a user in a visual manner; g) deriving a data problem removal method (S401a) for deriving a method for removing the problem; And h) providing the method of removing the data problem derived by the user (S402a).

The steps c) to e) may be performed in a reversed order, preferably simultaneously.

If the steps c) to e) are carried out simultaneously, the data can be rapidly purified.

3 is a flowchart for explaining a data refining method according to another embodiment of the present invention.

Referring to FIG. 3, the data refinement method according to another embodiment of the present invention may further comprise: i) after the step h), removing the data problem by obtaining and at least partially reflecting feedback of the user (S403b ).

Thus, refined data can be produced that more closely matches the area the user wants to utilize.

4 is a flowchart for explaining a data refinement method according to another embodiment of the present invention.

Referring to FIG. 4, a data refining method according to another embodiment of the present invention includes the steps of: h ') replacing the step h), removing the data problem by a pre-stored rule, and providing final data to a user (S402c).

This allows the production of refined data that can be utilized by the user directly without further processing without user intervention.

It will be understood by those skilled in the art that the foregoing description of the present invention is for illustrative purposes only and that those of ordinary skill in the art can readily understand that various changes and modifications may be made without departing from the spirit or essential characteristics of the present invention. will be. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is defined by the appended claims, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included within the scope of the present invention.

100: data acquisition unit 200:
201: data copy module 202: error judgment module of data value
203: Data format consistency determination module
204: Data structure problem determination module
300: Data problem indicator
400:

Claims (14)

A data acquisition unit for reading data;
A test unit for detecting a problem inherent in the data by testing the data;
A data problem display unit for displaying the detected problem to a user; And
And a data processing unit for performing post-processing of the data,
The test is performed by rules found in the data itself through analysis of the data,
Wherein the test unit includes a data value error determination module,
Wherein the rule performed by the error determination module of the data value comprises:
One area in which the test is performed in the data is set,
The field type of the entire one-area internal cells is recognized,
Among the field types, a minimum ratio field type is selected,
Wherein data input to a cell having the field type of the minimum ratio is classified as an error value, thereby being detected as the problem. A data acquisition unit for reading data;
A test unit for detecting a problem inherent in the data by testing the data;
A data problem display unit for displaying the detected problem to a user; And
And a data processing unit for performing post-processing of the data,
The test is performed by rules found in the data itself through analysis of the data,
Wherein the test unit includes a data format consistency determination module,
Wherein the rules performed by the consistency determination module of the data format include:
One area in which the test is performed in the data is set,
The format of the entire internal cells of one area is recognized,
Among the above formats, a maximum ratio format is selected,
Wherein data entered into a cell having a format different from the maximum ratio format is detected as the problem. A data acquisition unit for reading data;
A test unit for detecting a problem inherent in the data by testing the data;
A data problem display unit for displaying the detected problem to a user; And
And a data processing unit for performing post-processing of the data,
The test is performed by rules found in the data itself through analysis of the data,
Wherein the testing unit includes a data structure problem determination module,
Wherein the rules performed by the data structure problem determination module include:
One area in which the test is performed in the data is set,
Wherein discontinuous blank areas recognized in the one area internal cells are detected as the problem. A data acquisition unit for reading data;
A test unit for detecting a problem inherent in the data by testing the data;
A data problem display unit for displaying the detected problem to a user; And
And a data processing unit for performing post-processing of the data,
The test is performed by rules found in the data itself through analysis of the data,
Wherein the test unit includes a data copy module, a data value error determination module, a data format consistency determination module, and a data structure problem determination module,
A plurality of data copies are generated by the data copy module,
Wherein the data copy is allocated to each of a data value error determination module, a data format consistency determination module, and a data structure problem determination module,
Wherein the test is simultaneously performed in the error determination module of the data value, the consistency determination module of the data format, and the data structure problem determination module. 5. The method of claim 4,
Wherein the rule is performed by the error judgment module of the data value, wherein one field to be tested is set in the data, a field type of the entire one of the internal cells is recognized, Type is selected and data input to a cell having the field type of the minimum ratio is classified as an error value,
The rules performed by the consistency determination module of the data format are configured such that one area in which the test is performed in the data is set, a format of all the cells in one area is recognized, and a maximum ratio format is selected Data inputted to a cell having a format different from that of the maximum rate is detected as the problem,
Wherein the rules performed by the data structure problem determination module are such that one area for testing is set in the data and a discontinuous blank area recognized in the one area internal cells is detected as the problem. . 6. The method according to any one of claims 1 to 5,
The post-
A method for eliminating the above problem is derived and provided to a user,
And receives the feedback from the user and at least partly reflects the feedback to eliminate the problem. 6. The method according to any one of claims 1 to 5,
The post-
A method for eliminating the above problem is derived,
And said final data is provided to a user by eliminating said problem by said method and pre-stored rules. The method according to claim 1,
The post-
Selecting a field type having a maximum ratio among the field types,
And replaces the error value with a specific value having the field type of the maximum rate. 3. The method of claim 2,
The post-
And changes the detected data format to the maximum ratio format. The method of claim 3,
The post-
And removes the discontinuous blank area detected as the problem. a) a data acquiring step of reading data;
b) a data copying step of generating a plurality of copies of data;
c) analyzing a field type of the data cell with respect to one area by receiving the data copy, classifying the data input into the cell having the minimum ratio field type as an error value, ;
d) analyzing a format of a data cell with respect to one area, and detecting data inputted to a cell having a format different from a maximum ratio format as a problem;
e) a data structure problem determination step of detecting a discontinuous blank area as a problem for one area;
f) displaying a data problem for displaying the problem to a user in a visual manner;
g) deriving a data problem removal method for deriving a method for eliminating the problem; And
h) providing the user with a method for removing the data problem derived from the data. 12. The method of claim 11,
After the step h)
i) removing the data problem by obtaining and at least partially reflecting the feedback of the user. 12. The method of claim 11,
Instead of the step h)
h ') removing the data problem by pre-stored rules, and providing final data to the user. delete

Images