KR102262279B1 - Data processing for detecting outlier method and device thereof - Google Patents

Abstract

According to an embodiment of the present invention, a data processing method includes the steps of acquiring data to be collected, performing pre-processing on the data to be collected, and detecting normal values, outliers and missing values, the detected normal values, outliers, and Visualizing the missing value through an interface and providing metadata information that is a criterion for detecting the outlier through the interface in response to a user's request.

Description

DATA PROCESSING FOR DETECTING OUTLIER METHOD AND DEVICE THEREOF

The present invention relates to the field of computer information processing, and to a data processing method, apparatus and system for detecting outliers or missing values.

The content described below merely provides background information related to an embodiment according to the present invention and does not constitute the prior art.

In modern society, data has penetrated into various businesses and has become an important factor, and the era of big data has arrived. The explosive growth of large-capacity and diverse big data requires the data processing and analysis capabilities of modern companies, and provides opportunities for companies to accurately predict the future. Companies actively used data to find ways to build their own business strategies and make tactical decisions, and the concept of business intelligence (BI) was born. The most important thing in modern society is to process and analyze such big data, and only through processing and analysis of big data can intelligent and valuable information be obtained.

With the development of big data analysis technology, augmented analysis technology is emerging. It refers to predicting what will happen in the future by grafting artificial intelligence machine learning technology to existing business intelligence, and AI finds and presents optimized alternatives accordingly. In other words, augmented analysis automatically processes a series of processes from data preprocessing to feature engineering by machine learning rather than human judgment, and can perform predictive analysis by finding larger amounts of data and hidden patterns.

In order to use the augmented analysis technology in this way, it is necessary to supplement the outliers and missing values in the data preprocessing process. At this time, in order to accurately supplement the metadata, it is necessary to check the metadata, which is the standard for detecting outliers, and to check whether an appropriate command is included. However, it is not easy to check and correct such metadata information.

The problem to be solved by the present invention, conceived in this regard, can be analyzed by processing and processing big data that can refine/select data to enable augmented analysis from large-capacity raw and unstructured data produced in the field. An algorithm is needed.

In addition, it is necessary to provide a user interface so that metadata information, which is a standard for detecting outliers or missing values, can be checked and corrected in the data preprocessing process.

According to an embodiment of the present invention, a data processing method includes the steps of acquiring data to be collected, performing pre-processing on the data to be collected, and detecting normal values, outliers and missing values, the detected normal values, outliers, and The method may include visualizing the missing value through an interface and providing metadata information, which is a criterion for detecting the outlier, through the interface in response to a user's request.

According to an embodiment, the metadata information may be index information including at least one of a data dictionary and a regular expression.

According to an embodiment, the interface may provide a search window so that the user can search for the index information.

According to an embodiment, the index information may be added/deleted/modified in response to the user's command through the interface.

According to an embodiment of the present invention, a data processing apparatus includes a data collection unit for acquiring data to be collected, a data preprocessor for performing preprocessing on the data to be collected, normal values, outliers, and missing values of the preprocessed data It may include a data quality control unit for detecting and an interface providing unit for visualizing the detected normal values, outliers and missing values through an interface.

According to an embodiment, the interface providing unit may provide metadata information that is a criterion for detecting the outlier in response to a user's request.

According to an embodiment, the metadata information may be index information including at least one of a data dictionary and a regular expression.

According to an embodiment, the interface providing unit may provide a search window so that the user can search for the index information.

According to an embodiment, the index information may be added/deleted/modified in response to the user's command through the interface providing unit.

According to an embodiment of the present invention, an algorithm capable of processing, processing and analyzing big data capable of refining/selecting data to enable augmented analysis from a large amount of raw and unstructured data may be provided.

In addition, a user interface is provided to check metadata information, which is a standard for detecting outliers or missing values, during data pre-processing, so that a user can easily understand the information and, if necessary, modify it. This may facilitate adding new data or deleting existing information.

1 is a diagram for explaining a data processing apparatus for detecting an outlier according to an embodiment of the present invention.
2 is a diagram for explaining an operation sequence of a data processing apparatus for detecting an outlier according to an embodiment of the present invention.
3 to 5 are diagrams for explaining an interface provided by a data processing apparatus for detecting an outlier according to an embodiment of the present invention.
6 is a flowchart illustrating a data processing method for detecting outliers according to an embodiment of the present invention.

Specific structural or functional descriptions of the embodiments according to the concept of the present invention disclosed herein are only exemplified for the purpose of explaining the embodiments according to the concept of the present invention, and the embodiment according to the concept of the present invention These may be embodied in various forms and are not limited to the embodiments described herein.

In the entire specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

Since the embodiments according to the concept of the present invention may have various changes and may have various forms, the embodiments will be illustrated in the drawings and described in detail herein. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes all changes or replacements included in the spirit and scope of the present invention.

Terms such as first or second may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one element from other elements, for example, without departing from the scope of rights according to the concept of the present invention, a first element may be named a second element, Similarly, the second component may also be referred to as the first component.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. does not

Hereinafter, embodiments of a data processing apparatus and method for detecting an outlier will be described in detail with reference to the accompanying drawings.

1 is a diagram for explaining a data processing apparatus for detecting an outlier according to an embodiment of the present invention.

Referring to FIG. 1 , the data processing apparatus 130 may be connected to the user terminal 110 through the network 120 . In this case, the data processing device 130 may be implemented in the form of a server and provided to the user terminal through a web browser.

According to an embodiment, the user terminal 110 may include any type of user terminal capable of accessing a web browser. For example, it may include a personal computer, a laptop computer, a PDA, a smartphone, a smart pad, a tablet PC, and the like.

The network 120 may be wired and/or wireless and may be or include one or more wide area networks (WANs) and/or local area networks (LANs) capable of supporting secure communication. In some examples, the network 120 may include one or more cellular networks and/or the Internet, among other networks. The communication network 104 may operate according to one or more communication protocols such as LTE, CDMA, GSM, LPWAN, WiFi, Bluetooth, Ethernet, HTTP/S, TCP, CoAP/DTLS, and the like. Although network 120 appears to be a single network, it should be understood that network 120 may be a number of separate networks to which they are communicatively linked. Further, in example cases, the network 120 may facilitate secure communications between network components (eg, via encryption or other secure means). The network 120 may likewise take other forms.

According to an embodiment, the data processing device 130 for detecting outliers includes a central processor 131 , a data collection unit 132 , a data preprocessor 133 , a data quality management unit 134 , and an interface providing unit 135 . and a communication unit 136 .

The central processor 131 is a device that controls each component of the data processing device 130 , and in fact, may perform a role of each component in one central processor.

According to an embodiment, the data collection unit 132 may collect raw data to be collected. Here, the raw data may include unrefined data, unstructured data, etc. prepared in an Excel or spreadsheet format file.

According to an embodiment, the data preprocessor 133 may perform a preprocessing process on the collected raw data. The preprocessing process may include data purification, data formalization, and data transformation processes.

According to an embodiment, the data quality management unit 134 may detect normal values, outliers, and missing values of the preprocessed data. In this case, pre-collected metadata on data quality may be used. In addition, metadata may be modified or added when a new type is created. The metadata information may be index information including at least one of a data dictionary and a regular expression.

Metadata is data attribute information, and refers to data given to content according to a certain rule in order to efficiently find and use the information that is being sought from among a large amount of information. Here, the location and content of the content, information about the creator, rights conditions, conditions of use, history of use, etc. may be recorded. In computers, metadata is usually used for the purpose of expressing data and for the purpose of finding data quickly.

Here, the index means an index or a list, and when data is recorded, the attributes such as the name and data size of the data and the recording location are indicated in a table, that is, for reference. The data is called an index.

According to an embodiment, the interface providing unit 135 may visualize the detected upper values, outliers, and missing values and provide them to the user terminal. According to an embodiment, the interface providing unit 135 may provide metadata information that is a criterion for detecting the outlier in response to a user's request.

A more detailed description of the visualization will be provided with reference to FIGS. 3 and 4 .

According to an embodiment, the interface providing unit 135 may provide a search window so that the user can search for the index information.

According to an embodiment, the index information may be added/deleted/modified in response to the user's command through the interface providing unit.

According to an embodiment, the communication unit 136 may transmit an interface to the user terminal and receive a command from the user.

2 is a diagram for explaining an operation sequence of a data processing apparatus for detecting an outlier according to an embodiment of the present invention.

Referring to FIG. 2 , the data processing device 200 for detecting outliers includes a gateway 220 , a data collection unit 230 , a data preprocessor 240 , a data quality management unit 250 , and a user interface 260 . may include

According to an embodiment, the gateway 220 may receive the raw data 210 .

According to an embodiment, the data collection unit 230 may collect original data.

According to an embodiment, the data preprocessor 240 may perform a preprocessing process on the collected original data.

According to an embodiment, the data quality management unit 250 may detect normal values, outliers, and missing values of the preprocessed data. In this case, the data quality management unit 250 may evaluate the quality of data according to at least one of a predetermined data dictionary and a regular expression.

According to an embodiment, the user interface 260 may visualize the detected normal values, outliers, and missing values and provide them to the user 290 .

The user 290 may view the data provided through the user interface and determine whether the metadata is abnormal.

According to an embodiment, the user 290 may search and add/delete/modify metadata that is a criterion for detecting an outlier through the provided interface.

3 to 5 are diagrams for explaining an interface provided by a data processing apparatus for detecting an outlier according to an embodiment of the present invention.

Referring to FIG. 3 , an interface provided through a table by preprocessing raw data can be checked.

In this case, green marks indicate data detected as normal values, red marks indicate data detected as outliers, and white marks indicate data detected as missing values. I used green, red, and white for the colors here, but you can use any other color. A color bar displayed at the bottom of the title column may indicate the ratio of data to normal/outlier/missing values. The longer the green bar, the more normal values, the longer the white bar, the more missing values, and the longer the red bar, the more outliers.

Here, the color bar may be a data quality check bar, and may be an indicator for determining how valid the data value of the corresponding column is.

According to one embodiment, in the identifier column, the id value should have only an integer value according to a regular expression, but if a special character (eg, @) is included or an alphabet (eg, A) is included, It can be judged as an outlier.

According to another embodiment, in the email column, the format of the email has a format specified in the regular expression (eg, adh@digitalship.co.kr), and data that does not match the regular expression corresponds to an outlier. can judge

According to another embodiment, in the city column, the city name may be stored in a pre-stored data dictionary. If data not registered in the pre-stored data dictionary is detected, it may be determined that it corresponds to an outlier.

According to FIG. 3, interfaces are provided as no outliers in the name column, last name column, job column, company column, city column, and date column, and the interface is provided as outliers exist in the id column, email column, and main column. that can be checked

Referring to FIG. 4 , an id column 410 , a first name column 420 , a last name column 430 , an email column 440 , and a job name column 450 may be identified. In this case, it can be seen that a color bar related to data quality is provided at the lower portion of the email column 440 . In the case of e-mail, a lot of data corresponds to the normal value 441, but it can be seen that the missing value 442 and the outlier value 443 are also mixed.

In this case, the user may wonder what the criteria for detecting outliers are. However, it was very difficult to analyze data because other data processing devices do not disclose or modify/add/delete metadata.

Referring to FIG. 5 , if an e-mail column is selected, data showing that the data column has a 99.8% probability of being an e-mail and if it is clear that the data is different from the user's view, it can be modified to other data. That is, in order to increase the accuracy of data processing, the user can modify the properties of the corresponding column.

In addition, data dictionaries and regular expressions may be modified in response to user commands. This is because immediate reflection is required even when a new city is created or the email address format is changed due to a new rule.

6 is a flowchart illustrating a data processing method for detecting outliers according to an embodiment of the present invention.

Referring to FIG. 6 , in operation S610 , the data processing apparatus according to an embodiment may collect raw data.

In step S620, the data processing apparatus according to an embodiment may pre-process the collected data.

In operation S630, the data processing apparatus according to an embodiment may provide a visualization interface for outliers and missing values of data. The data processing apparatus may detect normal values, outliers, and missing values through the preprocessed data. In this case, normal values, outliers, and missing values can be detected according to at least one of a data dictionary and a regular expression.

In operation S640 , the data processing apparatus according to an embodiment may provide an interface for metadata that is a criterion for detecting outliers. Here, the metadata information may be index information including at least one of a data dictionary and a regular expression.

According to an embodiment, the interface may provide a search window so that the user can search for the index information.

According to an embodiment, the index information may be added/deleted/modified in response to the user's command through the interface.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

110: user terminal
120: network
130: data processing unit
131: central processor
132: data collection unit
133: data preprocessor
134: Data Quality Management Department
135: interface providing unit
136: communication department

Claims (9)

acquiring data to be collected, performing pre-processing on the data to be collected, and detecting normal values, outliers, and missing values;
visualizing the detected normal values, outliers and missing values through an interface; and
In response to a user's request, comprising the step of providing metadata information, which is a criterion for detecting the outlier, through the interface,
Visualizing the detected normal values, outliers, and missing values through an interface includes displaying the preprocessed data in a tabular format consisting of rows and columns through the interface,
The title row of the table includes color bars indicating normal values, outliers, and missing values among data of each column,
Each of the color bars is partitioned into partial bars having a length corresponding to the ratio of each of the normal value, the outlier value, and the missing value among the data of each column,
The partial bars are displayed in different colors corresponding to normal values, outliers, and missing values, respectively,
The step of providing the metadata information through the interface comprises:
When the user's request includes a selection input for any one of the columns, displaying metadata information including a first type as a criterion for detecting an outlier for the data of the one column, and ,
The metadata information includes a probability that the one column corresponds to the first type and a second type changeable from the first type. According to claim 1,
The data processing method, characterized in that the metadata information is index information including at least one of a data dictionary and a regular expression. 3. The method of claim 2,
The interface provides a search window so that the user can search for the index information. 3. The method of claim 2,
The index information can be added/deleted/modified in response to the user's command through the interface. delete a data collection unit for acquiring data to be collected;
a data pre-processing unit that pre-processes the collection target data;
a data quality management unit for detecting normal values, outliers, and missing values of the preprocessed data; and
and an interface providing unit for visualizing the detected normal values, outliers, and missing values through an interface,
The interface providing unit,
In response to a user's request, providing metadata information that is a criterion for detecting the outlier,
The interface providing unit
Displaying the pre-processed data in a table format consisting of rows and columns through the interface,
The title row of the table includes color bars indicating normal values, outliers, and missing values among data of each column,
Each of the color bars is partitioned into partial bars having a length corresponding to a ratio of each of the normal values, outliers, and missing values among the data of each column,
The partial bars are displayed in different colors corresponding to normal values, outliers, and missing values, respectively,
The interface providing unit
When the user's request includes a selection input for any one of the columns, displaying metadata information including a first type as a criterion for detecting outliers for the data of the one column,
The metadata information includes a probability that the one column corresponds to the first type and a second type changeable from the first type. 7. The method of claim 6,
The data processing apparatus, characterized in that the metadata information is index information including at least one of a data dictionary and a regular expression. 8. The method of claim 7,
and the interface providing unit provides a search window so that the user can search for the index information. 8. The method of claim 7,
The data processing apparatus, characterized in that the index information can be added/deleted/modified in response to the user's command through the interface providing unit.

Images