KR101965598B1 - APPARATUS AND METHOD for DETECTING DATA OUTLIERS IN FRAUN DETECTION SYSTEM - Google Patents

Abstract

The method of detecting an abnormality of data detects an abnormality of the first data through a first artificial intelligence algorithm based on the first reference information and the second reference information among a plurality of reference information including a data type, metadata, and domain. And an abnormality of the second data based on the second reference information and the third reference information and the storage result among the plurality of reference information including the data type, metadata, and domain. Detecting whether through the second AI algorithm.

Description

Apparatus and method for detecting abnormality in data in abnormal detection system {APPARATUS AND METHOD for DETECTING DATA OUTLIERS IN FRAUN DETECTION SYSTEM}

The present invention relates to an apparatus and method for detecting abnormalities in data, and to an apparatus and method for automatically detecting abnormalities for data based on artificial intelligence.

As the demand for securing the quality of the collected big data is increasing, securing the quality of the big data has emerged as an important topic, but the amount of data has exploded at a rate corresponding to Moore's law. It is difficult to measure quality with the experience and effort of data engineers. Moore's law states that the performance of semiconductor integrated circuits doubles every 18 months. In addition, as the complexity of the business increases, the complexity of the information system is increasing. The type of data is also expanding from structured data to unstructured data, and the amount of data is increasing as interest in the use of big data increases.

Data management technologies are mainly developed for data storage and access technologies. On the other hand, the awareness of data quality is very low, and there is a need for data quality management due to damages and costs of low quality data. For example, as of May 2009, there have been more than 300,000 cases where the national pension subscriber's registration number does not match the name. There is an example of sending.

As such, incidents related to databases used in public institutions or corporations in Korea and abroad are gradually increasing, and a study by NASA in the United States is automated to satisfy data quality requirements that meet the increasing data volume in the case of big data. There is an increasing demand for technology that automatically manages and measures the quality of databases that cannot be solved by human resources, including the need for data quality measurement tools.

Currently, the database is in the low awareness of the data quality of the database, such as low awareness of the importance and inaccurate data, lack of awareness of data quality management costs, and lack of data quality experts, the lack of data quality experts, Application techniques considered are required.

The background technology of the present application is disclosed in Korean Patent Laid-Open No. 2000-0055986.

The present invention is to solve the above-mentioned problems of the prior art, and an object of the present invention is to provide a system for automatically detecting anomalies or abnormalities for data in a big data environment. In addition, an object of the present invention is to provide an artificial intelligence-based outlier detection system that can automatically identify and learn outliers for data in a big data environment.

In addition, an object of the present invention is to provide a system for detecting abnormalities for each type of data by customizing according to various types of data, such as numbers, letters, strings, photos, etc. that occur in a big data environment.

However, the technical problem to be achieved by the embodiments of the present application is not limited to the technical problems as described above, and other technical problems may exist.

As a technical means for achieving the above technical problem, a method for detecting anomaly of data includes first data based on first reference information and second reference information among a plurality of reference information including a data type, metadata, and domain. Detecting whether an abnormality is detected by using a first artificial intelligence algorithm, storing a result of detecting abnormality of the first data, and second reference information and third reference among a plurality of reference information including data types, metadata, and domains; The method may include detecting whether the second data is abnormal based on the information and the storage result through the second artificial intelligence algorithm.

In addition, based on the first reference information and the third reference information and the storage result of the plurality of reference information including the data type, metadata, domain according to an embodiment of the present application through the third artificial intelligence algorithm It may be to detect whether there is an abnormality.

 In addition, the data type according to an embodiment of the present application is to classify the data into one of the unstructured data type or unstructured data type, and the step of detecting whether the first data is abnormal or not based on the data type whether the first data is abnormal or not. May be detected through the first AI algorithm.

In addition, detecting whether the first data is abnormal according to an embodiment of the present disclosure, when the first reference information is a data type and the second reference information is a domain, the data type is a formal data type, and the domain is an amount of money. If the domain, the first data may be detected through a boxplot algorithm.

In addition, detecting whether the first data is abnormal according to an embodiment of the present disclosure, when the first reference information is a data type, the second reference information is metadata, the data type is an unstructured data type, JPEG ( In the Joint Photographic Experts Group) format, the first data may be detected through a deep learning algorithm.

In addition, the detecting of the abnormality of the second data according to an embodiment of the present application, the second reference information and the third reference information of the plurality of reference information including the data type, metadata, domain, storage result and first The abnormality of the second data may be detected through the second artificial intelligence algorithm based on the first data.

In addition, detecting whether the second data is abnormal according to an embodiment of the present disclosure, when the second reference information is a data type and the second reference information is a domain, the second reference information is a standard data type, and the domain In this case, the second domain may be detected by a linear regression algorithm based on the relationship information with the first data.

The method may further include storing an abnormality detection result of the second data after detecting the abnormality of the second data according to the exemplary embodiment of the present application.

In addition, detecting whether an abnormality of the second data according to an embodiment of the present application may normalize the second data when an abnormality of the second data is detected.

In addition, detecting whether the second data is abnormal according to an embodiment of the present disclosure, when the second reference information is a data type and the third reference information is a domain, the data type is a formal data type, and the domain is a name. In the domain, whether the abnormality of the second data is detected through a similarity algorithm, and when an abnormality of the second data is detected, the second data may be normalized to data having a high similarity with the second data.

In addition, the apparatus for detecting an abnormality of the data according to an embodiment of the present application whether the first data is abnormal based on the first reference information and the second reference information of a plurality of reference information including the data type, metadata, domain Includes a database for storing the abnormality detection result of the first data and the abnormality detection unit for detecting through the first artificial intelligence algorithm, the abnormality detection unit a second criterion of the plurality of reference information including the data type, metadata, domain The abnormality of the second data may be detected through the second artificial intelligence algorithm based on the information, the third reference information, and the storage result.

The above-mentioned means for solving the problems are merely exemplary and should not be construed as limiting the present application. In addition to the above-described exemplary embodiments, additional embodiments may exist in the drawings and detailed description of the invention.

According to the above-described problem solving means of the present application, it is possible to automatically detect outliers of big data, thereby enabling data quality management without the involvement of engineers and personnel. In addition, it is possible to systematically manage big data quality by automatically learning new types of data according to the increasing amount of data in a big data environment. Quality management of big data can be provided through continuous learning about big data type determination result and big data outlier detection result.

In addition, it can intuitively provide outlier detection results according to the newly added big data type considering the characteristics of big data, and AI-based outlier detection that can automatically identify and learn outliers for data in a big data environment. A system can be provided.

The present invention can detect outliers for each type of data by customizing it according to various types of data such as numbers, letters, strings, pictures, etc., generated in a big data environment.

1 is a block diagram of a system for detecting anomaly of data according to an embodiment of the present application.
2 is a view showing the configuration of the abnormality detection apparatus according to an embodiment of the present application.
3A to 3C illustrate anomaly detection of data according to an embodiment of the present application.
4 is a flowchart illustrating a process of detecting abnormality of data according to an embodiment of the present application.
5 is a flowchart illustrating a method of detecting an abnormality of data according to an exemplary embodiment of the present application.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present disclosure. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted for simplicity of explanation, and like reference numerals designate like parts throughout the specification.

Throughout this specification, when a portion is "connected" to another portion, this includes not only "directly connected" but also "electrically connected" with another element in between. do.

Throughout this specification, when a member is said to be located on another member "on", "upper", "top", "bottom", "bottom", "bottom", this means that any member This includes not only the contact but also the presence of another member between the two members.

Throughout this specification, when a part is said to "include" a certain component, it means that it can further include other components, without excluding the other components unless specifically stated otherwise.

1 is a block diagram of a system for detecting anomaly of data according to an embodiment of the present application. Referring to FIG. 1, a system for detecting abnormality of data may include an abnormality detecting apparatus 10, a reference information dictionary 20, and a detection target database. However, the configuration shown in FIG. 1 is not limited to the above configuration.

The abnormality detection apparatus 10 may perform prior learning through artificial intelligence based on the reference information dictionary 20 and perform abnormality detection on the detection target database 30 based on the learning result. The abnormality detection apparatus 10 may perform detection of data that is out of the normal category according to various algorithms with respect to columns or data of the detection target database 30, and by learning the detection result, the other detection target database ( 30) can be used to perform anomaly detection. The configuration of the abnormality detecting device 10 is described in detail with reference to FIG. 2.

The reference information dictionary 20 may include a preceding data or a source for enabling the abnormality detecting apparatus 10 to perform the preceding learning. For example, the reference information dictionary 20 may include databases for various fields, and in more detail, for example, normal blood pressure level database by age group, student average height in Seoul, weight database, and Korean average body database For example, the index may include a database or an average value, which may be a reference in various fields, such as a 2-digit and 3-digit code database for each country and a representative name database of companies.

The detection target database 30 is a database for abnormal detection, and the abnormality detection apparatus 10 may detect whether or not an abnormality is detected in the data included in the detection target database 30 based on the learning result.

The abnormality detection of the data through the abnormality detection apparatus 10 may have an effect of managing the quality of the data. Data quality management may refer to activities for data management and improvement that are carried out continuously to meet the expectations of information systems or database users, both inside and outside an organization or organization. Data may mean any data or information related to an information system that is built and operated to achieve its purpose. In addition to data values stored in the database, data may be structural output such as data models or standard data and outputs in the form of documents. In general, the data may mean digital data stored in an information system.

In addition, data quality management may refer to a procedure of measuring the quality of data contained in an information system currently being operated or managed to evaluate the current level and to analyze the factors of quality degradation. Data quality management can mean diagnosing the quality of data by analyzing data values based on tables, columns, codes, relationships, and business rules in the operational database. It can refer to the process of calculating the details and analyzing the cause of the error.

2 is a view showing the configuration of the abnormality detection apparatus 10 according to an embodiment of the present application. Referring to FIG. 2, the abnormality detecting apparatus 10 may include a database 110 and an abnormality detecting unit 120. However, the configuration of the abnormality detecting apparatus 10 shown in FIG. 2 is not limited to those described above.

The abnormality detection unit 120 may detect whether the first data is abnormal through the first artificial intelligence algorithm based on the first reference information and the second reference information among a plurality of reference information including a data type, metadata, and domain. Can be. For example, when the first reference information is a data type and the second reference information is a domain, the abnormality detection unit 120 determines whether the first data is abnormal when the data type is a standard data type and the domain is an amount domain. This can be detected using a Boxplot algorithm.

At this time, the outliers may refer to all abnormal data or data formats that degrade the quality of the wrong input data when the format or the number of digits of the data are different.

The data type is classified into one of a structured data type and an unstructured data type. In this case, the structured data type is standardized data and may mean data stored in a fixed field. For example, it may mean data stored in a certain format consisting of fixed fields in a database such as name, address, contact, shipping address, and payment information. On the other hand, unstructured data may mean data that is not stored in a fixed field, for example, video data uploaded to a video streaming site, photos and audio data stored on SNS, conversation data exchanged via messenger, smartphone Etc., location information recorded through the data may be included.

Metadata is called data about data and can be defined as data created for any purpose. In more detail, metadata is structured data about data, and may refer to data describing other data. In other words, metadata is data that is given to content or data according to a predetermined rule in order to efficiently find and use information that is being searched among a large amount of information, and to analyze and classify structured information and add additional information. It can refer to information that follows the data together.

A domain generally refers to a characteristic of a column, which is the smallest unit of data given when designing a database. If a data business rule suitable for a domain is applied, the domain can have the effect of maintaining the integrity of the data. In more detail, a domain is a unique property that is applied to each column during the design stage of a table in a database, and thus, a definition of a characteristic of a column, which is the smallest unit of data managed by the database. Examples of domains may include, but are not limited to, amounts, numbers, rates, codes, IDs, names, contents, dates, flags, numbers, and contacts.

The amount domain may refer to a number related to money, such as sales, sales amount, or cost, among data whose data type is numeric type. In more detail, the money domain is a value representing the amount of money, and may be composed of values of a numeric type corresponding to a national currency. Ongoing management of monetary domains ensures that the data stored in the monetary domains is always in valid form and that the scope of the data itself is valid. Examples of an amount domain may include an amount, tax, price, unit price, cost, fee, balance or total amount.

The number domain may mean a number that is not an amount of money such as the number of customers, the number of products, and the number of audiences among the data whose data type is a numeric type. In more detail, the number domain may mean an item managed numerically, such as the number, size, and number of times. Ongoing management of the quantity domain allows you to maintain an effective range of maximum and minimum values for data stored in the number domain. Examples of the water domain may include a number, number of copies, turn, number, distance, scale, length, weight, speed, number of times, equilibrium, area, or temperature.

The rate domain may refer to a number including a ratio, such as achievement rate, accuracy, and cost rate, among data whose data type is numeric type. More specifically, the rate domain can be defined and used in various ways such as progress rate, growth rate, rate of return, rate of change, interest rate, addition rate, and rate, and the rate of increase, rate of return, rate of change, etc. , Interest rates, addition rates, rates, etc. may be managed as reference information applied to the calculation of other numerical data. Examples of rate domains may include interest rates, interest rates, rates, exchange rates or percentages.

The code domain is a predefined item of data whose data type is numeric or character type, and may include codes and values. For example, if a woman expresses 'F' and a man expresses 'M', 'F' and 'M' can be codes, and woman and man can be values. More specifically, a code domain can mean data that has been replaced by a short code value to limit the data available or to manage data with the same meaning in the same representation, and code domains are generally managed by code and code values. Can be. For example, the code may mean a gender classification code, a customer rating code, a department code, a product code, a region code, and the like, and the code value may include data in a corresponding information item such as 'M' or 'F' of the gender classification code. It can mean a value to represent or limit. The code domain may be managed by predefined standard codes.

The ID domain may mean a unique value defined by a user, such as a USER ID.

The name domain includes a character string having a predetermined length or less and may mean a customer name, a product name, or the like. In more detail, the name domain is a name attached to an object, a person, a group, or the like to distinguish it from others, and may include a name, a place, a customer name, an English customer name, a URL, an IP, and the like.

The content domain includes a string having a predetermined length or more, and may mean a post content or a cover letter. In more detail, the value of the content domain may refer to data describing a description or reference of an object or action, and examples of definition, description, remark, content, summary, and the like may exist. The content domain may have properties that consist of unstructured characters.

The date domain may mean a date such as year, year, month, date, and date. In more detail, the date domain may mean an item managed by a date, and may include data representing a date and time such as a reception date, a registration date and time, a settlement date month, and a transmission time. Data type of date domain can use character type and method using date data type provided by DBMS. When using the date data type provided by the DBMS, there is almost no error in the date value because the invalid date value is checked by the DBMS itself. However, when used as a character type, an incorrect date value may be entered.

The flag domain indicates whether the flag domain is 0, 1, 'Y', 'N', 'true', false ',' True ', False' may be composed of two opposing values. In more detail, the flag domain is one of the domains for which standardization and management of data is simple, and may consist of two or three simple classification values such as whether or not, 'Y', 'N' or 1, 0, etc. Can be defined as a separate value. Data stored in the flag domain is always managed in the same form to maintain consistency between information systems.

The number domain may refer to a numerical value that does not mean to increase sequentially, such as customer number and product number. In more detail, a number domain is composed of a combination of letters or numbers and can be managed mostly according to the internal and external number system. No. Domain is data that complies with domestic and international standard number system such as social security number, business registration number, zip code, corporation number, IP address, international standard data number (ISBN / ISSN), or user number, account number, permission There may be data managed according to the company's internal standard number management system such as number, approval number, registration number and product number.

The contact domain may refer to data including information such as an address, an email, and a contact.

The abnormality detection unit 120 may detect whether the first data is abnormal based on the data type through the first artificial intelligence algorithm. For example, when the first reference information is a data type and the second reference information is a domain, the abnormality detection unit 120 determines whether the first data is abnormal when the data type is a standard data type and the domain is an amount domain. This can be detected using a Boxplot algorithm.

That is, when the data of the detection target database 30 is the stereotyped data and the value domain, the abnormality detection unit 120 has an outlier that is outside the normal range with respect to the data of the detection target database 30 through a box plot algorithm. Can be detected.

The box plot algorithm is a box-shaped algorithm that represents the distribution of data in the form of a box. The upper value, the upper limit value, the lower limit value, and the quartile range may be measured, and data exceeding the upper limit value or data below the lower limit value may be detected as having an abnormal value or an abnormal value. In this case, the quartile score is the range of the upper quartile value minus the lower quartile value.In general, the upper limit value is determined by multiplying the upper quartile value by 1.5 and the lower limit by multiplying the lower quartile value by 1.5. Can be.

3A to 3C illustrate anomaly detection of data according to an embodiment of the present application. For a more detailed example, referring to FIG. 3A, an operation of the abnormality detector 120 for mathematical grade data in a high school grade database is described. The anomaly detection unit 120 may detect an anomaly through a boxplot algorithm because the data is structured data and is domain. The abnormality detection unit 120 may determine that the average of the mathematical grade data (or the column including the mathematical grade data) is 46.6 points, the upper third percentile value is 53.8, and the lower first percentile value is 38.6 based on the box plot algorithm. Based on this, the upper limit value can be determined as 80.7 and the lower limit value can be determined as 25.4. The abnormality detector 120 may detect that the data has an outlier when data exceeding 80.7 or less than 25.4 is located in the detection target database 30.

If the data follow a normal distribution, the probability that there is a value above or above the lower limit of the boxplot algorithm may be as small as 0.35%, respectively.

Meanwhile, when the first reference information is structured data and the second reference information is several domains, the abnormality detection unit 120 may detect whether the data is abnormal through the histogram algorithm. The histogram algorithm is a graphical representation of a table of frequency distributions, which may mean a graphical representation of the frequency distribution table. At this time, the abnormality detection unit 120 may detect whether the corresponding data is abnormal based on the frequency distribution.

In another example, when the first reference information is a data type and the second reference information is metadata, the abnormality detection unit 120 may include a first type when the data type is an unstructured data type and a JPEG (Joint Photographic Experts Group) format. The abnormality of the data can be detected through a deep learning algorithm.

For example, when the data is an unstructured data type and a picture file format, the abnormality detector 120 may detect whether the data is abnormal based on a deep learning algorithm. 3B, the anomaly detection unit 120 divides an image into cells having a predetermined size, extracts feature points (edges) for each cell, obtains a histogram of directions of the extracted feature points, and the like. The outliers for the image may be detected based on the similarity between the histograms or the similarity between the direction vectors of the extracted feature points or the direction vectors. The similarity between the direction vectors of the extracted feature points is higher as the angle difference between the direction vectors of the feature points is smaller, and the similarity is lower as the angle difference between the direction vectors of the feature points is larger.

In more detail, for example, when the detection target database 30 is an automobile insurance database, the abnormality detection unit 120 may detect images having low similarity as outliers based on the contact accident images between cars. The abnormality detector 120 may detect images that are not related to a contact accident between automobiles including an automobile photograph as outliers. At this time, the abnormality detection unit 120 learns the detection result based on the deep learning algorithm, and as the number of learning on the image is accumulated, the accuracy of the detection can be increased, thereby improving the reliability of the outlier detection. Can be.

The abnormality detection unit 120 stores the abnormality detection result of the first data, so that the stored abnormality detection result can also be used as one of the reference information. In other words, in addition to the reference information learned through the abnormality detection unit 120 reference information dictionary 20, the detection result detected by the abnormality detection unit 120 may be learned to increase the accuracy and reliability of the abnormality detection. .

The abnormality detection unit 120 determines whether the second data is abnormal based on the second reference information, the third airborne information, and the storage result among the plurality of reference information including the data type, metadata, and domain. The third data may be detected through the third artificial intelligence algorithm based on the first reference information, the third reference information and the storage result among the plurality of reference information.

For example, when the second reference information is a data type and the third reference information is a domain, when the data type is a formal data type and the domain is a name domain, whether the second data is abnormal is detected through a similarity algorithm. When abnormality of 2 data is detected, it can normalize to data with high similarity with 2nd data.

In other words, when the data type is a formal data type and the name domain, the abnormality detection unit 120 determines the name having the most duplicate value as standardized data based on the similarity of the data, and is different from the data determined as the standardized data. Data can be detected as outliers.

For example, referring to FIG. 3C, when the trademark name database is the detection target database 30, the abnormality detection unit 120 may bundle duplicate data and high similarity data with respect to the data of the corresponding column. The name with the most duplicate values can be determined as standardized data. In more detail, the anomaly detection unit 120 may detect 105 pieces of data in two patterns, including 100 'metrocity' and 5 'metro city' based on the similarity between the data in the trademark name database. In this case, five “meta-City” can be detected as an outlier and determined. At this time, the abnormality detection unit 120 may determine the 'metrocity' from which the most data is searched as standardization data, and standardize the change by changing the metro city, which has an outlier value, to metrology, which is standardization data. .

Meanwhile, similarly to the above, the abnormality detection unit 120 may determine the 'bin pole' having the most duplicate values among 'bin pole' or 'empty pole' as standardized data, and may standardize the 'empty pole' as 'bean pole'. The standardized data may determine 'MONTBLANC' in which the most duplicate values among 'MONTBLANC', 'MONT BLANC' or 'MONTBLANCC' are detected.

The abnormality detection unit 120 determines whether the second data is abnormal based on the second reference information, the third reference information, the storage result, and the first data among the plurality of reference information including the data type, metadata, and domain. It can be detected through AI algorithms. For example, when the second reference information is a data type and the second reference information is a domain, the abnormality detection unit 120 has a relation with the first data when the second reference information is a standard data type and the domain is a few domains. The abnormality of the second data may be detected through a linear regression algorithm based on the information.

In more detail, when the detection target database 30 is a database including a high school student's standard height and weight, the abnormality detection unit 120 may detect an outlier through a linear regression algorithm in consideration of the relationship between the height and the weight. Can be. That is, when the height is 140cm and the weight is 100kg, the abnormality detection unit 120 does not detect the outliers, but when the student having the height of 140cm has the weight of 100kg, the abnormality detection may be performed based on the linear regression algorithm. Can be.

The anomaly detection unit 120 automatically detects an anomaly for data based on artificial intelligence in the detection target database 30 in consideration of the characteristics of big data that are continuously changed and added, and detects a detection result of a user (operator or It can be delivered intuitively to administrators and displayed in an easy-to-understand form. In other words, the abnormality detection unit 120 may display the abnormality detection result to the user, thereby performing a confirmation procedure from the user to reflect the abnormality detection result and the detection result in the system.

The abnormality detection unit 120 may store the abnormality detection result of the second data in the database 110 and use it for abnormality detection of the detection target database 30 as new reference information.

The database 110 may include big data and may include both structured data and unstructured data. Database 110 may include semi-structured data.

The anomaly detection unit 120 may detect an anomaly for the detection target database 30 based on a plurality of reference information including a data column, which is a learning result stored in the database 110, and based on this, various algorithms may be used. Based on the detected outlier detection results, iterative learning can be performed to gradually improve the accuracy and reliability of the outlier detection results. In other words, the anomaly detection unit 120 may store the anomaly detection result in the database 110 and use the stored anomaly detection result as one reference information to detect an anomaly of other data.

domain Usage algorithm Detail Price Boxplot
Histogram Outlier detection using box plots and histograms Number Boxplot
Histogram Outlier detection using box plots and histograms rate Boxplot
Histogram Outlier detection using box plots and histograms Picture Deep learning Similarity-based Image Clustering through Prior Learning by Photo Type text Deep learning Determine existence of required metadata and value by text type using metadata

Table 1 illustrates an artificial intelligence algorithm that the anomaly detection unit 120 determines for anomaly detection according to a data type and a type of domain. The anomaly detection unit 120 may detect an anomaly through a box plot algorithm or a histogram algorithm in the case of domains related to numbers, such as an amount, a number, a rate domain, and the like. In the case of domains, an outlier may be detected through a similarity algorithm between image similarity, text type, and metadata according to a deep learning algorithm. However, the ones shown in Table 1 are not limited thereto because they are only one of various embodiments of the present disclosure.

That is, the anomaly detection unit 120 may determine an outlier based on various reference information such as whether the detection target data is structured data or unstructured data, the nature of the domain or domain determined for the data, metadata, domain, column, or relationship information between the data. An AI algorithm for detection may be determined, and anomaly detection may be performed on the detection target data or database based on the determined AI algorithm. In relation to the AI algorithm, algorithms such as a box plot algorithm, a histogram algorithm, a linear regression algorithm, and a deep learning algorithm may be included, but are not limited thereto. There may be various algorithms.

4 is a flowchart illustrating a process of detecting abnormality of data according to an embodiment of the present application. Referring to FIG. 4, in step S401, the abnormality detection apparatus 10 learns data about reference information for outlier detection based on the reference information dictionary 20, and stores the learning result in the database 110 in step S402. Can be. Subsequently, in step S403, the abnormality detection apparatus 10 performs abnormality detection on the data included in the detection target database 30 based on the learning result, and in step S404, the abnormality detection apparatus 10 applies to the detection result, the reference information used for the detection, or the detection. Information related to the detection result, such as the AI algorithm used, may be stored in the database 110. In operation S405, the abnormality detecting apparatus 10 may detect an abnormality value for other data using various artificial intelligence algorithms based on the plurality of reference information and the storage result stored in the database 110.

That is, the anomaly detection apparatus 10 according to an embodiment of the present application may perform an outlier detection on data through a plurality of reference information and a plurality of artificial intelligence algorithms with respect to the detection target database 30. As the number of times increases, the accuracy of outlier detection through learning increases, which increases the accuracy and reliability of data quality management.

5 is a flowchart illustrating a method of detecting an abnormality of data according to an exemplary embodiment of the present application. The abnormality detecting method illustrated in FIG. 5 performs the operation of the abnormality detecting apparatus 10 described with reference to FIGS. 1 to 4. Therefore, since the content not described in FIG. 5 also applies to the description of the abnormality detecting apparatus 10 described with reference to FIGS. 1 to 4, detailed description thereof will be omitted.

Referring to FIG. 5, in operation S501, the abnormality detecting apparatus 10 determines whether the first data is abnormal based on the first reference information and the second reference information among a plurality of reference information including a data type, the metadata, and a domain. Is detected through the first artificial intelligence algorithm, and the abnormal detection result of the first data is stored in step S502.

In operation S503, the abnormality detecting apparatus 10 determines whether the second data is abnormal based on the second reference information, the third reference information and the storage result among the plurality of reference information including the data type, metadata, and domain. 2 Can be detected by AI algorithm.

The present application can provide an apparatus and method for improving the reliability of quality for big data by performing AI-based data outlier detection in a system to which big data is applied. To this end, the present application may determine an artificial intelligence algorithm for outlier detection based on various reference information to detect outliers for data through an appropriate method.

The present application may perform prior learning through artificial intelligence based on a reference information dictionary, and perform outlier detection on a target database to be detected based on prior learning results. Outlier detection results can also be stored and learned to increase the accuracy of outlier detection. This company can automate the manual work of engineers for data quality management, and by doing so, it can minimize the manpower and perform continuous data quality management without human intervention.

The foregoing description of the application is intended for illustration, and it will be understood by those skilled in the art that the present invention may be easily modified in other specific forms without changing the technical spirit or essential features of the present application. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present application is indicated by the following claims rather than the above description, and it should be construed that all changes or modifications derived from the meaning and scope of the claims and their equivalents are included in the scope of the present application.

10: abnormal detection device
110: database
120: abnormal detection unit
30: Target database detected

Claims (11)

In the method of detecting abnormality of data,
Determining a first artificial intelligence algorithm based on first reference information and second reference information among a plurality of reference information including a data type, metadata, and domain related to the first data;
Detecting whether the first data is abnormal through the first artificial intelligence algorithm by referring to a reference information dictionary;
Storing the abnormality detection result of the first data;
Determining a second artificial intelligence algorithm based on the second reference information and the third reference information among the plurality of reference information including the data type, the metadata, and the domain related to the second data and the storage result; And
Detecting whether the second data is abnormal through the second artificial intelligence algorithm by referring to a reference information dictionary and the storage result. The method of claim 1,
Determining a third artificial intelligence algorithm based on first reference information and third reference information among a plurality of reference information including the data type, the metadata, and the domain related to the third data; And
And detecting whether the third data is abnormal through the third artificial intelligence algorithm by referring to a reference information dictionary and the storage result. The method of claim 1,
The data type includes a structured data type and an unstructured data type,
Detecting whether the first data is abnormal
And detecting the abnormality of the first data based on the data type through the first artificial intelligence algorithm. The method of claim 3, wherein
Determining the first artificial intelligence algorithm,
When the first reference information is the data type and the second reference information is the domain, when the data type is the formal data type and the domain is an amount domain, a boxplot algorithm is applied to the first artificial information. Anomaly detection method that is determined by an intelligent algorithm. The method of claim 3, wherein
Determining the first artificial intelligence algorithm,
If the first reference information is the data type and the second reference information is the metadata, if the data type is an unstructured data type and a JPEG (Joint Photographic Experts Group) format, a deep learning algorithm is applied. Determination by the first artificial intelligence algorithm, the abnormality detection method. The method of claim 1,
Detecting whether the second data is abnormal,
And detecting the abnormality of the second data through the second artificial intelligence algorithm by referring to a reference information dictionary, the storage result, and the first data. The method of claim 6,
Determining the second AI algorithm,
If the second reference information is the data type and the second reference information is the domain, if the data type is a formal data type and the domain is a number domain, the linear regression algorithm is determined as the second artificial intelligence algorithm. and,
Detecting whether the second data is abnormal,
And detecting the abnormality of the second data through the second artificial intelligence algorithm based on a reference information dictionary, the storage result, and relationship information with the first data. The method of claim 1,
After detecting whether the second data is abnormal,
And storing an abnormality detection result of the second data. The method of claim 1,
Detecting whether the second data is abnormal
And when the abnormality of the second data is detected, normalizing the second data. The method of claim 9,
Determining the second AI algorithm,
If the second reference information is the data type and the third reference information is the domain, if the data type is a formal data type and the domain is a name domain, a similarity algorithm is determined as the second artificial intelligence algorithm. ,
Detecting whether the second data is abnormal,
And when the abnormality of the second data is detected, normalizing the second data based on the similarity algorithm. In the device for detecting abnormality of data,
A first AI algorithm is determined based on the first reference information and the second reference information among a plurality of reference information including data types, metadata, and domains, and the first AI algorithm is referenced by referring to a reference information dictionary. An abnormality detecting unit detecting whether the first data is abnormal; And
Including a database for storing the abnormality detection result of the first data,
The abnormality detection unit determines a second artificial intelligence algorithm based on the second reference information, the third reference information, and the storage result among a plurality of reference information including the data type, the metadata, and the domain, and the reference information. And detecting the abnormality of the second data through the second artificial intelligence algorithm by referring to a dictionary and the storage result.

Images