KR20180097895A - APPARATUS AND METHOD for DETECTING DATA OUTLIERS IN FRAUN DETECTION SYSTEM - Google Patents

Abstract

A method for detecting the abnormality of data includes the steps of: detecting whether or not first data is abnormal based on first reference information and second reference information among a plurality of reference information including a data type, metadata, and a domain through a first artificial intelligence algorithm; storing an abnormality detection result of the first data; and detecting whether or not second data is abnormal based on the second reference information and third reference information among the plurality of reference information including the data type, the metadata, and the domain through a second artificial intelligence algorithm. Accordingly, the present invention can manage data quality without the involvement of engineer experts and a director.

Description

[0001] APPARATUS AND METHOD FOR DETECTING DATA OUTLIERS IN FRAUN DETECTION SYSTEM [0002]

The present invention relates to an apparatus and method for detecting abnormalities in data, and an apparatus and method for automatically detecting an abnormal value for data based on artificial intelligence.

As the demand for securing the quality of the collected big data is increasing, securing the quality of the big data is becoming an important issue, but the amount of data is explosively increased at a rate corresponding to Moore's law And it is difficult to measure the quality due to the experience and efforts of the data engineer. Moore's Law states that the performance of semiconductor integrated circuits doubles every 18 months. In addition, the complexity of information systems is increasing due to the increasing complexity of business. Data types are also expanding from regular data to unstructured data, and the amount of data is increasing due to the increased interest in utilization of big data.

Data management techniques are mainly developed for data storage and access technology. On the other hand, the recognition of data quality is very low, and there is a need for data quality management according to damage cases of low quality data and cost incurrence. For example, in May 2009, there were about 300,000 cases in which the resident registration number and the name of the national pension subscriber do not match. In foreign countries, the government office issued a notice to parents of children who have already died, .

As a result, NASA's research in the United States is increasingly concerned with the use of databases in domestic and foreign public institutions and corporations, and in the case of Big Data, automation is required to meet data quality requirements that meet the increasing amount of data. There is a growing demand for a technology to automatically manage and measure the quality of a database that can not be solved by the manpower.

At present, the database is not well aware of the data quality of the database, such as low awareness of importance and inaccurate data, lack of awareness of data quality management costs, and lack of data quality experts. The application technique considered is required.

The background technology of the present application is disclosed in Korean Patent Laid-Open Publication No. 2000-0055986.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a system for automatically detecting abnormalities or abnormalities in data in a big data environment. It is another object of the present invention to provide a system for detecting an abnormal value based on artificial intelligence capable of automatically identifying and learning an abnormal value for data in a big data environment.

It is another object of the present invention to provide a system for detecting an abnormal value for each type of data by customizing it according to various types of data such as numbers, letters, strings, pictures, etc., occurring under a big data environment.

It should be understood, however, that the technical scope of the embodiments of the present invention is not limited to the above-described technical problems, and other technical problems may exist.

According to an aspect of the present invention, there is provided a method for detecting abnormality in data, the method comprising the steps of: detecting first abnormality of data based on first reference information and second reference information among a plurality of reference information including a data type, Detecting an abnormality of the first data through the first artificial intelligence algorithm, storing an abnormality detection result of the first data, and storing second reference information and third reference information among a plurality of reference information including a data type, And detecting the abnormality of the second data based on the information and the storage result through the second artificial intelligence algorithm.

In addition, the first and second reference information among the plurality of reference information including the data type, the metadata, and the domain according to an embodiment of the present invention, and the third reference information, It may be to detect anomalies.

 In addition, the data type according to an embodiment of the present invention classifies the data into either a formal data type or an unstructured data type, and the step of detecting the abnormality of the first data may include determining whether the first data is abnormal Lt; RTI ID = 0.0 > AI < / RTI > algorithm.

In addition, the step of detecting the abnormality of the first data according to an embodiment of the present invention may include the steps of detecting whether the first reference information is a data type, the second reference information is a domain, Domain, it may be to detect the abnormality of the first data through a Boxplot algorithm.

In addition, the step of detecting abnormality of the first data according to an exemplary embodiment of the present invention may include the steps of detecting whether the first reference information is a data type, the second reference information is metadata, the data type is an irregular data type, Joint Photographic Experts Group) format, it may be to detect the abnormality of the first data through a Deep Learning algorithm.

In addition, the step of detecting the abnormality of the second data according to the embodiment of the present invention may include the steps of detecting the second reference information and the third reference information among the plurality of reference information including the data type, metadata, domain, 1 < / RTI > data based on the second artificial intelligence algorithm.

In addition, the step of detecting the abnormality of the second data according to an embodiment of the present invention may include the steps of detecting whether the second reference information is a data type, the second reference information is a domain, In the case of the number domain, it may be detected through the linear regression algorithm whether the second data is abnormal based on the relationship information with the first data.

In addition, the method may further include storing an abnormality detection result of the second data after the step of detecting abnormality of the second data according to an embodiment of the present invention.

In addition, the step of detecting the abnormality of the second data according to an embodiment of the present invention can standardize the second data when an abnormality of the second data is detected.

If the second reference information is a data type and the third reference information is a domain, the step of detecting the abnormality of the second data according to an embodiment of the present invention may include the steps of: Domain, it may be detected through the similarity algorithm whether the second data is abnormal, and when the abnormality of the second data is detected, the data may be standardized to the data having high similarity to the second data.

In addition, an apparatus for detecting abnormality of data according to an embodiment of the present invention may be configured to detect abnormality of first data based on first reference information and second reference information among a plurality of reference information including a data type, metadata, And a database for storing an abnormality detection result of the first data, wherein the abnormality detection unit is configured to detect an abnormality detection result of the first reference data among the plurality of reference information including the data type, The second artificial intelligence algorithm can detect the abnormality of the second data based on the information, the third reference information, and the storage result.

The above-described task solution is merely exemplary and should not be construed as limiting the present disclosure. In addition to the exemplary embodiments described above, there may be additional embodiments in the drawings and the detailed description of the invention.

According to the above-mentioned problem solving means of the present invention, it is possible to automatically detect the outliers of the big data, and it is possible to manage the data quality without involvement of the engineer specialist and the person in charge. Also, systematic big data quality management is possible by automatically learning new types of data according to the amount of data that grows under the big data environment. As a result of the determination of the big data type, it is possible to provide the quality control for the big data through continuous learning about the result of detection of the big data outliers.

In addition, it is possible to intuitively provide the outlier detection result according to the newly added big data type considering the characteristics of the big data, and to detect the abnormal value based on the artificial intelligence based on the intelligent detection System can be provided.

The present invention can customize the data according to various types of data such as numbers, letters, strings, pictures, and the like, which occur under a big data environment, and detect abnormal values for each type of data.

1 is a block diagram of a system for detecting abnormality of data according to an embodiment of the present invention.
2 is a diagram showing a configuration of an anomaly detection device according to an embodiment of the present invention.
3A to 3C are diagrams illustrating abnormal detection of data according to an embodiment of the present invention.
FIG. 4 is a flowchart illustrating a process of detecting abnormality of data according to an embodiment of the present invention.
5 is an operational flow diagram illustrating a method for detecting anomalies in data in accordance with one embodiment of the present application.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. It should be understood, however, that the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In the drawings, the same reference numbers are used throughout the specification to refer to the same or like parts.

Throughout this specification, when a part is referred to as being "connected" to another part, it is not limited to a case where it is "directly connected" but also includes the case where it is "electrically connected" do.

It will be appreciated that throughout the specification it will be understood that when a member is located on another member "top", "top", "under", "bottom" But also the case where there is another member between the two members as well as the case where they are in contact with each other.

Throughout this specification, when an element is referred to as "including " an element, it is understood that the element may include other elements as well, without departing from the other elements unless specifically stated otherwise.

1 is a block diagram of a system for detecting abnormality of data according to an embodiment of the present invention. Referring to FIG. 1, a system for detecting abnormality of data may include an anomaly detection device 10, a reference information dictionary 20, and a detection object database. However, the configuration shown in Fig. 1 is not limited to the configuration described above.

The anomaly detection device 10 can perform the anterior detection based on the reference information dictionary 20 and the anomaly detection on the detection subject database 30 based on the learning result. The anomaly detection device 10 can perform detection of data that deviates from the normal category according to various algorithms on the column or data of the detection subject database 30. By learning the detection result, 30). ≪ / RTI > The configuration of the abnormality detection device 10 is described in detail with reference to FIG.

The reference information dictionary 20 may include preceding data or a source that enables the anomaly detection device 10 to perform the preceding learning. For example, the reference information dictionary 20 may include databases for various fields, and more specifically, for example, a normal blood pressure database by age, an average height of a student in Seoul, a weight database, , A 2-digit code for a country and a 3-digit code database, and a representative database of corporations.

The detection target database 30 is a database to be subjected to anomaly detection, and the aptitude detection device 10 can detect an abnormality with respect to the data included in the detection target database 30 based on the learning results.

The abnormality detection of the data through the anomaly detection device 10 can have an effect of managing the quality of the data. Data quality management can mean data management and improvement activities that are continuously performed to meet the expectations of an information system or database user within an organization or an organization. Data may refer to all data or information related to the information system that is constructed and operated to achieve the purpose. Data may include data values stored in the database, structure information such as data model or standard data, , But generally the data may refer to digital data stored in the information system.

Data quality management can also refer to the process of evaluating the current level by measuring the quality of the data contained in the information system currently being operated or managed and analyzing the factors of the quality degradation. Data quality management can be used to diagnose the quality of data by analyzing the data values based on the tables, columns, codes, relationships, and business rules of the operational database. By applying quality standards related to data values, This may mean a procedure for calculating the breakdown and analyzing the cause of the breakdown.

2 is a diagram showing a configuration of an anomaly detection device 10 according to an embodiment of the present invention. Referring to FIG. 2, the anomaly detection apparatus 10 may include a database 110 and an anomaly detection unit 120. However, the configuration of the anomaly detection device 10 shown in Fig. 2 is not limited to those described above.

The anomaly detection unit 120 detects whether the first data is abnormal based on the first reference information and the second reference information among the plurality of reference information including the data type, the metadata, and the domain through the first artificial intelligence algorithm . For example, if the first reference information is a data type, the second reference information is a domain, the data type is a fixed data type, and the domain is an amount domain, the abnormality detection unit 120 determines whether the first data is abnormal It can be detected through the Boxplot algorithm.

At this time, if the format of the abnormal data is different or the number of digits is different, the abnormal data or the format of the data which degrades the poorly inputted data quality can be described.

The data type may be classified into either a regular data type or an unstructured data type. In this case, the fixed data type may be formatted data and may be data stored in a fixed field. For example, it may refer to data stored in a fixed format composed of fixed fills in the database such as name, address, contact, shipping address, and payment information. On the other hand, unstructured data may mean data that is not stored in a fixed field. For example, moving picture data uploaded to a video streaming site, pictures and audio data stored in a SNS, conversation data exchanged through a messenger, And the like, and the like.

Metadata is data about the data, which can be defined as data created for some purpose. More specifically, the metadata may be structured data about the data and may mean data describing other data. In other words, metadata is data given to contents or data according to a certain rule in order to efficiently find and use information that is found in a large amount of information, and to analyze and classify structured information and add additional information It may mean the information that follows along with the data.

A domain generally refers to a characteristic of a column, which is the smallest unit of data assigned when designing a database. By applying a data service rule conforming to a domain, the integrity of data can be maintained. More specifically, a domain is a definition of the characteristics of a column, the smallest unit of data managed by a database, with a unique nature that applies to each column at the design stage of the table in the database. Examples of the domain may include an amount, a number, a rate, a code, an ID, a name, a content, a date, a flag, a number and a contact, but the present invention is not limited thereto.

An amount domain can mean a number related to money such as sales amount, sales amount, cost, and the like, in which the type of data is a numeric type. More specifically, the value domain is a value representing the amount of money, and may be composed of a value of a numeric type suited to a country's currency unit. By constantly managing the money domain, the data stored in the money domain can always be managed in a valid form, and the range of the data itself can be validated. Examples of money domain are money, tax, price, unit price, cost, charge, balance or total amount.

The number domain can mean a number, not an amount, such as the number of customers, the number of products, the number of audiences, among the data in which the type of data is a numeric type. More specifically, a numeric domain may mean numerically managed items such as number, scale, number of times, and so on. If you manage the quantity domain continuously, you can maintain the maximum value and the minimum value range for the data stored in the number domain. Examples of numerical domains may include the number of rows, the number of rows, the number of rows, the number, the distance, the scale, the length, the weight, the speed, the number, the equilibrium, the area or the temperature.

A rate domain can mean a number that includes the rate of accomplishment, accuracy, cost ratio, etc., of data whose type of data is a numeric type. More specifically, the rate domain can be defined and used in various ways, such as progress rate, growth rate, profit rate, rate of change, interest rate, addition rate, rate, etc. At this time, the rate of increase, , The interest rate, the addition rate, the rate, and the like can be managed as the reference information applied to the calculation of the other numerical data. Examples of rate domains are interest rates, interest rates, rates, exchange rates or percentages.

A code domain can contain code and values as predefined items of data whose type is numeric or character type. For example, if a woman is represented by an 'F' man and an 'M' by a woman, then 'F' and 'M' can be codes and women and men can be values. More specifically, a code domain may refer to data that has been replaced with a short code value to limit the data available or to manage the same semantically equivalent data. In general, the code domain is managed by code and code values . For example, the code may refer to a gender identification code, a customer rating code, a department code, a product code, a region code, and the code value may include data such as 'M', 'F' May represent values for representing or limiting. The code domain can be predefined and managed with standardized code.

The ID domain can mean a user-defined unique value, such as a USER ID.

The name domain includes a character string of a predetermined length or shorter, and may mean a customer name, a product name, and the like. More specifically, the name domain may include a name, a place, a customer name, an English customer name, a URL, an IP, and the like as a name attached to an object, a person,

The content domain includes a character string of a predetermined length or more, and may mean the content of a post or a self-introduction document. More specifically, the value of the content domain may refer to data describing a description of an object or an action or contents to be referred to, and definitions, descriptions, remarks, contents, abstracts, and the like may exist. The content domain may have properties that are composed of atypical characters.

The date domain can be a date, such as year, month, year, month, date, and the like. More specifically, the date domain may refer to an item managed by a date, and may include data indicating a date and time, such as a reception date, a registration date, a settlement date, and a transmission time. The date domain data type can use the date data type provided by the DBMS and the character type. When using the date data type provided by the DBMS, there is almost no error in the date value because the DBMS itself checks the invalid date value, but when the character type is defined, the wrong date value can be inputted.

The flag domain means whether there are two opposing values such as 0, 1, 'Y', 'N', 'True', 'False', 'True', and False '. More specifically, the flag domain may be composed of two or three simple classification values such as whether or not the data is normalized and managed in a plain domain, such as whether or not there is 'Y', 'N' But can be defined as a separate value. The data stored in the flag domain is always managed in the same form so that the consistency between the information systems can be maintained.

The number domain can mean a numerical value that has no meaningful increase in order, such as a customer number and a product number. More specifically, the number domain is composed of a combination of letters or numbers and can be managed according to the internal and external numbering system. The number domain is the data that commonly conforms to the national and international standard numbering system such as resident registration number, business registration number, postal code, corporation number, IP address, international standard number (ISBN / ISSN) There may be data managed according to the company's internal standard number management system such as number, approval number, registration number, and product number.

A contact domain may refer to data that includes information such as an address, an email, or a contact.

The anomaly detection unit 120 can detect the abnormality of the first data based on the data type through the first artificial intelligence algorithm. For example, if the first reference information is a data type, the second reference information is a domain, the data type is a fixed data type, and the domain is an amount domain, the abnormality detection unit 120 determines whether the first data is abnormal It can be detected through the Boxplot algorithm.

That is, when the data of the detection subject database 30 is the stereotyped data and the money amount domain, the abnormality detection unit 120 has an abnormal value that deviates from the normal range with respect to the data of the detection subject database 30 through the box plot algorithm It can detect.

The box plot algorithm is an algorithm that shows the degree of distribution of the data in a box form. It has a mean value, a median value, a lower first decile value indicating a value of the lower 25%, and an upper third value The upper limit value, the upper limit value, the lower limit value and the quadrature range can be measured, and data exceeding the upper limit value or data below the lower limit value can be detected as having an abnormal value or an abnormal value. In this case, the quartile score means a range obtained by subtracting the lower third value from the upper third value. Generally, the upper value is determined by multiplying the upper third value by 1.5, and the lower value is determined by multiplying the lower first value by 1.5 .

3A to 3C are diagrams illustrating abnormal detection of data according to an embodiment of the present invention. For example, referring to FIG. 3A, the operation of the abnormality detection unit 120 for mathematical sexual data in a sexual database of a high school is described. The anomaly detection unit 120 can detect the abnormal value through the box plot algorithm because the corresponding data is the fixed data and the number domain. Based on the box plot algorithm, the abnormality detection unit 120 can determine the average of the mathematical performance data (or the column including the mathematical performance data) to be 46.6 points, the upper third value to 53.8, and the lower first decile value to 38.6, Based on this, the upper limit value can be determined as 80.7, and the lower limit value can be determined as 25.4. The abnormality detection unit 120 can detect that the data has an abnormal value when data exceeding 80.7 or less than 25.4 is located in the detection subject database 30. [

If the data follows a normal distribution, the probability of exceeding the upper limit value of the box plot algorithm or having a value lower than the lower limit value may be as small as 0.35%, respectively.

On the other hand, if the first reference information is the fixed data and the second reference information is the number domain, the abnormality detection unit 120 may detect abnormality of the data through the histogram algorithm. The histogram algorithm is a graph showing the distribution of frequencies in a table as information graphs, which can be expressed as a graph of the frequency distribution table. At this time, the abnormality detection unit 120 can detect abnormality of the data based on the frequency distribution.

In another example, if the first reference information is a data type, the second reference information is metadata, the data type is an unstructured data type, and the JPEG (Joint Photographic Experts Group) format is used, It is possible to detect the abnormality of data through the Deep Learning algorithm.

For example, the abnormality detection unit 120 can detect abnormality of data based on the deep learning algorithm when the data is an unstructured data type and a picture file format. 3B, the anomaly detection unit 120 divides an image into cells of a predetermined size, extracts feature points (edge, outline) for each cell, obtains a histogram of the directions of the extracted feature points , An ideal value for the image can be detected based on the similarity between the histograms or the similarity between the direction of the extracted feature points or the direction vector. The degree of similarity between the direction vectors of the extracted minutiae points is higher as the angle difference between the directional vectors of the minutiae points is smaller, and it can be judged that the degree of similarity is lower as the angle difference between the direction vectors of the minutiae points is larger.

More specifically, for example, when the detection subject database 30 is an automobile insurance database, the abnormality detection unit 120 can detect images with low similarity as outliers based on the automobile contact accident image. The abnormality detection unit 120 may detect abnormal images that are not related to a contact accident between the automobile including the automobile photograph. At this time, the abnormality detection unit 120 learns the detection result based on the deep learning algorithm, and as the number of learning about the image accumulates, the accuracy of the detection can be increased, thereby improving the reliability of the detection of the outliers .

The anomaly detection unit 120 may store the anomaly detection result of the first data so that the stored anomaly detection result may be used as one of the reference information. In other words, in addition to the reference information learned through the reference information dictionary 20 of the abnormality detection unit 120, the detection result detected through the abnormality detection unit 120 can be learned to increase the accuracy and reliability of the abnormality detection .

The abnormality detection unit 120 may determine whether the second data is abnormal based on the second reference information and the third suspension information among the plurality of reference information including the data type, the metadata, the domain, and the storage result, And may detect abnormality of the third data through the third artificial intelligence algorithm based on the first reference information, the third reference information, and the storage result of the plurality of reference information.

For example, if the second criterion information is a data type, the third criterion information is a domain, the data type is a formal data type, and the domain is a name domain, whether the second data is abnormal is detected through a similarity algorithm, 2 data abnormality is detected, it is possible to standardize the data with high degree of similarity to the second data.

In other words, if the data type is a regular data type and is a name domain, the abnormality detection unit 120 determines the name having the largest number of redundant values as the standardized data based on the similarity of the data, The data can be detected as being abnormal.

Referring to FIG. 3C, for example, when the trademark name database is the detection subject database 30, the abnormality detection unit 120 can bundle the duplicated data and the highly similar data with respect to the data of the corresponding column, The name having the largest number of duplicate values can be determined as the standardized data. More specifically, the abnormality detection unit 120 can detect 105 data of two patterns of 'metro city' and 'metro city' based on the degree of similarity between data in the trademark name database , And five cases of 'Metro City' can be determined by detecting an outlier. At this time, the abnormality detection unit 120 may determine the metric city in which the largest amount of data is searched as the standardized data, and change the metric city, which is the data having the abnormal value, to the metric city, which is the standardized data, .

Similarly, the abnormality detection unit 120 may standardize the 'empty pole' to 'beanpole' by determining 'beanpole' having the greatest duplicate value among 'beanpole' or 'beanpole' as standardized data, MONTBLANC ',' MONTBLANC ', or' MONTBLANCC ', which is the most redundant value detected, can be determined as the standardized data.

The abnormality detection unit 120 may determine abnormality of the second data based on the second reference information and the third reference information among the plurality of reference information including the data type, the metadata, the domain, and the storage result and the first data, It can be detected through artificial intelligence algorithms. For example, if the second criterion information is a data type, the second criterion information is a domain, the second criterion information is a fixed data type, and the domain is a number domain, Based on the information, it is possible to detect the abnormality of the second data through the linear regression algorithm.

More specifically, when the detection object database 30 is a database including a standard key and a weight of a high school student, the abnormality detection unit 120 detects an abnormal value using a linear regression algorithm in consideration of the relationship between the key and the weight . That is, in the case where the height is 140 cm and the body weight is 100 kg, the abnormality detection unit 120 does not detect an abnormal value, but when a student having a height of 140 cm has a body weight of 100 kg, .

The abnormality detection unit 120 automatically detects an abnormal value for the data based on the artificial intelligence in the detection subject database 30 in consideration of the characteristics of the big data that is continuously changed and added and outputs the detection result to the user Manager) to display it in an easy-to-understand format. In other words, the anomaly detection unit 120 can perform a confirmation process from the user to reflect the outlier detection result and the detection result to the system by displaying the outliers detection result to the user.

The abnormality detection unit 120 may store the abnormality detection result of the second data in the database 110 and use it as abnormality detection of the detection subject database 30 as new reference information.

The database 110 may include big data and may include both structured data and unstructured data. The database 110 may include semi-structured data.

The abnormality detection unit 120 can detect an abnormal value for the detection subject database 30 based on a plurality of reference information including data columns or the like which are learning results stored in the database 110. Based on this, It is possible to gradually improve the accuracy and reliability of the outlier detection result by performing the iterative learning based on the outlier detection result detected through the detection. In other words, the anomaly detection unit 120 may store the outliers detection result in the database 110, and use the stored outliers detection result as one reference information to detect outliers for other data.

domain Usage algorithm Detail Price Boxplot
Histogram Outlier detection using box picture and histogram Number Boxplot
Histogram Outlier detection using box picture and histogram rate Boxplot
Histogram Outlier detection using box picture and histogram Picture Deep Learning Similarity-based image clustering through prior learning by photo type text Deep Learning Determine whether necessary metadata and values exist for each type of text using metadata

Table 1 illustrates an artificial intelligence algorithm that is determined for anomaly detection by the anomaly detection unit 120 according to a data type and a domain type. The abnormality detector 120 can detect an abnormal value through a box plot algorithm or a histogram algorithm in the case of a domain related to a number such as a money amount, a number rate domain, and the like, and can detect an irregular data type such as a photograph or text, In the case of a domain, it is possible to detect an abnormal value through a similarity algorithm between image similarity, text type, and metadata according to a deep learning algorithm. However, those shown in Table 1 are only examples of various embodiments of the present invention, and thus the present invention is not limited thereto.

That is, the abnormality detection unit 120 determines whether or not the detection target data is abnormal data based on various standard information such as whether the detection target data is fixed data or non-fixed data, the domain or domain determined for the data, metadata, domain, Determine an artificial intelligence algorithm for detection, and perform anomaly detection on the object data or database based on the determined artificial intelligence algorithm. Algorithms such as a box plot algorithm, a histogram algorithm, a linear regression algorithm, and a deep running algorithm may be included in the artificial intelligence algorithm, but the present invention is not limited thereto and various algorithms may be present.

FIG. 4 is a flowchart illustrating a process of detecting abnormality of data according to an embodiment of the present invention. Referring to FIG. 4, in step S401, the anomaly detection device 10 learns data on reference information for abnormal value detection based on the reference information dictionary 20, and stores the learning result in the database 110 in step S402 . Thereafter, in step S403, the abnormality detection device 10 performs abnormality detection on the data included in the detection subject database 30 based on the learning result, and in step S404, the detection result, the reference information used for detection, Information related to the detection result such as the artificial intelligence algorithm used may be stored in the database 110. [ In step S405, the anomaly detection apparatus 10 can detect abnormal values for other data using various artificial intelligence algorithms based on a plurality of reference information and stored results stored in the database 110. [

That is, the anomaly detection apparatus 10 according to an embodiment of the present invention can perform an outlier detection on data through a plurality of reference information and a plurality of artificial intelligence algorithms with respect to the detection subject database 30, As the frequency increases, the accuracy of detection of outliers increases through learning, and the accuracy and reliability of data quality management can be increased.

5 is an operational flow diagram illustrating a method for detecting anomalies in data in accordance with one embodiment of the present application. The anomaly detection method shown in Fig. 5 performs the operation of the explanatory error detection apparatus 10 through Figs. 1 to 4. Fig. Therefore, the description not illustrated in FIG. 5 is also applied to the description of the anomaly detection apparatus 10 described with reference to FIGS. 1 to 4, so that a detailed description thereof will be omitted.

Referring to FIG. 5, in step S501, the anomaly detection device 10 determines whether or not the first data is abnormal based on the first reference information and the second reference information among the plurality of reference information including the data type, the metadata, Through the first artificial intelligence algorithm, and stores an abnormality detection result of the first data in step S502.

Thereafter, in step S503, the abnormality detection apparatus 10 determines whether or not the second data is abnormal based on the second reference information and the third reference information among the plurality of reference information including the data type, the metadata, and the domain, 2 artificial intelligence algorithms.

The present invention can provide an apparatus and method for improving reliability of quality of big data by performing artificial intelligence based data outlier detection in a system to which big data is applied. To this end, we can determine an artificial intelligence algorithm for detection of anomaly based on various standard information and detect abnormal values for data through appropriate methods.

We can perform preliminary learning through artificial intelligence based on the reference information dictionary and perform outlier detection on the detection target database based on the preliminary learning result. By storing and learning outlier detection results, accuracy of outlier detection can be increased. By automating and automating the tasks that the engineer carries on manually for data quality management, this department can minimize the input workforce and perform continuous data quality management without human intervention.

It will be understood by those of ordinary skill in the art that the foregoing description of the embodiments is for illustrative purposes and that those skilled in the art can easily modify the invention without departing from the spirit or essential characteristics thereof. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is defined by the appended claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be interpreted as being included in the scope of the present invention.

10: Abnormal detection device
110: Database
120: abnormality detection unit
30: Database to be detected

Claims (11)

A method for detecting anomalies in data,
Detecting, through a first artificial intelligence algorithm, an abnormality of the first data based on first reference information and second reference information among a plurality of reference information including a data type, metadata, and domain;
Storing an abnormality detection result of the first data; And
The second artificial intelligence algorithm detects whether the second data is abnormal based on the second reference information and the third reference information among the plurality of reference information including the data type, the metadata, the domain, and the storage result The method comprising the steps of: The method according to claim 1,
The third artificial intelligence algorithm detects an abnormality of the third data based on the data type, the metadata, the first reference information and the third reference information among the plurality of reference information including the domain, and the storage result The method comprising the steps of: The method according to claim 1,
Wherein the data type classifies the first data into either a structured data type or an unstructured data type,
The step of detecting the abnormality of the first data
And detects whether the first data is abnormal based on the data type through the first artificial intelligence algorithm. The method of claim 3,
Wherein the step of detecting the abnormality of the first data comprises:
Wherein if the first reference information is the data type, the second reference information is the domain, the data type is the regular data type, and the domain is the money domain, Boxplot) algorithms. The method of claim 3,
Wherein the step of detecting the abnormality of the first data comprises:
Wherein if the first reference information is the data type and the second reference information is the metadata, if the data type is an unstructured data type and the Joint Photographic Experts Group (JPEG) format, An anomaly detection method, which is detected through a Deep Learning algorithm. The method according to claim 1,
Wherein the step of detecting the abnormality of the second data comprises:
The second data, the meta data, the second reference information and the third reference information among the plurality of reference information including the domain, and the storage result and the first data, An anomaly detection method that detects through an intelligent algorithm. The method according to claim 6,
Wherein the step of detecting the abnormality of the second data comprises:
Wherein when the second reference information is the data type, the second reference information is the domain, the data type is the fixed data type, and the domain is the number domain, And detecting whether the second data is abnormal through a linear regression algorithm. The method according to claim 1,
After detecting the abnormality of the second data,
And storing an abnormality detection result of the second data. The method according to claim 1,
The step of detecting the abnormality of the second data
And normalizes the second data when an abnormality of the second data is detected. 10. The method of claim 9,
Wherein the step of detecting the abnormality of the second data comprises:
And if the second reference information is the data type, the third reference information is the domain, the data type is the fixed data type, and the domain is the name domain, whether the second data is abnormal Detect,
And normalizes the second data based on the similarity algorithm when an abnormality of the second data is detected. An apparatus for detecting abnormality in data,
An abnormality detection unit for detecting an abnormality of the first data based on first reference information and second reference information among a plurality of reference information including a data type, metadata, and a domain through a first artificial intelligence algorithm; And
And a database for storing an abnormality detection result of the first data,
The abnormality detection unit may determine abnormality of the second data based on the data type, the metadata, the second reference information and the third reference information among the plurality of reference information including the domain, Which is an anomaly detection device.

Images