KR102110350B1 - Domain classifying device and method for non-standardized databases - Google Patents

Abstract

The present invention relates to a domain discrimination device for a non-standardized database. The domain discrimination device for non-standardized database comprises: a database containing a plurality of non-standardized data; a derived variable generation unit for generating a derived variable using a plurality of non-standardized data included in the database; a model learning unit that generates a domain discrimination model using a derived variable data set; a domain discrimination unit for determining a domain for a first data set using the domain discrimination model; and a data result unit that stores data in the database by linking the data included in the first data set with the domain discrimination result determined by the domain discrimination unit.

Description

Domain determination device and method for non-standardized database {DOMAIN CLASSIFYING DEVICE AND METHOD FOR NON-STANDARDIZED DATABASES}

The present invention relates to a domain discrimination apparatus and method for a non-standardized database.

In modern society, the value of big data is increasing, and not only IT, but also public institutions, legal, medical, and financial fields are using big data to create new values. In response to this move, the government has also started to invest heavily in big data-related industries. In particular, after 2017, public data was opened to promote the 4th industrial revolution.

To create new value using big data, highly reliable data must be premised. Analysis based on low-reliability data not only causes problems from the analysis process, but can also make errors in judgment based on the analysis results. For this reason, the private sector worldwide spends more than $600 billion a year to secure data reliability and quality, and studies on indicators for evaluating the quality control level are also under way. In Korea, public institutions are making investments to improve the quality of public data.

Data quality issues can be a problem across the AI industry and the creation of value from data mining. Data quality is defined as <a level that secures the latest, accurate, and interrelated data, and uses it to give useful value to users>. The purpose of conducting data quality diagnosis is to maintain and improve data quality systematically and continuously.

In the case of the domain classification work, since the user checks the data one by one and proceeds by hand, human errors frequently occur, and the work time also takes a long time. In order to solve this problem, a domain discrimination device (No. 10-2018-0097892) was proposed, but there was a disadvantage that the device can be applied only to a standardized database, and an algorithm with an overfitting problem was used. .

The background technology of the present application is disclosed in Korean Patent Registration No. 10-1930034.

The present invention is to solve the problems of the prior art described above, and to generate a derived variable for each column from a table of the database to be determined, and to provide a domain determination apparatus and method for automatically determining the domain of data using a machine learning algorithm It is aimed at.

An object of the present invention is to solve the above-described problems of the prior art, and to provide a stable domain discrimination apparatus and method using a random forest algorithm instead of a decision tree algorithm having an overfitting problem.

However, the technical problems to be achieved by the embodiments of the present application are not limited to the technical problems as described above, and other technical problems may exist.

As a technical means for achieving the above technical problem, the domain discrimination apparatus for a non-standardized database according to an embodiment of the present application includes a database including a plurality of non-standardized data and a plurality of non-standardized data included in the database. Derived variable generation unit for generating a derived variable using, a model learning unit for generating a domain discrimination model using a derived variable data set, a domain discrimination unit for determining a domain for the first data set using the domain discrimination model, and And a data result unit that stores data in the database by linking the data included in the first data set with the domain discrimination result determined by the domain discrimination unit.

According to an embodiment of the present application, the derived variable generator may generate a derived variable by extracting a characteristic value or a representative value of a plurality of non-standardized data included in the database.

According to one embodiment of the present application, the derived variable generator, whether the non-standardized data included in the database is a data type, a data maximum length, a data minimum length, a data length change, a length under the decimal point, a date format, or a contact format , Derivative variables corresponding to at least one of a blank ratio, whether to include enter, whether to write in English, whether to write a number, a ratio in hundreds or less, a grouping ratio, or PK can be generated.

According to one embodiment of the present application, the model learning unit may generate the domain discrimination model by applying a set of derived variable data whose domain is labeled to the derived variable generated by the derived variable generation unit to an artificial intelligence algorithm.

According to one embodiment of the present application, the domain discrimination unit uses the domain discrimination model to number a domain for the first data set among numbers, amounts, codes, numbers, dates, contents, rates, names, flags, and contacts It can be determined by at least one.

According to one embodiment of the present application, the data result unit may perform an analysis of the influence of the generated derivative variable on the domain discrimination result.

According to an embodiment of the present application, the domain discrimination apparatus for a non-standardized database includes a data providing unit providing a selection item related to domain discrimination input information to a user terminal and a user input receiving unit receiving domain discrimination input information from the user terminal. It may further include.

According to an embodiment of the present application, a domain discrimination method for a non-standardized database includes generating a derived variable using a plurality of non-standardized data included in a database including a plurality of non-standardized data, and a set of derived variable data. Generating a domain discrimination model using the domain discrimination model, determining a domain for the first data set using the domain discrimination model, and linking the data included in the first data set with the discriminated domain discrimination results to the database. And storing.

The above-described problem solving means are merely exemplary and should not be construed as limiting the present application. In addition to the exemplary embodiments described above, additional embodiments may exist in the drawings and detailed description of the invention.

According to the above-described problem solving means of the present application, the domain discrimination device and method for the non-standardized database can minimize the input manpower by automating the work that the existing data architecture engineer has manually performed using tools, and the data engineer expert Continuous data quality management can be facilitated without human intervention.

According to the above-described problem solving means of the present application, by automatically determining the domain of the data in the big data environment, it is possible to prevent data quality management that may be wrong due to an engineer's mistake.

According to the above-described problem solving means of the present application, regardless of whether or not the database is standardized, it is possible to broaden the versatility of using the device by generating a derived variable using data in the database and learning an automation model.

However, the effects obtainable herein are not limited to the above-described effects, and other effects may exist.

1 is a schematic configuration diagram of a domain discrimination apparatus for a non-standardized database according to an embodiment of the present application.
2 is an example of a data set extracted as a derived variable of a domain discrimination device for a non-standardized database according to an embodiment of the present application.
3 is a table showing a part of query statements for extracting derived variables of a domain discrimination device for a non-standardized database according to an embodiment of the present application.
FIG. 4 is a diagram illustrating the principle of a random forest model among supervised learning algorithms of a domain discrimination apparatus for a non-standardized database according to an embodiment of the present application.
FIG. 5 is a diagram exemplarily showing a result of a confusion matrix that can be confirmed in a data result unit of a domain discrimination device for a non-standardized database according to an embodiment of the present application.
FIG. 6 is a diagram schematically showing a domain discrimination result of a domain discrimination apparatus for a non-standardized database according to an embodiment of the present application.
7 is an operation flowchart for a domain discrimination method for a non-standardized database according to an embodiment of the present application.

Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings so that those skilled in the art to which the present application pertains may easily practice. However, the present application may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in order to clearly describe the present application in the drawings, parts irrelevant to the description are omitted, and like reference numerals are assigned to similar parts throughout the specification.

Throughout this specification, when a part is "connected" to another part, it is not only "directly connected", but also "electrically connected" or "indirectly connected" with another element in between. "It also includes the case where it is.

Throughout the present specification, when a member is positioned on another member “on”, “on the top”, “top”, “bottom”, “bottom”, “bottom”, this means that one member is attached to another member. This includes cases where there is another member between the two members as well as when in contact.

Throughout this specification, when a part “includes” a certain component, it means that the component may further include other components, not to exclude other components, unless otherwise stated.

Hereinafter, a domain discrimination apparatus for a non-standardized database according to an embodiment of the present application will be referred to as a domain discrimination apparatus 100 for convenience of description. In addition, columns and columns used interchangeably hereinafter are the same name of the column, and mean information located in each column constituting the table.

In addition, the domain described below generally refers to a characteristic of a column, which is the smallest unit of data given when designing a database, and if data business rules that conform to the domain are applied, the integrity of data can be maintained. . A domain is a unique property that is applied to each column in the design stage of a table in the database. It can be said to be a definition of the characteristics of a column, the smallest unit of data managed by the database.

According to an embodiment of the present application, in order to perform domain classification of columns included in a database in which the domain discrimination apparatus 100 is not standardized, derived variables are generated using data of each column, and a domain discrimination model and It may be a device for determining a domain using a data set generated as a derived variable.

In addition, the domain discrimination apparatus 100 performs pre-learning through artificial intelligence based on the domain and the derived variable of the column already included in the database in which the domain has been determined, and sets the derived variable data of the database to be determined based on the learning result. It is possible to perform domain discrimination. Also, the domain determination device 100 may determine a domain for each column included in the determination target database based on the learning result.

In addition, the domain discrimination apparatus 100 may discriminate the domain of the column into 10 domains such as number, amount, code, number, date, content, rate, name, flag, and contact information for the column included in the database. By storing and learning the type of the domain, it can be used to determine the domain of the data set in which the domain is not discriminated. Table 1 schematically shows domain classification, domain examples, and check contents determined by the domain discrimination apparatus 100.

Domain classification Domain example Inspection contents number Resident registration number, business registration number, postal code, customer number, account number Diagnosis of pattern and check bit of number related data Price Amount, tax, price, unit price, cost, charge, balance, total amount Diagnosis of allowable range of amount-related data designation Name, address, ID, place, customer name, English customer name, URL, email, IP Diagnosis of pattern and length of name-related data Quantity Number, number, round, count, distance, scale, length, weight, speed, count, balance, area, temperature Diagnosis of allowable range of data related to quantity flag Presence, presence, classification, status Diagnosis of standard definition values of classification-related data date Year, Year, Year Date, Hour, Minute, Second, Day, Half Year, Quarter Diagnosis of allowable range and effective value of date related data rate Interest rate, interest rate, rate, exchange rate, percentage Diagnosis of allowable range of data related to percentage (%) Contents Content, remarks, explanation, information, summary Diagnosis of applied language patterns of content-related data Contact Address, phone number, email Diagnose patterns in contact data code Classification, type Diagnose whether code-related data is duplicated or repeated

According to one embodiment of the present application, the domains used in the domain discrimination device 100 are 10 types specified in Table 1, but the types of domains are increased in accordance with the needs of the user or the characteristics of the business in addition to the domains specified in the specification. It can be reduced or modified. However, for this, the labeling of the derived variable data set used in prior learning must be modified. In addition, the domain discrimination apparatus 100 may automatically determine or learn a new type of domain based on artificial intelligence in a big data environment, thereby enabling quality control for the domain. More specifically, prior learning of the machine learning algorithm model is performed based on the domain and the derived variable of the column of the database where the domain has already been determined, and discrimination of the data domain in the database to be determined is performed based on the results of the prior learning. It can be done.

In general, data quality management may refer to a procedure for evaluating the current level by analyzing the quality of data contained in an information system currently being operated or managed, and analyzing factors of quality degradation. Data quality management can mean diagnosing the quality of data by analyzing the values of data based on tables, columns, codes, relationships, business rules, etc. of the operational database, and applying quality standards related to data values to error It can mean the process of calculating the details and analyzing the cause of the error.

1 is a schematic configuration diagram of a domain discrimination apparatus for a non-standardized database according to an embodiment of the present application.

Referring to FIG. 1, the domain discrimination apparatus 100 includes a database 110, a derivative variable generation unit 120, a model learning unit 130, a domain discrimination unit 140, a data result unit 150, and a data providing unit It may include a 160 and the user input receiving unit 170.

The database 110 may include a plurality of non-standardized data. In other words, the database 110 may include a database (determination target database) including data that is a target of domain determination. Here, the non-standardized data is data that has not been standardized.

In addition, the database 110 may include a database storing the derived variable generated by the derived variable generator 120 and the domain result determined by the domain determining unit 130. In addition, the database 110 may store data as a database by linking data for domain determination, derived variables, and domain determination results. Also, the database 110 may store the result of discrimination of the domain of the target database (eg, the first data set) and the domain of the derived variable and the column generated for each column of the target database.

For example, the database 110 may include a plurality of data sets including numeric data and/or character data. The database may contain unstructured data. Unstructured data, unstructured data, and unstructured data may mean information that does not have a predefined data model or is not organized in a predefined manner. Unstructured data may mean unstructured data having different shapes and structures, such as pictures, images, and documents, unlike numeric data having a certain standard or shape. For example, the data set may include a set of data derived from the medical field, financial field, and the like. A data set can store data in a form of a table, called rows (rows, rows) and columns (columns: columns, columns). Column items included in the data set can be divided into a representative key and a general column (data). The representative key is a representative item of the column item. For example, referring to FIG. 6, items such as POLY_NO, CUST_ID, and CUST_ROLE included in the column name may be referred to as a representative key, but are not limited thereto. In addition, the data set may include a plurality of column items. A plurality of column items may be divided into a plurality of derived variables, domains, and the like. Each column item may be stored in the first row data, and data included in the column item may be stored in each row of the column items.

According to an embodiment of the present application, the derived variable generator 120 may generate a derived variable using a plurality of non-standardized data included in the database 110. In other words, the derived variable generator 120 may generate a derived variable from the database to be determined. That is, the derived variable generator 120 may generate a derived variable using data of the database to be determined. The derived variable generator 120 may generate and store a derived variable required for domain determination based on a plurality of columns included in the database 110.

In addition, the derived variable generator 120 may generate a derived variable by extracting a characteristic value or a representative value of a plurality of non-standardized data included in the database 110. For example, the derived variable generator 120 may extract characteristic values or representative values of data included in the column as a query statement in the database to be determined. The derived variable generator 120 may store the extracted results corresponding to the derived variable table in the database 110. At this time, in the case of a column in which the domain is already determined, the derived variable generator 120 may also store the domain.

2 is an example of a data set extracted as a derived variable of a domain discrimination device for a non-standardized database according to an embodiment of the present application.

2, the derived variable data set 1 may include a plurality of derived variable column items 12 and a domain column item 13. The derived variable generator 120 may store derived variables corresponding to each derived variable column item 12. In addition, when the domain data exists, the derived variable generator 120 may store the data in the domain column item 13.

3 is a table showing a part of query statements for extracting derived variables of a domain discrimination device for a non-standardized database according to an embodiment of the present application.

According to one embodiment of the present application, the derived variable generation unit 120 may change the non-standardized data included in the database 110 into a data type, a data maximum length, a data minimum length, a data length change, a length under the decimal point, or a date format, Derived variables corresponding to at least one of a contact type, a blank ratio, whether to include an enter, whether to write in English, whether to write a number, a ratio in hundreds or less, a grouping ratio, or a PK can be generated. The plurality of derived variables described above are derived variables determined based on whether the user can determine the domain of the data column based on the derived variables.

As an example, the derived variable generator 120 uses a SQL query statement in the database to be determined, data type, data maximum length, data minimum length, data length change, length under decimal point, date format, contact format, space Ratios, whether to include enter, whether to write in English, whether to write numbers, ratios in hundreds or less, grouping ratios, PK or not can generate at least one derived variable and store it in the database 110.

Referring to FIG. 3 as an example, the derived variable generator 120 may write an SQL statement to generate the derived variable. In FIG. 3, only an example of an SQL statement for generating a derived variable of whether the maximum length, the data length is variable, or the contact type is shown, but is not limited thereto. In addition, if the same result is generated with SQL query statements for derived variables regarding maximum length, variable data length, and contact type, which are three of the SQL statements for generating derived variables, you can write different query statements.

Table 2 shows the types and descriptions of the derived variables generated in the derived variable generator 120.

variable Explanation Data type Variable that can distinguish data values such as INT, CHAR, VARCHAR, etc. Maximum data length The length of the data in the column that has the maximum length Minimum data length The length of the data in the column that has the minimum length Data length change Whether the data length in the column is variable Length below decimal point Length of data in column below decimal point Date format Date format data, not data type Whether the contact is formatted Data using patterns used in contacts and addresses such as @,-
Presence Blank ratio Whether the space is occupied by the entire data Whether to include enter Whether a line break occurs in the data in the column English writing Whether the data was written in English only Whether to write numbers Whether the data is written only as numbers Percentage of hundreds or less Percentage of less than 100 units of data in the column as 000 Grouping rate The ratio that can be grouped among the data in the column PK or not Whether the column is set as the primary key

However, the derived variable described in Table 2 above is only an example, and the derived variable is not limited thereto. That is, the derived variable may include representative values or characteristic values that can be extracted from data of columns not specified herein. According to an embodiment of the present application, the data type of the derived variable determined by the derived variable generator 120 may mean the type of data stored in the column in the database to be determined. For example, it may be classified as a number, character, or string type, and may be classified and stored as'NUMBER','CHAR', or'VARCHAR'.

In addition, according to an embodiment of the present application, among the derived variable determined by the derived variable generator 120, the minimum and maximum data length and whether the data length is changed can determine the actual length of data stored in a column in the database to be determined. have. This is not the length of the data type stored in the schema, but the length of each data used in the actual data and whether the length is changing in the data can be identified and stored.

According to one embodiment of the present application, the model learning unit 130 may generate a domain discrimination model using the derived variable data set 1. In other words, the model learning unit 130 may generate a domain discrimination model by applying the derived variable data set 1 having the domain label to the derived variable generated by the derived variable generation unit 120 to the artificial intelligence algorithm. . Here, the AI algorithm may be a random forest algorithm, but is not limited thereto, and the AI algorithm may be replaced by at least one of the classification algorithms. As an example, the model learning unit 130 may generate a domain discrimination model by training a random forest algorithm model using a derived variable labeled with a domain stored in the database 110.

FIG. 4 is a diagram illustrating the principle of a random forest model among supervised learning algorithms of a domain discrimination apparatus for a non-standardized database according to an embodiment of the present application.

The random forest in machine learning is a kind of ensemble learning method used for classification, regression analysis, etc., and operates by outputting class or average prediction values from a plurality of decision trees constructed in a training process.

For example, referring to FIG. 4, the random forest creates random n decision trees defined by the user and classifies the values determined therein through voting in the algorithm. Therefore, the model learning unit 130 may provide a user with a correct answer probability for each classification target (eg, domain classification).

According to one embodiment of the present application, the model learning unit 130 may train the artificial intelligence algorithm using only a data set labeled as a domain. In addition, the model learning unit 130 may automatically determine the domain in the case of an unlabeled data set by the domain determination unit 140. In addition, in the case of an unlabeled data set, the model learning unit 130 may determine the domain based on the user input information provided from the user input receiving unit 170.

In addition, the model learning unit 130 may develop the domain discrimination model by retraining the model using the result of determining the new data set (eg, the first data set).

For example, the types of domains to be trained and predicted by the model learning unit 130 and the domain discrimination unit 140 include numbers, amounts, codes, numbers, dates, contents, rates, names, flags, and contacts. It may include, but is not limited to, the type of domain may be reduced or increased according to the user's convenience and the type of business.

According to an embodiment of the present application, the domain discrimination unit 140 may determine a domain for the first data set using a domain discrimination model. Here, the first data set may be an unused data set when generating a domain discrimination model. Also, the first data set may be a derived variable data set that is not domain labeled. For example, the domain discrimination unit 140 may use the domain discrimination model generated by the model learning unit 130 to determine the domain for the unlabeled derived variable data set.

In addition, according to an embodiment of the present application, the domain discrimination unit 140 uses the domain discrimination model to number the domain for the first data set by number, amount, code, number, date, content, rate, name, flag, It can be determined by at least one of the contacts. However, the domains classified above are not limited thereto, and various domains may exist.

Also, the number domain may mean a numerical value that does not have a meaning that increases sequentially, such as a customer number or a product number. More specifically, the number domain is composed of a letter or a combination of numbers and can be managed mostly according to internal and external numbering systems. The number domain is data that commonly complies with domestic and foreign standard numbering systems such as social security number, business registration number, postal code, corporate number, IP address, and international standard data number (ISBN/ISSN), or user number, account number, and permission. There may be data managed according to the standard number management system internal to the company, such as number, approval number, registration number, and product number. Although ID domains are distinguished, they can be integrated into number domains.

In addition, the money domain may mean numbers related to money, such as sales, sales, and cost, among data in which the data type is a numeric type. In more detail, the money domain is a value representing the amount of money, and may be configured as a number type value suitable for each country's currency unit. By continuously managing the money domain, the data stored in the money domain can always be managed in a valid form and the validity of the scope of the data itself can be maintained. Examples of the money domain may include money, tax, price, unit price, cost, fee, balance or total amount.

In addition, the code domain may include codes and values as predefined items among data in which the data type is numeric or character type. For example, if a woman expresses'F' as a man and'M','F' and'M' can be codes, and a woman and a man can be values. More specifically, code domains can mean data that is replaced by simple code values to limit the data that can be used or to manage data of the same meaning in the same expression, and code domains are generally managed by code and code values. Can be. As an example, the code may mean a gender classification code, a customer grade code, a department code, a product code, a region code, etc., and the code value may include data in the corresponding information item such as'M','F' of the gender classification code. It can mean a value to represent or limit. In the code domain, standardized codes may be predefined and managed.

In addition, the number domain may mean a number that is not an amount of money such as the number of customers, the number of products, or the number of audiences among data in which the data type is a number type. In more detail, the number domain may mean items managed by numbers, such as the number, size, and number of times. By continuously managing the quantity domain, it is possible to maintain the effective range of the maximum and minimum values for data stored in the quantity domain. Examples of the number domain may include the number of cases, number of copies, number of rounds, number, distance, scale, length, weight, speed, number, balance, area, or temperature.

Also, the date domain may mean dates such as year, year, month, day, and date. In more detail, the date domain may mean an item managed as a date, and may include data indicating a date and time, such as a reception date, a registration date, a settlement month, and a transmission time. The data type of the date domain can use the method and character type using the date data type provided by the DBMS. When using the date data type provided by DBMS, there is almost no error in the date value because the DBMS itself checks for an invalid date value, but when it is defined and used as a character type, an incorrect date value may be entered.

In addition, the content domain includes a character string of a predetermined length or more, and may mean content in a post or self-introduction. In more detail, the value of the content domain may mean data describing an object or an action or content that can be referred to, and examples such as definition, description, remarks, content, and summary may exist. The content domain may have characteristics composed of atypical characters. The date domain may refer to dates such as year, year, month, day, and date. In more detail, the date domain may mean an item managed as a date, and may include data indicating a date and time, such as a reception date, a registration date, a settlement month, and a transmission time. The data type of the date domain can use the method and character type using the date data type provided by the DBMS. When using the date data type provided by the DBMS, there is almost no error in the date value because the DBMS itself checks for an invalid date value, but when defined and used as a character type, an incorrect date value may be entered.

In addition, the rate domain may mean a number including a ratio such as achievement rate, accuracy, and cost rate among data in which the data type is a numeric type. In more detail, the rate domain can be defined and utilized in various ways such as progress rate, growth rate, rate of return, rate of change, interest rate, addition rate, rate, etc. At this time, the rate of increase, rate of return, rate of change, etc. can be calculated by a calculation formula, The interest rate, addition rate, and rate can be managed as reference information applied to the calculation of other numerical data. Examples of rate domains include interest rates, interest rates, rates, exchange rates, or percentages.

Further, the name domain includes a character string of a predetermined length or less, and may mean a customer name, a product name, and the like. In more detail, the name domain is a name attached to an object, person, organization, etc. in order to distinguish it from others, and may include a name, a place, a customer name, an English customer name, a URL, an IP, and the like.

In addition, the flag domain indicates whether or not, and may be composed of two opposite values: 0, 1,'Y','N','true', false','True', False'. In more detail, the flag domain is one of the domains in which standardization and management of data is simple, and may be composed of 2 to 3 simple classification values, such as whether or not,'Y','N', or 1, 0, and the like. Can be defined as a separate value. Data stored in the flag domain is always managed in the same form to maintain consistency between information systems.

Further, the contact domain may refer to data including information such as an address, email, and contact information.

According to an embodiment of the present application, the data result unit 150 may store the data included in the first data set and the domain discrimination result determined by the domain discrimination unit in the database. In addition, the data result unit 150 may check, modify, and store the results of the model learning unit 130 and the domain discrimination unit 140. Also, the data result unit 150 may input a domain for the derived variable data set 1. The entered domain may be used as a label value of training data of the model learning unit 130.

In addition, the data result unit 150 may train the model discrimination model 130 by dividing it into training data and test data when the domain discrimination model is generated, and in case of training, test the domain discrimination model using the test data. The data result unit 150 may check the performance of the model such as accuracy, precision, reproducibility, and F1 score by testing the domain discrimination model using test data. The data result unit 150 may be identified by an indicator of performance of the domain discrimination model and a confusion matrix of the number of correct and incorrect answers for each domain.

FIG. 5 is a diagram exemplarily showing a result of a confusion matrix that can be confirmed in a data result unit of a domain discrimination device for a non-standardized database according to an embodiment of the present application.

Referring to FIG. 5 as an example, as possible, the Y-axis is a classification of real domains in the drawing, and the X-axis is a classification of domains predicted through a domain discrimination model. Through the combination of the X-axis and the Y-axis, it is possible to check the ratio of the correct answer of each domain and the pattern of the incorrect answer.

The data result unit 150 may perform an analysis of the influence of the generated derived variable on the domain determination result. In other words, the data result unit 150 may check the influence of the variable as a result of the model learning unit 130. The degree of influence of variables can be expressed as a percentage of how much each derived variable affects domain classification.

Table 3 shows the degree of influence of the derived variables in percentage.

variable Impact Data length change 19.60% Date format 15.40% Maximum data length 15.30% Grouping rate 12.70% Minimum data length 8.90% Data type 7.70% Blank ratio 5.60% Whether the contact is formatted 4.40% Whether to write numbers 4.30% Percentage of hundreds or less 3.20% PK or not 2.10% Whether to include enter 0.40% Length below decimal point 0.20% English writing 0.20%

In addition, the data result unit 150 may check and correct the discrimination result of the domain discrimination unit 140. The data result unit 150 may correct the result when the domain determination result is wrong. In addition, the data result unit 150 may provide the determination result of the domain determination unit 140 to the user terminal (not shown) through the data providing unit 160. The user input receiving unit 170 may receive a user input for modifying the judgment result and provide a user input to the data result unit 150. The data result unit 150 may correct the discrimination result of the domain discrimination unit 140 based on the user input information and store it in the database 110. In addition, the data result unit 150 stores the domain discrimination result in the database 110, but when the domain discrimination result is stored, the data discrimination result may be stored in connection with the derived variable data. In addition, the data result unit 150 may retrieve and modify the domain determination result stored in the database 110.

According to one embodiment of the present application, the data providing unit 160 may provide a selection item related to domain discrimination input information to a user terminal (not shown). The user may check the items related to the domain determination displayed on the user terminal (not shown), and select a modification. As an example, a selection item related to domain determination may include a selection item related to confirmation and modification of a domain determination result according to a request of the data result unit 150.

According to an embodiment of the present application, the user input receiver 170 may receive domain discrimination input information from a user terminal (not shown). In other words, the user input receiver 170 may receive user input information related to domain selection selected by the user from a user terminal (not shown). The user input receiving unit (not shown) may provide corresponding information to each unit (unit) that has requested a selection item related to data prediction input information. As an example, the domain discrimination input information may be user input information for an item provided in a confirmation and modification menu of a domain discrimination result.

According to one embodiment of the present application, the domain discrimination apparatus 100 may provide a selection menu required for domain discrimination to a user terminal (not shown). For example, a user terminal (not shown) downloads and installs an application program provided by the domain discrimination device 100, and a domain discrimination menu may be provided through the installed application. The selection menu required for domain identification may include a final domain selection menu, but is not limited thereto.

The domain discrimination apparatus 100 may include a user terminal (not shown) and transmits/receives data, contents, and various communication signals through a network, and includes all types of servers, terminals, or devices having functions of data storage and processing. .

The user terminal (not shown) is a device interworking with the domain discrimination device 100 through a network, for example, a smart phone, a smart pad, a tablet PC, a wearable device, and a PCS (Personal Communication System). ), Global System for Mobile Communication (GSM), Personal Digital Cellular (PDC), Personal Handyphone System (PHS), Personal Digital Assistant (PDA), International Mobile Telecommunication (IMT)-2000, Code Division Multiple Access (CDMA)-2000 , W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet), and all kinds of wireless communication devices and desktop computers, fixed terminals such as smart TVs.

Examples of networks for sharing information between the domain discrimination device 100 and a user terminal (not shown) include a 3rd Generation Partnership Project (3GPP) network, a Long Term Evolution (LTE) network, a 5G network, and World Interoperability for Microwave Access (WIMAX). ) Network, wired and wireless Internet (LAN), Local Area Network (LAN), Wireless Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), Bluetooth network, Wifi network, NFC ( Near Field Communication (Near Field Communication) network, satellite broadcasting network, analog broadcasting network, DMB (Digital Multimedia Broadcasting) network, and the like may be included, but are not limited thereto.

FIG. 6 is a diagram schematically showing a domain discrimination result of a domain discrimination apparatus for a non-standardized database according to an embodiment of the present application.

For example, referring to FIG. 6, the domain discrimination apparatus 100 may determine a data set to perform domain discrimination. For example, the domain discrimination apparatus 100 may perform a domain discrimination of data included in CNTT_DATA, and the schema name of METADB of the DBMS is FINTECH.

The domain discrimination apparatus 100 may classify and provide a domain discrimination result corresponding to each column name into a recommended domain 14, a recommended domain probability 15, and a final domain 16. The recommended domain 14 may be a result of determining the domain of the data using the artificial intelligence algorithm in the domain determination device 100. In addition, the recommended domain probability 15 may be accuracy for domain discrimination determined by the domain discrimination apparatus 100. Also, the final domain 16 may be a domain determination result modified by a user, not a domain determined by the domain determination apparatus 100. In other words, the final domain 16 may be information that can be changed based on user input information.

As described above, the domain discrimination apparatus 100 may perform domain discrimination through a supervised learning algorithm learned by using a derived variable generated by using data in the database with respect to the database 110 to be detected. As the number of times increases, the accuracy of domain discrimination increases through learning, thereby increasing the reliability of data quality management.

In addition, the domain discrimination apparatus 100 may provide an apparatus and method for improving reliability of quality for big data by performing automatic domain identification based on artificial intelligence in a system to which big data is applied. To this end, the domain discrimination apparatus 100 may discriminate domains through various classification criteria for automatically discriminating domains defined in the data quality guidelines.

In addition, the domain discrimination apparatus 100 may perform pre-learning through artificial intelligence based on a standard data dictionary, and perform domain discrimination on a supported database based on the pre-learning results. The domain discrimination apparatus 100 can increase the accuracy of domain discrimination by storing and learning the domain discrimination results, and such a domain discrimination apparatus 100 can automate the work manually performed by engineers for data quality management. In addition, by automating, it is possible to minimize input manpower and perform continuous data quality management without human intervention.

Hereinafter, based on the details described above, the operation flow of the present application will be briefly described.

7 is an operation flowchart for a domain discrimination method for a non-standardized database according to an embodiment of the present application.

The domain determination method for the non-standardized database illustrated in FIG. 7 may be performed by the domain determination apparatus 100 for the non-standardized database described above. Therefore, even if omitted, the description of the domain discrimination apparatus 100 for a non-standardized database can be applied to the description of the domain discrimination method for a non-standardized database.

In step S701, the domain determination apparatus 100 may generate a derived variable using a plurality of non-standardized data included in the database 110 including a plurality of non-standardized data.

In step S702, the domain discrimination apparatus 100 may generate a domain discrimination model using the derived variable data set.

In step S703, the domain discrimination apparatus 100 may discriminate the domain for the first data set using the domain discrimination model.

In step S704, the domain determination device 100 may store the data included in the first data set and the determined domain determination result in association with the database 110.

In the above description, steps S701 to S704 may be further divided into additional steps, or combined into fewer steps, according to an embodiment of the present application. Also, some steps may be omitted if necessary, and the order between the steps may be changed.

The domain discrimination method for a non-standardized database according to an embodiment of the present application may be implemented in a form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and magnetic media such as floptical disks. -Hardware devices specifically configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language code that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

In addition, the above-described domain discrimination method for a non-standardized database may also be implemented in the form of a computer program or application executed by a computer stored in a recording medium.

The foregoing description of the present application is for illustration, and a person having ordinary knowledge in the technical field to which the present application belongs will understand that it is possible to easily change to other specific forms without changing the technical spirit or essential characteristics of the present application. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present application is indicated by the claims below, rather than the detailed description, and it should be interpreted that all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof are included in the scope of the present application.

100: domain determination device
110: database
120: derived variable generator
130: model learning department
140: domain discrimination unit
150: data result unit
160: data providing unit
170: user input receiver

Claims (8)

In the domain discrimination device for a non-standardized database,
A database including a plurality of non-standardized data;
A derived variable generator for generating derived variables using a plurality of non-standardized data included in the database;
A model learning unit generating a domain discrimination model using a derived variable data set;
A domain discrimination unit for determining a domain for a first data set using the domain discrimination model; And
A data result unit that stores data in the database by linking data included in the first data set and domain determination results determined by the domain determination unit;
Including,
The model learning unit,
Re-learning the domain discrimination model using the new data set stored by linking the data included in the first data set with the domain discrimination result determined by the domain discrimination unit,
The data result section,
A domain discrimination apparatus that checks and corrects the discrimination results of the model learning unit and the domain discrimination unit, and stores them in the database. According to claim 1,
The derived variable generator,
The domain discrimination apparatus is to generate a derived variable by extracting a characteristic value or a representative value of a plurality of non-standardized data included in the database. According to claim 2,
The derived variable generator,
The non-standardized data included in the database is the data type, the maximum data length, the minimum data length, the data length change, the length under the decimal point, the date format, the contact format, the blank ratio, whether to include the enter, whether to write in English, or to write numbers A domain discrimination device that generates a derivative variable corresponding to at least one of whether or not, a percentage of a hundred or less, a grouping rate, or a PK. According to claim 3,
The model learning unit,
The domain discrimination device is to generate the domain discrimination model by applying a set of derived variable data whose domain is labeled to the derivation variable generated by the derivation variable generation unit to an artificial intelligence algorithm. According to claim 1,
The domain discrimination unit,
The domain discrimination apparatus is to determine a domain for the first data set using at least one of a number, an amount, a code, a number, a date, a content, a rate, a name, a flag, and a contact information, using the domain discrimination model. According to claim 1,
The data result section,
A domain discrimination apparatus that performs an analysis of the influence of the generated derived variable on the domain discrimination result. According to claim 1,
A data providing unit that provides a selection item related to domain discrimination input information to a user terminal; And
And a user input receiving unit receiving domain determination input information from the user terminal. In the domain discrimination method for a non-standardized database,
Generating a derived variable using a plurality of non-standardized data included in a database including a plurality of non-standardized data;
Generating a domain discrimination model using the derived variable data set;
Determining a domain for a first data set using the domain discrimination model; And
Storing the data included in the first data set and the determined domain discrimination result in the database,
Including,
Generating the domain discrimination model,
Re-learning the domain discrimination model using a new data set stored in association with the data included in the first data set and the discriminated domain discrimination result,
The step of storing in the database,
The method for determining a domain, wherein the generated domain discrimination model and the discriminated domain discrimination result are checked and corrected and stored in the database.

Images