KR102251139B1 - A missing value correction system using machine learning and data augmentation - Google Patents

Abstract

The present invention relates to a missing value correction system using data augmentation-based machine learning, configured to create new record data by synthesizing existing record data, with respect to table data composed of a plurality of records, add the created new data to augment a record set, and train a neural network model with the augmented record set. The missing value correction system using data augmentation-based machine learning comprises: a basic data input unit which receives basic data and configures the received basic data as a dataset (hereinafter, referred to as a first record set) of a record structure; a missing data detection unit which detects missing records from the first record set and distinguishes missing records from integrity records; a record augmentation unit which creates a new record by synthesizing at least two records in the first record set, adds the created new record to the first record set, and augments the record set; a model learning unit which configures a training dataset using the augmented record set (hereinafter, referred to as a second record set) and trains a machine learning model with the training dataset; a missing value prediction unit which inputs the missing record of the first record set into the machine learning model and predicts a missing value of the corresponding missing record with an output value; and a correction data creation unit which corrects the missing record of the first record set by correcting the missing value with a predicted value. By means of the system described above, new record data is created by synthesizing existing record data so that even when there is little basic data, training data can be sufficiently created, and through this, missing values can be predicted more accurately.

Description

{A missing value correction system using machine learning and data augmentation}

The present invention generates new record data by synthesizing existing record data for table data composed of a plurality of records, augments a record set by adding the generated new data, and trains a neural network model with the augmented record set It relates to a system for correcting missing values in a machine learning method based on data augmentation.

In general, an online analytical processing system, that is, an OLAP system, is a representative example of a decision support system, and enables a multidimensional analysis that can analyze data using multiple criteria (dimensions). In other words, the online analysis processing system provides data that has already been built in various aspects and provides it in a standardized form.

However, if some data is missing in the constructed data, the analyzed result value may be analyzed completely differently. Therefore, in the OLAP system, the processing of missing values in tabular data is very important in the decision-making process. In addition, the results may vary depending on the treatment method.

Therefore, a lot of research is being conducted on how to correct missing values in the OLAP system. Statistically, the method of solving missing values is as follows.

First, you can delete and analyze the missing row. This method has a problem in that the analysis result may be distorted when there are many missing values.

In addition, it is possible to fill in a column-based mean or median. This way, the calculation process is easy. However, this method can be applied in the case of a small amount of data, but there is a disadvantage that the correlation between features is not considered and category data cannot be used. In particular, when the data is time series data, a technique for predicting missing values using an interpolation technique has also been proposed [Patent Document 1].

In addition, it is possible to fill in the mode (Most Frequent) or 0 (zero). This approach also does not take into account the correlation between features. In particular, this method can be used in case of category data.

In addition, a technique for predicting missing values using the k-NN technique has been proposed [Patent Document 2]. As an example, the missing value of clinical data is predicted by k-NN (k-nearest neighbor algorithm). This method is more accurate than the statistical method, but the computational amount is relatively increased.

In addition, a technique for solving missing values by machine learning has been proposed [Patent Document 3]. That is, in the prior art, a machine learning model is trained using integrity data of basic data, and a missing value is predicted by applying row data of the missing data to the learned machine learning model.

Compared to other mathematical methods, machine learning methods are more efficient and accurate when there are many basic data. However, there is a problem in that table-type data cannot be learned with a small amount of data because deep learning is performed.

Korean Patent Laid-Open Patent No. 10-2020-0082893 (published on July 8, 2020) Korean Patent Publication No. 10-2019-0021471 (published on Mar 5, 2019) Korean Patent Publication No. 10-2020-0030303 (published on March 20, 2020) Korean Patent Registration No. 10-2058124 (announced on December 23, 2019)

An object of the present invention is to solve the above-described problem, for table data composed of a plurality of records, a record set by synthesizing existing record data to generate new record data, and adding the generated new data. It is to provide a system for correcting missing values based on data augmentation-based machine learning, which augments and trains a neural network model with an augmented record set.

In addition, it is an object of the present invention to generate a new record by synthesizing at least two records with random weights from an existing record set, but when the field value of the record is categorical data, the weight is set as a probability vector of the categorical value. It is to provide a system for correcting missing values in a machine learning method based on data augmentation.

In addition, it is an object of the present invention to generate a new record by synthesizing at least two records from an existing record set with an arbitrary weight, but synthesizing a field with a missing value among the records as having no corresponding value. It is to provide a system for correcting missing values of the method.

In order to achieve the above object, the present invention relates to a system for correcting missing values based on a data augmentation-based machine learning method, comprising: a basic data input unit configured to receive basic data and configure a data set of a record structure (hereinafter, referred to as a first record set); A missing data detection unit detecting missing records in the first record set and distinguishing between missing records and incomplete records; A record augmenting unit for synthesizing at least two records from the first record set to create a new record, and adding and enhancing the generated new record to the first record set; A model learning unit that constructs a training data set using the augmented record set (hereinafter referred to as a second record set) and trains a machine learning model using the training data set; A missing value predictor for inputting the missing record of the first record set into a machine learning model and predicting a missing value of the corresponding missing record using the output value; And a correction data generation unit correcting the missing record in the first record set by correcting the missing value to a predicted value.

In addition, in the present invention, in the data enhancement-based machine learning system for correcting missing values, the record enhancement unit extracts at least two records (hereinafter, synthesized records) from the first record set, and weights the extracted at least two records. It is characterized in that a new record is created by weighting by the sum.

In addition, the present invention is characterized in that the sum of the weights is 1 in the missing value correction system based on data augmentation-based machine learning.

In addition, in the system for correcting missing values based on a data augmentation-based machine learning method, the present invention is characterized in that the record augmentation unit randomly sets the weight each time it is synthesized.

Further, in the present invention, in the system for correcting missing values based on a data augmentation-based machine learning method, when the record augmenting unit extracts the composite record from the first record set, the field value having a missing value if the composite record is a missing record. Is characterized in that it is set to have a value of 0.

In addition, the present invention is a system for correcting missing values based on a data augmentation-based machine learning method, when a field of the composite record is categorical data, a field value of the corresponding field is set as a categorical probability vector, and the categorical probability vector is the The category probability element of the field of the composite record has a probability value of 1, and the remaining category probability elements have a probability value of 0.

In addition, the present invention is a system for correcting missing values based on a data augmentation-based machine learning method, wherein the model learning unit extracts and configures a training data set from the remaining records excluding the missing records of the first record set from the second record set. It is characterized by that.

As described above, according to the missing value correction system of the machine learning method based on data augmentation according to the present invention, by synthesizing the existing record data to generate new record data, it is possible to sufficiently generate the learning data even if the basic data is small, Through this, the effect of predicting the missing value more accurately is obtained.

In addition, according to the data augmentation-based machine learning system according to the present invention, when the field value of the record is categorical data, the weight is set as a probability vector, so that a new record can be synthesized even if the field of the record is categorical data. A possible effect is obtained.

In addition, according to the data augmentation-based machine learning method of the missing value correction system according to the present invention, a field with a missing value in a record is synthesized as having no corresponding value, so that a more diverse new record can be generated by using the missing record for synthesis. A good effect is obtained.

1 is a block diagram of an entire system for implementing the present invention.
2 is a block diagram of a configuration of a system for correcting missing values in a machine learning method based on data augmentation according to an embodiment of the present invention.
3 is a diagram illustrating a data set of a record structure according to an embodiment of the present invention.
4 is an exemplary view of a pivot table according to an embodiment of the present invention.

Hereinafter, specific details for the implementation of the present invention will be described with reference to the drawings.

In addition, in describing the present invention, the same portions are denoted by the same reference numerals, and repeated explanations thereof are omitted.

First, a configuration of an entire system for implementing the present invention will be described with reference to FIG. 1.

1A or 1B, the missing value correction system according to the present invention may be implemented as a server system (or server-client system) on a network or a program system on a computer terminal.

As shown in FIG. 1A, an example of the overall system for the implementation of the present invention is composed of a client terminal 10 and a server 30 and connected to each other through a network 80. In addition, a database 40 for storing necessary data may be further provided.

The client terminal 10 is a general computing terminal such as a PC, a notebook computer, a netbook, a PDA, a tablet PC, a smart phone, and a mobile device used by the user. The user transmits basic record data to the server 30 using the client terminal 10 or receives the corrected record data from the server 30. In addition, an application (or a mobile application, etc.) may be installed on the client terminal 10.

The server 30 is a normal server and is connected to the network 20 to provide a service for correcting missing values. Meanwhile, the server 30 may be implemented as a web server or a web application server that provides each of the services as web pages on the Internet. Alternatively, the server 30 may be a general server that provides a service by interworking with an application.

In addition, the client terminal 10 and the server 30 may be implemented as a server-client system. That is, the functions of the entire system may be divided according to the performance of the client or the amount of communication with the server. In addition, the server 30 may be implemented as a cloud system.

In addition, the database 40 includes a basic data DB 41 that stores basic data or table data composed of basic records, an augmented data DB 42 that stores an augmented record set from the basic records, and record data in which missing values are corrected. It includes a correction data DB (43) and the like to store. However, the configuration of the database 40 is only a preferred embodiment, and may be configured in a different structure according to the database construction theory in consideration of the ease and efficiency of access and search in developing a specific device.

As shown in FIG. 1B, another example of the overall system for implementing the present invention includes a missing value correction system 30 in the form of a program installed in the computer terminal 10. That is, each function of the missing value correction system 30 is implemented as a computer program and installed in the computer terminal 10, and records basic data, etc., are inputted by the user through the input device of the computer terminal 10 and registered or corrected. The result is output through the output device of the computer terminal (10). Meanwhile, the data required by the counseling system 30 are stored and used in a storage space 40 such as a hard disk of the computer terminal 10. Data can be stored in the form of a database in the storage space.

Next, a system for correcting missing values in a data augmentation-based machine learning method according to the present invention will be described with reference to FIG. 2.

As shown in FIG. 2, the missing value correction system 30 of the machine learning method based on data augmentation according to an embodiment of the present invention receives basic data and includes a basic data input unit 31 configured as record data, and a missing record. Missing data detection unit 32 to detect, record augmentation unit 33 to augment record data, model learning unit 34 to train a neural network, missing value prediction unit 35 to predict missing values, and correcting missing values to predicted values It consists of a correction data generation unit 36 that generates a corrected table. In addition, it is configured to further include a neural network model (38).

First, the basic data input unit 31 receives basic data and configures it into record data (or record set). That is, it is composed of a data set (or record set) of a record structure.

The basic data is a data set generated according to OLAP (Online Analytical Processing) analysis, and is a data set having a table or pivot table structure.

First, when the basic data is a table, the table data is composed of a plurality of record data (or record set). An example of table data is shown in FIG. 3.

As shown in FIG. 3, table data is composed of a plurality of rows and columns, and data values are placed in cells where each row and column meet. At this time, the data of each row is called a record or record data. In the example of Fig. 3, record data include R1, R2, ..., R8, and the like. Table data consists of these record data (or record sets).

In addition, each column in the table is called a field. In the example of FIG. 3, a field of a table or a field of each record includes <sales date>, <sales classification name>, <business division classification>, <sales quantity>, and <KRW unit price>. That is, the name of each field is indicated in the top record of the table.

On the other hand, each record is composed of a plurality of field data. In the example of FIG. 3, record R6 is composed of field data such as <20180102>, <domestic demand>, <home entertainment division>, <25>, and <200>.

Also, in table data, each column has a data value of the same format. For example, for the field <Sales Quantity>, the corresponding field data of all records has the same data format of 0 or natural number representing the quantity.

On the other hand, the data format is largely divided into continuous data and categorical data. Continuous data represents data that can have a continuous value by numerical values such as <Sales Quantity> and <Unit Price in Won>, and <Sales Classification Name> or <Division Classification Name> refers to data having values classified into categories. . <Sales Date> is 20180131, 20180201, etc., and the format of the date is not continuous, but it can be converted into continuous data if it is expressed as the number of days based on a specific date. That is, the <sales date> can be classified as continuous data.

Next, when the basic data is a pivot table, the pivot table is converted into a basic table composed of records. An example of a pivot table is shown in FIG. 4.

As shown in Fig. 4, the pivot table is formed in a data area that is formed of rows and columns to display a dataset, and is formed on one side of a row (preferably to the left) or one end of a column (preferably at the top) to display the dimension names. It is in the form of a table composed of dimensional areas. Data values or statistics (eg, number, sum, average, etc.) based on a combination of row and column dimensions are input and displayed in the cells of the data area.

A specific technique for converting from a pivot table to a data set of a record structure, or vice versa, from a data set of a record structure to a pivot table structure, is presented in [Patent Document 4] filed by the present applicant.

Next, the missing data detection unit 32 detects the missing record in the record set. In other words, it is divided into missing records (missing records) and non-missing records (intact records) in the entire record set. In addition, the missing data detection unit 32 detects a field having a missing value in the missing record.

That is, the missing data detection unit 32 detects a missing field (or missing value) that does not have a data value (or field value) in the record set, and detects a record having a missing field. Preferably, the missing data detection unit 32 searches all cells of the table to check whether they are missing, and detects records having cells (or fields) having missing values.

In addition, through this, the missing data detection unit 32 classifies all records of the record set into missing records and incomplete records.

As shown in Figure 3, the record set is composed of a plurality of records. At this time, the field <Division Classification Name> of record R1 does not have a data value and is therefore missing. That is, record R1 is a record with missing values. Records with missing values are R1, R3, R4, and R7. In particular, record R4 has missing values in two fields: <sales date> and <sales quantity>.

Next, the record enhancement unit 33 synthesizes at least two records (hereinafter, synthesized records) from the record set to create a new record, and adds the generated new record to the data set. Through this, the number of record sets (or record sets) is increased and augmented.

First, at least two records (or composite records) are extracted from the record set. Preferably, it is extracted randomly. Also, the record set includes not only intact records but also missing records. That is, a record having a missing value can be included in the record set and used to create a new record.

Next, the weight of each composite record is set. At this time, the weight is set so that the sum of the weights becomes 1.

Further, preferably, the weights are set randomly. In particular, the weight is newly set at random each time it is synthesized, thereby increasing the randomness.

The case of two composite records will be described in more detail.

A new new record (or composite record) r'is generated from two composite records r _i and r _{j according to Equation 1 below.}

[Equation 1]

Here, d(r,k) represents a field value (data value) of field k of record r. In particular, d(r',k) represents the field value of field k of the new record r', and d(r _i ,k) and d(r _j ,k) are fields k of the _{existing records r i} and r _{j, respectively.} Represents the field value of.

In addition, λ ₁ and λ ₂ represent weights. The sum of the weights λ ₁ and λ _{2 is 1.} Since there are two composite records, if one weight λ ₁ is set randomly, the other weight λ ₂ is automatically calculated.

Meanwhile, the above case corresponds to the case where the field of the record is continuous data.

When a field of a record is categorical data, the field value of the field is set as a probability vector of categorical data. That is, d(r,k) has a probability vector value (or a category probability vector) of the category data.

That is, the categorical field data d(r,k) is as follows.

[Equation 2]

d(r,k) = [d ₁ (r,k), d ₂ (r,k), ... d _L (r,k)]

Here, d _l (r,k) is a probability value of the lth category of the field, and L is the number of categories of the field.

That is, the size of the vector is the number of categories of categorical data, and each element of the vector is a probability value of each category.

In the example of FIG. 3, the <Sales Category> field is categorical data, and has two categories: <Domestic> and <Export>. Therefore, the <Sales Division> field is represented by two elements, that is, a two-dimensional vector.

d(r,k) = [d ₁ (r,k), d ₂ (r,k) ], d ₁ represents <domestic water>, and d ₂ represents <export>.

In particular, the category to which the field of the record belongs has 1 as a probability value, and the other categories have 0. In the example of Fig. 3, for records R2 and R8, the field <sales division> is indicated as follows.

d(R2,2) = [1, 0 ], d(R8,2) = [0, 1]

That is, the probability value of <domestic number> of record R2 is 1, and the probability value of <export> is 0. In addition, the probability value of <domestic number> of record R8 is 0, and the probability value of <export> is 1.

If λ ₁ is 0.7, the vector value of the composite record is calculated as [0.7, 0.3].

That is, when the field is categorical data, the field is expressed as a vector, and Equation 1 may be expressed as follows.

[Equation 3]

Meanwhile, the composite record is a missing record, and a specific field may have a missing value. In this case, the field value (data value) of a specific field is regarded as 0. In particular, when a specific field is categorical data, the entire vector of the field is set to 0.

In the example of Fig. 3, it is assumed that a new record is created by synthesizing the synthetic records R2 and R4. At this time, the <Sales Quantity> field of R4 is a missing value, and the value is regarded as 0 and set. Therefore, if λ ₁ is 0.7, the <Sales Quantity> field of the composite record is calculated as 14 (= 20x0.7).

In the example of Fig. 3, it is assumed that a new record is created by synthesizing the synthetic records R2 and R3. At this time, the <sales division> field of R3 is a missing value, and the vector is [0,0]. Therefore, if λ ₁ is 0.7, the vector value of the composite record is calculated as [0.7, 0].

In addition, when there are a plurality of synthesized records, in particular, if the number of synthesized records is m, [Equation 1] can be expressed as follows.

[Equation 4]

The weight λ _i is the weight for each composite record r _i . Also, the sum of their weights is 1.

Next, the model learning unit 34 constructs a training dataset with an augmented record set, and trains a machine learning model with the configured training dataset.

In particular, a training data set is constructed from the remaining records except for the missing record of the basic record set among the augmented record sets. That is, missing records are not included in the training dataset.

Machine learning models are composed of neural networks, etc., and internal variables are learned by training data. The input of the machine learning model is record data, and the output is predicted record data. Therefore, the machine learning model is trained, and when the values of input variables (or record data) are input, predicted values of each field of the corresponding record data are output.

As an example, in the example of FIG. 3, since the fields are divided into five fields, such as <sales date>, <sales classification name>, <business division classification>, <sales quantity>, and <KRW unit price>, the input of the machine learning model is 5 It is a record of four fields, and the output is also a record of five fields.

In particular, when an output field of an output record of a machine learning model has continuous data, the corresponding output field is output as continuous data. If the output field of the record is categorical data, it is output as a vector of the corresponding category, and the vector is represented by the probability of each category.

Also, the input of the machine learning model is record data. That is, when one record data is input, estimated values of the record are output. At this time, an estimated value (or a predicted value) or an estimated probability of a vector is output to each field of the output record.

Next, the missing value predictor 35 inputs the missing record into the machine learning model, and predicts the missing value using the output value.

As described above, the input of the machine learning model is record data, and the output is predicted record data. When the machine learning model receives input variable values (or record data), it outputs a predicted value of each field of the corresponding record data. In this case, the predicted value of the missing field is used instead of the missing value of the corresponding field.

In particular, when an output field of an output record of a machine learning model has continuous data, the corresponding output field is output as continuous data. Therefore, when a field with a missing value (or a missing field) is continuous data, the output value of the corresponding output field is used instead of the missing value. That is, the missing value is predicted by the output value of the corresponding output field.

In addition, when the missing field is categorical data, the field in the output record of the machine learning model is output as a vector of the corresponding category, and the vector is represented by the probability of each category. Therefore, the category with the largest probability among the probabilities of each category is estimated as the predicted category.

Next, the correction data generation unit 36 corrects the missing values to predicted values to generate a corrected table.

That is, the basic record set (the record set of the basic data) is corrected by replacing the missing value with the predicted value. In this way, the table is corrected with the corrected record set.

In particular, when the basic data is a pivot table, the corrected data of the recordset structure is restored to the pivot table [Patent Document 4].

In the above, the invention made by the present inventor has been described in detail according to the above embodiment, but the invention is not limited to the above embodiment, and it goes without saying that various changes can be made without departing from the gist of the invention.

10: client terminal 30: server
31: basic data input unit 32: missing data detection unit
33: record augmentation unit 34: model learning unit
35: missing value prediction unit 36: correction data generation unit
38: learning model
40: database 41: basic data DB
42: Augmented data DB 43: Corrected data DB
80: network

Claims (7)

In the data augmentation-based machine learning method of missing value correction system,
A basic data input unit configured to receive basic data into a data set having a record structure (hereinafter, referred to as a first record set);
A missing data detection unit detecting missing records in the first record set and distinguishing between missing records and incomplete records;
A record augmenting unit for synthesizing at least two records from the first record set to create a new record, and adding and enhancing the generated new record to the first record set;
A model learning unit that constructs a training data set using the augmented record set (hereinafter referred to as a second record set) and trains a machine learning model using the training data set;
A missing value predictor for inputting the missing record of the first record set into a machine learning model and predicting a missing value of the corresponding missing record using the output value; And,
Comprising a correction data generating unit for correcting the missing record of the first record set by correcting the missing value to a predicted value,
The record enhancement unit extracts at least two records (hereinafter, synthesized records) from the first record set, and generates a new record by weighting the extracted at least two records by a weight sum,
When the field of the composite record is categorical data, the field value of the field is set as a categorical probability vector, and the categorical probability vector has a probability value of 1 for the categorical probability element of the field of the composite record, and the remainder The categorical probability element has a probability value of 0. A system for correcting missing values based on a data enhancement-based machine learning method.
delete The method of claim 1,
The sum of the weights is 1, the missing value correction system of the data augmentation-based machine learning method, characterized in that.
The method of claim 1,
The data augmentation-based machine learning method for correcting missing values, wherein the record augmentation unit randomly sets the weights each time they are synthesized.
The method of claim 1,
When the record enhancement unit extracts the composite record from the first record set, if the composite record is a missing record, the field value having a missing value is set to have a value of 0. Method of missing value correction system.
delete The method of claim 1,
And the model learning unit extracts and configures a training data set from the remaining records excluding the missing record of the first record set from among the second record set.

Images